Rendering network training method, rendering method and related device

Abstract

The invention provides a rendering network training method, a rendering method and a related device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning and the like, and can be applied to scenes such as content generation based on artificial intelligence. According to the specific implementation scheme, a spherical depth estimation network based on a three-dimensional Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic images; processing the spherical depth map and the reference view based on a three-dimensional Gaussian model of a three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; rendering the three-dimensional Gaussian point to a target visual angle to obtain a rendering result; determining a loss value based on the spherical depth map, the rendering result and a target truth value of the target visual angle; and optimizing model parameters of the three-dimensional Gaussian rendering network based on the loss value.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision and deep learning, and can be applied to scenarios such as content generation based on artificial intelligence. Background Art

[0002] 3DGS (3D Gaussian Splatting) is an emerging 3D scene representation and rendering technology. The core advantage of 3DGS is that it can provide fast and high-quality 3D scene reconstruction and rendering.

[0003] 3DGS is developing rapidly and being applied in many fields, such as virtual reality, digital human reconstruction, AIGC (Artificial Intelligence Generated Content), and autonomous driving, showing great application potential. Summary of the invention

[0004] The present disclosure provides a rendering network training method, a rendering method and related devices.

[0005] According to one aspect of the present disclosure, a rendering network training method is provided, comprising:

[0006] A spherical depth estimation network based on a three-dimensional Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic images;

[0007] A 3D Gaussian model based on a 3D Gaussian rendering network processes the spherical depth map and the reference view to obtain 3D Gaussian points;

[0008] Render the three-dimensional Gaussian points to the target viewing angle to obtain the rendering result;

[0009] Determine the loss value based on the spherical depth map, the rendering result and the target true value of the target perspective;

[0010] Optimize model parameters of 3D Gaussian rendering network based on loss value.

[0011] According to another aspect of the present disclosure, a rendering method is provided, which is applied to the three-dimensional Gaussian rendering network in the aforementioned aspect, comprising:

[0012] Acquire a first panoramic image and a second panoramic image;

[0013] Inputting the first panoramic image and the second panoramic image into a three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points;

[0014] Render the three-dimensional Gaussian points to the specified viewing angle to obtain the rendering result.

[0015] According to another aspect of the present disclosure, a training device for a rendering network is provided, comprising:

[0016] A first processing module is used to process a reference view and a source view based on a spherical depth estimation network of a three-dimensional Gaussian rendering network to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic views;

[0017] A second processing module is used to process the spherical depth map and the reference view based on a three-dimensional Gaussian model of a three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points;

[0018] A first rendering module, used for rendering the three-dimensional Gaussian points to a target viewing angle to obtain a rendering result;

[0019] A loss determination module, for determining a loss value based on the spherical depth map, the rendering result, and a target true value of a target perspective;

[0020] The optimization module is used to optimize the model parameters of the 3D Gaussian rendering network based on the loss value.

[0021] According to another aspect of the present disclosure, a rendering device is provided, which is applied to the three-dimensional Gaussian rendering network in the aforementioned aspect, and the device includes:

[0022] An acquisition module, used for acquiring a first panoramic image and a second panoramic image;

[0023] A fifth processing module, used for inputting the first panoramic image and the second panoramic image into a three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points;

[0024] The second rendering module is used to render the three-dimensional Gaussian points to a specified viewing angle to obtain a rendering result.

[0025] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0029] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0030] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0031] In the disclosed embodiment, the reference view and the source view are panoramas, and the depth of the panorama is predicted to reconstruct the 3D Gaussian points to render the new perspective image. In the whole process, the panorama is used to realize the depth prediction and 3D scene reconstruction, providing a new generalizable 3D Gaussian rendering method.

[0032] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0034] Figure 1 is a flowchart of a training method for a rendering network according to the first embodiment of the present disclosure;

[0035] Figure 2 is a schematic diagram of a three-dimensional Gaussian rendering network according to a second embodiment of the present disclosure;

[0036] Figure 3 is another schematic diagram of a three-dimensional Gaussian rendering network according to the third embodiment of the present disclosure;

[0037] Figure 4 is a schematic diagram of a process of obtaining a spherical depth map of a reference view according to a fourth embodiment of the present disclosure;

[0038] Figure 5 is a schematic diagram of a process of obtaining a first cost volume according to the fifth embodiment of the present disclosure;

[0039] Figure 6 is a schematic diagram of a training process of a three-dimensional Gaussian rendering network according to a sixth embodiment of the present disclosure;

[0040] Figure 7 is another schematic diagram of a three-dimensional Gaussian rendering network according to the third embodiment of the present disclosure;

[0041] Figure 8 is a schematic diagram of a process for obtaining a spherical depth map according to the seventh embodiment of the present disclosure;

[0042] Fig. 9 is a schematic diagram of a process of obtaining a second cost volume according to an eighth embodiment of the present disclosure;

[0043] Fig.10is another schematic diagram of the training process of the three-dimensional Gaussian rendering network according to the ninth embodiment of the present disclosure;

[0044] Fig.11 is a flowchart of a rendering method according to a tenth embodiment of the present disclosure;

[0045] Fig.12 is a structural schematic diagram of a training device for a rendering network according to an eleventh embodiment of the present disclosure;

[0046] Fig.13 is a schematic structural diagram of a rendering device according to a twelfth embodiment of the present disclosure;

[0047] Fig.14 It is a block diagram of an electronic device used to implement the training method / rendering method of the rendering network of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0049] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] In the related art, the new perspective synthesis method of panorama usually requires a lengthy single-scene optimization process. This limits the application of 3DGS sputtering in wide-baseline panoramas. Due to the sparse perspective problem in wide-baseline panoramas, the single-scene optimization method easily causes 3DGS to overfit to the input perspective, and the effect of synthesizing new perspectives is very poor.

[0051] Generalizable 3D Gaussian rendering is a technique for reconstructing and rendering 3D scenes from multiple perspectives. The core of this technique is to use 3D Gaussian points to explicitly represent points in the scene and generate high-quality 3D scene images by rendering these Gaussian points.

[0052] In the related art, the generalizable 3DGS methods are mainly oriented to ordinary perspective image inputs. For panoramas (spherical images), these methods still need to be optimized.

[0053] In view of this, the disclosed embodiments provide a novel joint learning framework (hereinafter referred to as a three-dimensional Gaussian rendering network), which is a generalizable 3DGS, mainly including a large-scale spherical depth estimation network (Large-scale SphericalDepth Estimation Network) and a general 3D Gaussian model for 360 images (360 degreegeneralizable Gaussian decoder). The disclosed embodiments can support 3DGS rendering of various types of panoramas, such as spherical panoramas, cylindrical panoramas, etc.

[0054] like Figure 1 As shown, it is a flowchart of a training method for a rendering network provided by an embodiment of the present disclosure, including the following contents:

[0055] S101, a spherical depth estimation network based on a three-dimensional Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic views.

[0056] During implementation, when there are panoramas from multiple camera perspectives, the panoramas from each camera perspective can be used as reference views, and the panoramas from other camera perspectives can be used as source views. For example, the panoramas from three camera perspectives are A, B, and C. Then, when A is used as the reference view, B and C are used as source views. When B is used as the reference view, A and C are used as source views. The disclosed embodiment is described by taking a pair of reference views and source views as an example, and each pair of reference views and source views is processed in the same manner.

[0057] The value of each pixel in the spherical depth map is a predicted depth value, which corresponds one-to-one to the pixel of the reference view.

[0058] S102, processing the spherical depth map and the reference view based on the three-dimensional Gaussian model of the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points.

[0059] The 3D Gaussian model reconstructs 3D Gaussian points in 3D space based on the input. The properties of the 3D Gaussian points include covariance matrix, spherical harmonic coefficients, position, and opacity. Through the reconstructed 3D Gaussian points, a panoramic image of any perspective can be rendered.

[0060] S103, rendering the three-dimensional Gaussian points to a target viewing angle to obtain a rendering result.

[0061] The target viewing angle may be determined according to actual needs, and the embodiments of the present disclosure do not limit this.

[0062] S104, determining a loss value based on the spherical depth map, the rendering result and the target true value of the target viewing angle.

[0063] Among them, the target truth value can be understood as the real rendering effect of the three-dimensional scene from the target perspective, so as to facilitate supervised training.

[0064] S105, optimizing the model parameters of the 3D Gaussian rendering network based on the loss value.

[0065] like Figure 2 As shown, the generalizable three-dimensional Gaussian rendering network provided by the embodiment of the present disclosure includes a spherical depth estimation network and a three-dimensional Gaussian model.

[0066] The reference view and the source view are input into the spherical depth estimation network to obtain a spherical depth map. The spherical depth map is then combined with the reference view Figure 1 By inputting the 3D Gaussian model, the rendering result of the target perspective can be rendered. The spherical depth map, the rendering result and the corresponding target true value can be compared to optimize the model parameters based on the loss value.

[0067] In the embodiment of the present disclosure, the reference view and the source view are panoramas, and the three-dimensional Gaussian points are reconstructed by predicting the depth of the panorama to render a new perspective image. Thus, a new generalizable three-dimensional Gaussian rendering method is provided. This method can support three-dimensional Gaussian rendering with a panorama as input, and can retain the information in the three-dimensional scene as much as possible, thereby improving the rendering effect. In addition, compared with the single-scene optimization process, the model parameter optimization process of the embodiment of the present disclosure is more efficient. In summary, the embodiment of the present disclosure provides an efficient and high-precision three-dimensional Gaussian rendering mode.

[0068] In some embodiments, in order to achieve high-quality depth estimation to improve the rendering quality of the new perspective image, in the embodiments of the present disclosure, Figure 3 As shown, the spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module.

[0069] based on Figure 3 The spherical depth estimation network shown in FIG. 1 , the process of obtaining the spherical depth map of the reference view can be implemented as follows: Figure 4 As shown:

[0070] S401 : Process a first rectangular projection image of a reference view and a second rectangular projection image of a source view based on a first cost volume prediction network to obtain a first cost volume.

[0071] The reference view may be converted from spherical coordinates to plane coordinates, and the obtained rectangular projection (Equirectangular Projection) is used as the first rectangular projection.

[0072] Similarly, the source view can be converted from spherical coordinates to plane coordinates, and the obtained rectangular projection image is used as the second rectangular projection image.

[0073] In stereo vision and multi-view stereo vision, cost volume is a core concept used to represent the matching cost under different depth assumptions. It can help calculate the depth information of each pixel in the image. The first cost volume in step S401, as the name implies, is a cost volume determined based on the first cost volume prediction network.

[0074] S402, processing the cube projection image of the reference view based on the initial depth estimation network to obtain an initial spherical depth.

[0075] The initial spherical depth is the initially predicted depth information. During implementation, any network capable of estimating depth is applicable to the embodiments of the present disclosure.

[0076] S403 : Fusing the initial spherical depth and the first cost volume based on the first fusion module to obtain a spherical depth map of the reference view.

[0077] During implementation, the first cost volume can also be used to predict a spherical depth map. However, there will still be a large depth error when predicting the spherical depth map based solely on the first cost volume, and in most outdoor 360 scenes, there is no depth supervision. Therefore, in the disclosed embodiment, on the basis of the first cost volume prediction network, the depth prior given by the initial depth estimation network is further combined to improve the accuracy of depth estimation. It can be understood that the entire depth estimation process basically relies on the rectangular projection of the reference view and the source view, which can minimize the lossy compression loss caused by splitting the cube projection for depth estimation, and thus provide high-quality depth estimation data for Gaussian point reconstruction of the three-dimensional Gaussian model, thereby ultimately improving the rendering quality of the model.

[0078] In some embodiments, the first cost volume may be obtained by performing the following operations based on the first cost volume prediction network:

[0079] Step A1, extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image.

[0080] During implementation, the first cost volume prediction network includes an encoder. A first rectangular projection image of a reference view and a second rectangular projection image of a source view are input into the encoder to obtain a first feature of the reference view and a second feature of the source view.

[0081] Step A2, based on the first feature and the second feature, determine the first feature matching cost of multiple pixel points in the reference view under multiple depth assumptions as a first cost volume; the first feature matching cost is determined based on the similarity between the first feature vector of the multiple pixel points in the first feature and the second feature vector in the second feature.

[0082] During implementation, the first cost volume prediction network may include a matching cost determination module for determining the first feature matching cost.

[0083] The first feature matching cost is composed of the matching costs of multiple pixels, which can be understood as a multi-dimensional matrix, that is, an initial cost volume. During implementation, the following operations can be performed for each pixel in the first rectangular projection of the reference view to obtain the matching cost corresponding to the pixel:

[0084] Step A21, uniformly sampling is performed within a preset depth sampling range to obtain multiple depth hypotheses for the pixel.

[0085] During implementation, the preset depth sampling range may vary according to the reconstructed scene. For example, in an indoor scene, the range is [1m, 10m]; in an outdoor scene, the sampling range may be appropriately expanded. During specific implementation, the sampling range of different scenes may be set based on prior knowledge, and the embodiments of the present disclosure do not limit this.

[0086] The number of depth hypotheses for each pixel can also be flexibly set according to demand. Assuming that each pixel is sampled D times within the corresponding preset depth sampling range, each pixel corresponds to D depth hypotheses, where D is a positive integer greater than 1.

[0087] Step A22, obtaining the projection point corresponding to the pixel point in the source view.

[0088] During implementation, the pixel points of the same position point in the panoramic images at different viewing angles can be obtained according to the conversion relationship between the plane coordinates of the rectangular projection view, the spherical coordinates and the Cartesian coordinates in the camera coordinate system.

[0089] The first rectangular projection of the reference view and the second rectangular projection of the source view both use plane coordinates to express the positions of different pixels. The conversion relationship between plane coordinates and spherical coordinates is shown in expression (1):

[0090]

[0091] In expression (1), (u, v) represents the plane coordinate position in the rectangular projection image (such as the first rectangular projection image or the second rectangular projection image). (θ, φ) represents the polar angle and azimuth angle in the spherical coordinates. W and H represent the width and height of the rectangular projection image.

[0092] The spherical coordinate system can be transformed into the Cartesian coordinate system to obtain the position expression of each pixel in the rectangular projection image in the camera coordinate system, as shown in expression (2):

[0093]

[0094] In expression (2), (r, θ, φ) are spherical coordinates; r is the radial distance, which is the straight-line distance from the pole of the spherical coordinate system to the point. In the spherical coordinate system, this distance is fixed and represents the straight-line distance from the point to the origin; xyz is the coordinate of the camera coordinate system.

[0095] Because the source view and the reference view belong to different camera perspectives in the same scene, the reference view and the source view can be associated based on the conversion relationship between different camera perspectives, as shown in expression (3):

[0096]

[0097] In expression (3), i represents the camera of the reference view angle, represents the position coordinate expression of the reference view perspective, W is the conversion relationship between different camera perspectives, j represents the source view perspective, A positional representation that represents the perspective of the source view.

[0098] Through the above mapping relationship, the same position point in the scene can be located in the first rectangular projection image and the second rectangular projection image, that is, the projection point of the pixel point in the first rectangular projection image in the second rectangular projection image is obtained.

[0099] Step A23, obtaining a feature vector of the pixel point from the first feature as the first feature vector; and obtaining a feature vector of the projection point of the pixel point in the second rectangular projection image of the source view from the second feature as the second feature vector.

[0100] Step A24, determining the similarity between the first feature vector and the second feature vector as the matching cost of the pixel point.

[0101] For each pixel (u, v) of the reference view ref , assuming its depth is t i (i=1,2,…,D, i.e., the i-th depth hypothesis), find its corresponding pixel (u, v) in the source view according to the spherical projection src Next, calculate (u, v) ref and (u, v) srcThe similarity between the local features at (u, v, i), that is, the similarity between the first feature vector and the second feature vector, is regarded as the value of the cost volume (first feature matching cost) at (u, v, i). Assuming the length of the feature vector is F, the process produces a 4D cost volume V after D depth hypothesis scans.

[0102] If the cost volume is too large, it will affect the reasoning speed. In order to improve the rendering speed of this network, in other embodiments, based on the first cost volume prediction network, the following operations are performed to obtain the first cost volume: Figure 5 As shown, the following steps are included:

[0103] S501, extracting a first feature of a first rectangular projection image and a second feature of a second rectangular projection image.

[0104] S502, based on the first feature and the second feature, determine the first feature matching cost of multiple pixel points in the reference view under multiple depth assumptions; the first feature matching cost is determined based on the similarity between the first feature vector of the multiple pixel points in the first feature and the second feature vector in the second feature.

[0105] The operations of steps S501 and S502 are the same as those of steps A1 and A2 above, and will not be described again here.

[0106] In some embodiments, the cosine similarity between the feature vectors may be used as the similarity between the two feature vectors. The similarity between the two may also be determined as shown in expression (4):

[0107]

[0108] In expression (4), C represents the similarity between two feature vectors, and F i A feature vector corresponding to a pixel point in a first feature of a first rectangular projection image representing a reference view, that is, a first feature vector; represents the second eigenvector mentioned above; the F in the denominator is the length of the eigenvector.

[0109] S503 , performing a dimensionality reduction operation on the first feature matching cost to obtain a first cost volume.

[0110] In the disclosed embodiment, the first feature matching cost is obtained for each pixel under multiple depth assumptions, which can accurately and comprehensively describe the matching cost of the reference view and the source view to improve the accuracy of the cost volume. Through further dimensionality reduction operations, the size of the cost volume can be reduced, improving the rendering efficiency of the entire network.

[0111] In some embodiments, a three-dimensional convolutional neural network (3DCNN) may be used to process the first feature matching cost to obtain a first cost volume.

[0112] 3DCNN can effectively reduce the cost volume and retain the key information in the cost volume, which can improve the overall processing efficiency while ensuring the accuracy and efficiency of spherical depth map prediction.

[0113] In addition, in order to further improve the processing efficiency, in the present embodiment, the dimensionality reduction operation can also be implemented as follows: for the first feature matching cost, an average operation is performed on the dimension of the first feature vector to obtain a first cost volume.

[0114] As described above, at each hypothetical depth of each pixel, the matching cost is (u, v, i), and the feature vector length is F, then the scale of the final first feature matching cost is W*H*D*F. During implementation, the F dimension can be averaged so that F becomes 1, and the scale of the final first cost volume is W*H*D*1.

[0115] This method can not only effectively reduce the cost volume size, but also retain key information and maintain the prediction accuracy of the spherical depth map. The method of finding the mean here is simpler and faster, which can effectively improve the rendering efficiency of the entire 3D Gaussian rendering network.

[0116] As explained above, although the first cost volume predicted by the first cost volume prediction network can estimate the depth, it cannot solve the spherical distortion problem of equirectangular projection. Moreover, many outdoor scenes do not have reference depths for supervised training. Therefore, using only the first cost volume prediction network still has a large depth error, and the applicable scenarios are very limited. Therefore, based on the first cost volume prediction network, an initial depth estimation network is introduced.

[0117] Among them, the initial spherical depth is obtained by processing the cube projection image of the reference view based on the initial depth estimation network, which can be implemented as follows:

[0118] Step B1, perform depth estimation on the cube projection image to obtain a depth point cloud image.

[0119] In some embodiments, depth estimation can be performed on the cube projection image based on a geometric three-dimensional vision network (Geometric 3D Vision Made Easy, DUSt3R,) to obtain a depth point cloud image.

[0120] DUSt3R is a new paradigm for dense and unconstrained stereo 3D reconstruction, without prior knowledge of camera calibration or viewpoint pose. The pairwise reconstruction problem is transformed into a regression of point maps, relaxing the hard constraints of the traditional projective camera model. Given multiple images (e.g., six images in a cube projection), a simple but effective global alignment strategy is proposed to express all pairs of point maps in a common reference frame. The network architecture of DUSt3R is based on standard Transformer encoders and decoders, which can leverage powerful pre-trained models.

[0121] In the disclosed embodiment, DUSt3R can provide a good depth prior to provide a high-quality depth point cloud map, thereby improving the estimation accuracy of the spherical depth map and ultimately improving the accuracy of generalized 3D Gaussian rendering.

[0122] During implementation, the initial depth estimation network can use a mature network model that can effectively provide depth priors (such as DUSt3R). During the optimization of the 3D Gaussian rendering network, the model parameters of the initial depth estimation network are frozen. That is, during the training phase, the model parameters of the initial depth estimation network are not optimized, thereby improving the efficiency of model training while ensuring the accuracy of depth estimation.

[0123] Step B2: Project the depth point cloud onto the spherical surface of the reference view to obtain an initial spherical depth.

[0124] In the disclosed embodiment, by estimating the depth point cloud map, a good depth prior is provided for the first cost volume, and by projecting it onto a spherical surface, the first cost volume and the depth prior can be better integrated. The combination of the two can provide effective depth information to each other, so as to convert the rough initial spherical depth into a high-precision spherical depth map through the first cost volume.

[0125] On the basis of obtaining the initial spherical depth and the first cost volume, the initial spherical depth and the first cost volume are fused based on the first fusion module to obtain a spherical depth map of the reference view, which can be implemented as follows:

[0126] Step C1, fusing the initial spherical depth and the first cost volume to obtain a first fused feature;

[0127] During implementation, concat (stitching) may be used to fuse the initial spherical depth map and the first cost volume.

[0128] Step C2: perform depth prediction on the first fused feature to obtain a spherical depth map.

[0129] During implementation, the first fused feature may be depth predicted based on a first two-dimensional convolutional network (2D Convolutional Neural Network, 2D CNN) to obtain a spherical depth map. 2D CNN is a neural network architecture specifically used for processing image data, and it has shown great potential in the fields of computer vision and image processing.

[0130] 2D CNN is used for fusion. Through its local perception and parameter sharing, 2D CNN can efficiently capture local features in the first cost volume and initial spherical depth, reduce the number of network parameters, and improve the rendering efficiency of the 3D Gaussian rendering network.

[0131] In summary, the first fusion module, by fusing the first cost volume and the initial spherical depth and performing depth prediction, can effectively use the first cost volume and the initial spherical depth to estimate the spherical depth map, so as to improve rendering efficiency and accuracy.

[0132] In the entire 3D Gaussian rendering network, it is necessary to first estimate the spherical depth map, and then reconstruct the 3D Gaussian points by combining the 3D Gaussian model with the reference view.

[0133] In the disclosed embodiment, the three-dimensional Gaussian model includes an encoder and a decoder. Thus, the three-dimensional Gaussian model based on the three-dimensional Gaussian rendering network processes the spherical depth map and the reference view to obtain the three-dimensional Gaussian points, which can be implemented as follows: inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the second fused feature into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder; wherein the second fused feature is obtained by fusing the spherical depth map, the first feature and the second feature.

[0134] The embodiment of the present disclosure introduces the first feature and the second feature to provide key feature information of the panoramic image for the three-dimensional Gaussian model, so as to improve the accuracy of reconstructing three-dimensional Gaussian points by the three-dimensional Gaussian model.

[0135] The first feature, the second feature and the spherical depth map may be concat-operated to obtain a second fused feature.

[0136] Here, the reference view input to the encoder mainly uses the RGB color value of each pixel, and the three-dimensional Gaussian point is reconstructed through the three-dimensional Gaussian model.

[0137] The reconstructed 3D Gaussian points can then be rendered to any viewing angle to facilitate loss-based optimization of model parameters.

[0138] During implementation, the target truth values ​​used to determine the loss may include depth truth, photometric truth, and color truth. The depth truth is used to optimize the depth accuracy estimated by the spherical depth estimation network. The photometric truth and color truth are used to enable the reasoning of the entire network to achieve a realistic level of visual effect in rendering images of any perspective.

[0139] Therefore, determining the loss value based on the spherical depth map, the rendering result and the true value of the target perspective can be implemented as follows: determining the loss value based on the depth loss between the spherical depth map and the depth true value in the target true value, the photometric loss between the photometric value of the rendering result and the photometric true value in the target true value, and the color value of the rendering result and the color true value in the true value.

[0140] When implemented, the loss is calculated as shown in expression (5):

[0141] L joint =2β·t·(L depth +L photo )+(1-2β·t)·L rgb (5)

[0142] In expression (5), t represents the number of training steps, L depth is the depth loss, L photo is the luminosity loss, L rgb is the RGB (red green, blue) color value loss; β is the hyperparameter set before training.

[0143] In the disclosed embodiment, the ability to evaluate the model in multiple aspects such as depth and rendering effect can improve the rendering accuracy of the model at different stages.

[0144] In summary, the architecture of a three-dimensional Gaussian rendering network provided by the embodiment of the present disclosure is as follows: Figure 6 As shown:

[0145] The first rectangular projection of the reference view and the second rectangular projection of the source view are input into the first cost volume prediction network, and the encoder of the network (not shown in the figure) extracts the first feature of the first rectangular projection and the second feature of the second rectangular projection. Figure 6 The multi-view matching in the method determines the pixel points of the same position point in the scene in two rectangular projection images, thereby finding the first feature vector of the position point in the first feature and the second feature vector in the second feature. The similarity between the first feature vector and the second feature vector corresponding to each identical position point is calculated to obtain the first matching cost (not shown in the figure). Then, the first cost volume is obtained by reducing the dimension of the first matching cost.

[0146] At the same time, if Figure 6 As shown in the figure, in another branch, the reference view is split into a cube projection map and input into the initial depth prediction network (such as Figure 6 Then the initial spherical depth is concatenated with the first cost volume (not shown in the figure) and input to the 2DCNN (i.e. Figure 6 The first fusion module Fusion in is used for fusion to predict a spherical depth map.

[0147] The predicted spherical depth map is concat- ed with the relevant features and then input into a 3D Gaussian model (Gaussian Decoder) in combination with the reference view to reconstruct the 3D Gaussian points in the scene.

[0148] Then render the 3D Gaussian points to the target perspective and get the rendering result under the new perspective ( Figure 6 not shown).

[0149] Finally, the loss can be calculated based on expression (5), and the model parameters of the 3D Gaussian rendering network can be optimized accordingly. It should be noted that when DUSt3R is used in the initial deep network, the model parameters can be frozen and not participate in the optimization process to improve the model training efficiency.

[0150] remove Figure 6 In addition to the architecture shown, another three-dimensional Gaussian rendering network is also provided in the embodiments of the present disclosure.

[0151] like Figure 7 As shown, the spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module.

[0152] based on Figure 7 The spherical depth estimation network based on the 3D Gaussian rendering network processes the reference view and the source view to obtain the spherical depth map of the reference view, which can be implemented as follows: Figure 8 As shown:

[0153] S801, processing a cube projection image of a reference view based on a monocular depth estimation network to obtain a monocular feature extracted by an intermediate layer of the monocular depth estimation network.

[0154] During implementation, the monocular depth estimation network can use the monocular depth estimation algorithm of DepthAnythingV2. The last layer of features can be used as monocular features, and of course, features output by other layers can also be used as monocular features according to actual conditions.

[0155] S802 , processing the monocular features, the first rectangular projection image of the reference view, and the second rectangular projection image of the source view based on the second cost volume prediction network to obtain a second cost volume.

[0156] Similar to what has been described above, there will still be a large depth error when predicting the spherical depth map based only on the second cost volume. Moreover, in outdoor 360 scenes, there is no depth supervision in most cases. Therefore, the disclosed embodiment introduces the monocular features extracted by the monocular depth estimation network to provide feature information of the depth, so as to provide an effective depth prior and improve the estimation accuracy of the final spherical depth map.

[0157] S803 : Perform depth prediction on the second cost volume based on the second fusion module to obtain a spherical depth map of the reference view.

[0158] In the disclosed embodiment, based on the second cost volume prediction network, the depth prior provided by the monocular depth estimation network is further combined to improve the accuracy of depth estimation. It can also be understood that the entire depth estimation process basically relies on the rectangular projection of the reference view and the source view, which can minimize the lossy compression loss caused by splitting the cube projection for depth estimation, and then provide high-quality depth estimation data for Gaussian point reconstruction of the three-dimensional Gaussian model, ultimately improving the rendering quality.

[0159] During implementation, the monocular depth extracted by the monocular depth estimation network carries key features for estimating depth. Therefore, it can be fused into the features extracted by the second cost volume prediction network to improve the accuracy of the cost volume determined by the second cost extraction network. Thus, based on the second cost volume prediction network processing the monocular features, the first rectangular projection of the reference view, and the second rectangular projection of the source view, the second cost volume is obtained, which can be implemented as follows Fig. 9 As shown:

[0160] S901, extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image.

[0161] Similar to the first cost volume prediction network, the second cost volume prediction network here can extract the first feature from the first rectangular projection image based on the encoder, and extract the second feature from the second rectangular projection image.

[0162] S902, fusing the monocular feature into the first feature to obtain a third feature; and fusing the monocular feature into the second feature to obtain a fourth feature.

[0163] During implementation, the monocular feature may be concat to the first feature and the second feature, thereby obtaining the third feature and the fourth feature.

[0164] S903, based on the third feature and the fourth feature, determine the second feature matching cost of multiple pixels in the reference view under multiple depth assumptions; the second feature matching cost is determined based on the similarity between the third feature vector of the multiple pixels in the third feature and the fourth feature vector in the fourth feature.

[0165] Similar to the first feature matching cost, the second feature matching cost is composed of the matching costs of multiple pixels in the reference view. The second feature matching cost can be understood as a multi-dimensional matrix, that is, an initial cost volume. During implementation, the following operations can be performed for each pixel in the first rectangular projection of the reference view to obtain the matching cost corresponding to the pixel:

[0166] Step A21, uniformly sampling is performed within a preset depth sampling range to obtain multiple depth hypotheses for the pixel.

[0167] During implementation, the preset depth sampling range may vary according to the reconstructed scene. For example, in an indoor scene, the range is [1m, 10m]; in an outdoor scene, the sampling range may be appropriately expanded. During specific implementation, the sampling range of different scenes may be set based on prior knowledge, and the embodiments of the present disclosure do not limit this.

[0168] The number of depth hypotheses for each pixel can also be flexibly set according to demand. Assuming that each pixel is sampled D times within the corresponding preset depth sampling range, each pixel corresponds to D depth hypotheses, where D is a positive integer greater than 1.

[0169] Step A22, obtaining the projection point corresponding to the pixel point in the source view.

[0170] During implementation, the pixel points of the same position point in the panoramic image at different viewing angles can be obtained according to the conversion relationship between the plane coordinates of the rectangular projection view, the spherical coordinates and the Cartesian coordinates in the camera coordinate system. The specific implementation process can be referred to the mapping relationship shown in the above expressions (1)-(3), which will not be repeated here.

[0171] Through the mapping relationship shown in the above expressions (1)-(3), the same position point in the scene can be located in the first rectangular projection image and the second rectangular projection image, that is, the projection point of the pixel point in the first rectangular projection image in the second rectangular projection image is obtained.

[0172] Step A23, obtaining a feature vector of the pixel point from the third feature as the third feature vector; and obtaining a feature vector of the projection point of the pixel point in the second rectangular projection image of the source view from the fourth feature as the fourth feature vector.

[0173] Step A24, determining the similarity between the third eigenvector and the fourth eigenvector as the matching cost of the pixel point.

[0174] The method for calculating the similarity between the third eigenvector and the fourth eigenvector may refer to the aforementioned expression (4), which will not be described in detail in the embodiment of the present disclosure.

[0175] For each pixel (u, v) of the reference view ref , assuming its depth is t i (i=1,2,…,D, i.e., the i-th depth hypothesis), find its corresponding pixel (u, v) in the source view according to the spherical projection src Next, calculate (u, v) ref and (u, v) src The similarity between the local features at (u, v, i), that is, the similarity between the first feature vector and the second feature vector, is regarded as the value of the cost volume at (u, v, i). Assuming the length of the feature vector is F, this process produces a 4D cost volume V after D depth hypothesis scans.

[0176] If the cost volume is too large, it will affect the inference speed. In order to improve the rendering speed, the second feature matching cost can also be compressed. See S904 for details.

[0177] S904: Perform a dimensionality reduction operation on the second feature matching cost to obtain a second cost volume.

[0178] In the disclosed embodiment, for each pixel point, the monocular depth extracted by the monocular depth estimation network is additionally used to determine the matching cost, and the first feature matching cost is obtained for multiple pixels under multiple depth assumptions, which can accurately and comprehensively describe the matching cost of the reference view and the source view to improve the accuracy of the cost volume. Through further dimensionality reduction operations, the size of the cost volume can be reduced, and the rendering efficiency of the entire network can be improved.

[0179] In some embodiments, the dimensionality reduction operation on the second feature matching cost may be implemented as follows: using a second three-dimensional convolutional neural network to process the second feature matching cost to obtain a second cost volume.

[0180] 3DCNN can effectively reduce the cost volume and retain the key information in the cost volume, which can improve the overall processing efficiency while ensuring the accuracy of spherical depth map prediction.

[0181] In addition, in order to further improve the processing efficiency, in the present embodiment, the dimensionality reduction operation can also be implemented as follows: for the second feature matching cost, a mean operation is performed in the dimension of the third feature vector to obtain a first cost volume.

[0182] As described above, at each hypothetical depth of each pixel, the matching cost is (u, v, i), and the feature vector length is F, then the scale of the final second feature matching cost is W*H*D*F. During implementation, the F dimension can be averaged so that F becomes 1, and the scale of the final second cost volume is W*H*D*1.

[0183] This method can not only effectively reduce the cost volume size, but also retain key information and maintain the prediction accuracy of the spherical depth map. The method of finding the mean here is simpler and faster, which can effectively improve the rendering efficiency of the entire 3D Gaussian rendering network.

[0184] In the disclosed embodiment, the three-dimensional Gaussian model includes an encoder and a decoder. Therefore, the three-dimensional Gaussian model based on the three-dimensional Gaussian rendering network processes the spherical depth map and the reference view to obtain the three-dimensional Gaussian points, which can be implemented as follows: inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the third fusion feature into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder; wherein the third fusion feature is obtained by fusing the spherical depth map, the third feature and the fourth feature.

[0185] The embodiment of the present disclosure introduces the third feature and the fourth feature to provide the three-dimensional Gaussian model with key feature information and depth information of the panoramic image, so as to improve the accuracy of reconstructing three-dimensional Gaussian points by the three-dimensional Gaussian model.

[0186] Among them, the third feature, the fourth feature and the spherical depth map can be concat-operated to obtain the third fusion feature.

[0187] Here, the reference view input to the encoder mainly uses the RGB color value of each pixel, and the three-dimensional Gaussian point is reconstructed through the three-dimensional Gaussian model.

[0188] The reconstructed 3D Gaussian points can then be rendered to any viewing angle to facilitate loss-based optimization of model parameters.

[0189] In summary, the architecture of the second 3D Gaussian rendering network provided by the embodiment of the present disclosure is as follows: Fig.10 As shown:

[0190] The first rectangular projection image of the reference view and the second rectangular projection image of the source view are input into a second cost volume prediction network, and the encoder (not shown in the figure) of the network extracts the first feature of the first rectangular projection image and the second feature of the second rectangular projection image.

[0191] At the same time, if Fig.10 As shown, in another branch, the reference view is split into cube projection images, which are input into a monocular depth estimation network to obtain the features of its last layer output as monocular features.

[0192] The monocular feature is fused (eg, concat) to the first feature and the second feature to obtain the third feature and the fourth feature.

[0193] Then, the pixel points of the same position point in the scene in the two rectangular projection images are determined, so as to find the third eigenvector of the position point in the third feature and the fourth eigenvector in the fourth feature. The similarity between the corresponding third eigenvector and fourth eigenvector is calculated for each identical position point, so as to obtain the second feature matching cost. Then, the second cost volume is obtained by reducing the dimension of the second feature matching cost (not shown in the figure).

[0194] The second cost volume is then fed into a 2DCNN for processing to predict a spherical depth map.

[0195] The predicted spherical depth map is concatted with the third feature and the fourth feature, and then input into the 3D Gaussian model in combination with the reference view to reconstruct the 3D Gaussian points in the scene.

[0196] Then the three-dimensional Gaussian points are rendered to the target perspective to obtain a rendering result under the new perspective (not shown in the figure).

[0197] Finally, the loss can be calculated based on expression (5), and the model parameters of the three-dimensional Gaussian rendering network can be optimized accordingly.

[0198] In addition, regardless of Figure 6 The 3D Gaussian rendering network shown is still Fig.10 Another three-dimensional Gaussian rendering network is shown. The three-dimensional Gaussian model used in the embodiment of the present disclosure is a 360Depth-shifter Gaussian Decoder. Its goal is to predict three-dimensional Gaussian points in 3D space. The properties of the three-dimensional Gaussian points include covariance matrix, spherical harmonic coefficients, position and opacity. These three-dimensional Gaussian points can be used to render new 360-degree perspective images. The 360Depth-shifter Gaussian Decoder uses an explicit 3D Gaussian point representation, which makes the rendering process more efficient and scalable. It inputs a spherical depth map and an RGB image of a reference view. The three-dimensional Gaussian model is based on an encoder-decoder module constructed based on a U-net (Convolutional Networks for Biomedical Image Segmentation, U-Net), which is used to predict each three-dimensional Gaussian point and obtain a three-dimensional Gaussian point set G i = {g k =(μ k ,∑ k , α k , S k )}, where μ kis the mean, indicating the position of the 3D Gaussian center point; ∑ k is the covariance matrix; α k is opacity; S k are the spherical harmonic coefficients. And, μ k It is directly converted from the spherical depth map. Figure 6 and Fig.10 The improved spherical depth shown can obtain higher quality Gaussian properties and render higher quality new perspective panoramas.

[0199] Based on the same technical concept, the embodiment of the present disclosure also provides a rendering method, which is applied to the three-dimensional Gaussian rendering network trained as above. Fig.11 As shown, including the following:

[0200] S1101, obtaining a first panoramic image and a second panoramic image;

[0201] Among the first panorama and the second panorama, one panorama is used as a reference view and the other panorama is used as a source view to participate in the reasoning process.

[0202] S1102: Input the first panoramic image and the second panoramic image into a three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points.

[0203] The specific structure and reasoning process of the 3D Gaussian rendering network have been described in the previous article and will not be repeated here. Figure 6 or Fig. 9 Any provided three-dimensional Gaussian rendering network is inferred to obtain three-dimensional Gaussian points constructed based on the first panoramic image and the second panoramic image.

[0204] S1103, rendering the three-dimensional Gaussian points to a specified viewing angle to obtain a rendering result.

[0205] In the disclosed embodiment, it is possible to support the implementation of rangeable three-dimensional Gaussian sputtering based on a panoramic image. The panoramic image can retain more image details, reduce image loss at other times of rendering, and improve rendering quality.

[0206] In the related technology, for example, the generalizable spherical NeRF (Neural Radiance Fields) method (such as PanoGRF (Generalizable Spherical Radiance Fields for Wide-baseline Panoramas)) requires a large number of points to complete, the inference rendering speed is very slow, and a pre-trained depth estimation network is required.

[0207] Other generalizable 3DGS methods (such as pixelsplat and multi-view Figure 3 Efficient 3D Gaussian Splatting from Sparse Multi-View Images (mvsplat) is designed for perspective images, not panoramas. They perform coordinate transformation based on perspective projection, ignoring the spherical characteristics of panoramas. To use these methods, the panorama needs to be split into perspective images first. However, this splitting process is completed by approximate interpolation, which is a lossy transformation.

[0208] The three-dimensional Gaussian rendering network provided by the embodiment of the present disclosure utilizes the characteristics of the full field of view of the panorama, adopts a spherical cost-volume to improve the rough depth information, and processes it by the 3D Gaussian model to output Gaussian points aligned with the input perspective view. This process can decompose the Gaussian generation process and the new perspective drawing process, and allows real-time rendering at a high frame rate (≥100 FPS (Frames Per Second)). Experimental verification shows that the rendering time for a 256*256 resolution is about 5*10 -3 seconds, and can support 200FPS.

[0209] In addition, the present disclosure uses the cost-volume fusion monocular features provided by the monocular depth estimation network or the depth prior predicted by DUSt3R to infer a more accurate scene geometric depth (i.e., spherical depth map). Based on the optimized geometric depth, more accurate 3D-GS properties can be inferred, rendering a more accurate new perspective panoramic image.

[0210] The method provided by the embodiment of the present disclosure can be applied to products or projects that require real-time 3D graphics rendering and computer vision integration. Figure 3D-GS framework. It is capable of reasoning at a rate of ≥5FPS and real-time rendering at a rate of ≥100FPS. In addition, the method has achieved state-of-the-art results on datasets such as Matterport3D (Learning from RGB-D Data in Indoor Environments, a large-scale RGB-D dataset), Front3D (3D Furnished Rooms with layOuts and semantics), and Carla (Learning to Act), and is able to provide real-time rendering for immersive experiences. Therefore, it can be applied to fields such as virtual reality (VR), augmented reality (AR), autonomous driving, robot navigation, 3D modeling and rendering, etc., which require real-time 3D graphics and computer vision technologies.

[0211] Based on the same technical concept, the present disclosure also proposes a training device 1200 for a rendering network, such as Fig.12 As shown, including:

[0212] The first processing module 1201 is configured to process a reference view and a source view based on a spherical depth estimation network of a three-dimensional Gaussian rendering network to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic views;

[0213] A second processing module 1202 is used to process the spherical depth map and the reference view based on the three-dimensional Gaussian model of the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points;

[0214] A first rendering module 1203 is used to render the three-dimensional Gaussian points to a target viewing angle to obtain a rendering result;

[0215] A loss determination module 1204, configured to determine a loss value based on the spherical depth map, the rendering result, and a target true value of the target perspective;

[0216] The optimization module 1205 is used to optimize the model parameters of the three-dimensional Gaussian rendering network based on the loss value.

[0217] In some embodiments, the spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module; a first processing module includes:

[0218] A first processing unit, configured to process a first rectangular projection image of a reference view and a second rectangular projection image of a source view based on a first cost volume prediction network to obtain a first cost volume;

[0219] A second processing unit is used to process the cube projection image of the reference view based on the initial depth estimation network to obtain an initial spherical depth;

[0220] The fusion unit is used to fuse the initial spherical depth and the first cost volume based on the first fusion module to obtain a spherical depth map of the reference view.

[0221] In some embodiments, the first processing unit is specifically configured to:

[0222] Extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image;

[0223] Based on the first feature and the second feature, determining a first feature matching cost of a plurality of pixels in the reference view under a plurality of depth assumptions; the first feature matching cost is determined based on a similarity between a first feature vector in the first feature and a second feature vector in the second feature of the plurality of pixels;

[0224] A dimensionality reduction operation is performed on the first feature matching cost to obtain a first cost volume.

[0225] In some embodiments, the first processing unit is specifically configured to:

[0226] For the first feature matching cost, a mean operation is performed on the dimension of the first feature vector to obtain a first cost volume.

[0227] In some embodiments, the first processing unit is specifically configured to:

[0228] A three-dimensional convolutional neural network is used to process the first feature matching cost to obtain a first cost volume.

[0229] In some embodiments, the second processing unit is specifically configured to:

[0230] Perform depth estimation on the cube projection image to obtain a depth point cloud image;

[0231] Project the depth point cloud onto the spherical surface of the reference view to obtain the initial spherical depth.

[0232] In some embodiments, the second processing unit is specifically configured to:

[0233] The depth of the cube projection image is estimated based on the geometric 3D vision network to obtain a depth point cloud image.

[0234] In some embodiments, the fusion unit is specifically configured to:

[0235] Fusing the initial spherical depth and the first cost volume to obtain a first fused feature;

[0236] Depth prediction is performed on the first fused feature to obtain a spherical depth map.

[0237] In some embodiments, the fusion unit is specifically configured to:

[0238] The first fused feature is subjected to depth prediction based on the first two-dimensional convolutional network to obtain a spherical depth map.

[0239] In some embodiments, during the process of optimizing the 3D Gaussian rendering network, the model parameters of the initial depth estimation network are frozen and remain unchanged.

[0240] In some embodiments, the second processing module is specifically configured to:

[0241] Inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the second fused feature into the decoder of the three-dimensional Gaussian model, to obtain the three-dimensional Gaussian point output by the decoder;

[0242] The second fusion feature is obtained by fusing the spherical depth map, the first feature and the second feature.

[0243] In some embodiments, the spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module; the first processing module includes:

[0244] A third processing unit is used to process the cube projection image of the reference view based on the monocular depth estimation network to obtain the monocular features extracted by the middle layer of the monocular depth estimation network;

[0245] A fourth processing unit is used to process the monocular feature, the first rectangular projection image of the reference view, and the second rectangular projection image of the source view based on the second cost volume prediction network to obtain a second cost volume;

[0246] The prediction unit is configured to perform depth prediction on the second cost volume based on the second fusion module to obtain a spherical depth map of the reference view.

[0247] In some embodiments, the fourth processing unit is specifically configured to:

[0248] Extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image;

[0249] The monocular feature is fused into the first feature to obtain a third feature; and the monocular feature is fused into the second feature to obtain a fourth feature;

[0250] Based on the third feature and the fourth feature, determining a second feature matching cost of multiple pixels in the reference view under multiple depth assumptions; the second feature matching cost is determined based on a similarity between a third feature vector of the multiple pixels in the third feature and a fourth feature vector in the fourth feature;

[0251] A dimensionality reduction operation is performed on the second feature matching cost to obtain a second cost volume.

[0252] In some embodiments, the fourth processing unit is specifically configured to:

[0253] For the second feature matching cost, an average operation is performed in the third feature vector dimension to obtain a second cost volume.

[0254] In some embodiments, the fourth processing unit is specifically configured to:

[0255] A second three-dimensional convolutional neural network is used to process the second feature matching cost to obtain a second cost volume.

[0256] In some embodiments, the second processing module is specifically used to:

[0257] Inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the third fused feature into the decoder of the three-dimensional Gaussian model, to obtain the three-dimensional Gaussian point output by the decoder;

[0258] The third fusion feature is obtained by fusing the spherical depth map, the third feature and the fourth feature.

[0259] In some embodiments, the loss determination module includes:

[0260] The loss determination unit is used to determine the loss value based on the depth loss between the spherical depth map and the depth truth value in the target truth value, the photometric loss between the photometric value of the rendering result and the photometric truth value in the target truth value, and the color value of the rendering result and the color truth value in the target truth value.

[0261] Based on the same technical concept, the embodiment of the present disclosure also provides a rendering device 1300, which is applied to the above-mentioned three-dimensional Gaussian rendering network, such as Fig.13 As shown, the device comprises:

[0262] An acquisition module 1301 is used to acquire a first panoramic image and a second panoramic image;

[0263] A fifth processing module 1302 is used to input the first panoramic image and the second panoramic image into a three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points;

[0264] The second rendering module 1303 is used to render the three-dimensional Gaussian points to a specified viewing angle to obtain a rendering result.

[0265] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0266] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0267] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0268] Fig.14 A schematic block diagram of an example electronic device 1400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0269] like Fig.14 As shown, the device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0270] A number of components in the device 1400 are connected to the I / O interface 1405, including: an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0271] The computing unit 1401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as the training method / rendering method of the rendering network. For example, in some embodiments, the training method / rendering method of the rendering network may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the training method / rendering method of the rendering network described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to execute the training method / rendering method of the rendering network in any other appropriate manner (for example, by means of firmware).

[0272] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0273] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0274] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0275] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0276] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0277] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0278] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0279] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a rendering network, comprising: Processing a reference view and a source view based on a spherical depth estimation network of the 3D Gaussian rendering network to obtain a spherical depth map of the reference view; The reference view and the source view are panoramic views; Processing the spherical depth map and the reference view based on the three-dimensional Gaussian model of the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; Rendering the three-dimensional Gaussian points to a target viewing angle to obtain a rendering result; Determining a loss value based on the spherical depth map, the rendering result, and a target true value of the target viewing angle; The model parameters of the three-dimensional Gaussian rendering network are optimized based on the loss value.

2. The method according to claim 1, wherein: The spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module; The spherical depth estimation network based on the three-dimensional Gaussian rendering network processes the reference view and the source view to obtain a spherical depth map of the reference view, including: Processing the first rectangular projection image of the reference view and the second rectangular projection image of the source view based on the first cost volume prediction network to obtain a first cost volume; Processing the cube projection image of the reference view based on the initial depth estimation network to obtain an initial spherical depth; The initial spherical depth and the first cost volume are fused based on the first fusion module to obtain a spherical depth map of the reference view.

3. The method according to claim 2, wherein: The step of processing the first rectangular projection image of the reference view and the second rectangular projection image of the source view based on the first cost volume prediction network to obtain a first cost volume includes: Extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image; Based on the first feature and the second feature, determining a first feature matching cost of a plurality of pixels in the reference view under a plurality of depth hypotheses; the first feature matching cost is determined based on a similarity between a first feature vector of the plurality of pixels in the first feature and a second feature vector of the second feature; A dimensionality reduction operation is performed on the first feature matching cost to obtain the first cost volume.

4. The method according to claim 3, wherein: The performing a dimensionality reduction operation on the first feature matching cost to obtain the first cost volume includes: For the first feature matching cost, a mean operation is performed on the dimension of the first feature vector to obtain the first cost volume.

5. The method according to claim 3, wherein: The performing a dimensionality reduction operation on the first feature matching cost to obtain the first cost volume includes: A three-dimensional convolutional neural network is used to process the first feature matching cost to obtain the first cost volume.

6. The method according to claim 2 or 3, wherein: The processing of the cube projection image of the reference view based on the initial depth estimation network to obtain an initial spherical depth includes: Performing depth estimation on the cube projection image to obtain a depth point cloud image; The depth point cloud image is projected onto the spherical surface of the reference view to obtain the initial spherical depth.

7. The method according to claim 6, wherein: The step of performing depth estimation on the cube projection image to obtain a depth point cloud image includes: Depth estimation is performed on the cube projection image based on a geometric three-dimensional vision network to obtain the depth point cloud image.

8. The method according to claim 2, wherein: The fusing the initial spherical depth and the first cost volume based on the first fusion module to obtain the spherical depth map of the reference view includes: fusing the initial spherical depth and the first cost volume to obtain a first fused feature; Depth prediction is performed on the first fused features to obtain the spherical depth map.

9. The method according to claim 8, wherein: The performing depth prediction on the first fusion feature to obtain the spherical depth map includes: Perform depth prediction on the first fused features based on a first two-dimensional convolutional network to obtain the spherical depth map.

10. The method according to any one of claims 2 to 9, wherein: During the process of optimizing the three-dimensional Gaussian rendering network, the model parameters of the initial depth estimation network are frozen and remain unchanged.

11. The method according to claim 3, wherein: The three-dimensional Gaussian model based on the three-dimensional Gaussian rendering network processes the spherical depth map and the reference view to obtain three-dimensional Gaussian points, including: Inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the second fused feature into the decoder of the three-dimensional Gaussian model, to obtain the three-dimensional Gaussian point output by the decoder; The second fusion feature is obtained by fusing the spherical depth map, the first feature and the second feature.

12. The method according to claim 1, wherein: The spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module; The spherical depth estimation network based on the three-dimensional Gaussian rendering network processes the reference view and the source view to obtain a spherical depth map of the reference view, including: Processing the cube projection image of the reference view based on the monocular depth estimation network to obtain monocular features extracted by the middle layer of the monocular depth estimation network; Processing the monocular feature, the first rectangular projection image of the reference view, and the second rectangular projection image of the source view based on the second cost volume prediction network to obtain a second cost volume; Performing depth prediction on the second cost volume based on the second fusion module to obtain a spherical depth map of the reference view.

13. The method according to claim 12, wherein: The step of processing the monocular feature, the first rectangular projection image of the reference view, and the second rectangular projection image of the source view based on the second cost volume prediction network to obtain a second cost volume includes: Extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image; The monocular feature is fused into the first feature to obtain a third feature; and the monocular feature is fused into the second feature to obtain a fourth feature; Based on the third feature and the fourth feature, determining a second feature matching cost of a plurality of pixels in the reference view under a plurality of depth hypotheses; the second feature matching cost is determined based on a similarity between a third feature vector of the plurality of pixels in the third feature and a fourth feature vector in the fourth feature; A dimensionality reduction operation is performed on the second feature matching cost to obtain the second cost volume.

14. The method according to claim 13, wherein: The performing a dimensionality reduction operation on the second feature matching cost to obtain the second cost volume includes: For the second feature matching cost, an average operation is performed in the third feature vector dimension to obtain the second cost volume.

15. The method according to claim 13, wherein: The performing a dimensionality reduction operation on the second feature matching cost to obtain the second cost volume includes: A second three-dimensional convolutional neural network is used to process the second feature matching cost to obtain the second cost volume.

16. The method according to any one of claims 13 to 15, wherein: The three-dimensional Gaussian model based on the three-dimensional Gaussian rendering network processes the spherical depth map and the reference view to obtain three-dimensional Gaussian points, including: Inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the third fused feature into the decoder of the three-dimensional Gaussian model, to obtain the three-dimensional Gaussian point output by the decoder; The third fusion feature is obtained by fusing the spherical depth map, the third feature and the fourth feature.

17. The method according to any one of claims 1 to 16, wherein: The determining of the loss value based on the spherical depth map, the rendering result and the target true value of the target viewing angle comprises: The loss value is determined based on a depth loss between the spherical depth map and a depth truth value in the target truth value, a photometric loss between a photometric value of the rendering result and a photometric truth value in the target truth value, and a color value of the rendering result and a color truth value in the target truth value.

18. A rendering method, applied to a three-dimensional Gaussian rendering network trained by the method according to any one of claims 1 to 17, the method comprising: Acquire a first panoramic image and a second panoramic image; Inputting the first panoramic image and the second panoramic image into the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; The three-dimensional Gaussian points are rendered to a specified viewing angle to obtain a rendering result.

19. A training device for a rendering network, comprising: A first processing module, configured to process a reference view and a source view based on a spherical depth estimation network of the three-dimensional Gaussian rendering network to obtain a spherical depth map of the reference view; The reference view and the source view are panoramic views; A second processing module, configured to process the spherical depth map and the reference view based on a three-dimensional Gaussian model of the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; A first rendering module, used for rendering the three-dimensional Gaussian points to a target viewing angle to obtain a rendering result; A loss determination module, configured to determine a loss value based on the spherical depth map, the rendering result, and a target true value of the target viewing angle; An optimization module is used to optimize the model parameters of the three-dimensional Gaussian rendering network based on the loss value.

20. The device according to claim 19, wherein The spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module; The first processing module comprises: A first processing unit, configured to process the first rectangular projection image of the reference view and the second rectangular projection image of the source view based on the first cost volume prediction network to obtain a first cost volume; A second processing unit, configured to process the cube projection image of the reference view based on the initial depth estimation network to obtain an initial spherical depth; A fusion unit is used to fuse the initial spherical depth and the first cost volume based on the first fusion module to obtain a spherical depth map of the reference view.

21. The device according to claim 20, wherein: The first processing unit is specifically configured to: Extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image; Determining, based on the first feature and the second feature, first feature matching costs of a plurality of pixels in the reference view under a plurality of depth hypotheses; The first feature matching cost is determined based on the similarity between a first feature vector of the plurality of pixels in the first feature and a second feature vector in the second feature; A dimensionality reduction operation is performed on the first feature matching cost to obtain the first cost volume.

22. The device according to claim 21, wherein The first processing unit is specifically configured to: For the first feature matching cost, a mean operation is performed on the dimension of the first feature vector to obtain the first cost volume.

23. The device according to claim 21, wherein The first processing unit is specifically configured to: A three-dimensional convolutional neural network is used to process the first feature matching cost to obtain the first cost volume.

24. The device according to claim 20 or 23, wherein: The second processing unit is specifically configured to: Performing depth estimation on the cube projection image to obtain a depth point cloud image; The depth point cloud image is projected onto the spherical surface of the reference view to obtain the initial spherical depth.

25. The device according to claim 24, wherein: The second processing unit is specifically configured to: Depth estimation is performed on the cube projection image based on a geometric three-dimensional vision network to obtain the depth point cloud image.

26. The device according to claim 20, wherein: The fusion unit is specifically used for: fusing the initial spherical depth and the first cost volume to obtain a first fused feature; Depth prediction is performed on the first fused features to obtain the spherical depth map.

27. The device according to claim 26, wherein: The fusion unit is specifically used for: Perform depth prediction on the first fused features based on a first two-dimensional convolutional network to obtain the spherical depth map.

28. The device according to any one of claims 20 to 27, wherein: During the process of optimizing the three-dimensional Gaussian rendering network, the model parameters of the initial depth estimation network are frozen and remain unchanged.

29. The device according to claim 21, wherein The second processing module is specifically used for: Inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the second fused feature into the decoder of the three-dimensional Gaussian model, to obtain the three-dimensional Gaussian point output by the decoder; The second fusion feature is obtained by fusing the spherical depth map, the first feature and the second feature.

30. The device according to claim 19, wherein: The spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module; The first processing module comprises: A third processing unit, configured to process the cube projection image of the reference view based on the monocular depth estimation network to obtain the monocular features extracted by the middle layer of the monocular depth estimation network; a fourth processing unit, configured to process the monocular feature, the first rectangular projection image of the reference view, and the second rectangular projection image of the source view based on the second cost volume prediction network to obtain a second cost volume; A prediction unit is configured to perform depth prediction on the second cost volume based on the second fusion module to obtain a spherical depth map of the reference view.

31. The device according to claim 30, wherein The fourth processing unit is specifically configured to: Extracting a first feature of the first rectangular projection image and a second feature of the second rectangular projection image; The monocular feature is fused into the first feature to obtain a third feature; and the monocular feature is fused into the second feature to obtain a fourth feature; Based on the third feature and the fourth feature, determining a second feature matching cost of a plurality of pixels in the reference view under a plurality of depth hypotheses; the second feature matching cost is determined based on a similarity between a third feature vector of the plurality of pixels in the third feature and a fourth feature vector in the fourth feature; A dimensionality reduction operation is performed on the second feature matching cost to obtain the second cost volume.

32. The device according to claim 31, wherein The fourth processing unit is specifically configured to: For the second feature matching cost, an average operation is performed in the third feature vector dimension to obtain the second cost volume.

33. The device according to claim 31, wherein The fourth processing unit is specifically configured to: A second three-dimensional convolutional neural network is used to process the second feature matching cost to obtain the second cost volume.

34. The device according to any one of claims 31 to 33, wherein: The second processing module is specifically used for: Inputting the reference view into the encoder of the three-dimensional Gaussian model, and inputting the third fused feature into the decoder of the three-dimensional Gaussian model, to obtain the three-dimensional Gaussian point output by the decoder; The third fusion feature is obtained by fusing the spherical depth map, the third feature and the fourth feature.

35. The device according to any one of claims 19 to 34, wherein: The loss determination module comprises: A loss determination unit is used to determine the loss value based on the depth loss between the spherical depth map and the depth truth value in the target truth value, the photometric loss between the photometric value of the rendering result and the photometric truth value in the target truth value, and the color value of the rendering result and the color truth value in the target truth value.

36. A rendering device, applied to a three-dimensional Gaussian rendering network trained by the device according to any one of claims 19 to 35, the device comprising: An acquisition module, used for acquiring a first panoramic image and a second panoramic image; A fifth processing module, configured to input the first panoramic image and the second panoramic image into the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; The second rendering module is used to render the three-dimensional Gaussian points to a specified viewing angle to obtain a rendering result.

37. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 18.

38. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-18.

39. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images