Image rendering method, device, equipment, storage medium and program product based on Gaussian splashing

Abstract

The present application discloses an image rendering method, apparatus, device, storage medium and program product based on Gaussian splattering, which relates to the field of computer vision technology, including: obtaining a training image, selecting its reference perspective, obtaining the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model, generating a multi-view stereo point cloud through filtering and fusion, inputting it into a three-dimensional Gaussian splattering model, and initializing the model; performing forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain a mixed loss of the training image; transmitting the loss back to the three-dimensional Gaussian splattering model to train and update the model; given a target perspective and camera parameters corresponding to the target perspective, performing image rendering under the target perspective through the trained three-dimensional Gaussian splattering model to generate a target image under the target perspective. The present application can alleviate the problem of overfitting under sparse perspectives through forward mapping.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image rendering method, apparatus, device, storage medium, and program product based on Gaussian splattering. Background Art

[0002] Novel View Synthesis (NVS) is a hot topic in computer vision and computer graphics. Given a set of photos with camera parameters, the goal of the NVS task is to use this known information to reconstruct the scene and synthesize photos of the scene from a new perspective.

[0003] Neural Radiance Fields (NeRF) has made significant progress in the task of synthesizing new viewpoints. NeRF is an implicit representation for 3D reconstruction. It uses a multi-layer perceptron to fit a set of input images to represent the geometric appearance of a scene, ultimately synthesizing the target new viewpoint through volume rendering. While NeRF has achieved promising results in this area, it requires a large number of dense input viewpoints for training. In real-world applications such as AR / VR and robotics, the available viewpoints are often sparse. With sparse input, NeRF is prone to overfitting to the training viewpoints, resulting in a significant decline in the quality of synthesized new viewpoints. Furthermore, due to NeRF's implicit representation and volume rendering, its training and rendering speeds are very slow. Recently, the 3D Gaussian Splatting (3DGS) model has achieved promising results in the NVS task. The 3D Gaussian Splatting model is a point-based representation of a display scene. It describes the geometry and appearance of a scene through a set of points parameterized as a 3D Gaussian distribution, and then renders the image through splatting. Thanks to this explicit scene representation and efficient splatter rendering, the 3D Gaussian splatter model is very fast and efficient to train, and can achieve fast real-time rendering at high resolution. However, the 3D Gaussian splatter model also requires dense viewpoints as input. When the number of input views is small, it is also prone to overfitting to the training viewpoints, resulting in a significant performance drop.

[0004] Therefore, it is urgent to propose a solution that can alleviate the overfitting problem of the three-dimensional Gaussian splatter model when the number of input perspectives is small.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide an image rendering method, device, equipment, storage medium and program product based on Gaussian splattering, aiming to solve the technical problem of overfitting of three-dimensional Gaussian splattering models during training.

[0007] To achieve the above objectives, the present application proposes an image rendering method based on Gaussian splatting, the method comprising:

[0008] Acquire a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model;

[0009] Filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud;

[0010] Inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model;

[0011] The training image and the multi-view stereo depth are forward mapped and loss calculated to obtain a mixed loss of the training image;

[0012] Passing the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model;

[0013] Given the target perspective and the camera parameters corresponding to the target perspective, the image under the target perspective is rendered using the trained three-dimensional Gaussian splatter model to generate the target image under the target perspective.

[0014] In one embodiment, the reference viewing angle of the training image is selected, and the multi-view stereo depth of the training image under the reference viewing angle is obtained by a multi-view stereo model:

[0015] Selecting one viewing angle of the training image as a reference viewing angle, and selecting viewing angles of the training image other than the reference viewing angle as source viewing angles;

[0016] The features of the training image at a reference perspective and a source perspective are extracted through a multi-view stereo model, and the multi-view stereo depth of the training image at the reference perspective is generated based on the features at the reference perspective and the source perspective.

[0017] In one embodiment, the step of filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud includes:

[0018] Calculating the multi-view stereo depth of the training image at each viewing angle;

[0019] Multi-view stereo depths from all perspectives are filtered through geometric consistency to generate depth consistency masks;

[0020] A multi-view stereo point cloud of the target viewpoint is generated by fusion according to the depth consistency mask.

[0021] In one embodiment, the step of forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image includes:

[0022] The training image will be rendered using the initialized 3D Gaussian splash model to generate a rendered image.

[0023] The training image is forward mapped to obtain a projection image of the training image under a virtual perspective;

[0024] Acquiring appearance information of the training image under the virtual perspective according to the projection image under the virtual perspective;

[0025] A hybrid loss is generated based on the appearance information and the rendered image by a hybrid loss function.

[0026] In one embodiment, after forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image, the method further includes:

[0027] Obtain the 3D Gaussian depth of the training image through the 3D Gaussian splash model;

[0028] Calculating a geometric consistency loss of the three-dimensional Gaussian depth based on the multi-view stereo depth;

[0029] The geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

[0030] In one embodiment, after the step of obtaining the three-dimensional Gaussian depth of the training image using the three-dimensional Gaussian splatter model, the method further includes:

[0031] Generate a monocular predicted depth of a training image through a pre-trained model, and calculate the sorting loss of the three-dimensional Gaussian depth based on the monocular predicted depth;

[0032] The sorting loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an image rendering device based on Gaussian splashing, wherein the image rendering device based on Gaussian splashing comprises:

[0034] a depth calculation module, configured to obtain a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model;

[0035] A point cloud generation module, configured to filter and fuse the multi-view stereo depths to generate a multi-view stereo point cloud;

[0036] An initialization module, configured to input the multi-view stereo point cloud into a three-dimensional Gaussian splash model and initialize position parameters of the three-dimensional Gaussian splash model;

[0037] A hybrid loss module, configured to perform forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain a hybrid loss of the training image;

[0038] A training and updating module, configured to transmit the mixed loss back to the initialized three-dimensional Gaussian splash model to perform training and updating on the three-dimensional Gaussian splash model;

[0039] The image rendering module is used to render the image under the target perspective using the trained three-dimensional Gaussian splatter model given the target perspective and the camera parameters corresponding to the target perspective, thereby generating the target image under the target perspective.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes an image rendering device based on Gaussian splashing, which includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, and the computer program is configured to implement the steps of the image rendering method based on Gaussian splashing as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the image rendering method based on Gaussian splattering as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the image rendering method based on Gaussian splattering as described above.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] The embodiment of the present application proposes a Gaussian splash image rendering method, device, equipment, storage medium and program product, including: obtaining a training image, selecting its reference perspective, obtaining the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model; filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud; inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize the position parameters of the three-dimensional Gaussian splash model; performing forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain a mixed loss of the training image; transmitting the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model; given a target perspective and the camera parameters corresponding to the target perspective, performing image rendering under the target perspective through the trained three-dimensional Gaussian splash model to generate a target image under the target perspective. The present application can alleviate the problem of overfitting of the three-dimensional Gaussian splash model under sparse perspectives during training through forward mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 A flowchart of the first embodiment of the image rendering method based on Gaussian splatting provided in this application;

[0048] Figure 2 Schematic diagram of the module structure of the image rendering device based on Gaussian splattering in an embodiment of the present application;

[0049] Figure 3 Schematic diagram of the device structure of the hardware operating environment involved in the image rendering method based on Gaussian splattering in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: obtain a training image, select its reference perspective, and obtain the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model; filter and fuse the multi-view stereo depth to generate a multi-view stereo point cloud; input the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize the position parameters of the three-dimensional Gaussian splash model; obtain the mixed loss of the training image through forward mapping and loss calculation of the training image and the multi-view stereo depth; pass the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model; given the target perspective and the camera parameters corresponding to the target perspective, perform image rendering under the target perspective through the trained three-dimensional Gaussian splash model to generate a target image under the target perspective.

[0054] Novel View Synthesis (NVS) is a hot topic in computer vision and computer graphics. Given a set of photos with camera parameters, the goal of the NVS task is to use this known information to reconstruct the scene and synthesize photos of the scene from a new perspective.

[0055] Neural Radiance Fields (NeRFs) have made significant progress in the task of synthesizing new views. NeRFs are an implicit representation for 3D reconstruction. They aim to represent the geometric appearance of a scene by fitting a set of input images using a multi-layer perceptron, ultimately synthesizing the target new viewpoint through volume rendering. While NeRFs have achieved promising results in this area, they require a large number of dense input views for training. In real-world applications such as AR / VR and robotics, the available viewpoints are often sparse. With sparse input, NeRFs tend to overfit to the training viewpoints, significantly reducing the quality of synthesized new views. Furthermore, due to their implicit representation and volume rendering, NeRFs are very slow to train and render. Recently, 3D Gaussian Splatting (3DGS) has achieved promising results in the task of synthesizing new views. 3D Gaussian Splatting is a point-based representation of a display scene. It describes the geometry and appearance of a scene using a set of points parameterized as a 3D Gaussian distribution, and then renders the image using a splattering method. Thanks to this explicit scene representation and efficient splat rendering, 3D Gaussian splatting is very fast and efficient to train, and can achieve fast, real-time rendering at high resolution. However, similar to neural radiance fields, 3D Gaussian splatting requires dense viewpoints as input. When the number of input views is small, 3D Gaussian splatting is also prone to overfitting to the training viewpoints, resulting in a significant performance degradation. Therefore, a fast, real-time rendering solution that can achieve high resolution is urgently needed.

[0056] Through forward mapping, this application can alleviate the problem of overfitting of the three-dimensional Gaussian splash model under sparse perspective during training; by generating a multi-view stereo point cloud through a multi-perspective stereo model, the quality of the point cloud can be improved and the accuracy of the model image rendering can be improved; and then the multi-view stereo point cloud is input into the three-dimensional Gaussian splash model to increase the image rendering rate.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, an image rendering device based on Gaussian splattering, etc. The following uses an image rendering device based on Gaussian splattering as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the embodiment of the present application provides an image rendering method based on Gaussian splashing, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the image rendering method based on Gaussian splatting of the present application.

[0059] In this embodiment, the image rendering method based on Gaussian splattering includes steps S10 to S60:

[0060] Step S10 : acquiring a training image, selecting a reference viewing angle of the training image, and obtaining a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model.

[0061] It should be noted that the training images refer to the dataset used to train the 3D Gaussian splatter model. These datasets contain a large number of images and their corresponding labels or annotations. During training, the model learns the characteristics of these images and optimizes its parameters based on these labels or annotations to improve its ability to recognize, classify, detect, or segment image content.

[0062] Additionally, it should be noted that the reference perspective refers to the perspective currently being processed or used as a reference for depth estimation. During the multi-view stereo (MVS) imaging process, to reconstruct a 3D scene, a perspective needs to be selected as a starting point. This perspective is known as the reference perspective. The image from the reference perspective is used as a benchmark for comparison and matching with images from other perspectives to estimate the depth value of each pixel at that perspective.

[0063] In addition, it should be noted that the multi-view stereo model refers to the MVSFormer model, which is a new multi-view stereo imaging method based on deep learning technology. It enhances the feature extraction capability by introducing a pre-trained ViT model, and combines an efficient depth prediction network and a temperature-based depth estimation strategy to achieve high-precision and high-robustness depth estimation. This model has broad application prospects and important research value in fields such as three-dimensional reconstruction. The multi-view stereo model enhances the feature extraction capability of multi-view stereo MVS (Multi-View Stereo) by introducing pre-trained ViT, thereby learning more reliable feature representations. The core of the model lies in its unique architectural design, which includes three main parts: feature extraction, correlation body construction, and depth estimation. The multi-view stereo model introduces a depth prediction network. Commonly used depth prediction networks include CNN and Transformer, which learn and predict the depth value of each pixel from the correlation body.

[0064] Additionally, it should be noted that the multi-view stereo depth refers to the MVS depth of pixels in an image estimated by a multi-view stereo model.

[0065] Specifically, first obtain a training image, then select a reference perspective of the training image, and then use the multi-view stereo depth of the target viewpoint under the reference perspective of the multi-view stereo model. The specific steps of using the multi-view stereo depth of the target viewpoint under the reference perspective of the multi-view stereo model include:

[0066] The first step is to extract features from the training images: first, the input training images are downsampled through a multi-view stereo model to save computational overhead; then the downsampled images are sent to the pre-trained visual transformation model ViT (Visual Transformer) for feature extraction. Different types of models can be selected, such as Twins (with an efficient attention mechanism and robust position encoding at different scales) or DINO (self-supervised pre-trained ViT model); finally, the multi-scale features extracted, such as 1 / 8, 1 / 16, 1 / 32, and 1 / 64 resolutions, are then used for subsequent processing. The second step is to construct a correlation body: first, according to the preset initial inverse depth value, the feature points are transformed from the reference perspective to other perspectives to construct feature matching pairs; then the similarity or correlation of feature points between different perspectives is calculated to construct a correlation body. This correlation body contains matching cost information under different depth assumptions. The third step is depth estimation: The multi-view stereo model introduces a depth prediction network. Common depth prediction networks include convolutional neural networks (CNNs) and transformers. It learns and predicts the multi-view stereo depth of each pixel from the correlation volume. In order to improve the accuracy and robustness of depth estimation, the multi-view stereo model may adopt a temperature-based depth prediction strategy to balance the accuracy and smoothness of the prediction by adjusting the temperature parameters.

[0067] Step S20 , filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud.

[0068] It should be noted that the multi-view stereo point cloud refers to the point cloud obtained through the multi-view stereo model MVS. The point cloud obtained using the multi-view stereo model is of higher quality than the original COLMAP point cloud of the three-dimensional Gaussian splash model, and can promote the optimization of parameters in the subsequent three-dimensional Gaussian splash model. Since the point cloud generated by the multi-view stereo model has higher density and accuracy, it can provide richer input data for the three-dimensional Gaussian splash model. These data can not only improve the accuracy of surface reconstruction, but also help optimize the parameter settings in the three-dimensional Gaussian splash model, such as the size, shape and position of the Gaussian distribution. In the three-dimensional Gaussian splash model, the optimization of parameters usually depends on the accuracy and richness of the input data. Using the point cloud generated by the multi-view stereo model as input can make the parameter optimization process more stable, reliable and efficient. By continuously optimizing these parameters, the quality and realism of the three-dimensional surface model can be further improved.

[0069] Specifically, the multi-view stereo depth of the target viewpoint at each reference viewpoint is calculated through a multi-view stereo model; the multi-view stereo depths at all viewpoints are filtered through geometric consistency to generate a depth consistency mask; and based on the depth consistency mask, a multi-view stereo point cloud of the target viewpoint is generated by fusion.

[0070] Step S30 : inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model.

[0071] Specifically, the multi-view stereo point cloud is first preprocessed by denoising and smoothing, and the point cloud is downsampled or upsampled according to the actual situation to meet the accuracy requirements of the three-dimensional Gaussian splash model; then the processed point cloud is used as input to initialize the position parameters in the three-dimensional Gaussian splash model. The mean (i.e., position) of the Gaussian body in each three-dimensional Gaussian splash model can be set according to the points in the point cloud.

[0072] Step S40 , forward mapping the training image and the multi-view stereo depth and performing loss calculation to obtain a mixed loss of the training image.

[0073] Specifically, the training image is first rendered using an initialized three-dimensional Gaussian splatter model to generate a rendered image. Then, given the training image and the training perspective of the training image, the projection image of the training image at the training perspective is obtained through forward mapping, and the appearance information of the training image at the training perspective is obtained based on the projection image. During the training process, some virtual perspectives are randomly taken as training perspectives to obtain the corresponding appearance information under the training perspective. Then, based on the appearance information and the rendered image, a mixed loss of the training image is generated through a mixed loss function. This mixed loss can provide additional constraints for the three-dimensional Gaussian splatter model and alleviate the overfitting problem.

[0074] Step S50: transmitting the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model.

[0075] It should be noted that the back propagation algorithm refers to updating the network weights by calculating the gradient of the loss function with respect to the network parameters, thereby minimizing the loss function.

[0076] In addition, it should be noted that the three-dimensional Gaussian splatting model refers to the 3DGS model (three-dimensional Gaussian splatting model, 3D Gaussian Splatting), which mainly uses Gaussian functions to represent points or volumes in three-dimensional space, thereby achieving efficient and accurate representation of three-dimensional scenes, and has technical characteristics such as high efficiency, detail retention, flexibility, and real-time performance.

[0077] Specifically, the mixed loss is transmitted back to the three-dimensional Gaussian splash model whose position parameters are initialized by the multi-view stereo point cloud through the back-propagation algorithm, and the three-dimensional Gaussian splash model is iteratively trained and updated.

[0078] Step S60 , given a target perspective and camera parameters corresponding to the target perspective, image rendering is performed under the target perspective using the trained three-dimensional Gaussian splatter model to generate a target image under the target perspective.

[0079] Specifically, during training, we calculate the loss between the image rendered from the corresponding perspective of the 3D Gaussian splatter model and the supervisory signal at the training perspective and the virtual perspective of the forward projection, thereby optimizing the parameters of the Gaussian model. After training, images from any perspective can be rendered. Therefore, given a target perspective and the corresponding camera parameters, the trained 3D Gaussian splatter model is used to render the image from the target perspective, generating the target image at the target perspective. More specifically, a scene input is first used to generate Gaussian points. The input scene is usually a collection of images of a static scene, and the corresponding camera parameters calibrated by motion recovery technology. A sparse three-dimensional point cloud is generated from these images and camera parameters. Then, this point cloud is converted into a collection of Gaussian points, each defined by parameters such as position (mean), covariance matrix (representing shape and orientation), opacity α, and color (represented by spherical harmonics SH). The position parameters in the three-dimensional Gaussian splatter model are then initialized using the multi-view stereo point cloud, and the Gaussian points are optimized. The optimized Gaussian points are then projected onto the image plane, and the position of each Gaussian point on the image is calculated based on the camera's perspective and projection matrix. The attributes of each Gaussian point, such as color and opacity, are accumulated to the corresponding pixel. After the projection and attribute accumulation of all Gaussian points, the final rendered image, i.e., the target image, is generated. This image contains the three-dimensional information of the scene and is presented in the form of a two-dimensional image.

[0080] Through the above-mentioned scheme, this embodiment can alleviate the problem of overfitting of the three-dimensional Gaussian splash model under sparse perspective during training through forward mapping; by generating a multi-view stereo point cloud through a multi-perspective stereo model, the quality of the point cloud can be improved, and the accuracy and richness of the three-dimensional Gaussian splash model can be improved.

[0081] Based on the above embodiment, in a feasible implementation manner, selecting the reference perspective of the training image in the above step S10 and obtaining the multi-view stereo depth of the training image under the reference perspective through the multi-view stereo model may include steps S11 to S12:

[0082] Step S11 : selecting a viewing angle of the training image as a reference viewing angle, and selecting viewing angles of the training image other than the reference viewing angle as source viewing angles.

[0083] It should be noted that the reference viewpoint refers to the viewpoint currently being processed or used as a reference for depth estimation. During the multi-view stereo (MVS) imaging process, to reconstruct a 3D scene, a viewpoint must be selected as a starting point. This viewpoint is known as the reference viewpoint. The image from the reference viewpoint is used as a benchmark for comparison and matching with images from other viewpoints to estimate the depth value of each pixel at that viewpoint.

[0084] Additionally, it should be noted that the source perspective refers to perspectives other than the reference perspective that are used for comparison and matching with the reference perspective image to provide depth information. In multi-perspective stereo model imaging, there are typically multiple source perspective images, which together provide the information required for depth estimation for the reference perspective. The source perspective images provide the necessary depth information for 3D reconstruction by performing feature matching and depth estimation with the reference perspective images. This information is then used to construct a 3D point cloud or mesh to represent the geometric structure of the 3D scene.

[0085] Specifically, there are many perspectives of the target viewpoint in the training image. One of the perspectives is selected as the reference perspective, and the remaining perspectives are used as source perspectives. The multi-view stereo depth under the reference perspective is calculated through the multi-view stereo model; then another perspective is selected as the reference perspective, and the remaining perspectives are used as source perspectives to calculate the multi-view stereo depth under this perspective; all perspectives are selected as reference perspectives, and the multi-view stereo depth under this perspective is calculated.

[0086] Step S12: extracting features of the training image at a reference perspective and a source perspective through a multi-view stereo model, and generating a multi-view stereo depth of the training image at the reference perspective based on the features at the reference perspective and the source perspective.

[0087] Specifically, first, data preparation is performed on the training images to determine the training images and the camera parameters of the training images; then the training images are downsampled, and the downsampled images are passed through the Vision Transformer (ViT) to extract features from the training images, and then the extracted features are subjected to feature dimensionality reduction and upsampling; then, the above features and related values ​​are used to construct a cost volume, which contains the similarity or difference measurement between the projected images of all source views under different depth assumptions and the reference view; finally, a deep neural network is used to regularize the cost volume to estimate the optimal depth of each pixel. The network finds the most likely depth value by minimizing the loss function (such as a loss function based on photometric consistency) to obtain the multi-view stereo depth (MVS depth) under the determined reference perspective.

[0088] Based on the above embodiment, in a feasible implementation manner, the above step S20 filters and fuses the multi-view stereo depth to generate a multi-view stereo point cloud, which may include steps S21 to S23:

[0089] Step S21 : calculating the multi-view stereo depth of the training image at each viewing angle.

[0090] Specifically, the training image has many perspectives. One of the perspectives is selected as the reference perspective, and the remaining perspectives are used as source perspectives. The multi-view stereo depth under the reference perspective is calculated through the multi-view stereo model; then another perspective is selected as the reference perspective, and the remaining perspectives are used as source perspectives to calculate the multi-view stereo depth under this perspective; all perspectives are selected as reference perspectives, and the multi-view stereo depth under this perspective is calculated.

[0091] In step S22 , the multi-view stereo depths under all viewing angles are filtered through geometric consistency to generate a depth consistency mask.

[0092] It should be noted that the depth consistency mask is a binary image (or matrix) used to identify geometrically consistent areas in a depth map. In this mask, the value of each pixel (usually 0 or 1) represents the geometric consistency state of the pixel in the corresponding depth map. Specifically, if the depth value of a pixel in the depth map is consistent with its projection under multiple perspectives, then the corresponding value of the pixel in the consistency mask will be set to 1 (indicating consistency), otherwise it will be set to 0 (indicating inconsistency). The generation of the mask usually includes three steps: depth estimation, geometric consistency check and mask generation.

[0093] Specifically, the multi-view stereo depth under all view angles is first calculated. Then a geometric consistency check is performed: based on the multi-view stereo depth of all pixels in the training image under all view angles, a depth map of the training image is generated. Through the depth map and camera parameters, the pixels in the two-dimensional image are back-projected into three-dimensional space to construct a three-dimensional point cloud of the scene. For each point in the three-dimensional point cloud, check whether its projection is consistent under different view angles. This usually involves comparing the depth values ​​of pixels at the same spatial position under different view angles. Taking into account factors such as occlusion, noise and error, a certain threshold can be set to judge the consistency of the depth value. If the difference between the depth values ​​under two view angles is within the threshold range, the depth estimates under the two view angles are considered to be consistent. During the inspection process, all inconsistent pixels or areas are recorded. These areas may be caused by occlusion, lighting changes, camera calibration errors, etc. Finally, a depth consistency mask is generated: a binary image (mask) of the same size as the MVS depth map is created, with all pixels initially set to 1 (indicating that all pixels are assumed to be consistent); based on the results of the geometric consistency check, the pixel values ​​of the identified inconsistent areas are set to 0 in the mask (indicating inconsistency); and finally, a depth consistency mask is output. This mask can be used in subsequent tasks such as 3D reconstruction and scene understanding to indicate which areas have reliable depth values ​​and which areas may need further processing or be ignored.

[0094] Step S23 : generating a multi-view stereo point cloud of the target viewpoint by fusing the depth consistency mask.

[0095] Specifically, based on the depth consistency mask, the multi-view stereo depth after consistency filtering can be obtained. Using the depth map and camera parameters, the pixels in the training image are back-projected into the three-dimensional space to generate a three-dimensional point cloud at each perspective. During the back-projection process, only the pixels marked as consistent in the depth consistency mask should be considered, and the inconsistent pixels should be ignored; the point cloud at each perspective is preprocessed, including denoising, filtering and other operations to improve the quality of the point cloud; then the preprocessed point clouds of each perspective are fused to generate a multi-view stereo point cloud.

[0096] Based on the above embodiment, in a feasible implementation manner, the above step S40 forward maps the training image and the multi-view stereo depth and calculates the loss to obtain the mixed loss of the training image, including steps S41 to S44:

[0097] In step S41 , the training image is rendered using the initialized three-dimensional Gaussian splatter model to generate a rendered image.

[0098] Specifically, the three-dimensional Gaussian splash model parameterizes the Gaussian points in the scene into a three-dimensional Gaussian distribution. The mean of the three-dimensional Gaussian distribution is the position x of the point in the world coordinate system. Its shape and size are represented by the three-dimensional covariance matrix ∑. The three-dimensional Gaussian distribution is represented as follows:

[0099]

[0100] Where G(X) is a three-dimensional Gaussian distribution; x is the position of the Gaussian point in the world coordinate system; and E is the three-dimensional covariance matrix of position x.

[0101] Among them, the three-dimensional covariance matrix can be decomposed into rotation and scaling to represent the size and direction of the three-dimensional Gaussian distribution, expressed as:

[0102] E=RSS T R T

[0103] Where E is the three-dimensional covariance matrix of position x; R is the rotation matrix; R T is the transpose of the rotation matrix R; S is the scaling matrix, S T is the transpose of the scaling matrix S.

[0104] If there are n Gaussian points in the scene, the complete three-dimensional Gaussian parameters are expressed as:

[0105]

[0106] Among them, i represents the i-th Gaussian point; α represents the opacity of the Gaussian point; c represents the color of the Gaussian point; g is the three-dimensional Gaussian parameter of the Gaussian point; x is the position of the pixel in the world coordinate system; S is the scaling matrix; R is the rotation matrix.

[0107] When rendering, the three-dimensional Gaussian distribution is projected to two dimensions. Specifically, the expression for projecting the three-dimensional covariance matrix ∑ to two dimensions is:

[0108] E'=JWEW T J T

[0109] Where J is the Jacobian matrix of the projection transformation; J T is the transpose of the Jacobian matrix; W is an element in the Jacobian matrix, which is used to adjust the value of the covariance matrix during the projection process; W T is the transpose of W; E is the three-dimensional covariance matrix; E' is the covariance matrix of the three-dimensional covariance matrix projected to two dimensions.

[0110] After projection, the color of the current perspective is rendered using opacity blending. The expression for rendering the color of the current perspective is:

[0111]

[0112] Among them, T i is the cumulative opacity, α represents the opacity of the Gaussian point, c represents the color of the Gaussian point, and C is the rendered color.

[0113] T i The calculation formula is:

[0114]

[0115] Through the above steps, a rendering image I' is rendered under a viewing angle.

[0116] The following L1 and SSIM mixed losses are calculated for the rendered image I' and the real photo I at this perspective to train the 3D Gaussian splatter model. The formula for calculating the mixed loss is:

[0117] L photo =λ1L1(I',I)+(1-λ1)(1-SSIM(I',I))

[0118] Among them, L photo is a hybrid loss; L1 is the average of the sum of the absolute values ​​of the pixel value differences between the two images; SSIM is used to measure the visual impact and structural similarity of the two images; λ1 is a weight parameter between 0 and 1, which is used to balance the L1 loss and SSIM.

[0119] Step S42: forward mapping the training image to obtain a projection image of the training image under a virtual viewing angle.

[0120] Specifically, the device obtains the training image, the training perspective, the camera pose, the multi-view stereo depth under the training perspective, and the camera pose under the target perspective, and obtains the projected image I after the training image is projected under the target perspective through forward mapping. tgt :

[0121]

[0122] Among them, I tgt is the projected image after projection, I src For training images, is the multi-view stereo depth under the training perspective, P src is the camera pose, P tgt The fwd() function usually refers to a perspective transformation operation that transforms an image from one perspective to another.

[0123] Specifically, the coordinate relationship between the projected image and the training image is:

[0124]

[0125] Among them, K is the camera intrinsic parameter matrix; p tgt refers to the pixel coordinates in the projected image after forward mapping; p src refers to the pixel coordinates in the training image.

[0126] Step S43: acquiring appearance information of the training image under the virtual viewing angle according to the projection image under the virtual viewing angle.

[0127] Specifically, some virtual perspectives are randomly adopted, and the obtained multi-view stereo depth is used to obtain the appearance information under the virtual perspective through forward mapping.

[0128] Step S44: generating a hybrid loss through a hybrid loss function based on the appearance information and the rendered image.

[0129] Specifically, based on the appearance information and the rendered image, a hybrid loss is generated by a hybrid loss function, wherein the calculation formula of the hybrid loss function is:

[0130] L fwd =λ2L1(I' f ,I f )+(1-λ2)(1-SSIM(I' f ,I f ))

[0131] Among them, L fwd is a mixed loss, λ2 is a weight parameter between 0 and 1, which is used to balance L1 loss and SSIM; I f is the appearance information of the training image under the virtual perspective; I' f is the rendered image; L1 is the average of the sum of the absolute values ​​of the pixel value differences between the two images; SSIM is used to measure the visual impact and structural similarity of two images.

[0132] This embodiment uses the above-mentioned scheme. Since the three-dimensional Gaussian splash model also requires dense perspectives as input, when the number of input perspectives is small, the three-dimensional Gaussian splash model is prone to overfitting on the training perspectives, resulting in a significant decrease in its performance. Therefore, this application generates a mixed loss through a mixed loss function based on the appearance information and rendered image under the virtual perspective, which can alleviate the overfitting problem of the three-dimensional Gaussian splash model on the training perspective and improve the rendering quality of the model.

[0133] Based on the above embodiment, in a feasible implementation manner, after forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image, the method further includes steps S70 to S90:

[0134] Step S70: Acquire the three-dimensional Gaussian depth of the training image through a three-dimensional Gaussian splash model.

[0135] Specifically, the three-dimensional Gaussian depth of the training image at one viewing angle is obtained by rendering the three-dimensional Gaussian splash model. The calculation formula is:

[0136]

[0137] Where d is the three-dimensional Gaussian depth of the Gaussian point and is the opacity of the Gaussian point.

[0138] Step S80: Calculate the geometric consistency loss of the three-dimensional Gaussian depth based on the multi-view stereo depth.

[0139] Specifically, a geometric consistency loss is applied to the rendered 3D Gaussian depth. The calculation formula for the geometric consistency loss is:

[0140] L CS =∑|D r -D mvs |⊙M

[0141] Among them, L CS is the geometric consistency loss; D r is the three-dimensional Gaussian depth; D mvs is the multi-view stereo depth; M is the consistency mask in the 3D Gaussian splatter model; ⊙ is the exclusive-OR operator.

[0142] Step S90: The geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm to train and update the three-dimensional Gaussian splash model.

[0143] Specifically, the geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is iteratively trained and updated.

[0144] This embodiment uses the above scheme to calculate the geometric consistency loss, pass the loss back to the three-dimensional Gaussian splash model, and iteratively train and update the model, so that the collective information of the model can maintain consistent perspective during the training process.

[0145] Based on the above embodiment, in a feasible implementation manner, the above step S70 further includes steps S100 to S110:

[0146] Step S100: Generate a monocular predicted depth of a training image through a pre-trained model, and calculate the sorting loss of the three-dimensional Gaussian depth based on the monocular predicted depth.

[0147] It should be noted that the monocular predicted depth refers to predicting the depth information of each pixel in the scene from a two-dimensional image captured by a single camera.

[0148] Specifically, the device generates the monocular predicted depth of each training view of the training image through the pre-trained model DPT, and then calculates the sorting loss. For example, each time the sorting loss is calculated, two batches of non-overlapping pixel points S1 and S2 with the same number (512) are randomly sampled. The calculation formula for calculating the sorting loss is:

[0149]

[0150] Among them, L mono Predict depth for a single camera; D r (S1) is the three-dimensional Gaussian depth of pixel S1; D r (S2) is the three-dimensional Gaussian depth of pixel S2; M'=∏D mono (S1)<D mono (S2).

[0151] Step S110 , transmitting the sorting loss back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and training and updating the three-dimensional Gaussian splash model.

[0152] Specifically, the sorting loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is iteratively trained and updated.

[0153] This embodiment uses the above scheme to calculate the sorting loss, pass the loss back to the three-dimensional Gaussian splatter model, and iteratively train and update the model. The monocular predicted depth can be used as prior information as supplementary supervision to ensure that the depth map of the missing area after filtering can still remain complete.

[0154] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image rendering method based on Gaussian splattering of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0155] This application also provides an image rendering device based on Gaussian splashing, please refer to Figure 2 , the image rendering device based on Gaussian splashing includes:

[0156] The depth calculation module 201 is configured to obtain a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle using a multi-view stereo model;

[0157] A point cloud generation module 202 is configured to filter and fuse the multi-view stereo depth to generate a multi-view stereo point cloud;

[0158] An initialization module 203 is configured to input the multi-view stereo point cloud into a three-dimensional Gaussian splash model and initialize position parameters of the three-dimensional Gaussian splash model;

[0159] A mixed loss module 204 is configured to perform forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain a mixed loss of the training image;

[0160] A training and updating module 205 is configured to transmit the mixing loss back to the initialized three-dimensional Gaussian splash model to perform training and updating on the three-dimensional Gaussian splash model;

[0161] The image rendering module 206 is used to perform image rendering at the target perspective using a trained three-dimensional Gaussian splatter model given a target perspective and camera parameters corresponding to the target perspective, thereby generating a target image at the target perspective.

[0162] The Gaussian splatter-based image rendering device provided in this application adopts the Gaussian splatter-based image rendering method in the above-mentioned embodiment, which can solve the technical problem of overfitting of the three-dimensional Gaussian splatter model during training. Compared with the prior art, the beneficial effects of the Gaussian splatter-based image rendering device provided in this application are the same as the beneficial effects of the Gaussian splatter-based image rendering method provided in the above-mentioned embodiment, and the other technical features of the Gaussian splatter-based image rendering device are the same as the features disclosed in the above-mentioned embodiment method, and are not further described here.

[0163] The present application provides an image rendering device based on Gaussian splashing, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image rendering method based on Gaussian splashing in the above-mentioned embodiment one.

[0164] Reference below Figure 3 , which shows a schematic structural diagram of an image rendering device based on Gaussian splattering suitable for implementing embodiments of the present application. The image rendering device based on Gaussian splattering in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3The Gaussian splatter-based image rendering device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0165] like Figure 3 As shown, the image rendering device based on Gaussian splattering may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the image rendering device based on Gaussian splattering are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the Gaussian splatter-based image rendering device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a Gaussian splatter-based image rendering device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have instead.

[0166] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0167] The Gaussian splatter-based image rendering device provided in this application, which employs the Gaussian splatter-based image rendering method of the above-mentioned embodiment, can resolve the technical problem of overfitting of the three-dimensional Gaussian splatter model during training. Compared with the prior art, the beneficial effects of the Gaussian splatter-based image rendering device provided in this application are the same as the beneficial effects of the Gaussian splatter-based image rendering method provided in the above-mentioned embodiment, and the other technical features of the Gaussian splatter-based image rendering device are the same as those disclosed in the method of the previous embodiment, and are not further described here.

[0168] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0169] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0170] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the image rendering method based on Gaussian splattering in the above embodiment.

[0171] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0172] The computer-readable storage medium may be included in the image rendering device based on Gaussian splattering, or may exist independently without being assembled into the image rendering device based on Gaussian splattering.

[0173] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an image rendering device based on Gaussian splashing, the image rendering device based on Gaussian splashing: obtains a training image, selects its reference perspective, and obtains the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model; filters and fuses the multi-view stereo depth to generate a multi-view stereo point cloud; inputs the multi-view stereo point cloud into a three-dimensional Gaussian splashing model to initialize the position parameters of the three-dimensional Gaussian splashing model; obtains a mixed loss of the training image through forward mapping and loss calculation of the training image and the multi-view stereo depth; transmits the mixed loss back to the initialized three-dimensional Gaussian splashing model to train and update the three-dimensional Gaussian splashing model; given a target perspective and camera parameters corresponding to the target perspective, performs image rendering under the target perspective through the trained three-dimensional Gaussian splashing model to generate a target image under the target perspective.

[0174] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0175] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0176] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0177] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned Gaussian splattering-based image rendering method, thereby resolving the technical issues surrounding Gaussian splattering-based image rendering. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the Gaussian splattering-based image rendering method provided in the aforementioned embodiments, and are not further elaborated upon herein.

[0178] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned Gaussian splatter-based image rendering method when executed by a processor.

[0179] The computer program product provided in this application can solve the technical problem of overfitting during training of a three-dimensional Gaussian splatter model. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the Gaussian splatter-based image rendering method provided in the above embodiment, and will not be repeated here.

[0180] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image rendering method based on Gaussian splattering, characterized in that: The method comprises: Acquire a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model; Filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud; Inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model; The training image and the multi-view stereo depth are forward mapped and loss calculated to obtain a mixed loss of the training image; Passing the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model; Given the target perspective and the corresponding camera parameters, the trained 3D Gaussian splatter model is used to render the image at the target perspective and generate the target image at the target perspective. The step of forward mapping the training image and the multi-view stereo depth and performing loss calculation to obtain a mixed loss of the training image comprises: The training image will be rendered using the initialized 3D Gaussian splash model to generate a rendered image. The training image is forward mapped to obtain a projection image of the training image under a virtual perspective; Acquiring appearance information of the training image under the virtual perspective according to the projection image under the virtual perspective; A hybrid loss is generated based on the appearance information and the rendered image by a hybrid loss function.

2. The method according to claim 1, wherein The step of selecting a reference perspective of the training image and obtaining a multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model includes: Selecting one viewing angle of the training image as a reference viewing angle, and selecting viewing angles of the training image other than the reference viewing angle as source viewing angles; The features of the training image at a reference perspective and a source perspective are extracted through a multi-view stereo model, and the multi-view stereo depth of the training image at the reference perspective is generated based on the features at the reference perspective and the source perspective.

3. The method according to claim 1, wherein The step of filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud comprises: Calculating the multi-view stereo depth of the training image at each viewing angle; Multi-view stereo depths from all perspectives are filtered through geometric consistency to generate depth consistency masks; A multi-view stereo point cloud of the target viewpoint is generated by fusion according to the depth consistency mask.

4. The method according to claim 1, wherein After forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image, the method further includes: Obtain the 3D Gaussian depth of the training image through the 3D Gaussian splash model; Calculating a geometric consistency loss of the three-dimensional Gaussian depth based on the multi-view stereo depth; The geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

5. The method according to claim 4, wherein After the step of obtaining the three-dimensional Gaussian depth of the training image by using the three-dimensional Gaussian splash model, the method further includes: Generate a monocular predicted depth of a training image through a pre-trained model, and calculate the sorting loss of the three-dimensional Gaussian depth based on the monocular predicted depth; The sorting loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

6. An image rendering device based on Gaussian splashing, characterized in that: The device comprises: a depth calculation module, configured to obtain a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model; A point cloud generation module, configured to filter and fuse the multi-view stereo depths to generate a multi-view stereo point cloud; An initialization module, configured to input the multi-view stereo point cloud into a three-dimensional Gaussian splash model and initialize position parameters of the three-dimensional Gaussian splash model; A hybrid loss module, configured to perform forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain a hybrid loss of the training image; A training and updating module, configured to transmit the mixed loss back to the initialized three-dimensional Gaussian splash model to perform training and updating on the three-dimensional Gaussian splash model; The image rendering module is used to generate the target image at the target perspective by using the trained 3D Gaussian splatter model given the target perspective and the corresponding camera parameters; The hybrid loss module is also used to render the training image using the initialized three-dimensional Gaussian splash model to generate a rendered image. The training image is forward mapped to obtain a projection image of the training image under a virtual perspective; Acquiring appearance information of the training image under the virtual perspective according to the projection image under the virtual perspective; A hybrid loss is generated based on the appearance information and the rendered image by a hybrid loss function.

7. An image rendering device based on Gaussian splashing, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Gaussian splattering-based image rendering method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image rendering method based on Gaussian splattering according to any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, which implements the steps of the Gaussian splattering-based image rendering method according to any one of claims 1 to 5 when executed by a processor.

Images