Image rendering method and device based on Gaussian splashing, equipment, storage medium and program product

Abstract

The invention discloses an image rendering method and device based on Gaussian splashing, equipment, a storage medium and a program product, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a training image, selecting a reference visual angle of the training image, obtaining the multi-view stereoscopic depth of the training image under the reference visual angle through a multi-view stereoscopic model, and obtaining a multi-view stereoscopic depth of the training image under the reference visual angle; generating a multi-view three-dimensional point cloud through filtering fusion, inputting the multi-view three-dimensional point cloud into a three-dimensional Gaussian splash model, and initializing the model; performing forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain mixed loss of the training image; the loss is returned to the three-dimensional Gaussian splash model, and the model is trained and updated; and giving a target visual angle and camera parameters corresponding to the target visual angle, performing image rendering under the target visual angle through the trained three-dimensional Gaussian splash model, and generating a target image under the target visual angle. According to the invention, through forward mapping, the problem of overfitting under a sparse view angle can be relieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image rendering method, apparatus, device, storage medium and program product based on Gaussian splatting. Background Art

[0002] Novel View Synthesis (NVS) is one of the hot topics in computer vision and computer graphics. Given a set of photos with camera parameters, the goal of the novel view synthesis task is to use this known information to reconstruct the scene and synthesize photos of the scene from a new perspective.

[0003] At present, Neural Radiance Fields (NeRF) has made significant progress in the task of synthesizing new viewpoints. NeRF is an implicit representation for 3D reconstruction, which aims to use a multi-layer perceptron to fit a set of input images to represent the geometric appearance information of the scene, and finally synthesize the target new viewpoint through volume rendering. Although NeRF has achieved good results in this field, it requires a large number of dense input perspectives for training. In practical application scenarios such as AR / VR, robots, etc., the perspectives we obtain are often sparse. In the case of sparse input, NeRF is prone to overfitting on the training perspective, and the quality of the synthesis of new perspectives is greatly reduced. In addition, due to the implicit representation and volume rendering of NeRF, its training speed and rendering speed are very slow. Recently, the 3D Gaussian Splatting model 3DGS (3D Gaussian Splatting) has achieved good results on the NVS task. The 3D Gaussian Splatting model is a point-based display scene representation. The geometric and appearance information in the scene is described by a set of points parameterized as a 3D Gaussian distribution, and the photo is rendered by splatting. Thanks to this explicit scene representation and efficient splash rendering, the 3D Gaussian splash model can be trained very quickly and efficiently, and can achieve fast real-time rendering at high resolution. However, the 3D Gaussian splash model also requires dense perspectives as input. When the number of input perspectives is small, it is also easy to overfit on the training perspectives, resulting in a significant drop in performance.

[0004] Therefore, it is urgent to propose a solution that can alleviate the overfitting problem of the three-dimensional Gaussian splatter model when the number of input perspectives is small.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide an image rendering method, device, equipment, storage medium and program product based on Gaussian splashing, aiming to solve the technical problem of overfitting of three-dimensional Gaussian splashing model during training.

[0007] To achieve the above objectives, the present application proposes an image rendering method based on Gaussian splatting, the method comprising:

[0008] Acquire a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model;

[0009] Filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud;

[0010] Inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model;

[0011] The training image and the multi-view stereo depth are forward mapped and loss calculated to obtain a mixed loss of the training image;

[0012] Passing the mixed loss back to the initialized three-dimensional Gaussian splash model, and training and updating the three-dimensional Gaussian splash model;

[0013] Given a target perspective and camera parameters corresponding to the target perspective, the trained three-dimensional Gaussian splash model is used to render the image under the target perspective to generate a target image under the target perspective.

[0014] In one embodiment, the reference viewing angle of the training image is selected, and the multi-view stereo depth of the training image under the reference viewing angle is obtained through a multi-view stereo model:

[0015] Selecting a viewing angle of the training image as a reference viewing angle, and selecting viewing angles of the training image other than the reference viewing angle as source viewing angles;

[0016] The features of the training image at a reference perspective and a source perspective are extracted through a multi-view stereo model, and the multi-view stereo depth of the training image at the reference perspective is generated based on the features at the reference perspective and the source perspective.

[0017] In one embodiment, the step of filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud includes:

[0018] Calculating the multi-view stereo depth of the training image at each viewing angle;

[0019] Multi-view stereo depths from all perspectives are filtered through geometric consistency to generate depth-consistent masks;

[0020] A multi-view stereo point cloud of the target viewpoint is generated by fusion according to the depth consistency mask.

[0021] In one embodiment, the step of forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image comprises:

[0022] The training image is rendered through the initialized three-dimensional Gaussian splash model to generate a rendered image;

[0023] The training image is forward mapped to obtain a projection image of the training image under a virtual viewing angle;

[0024] Acquiring appearance information of the training image under the virtual perspective according to the projection image under the virtual perspective;

[0025] Based on the appearance information and the rendered image, a hybrid loss is generated by a hybrid loss function.

[0026] In one embodiment, after forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image, the method further includes:

[0027] Obtain the three-dimensional Gaussian depth of the training image through the three-dimensional Gaussian splash model;

[0028] Calculating a geometrically consistent loss of the three-dimensional Gaussian depth based on the multi-view stereo depth;

[0029] The geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

[0030] In one embodiment, after the step of obtaining the three-dimensional Gaussian depth of the training image by using the three-dimensional Gaussian splash model, the step further includes:

[0031] Generate a monocular predicted depth of a training image through a pre-trained model, and calculate a sorting loss of the three-dimensional Gaussian depth based on the monocular predicted depth;

[0032] The sorting loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an image rendering device based on Gaussian splashing, and the image rendering device based on Gaussian splashing includes:

[0034] A depth calculation module, used to obtain a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model;

[0035] A point cloud generation module, used for filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud;

[0036] An initialization module, used for inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model;

[0037] A mixed loss module, used for forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain a mixed loss of the training image;

[0038] A training update module, used for transmitting the mixed loss back to the initialized three-dimensional Gaussian splash model, and performing training update on the three-dimensional Gaussian splash model;

[0039] The image rendering module is used to render the image under the target perspective by using the trained three-dimensional Gaussian splash model given the target perspective and the camera parameters corresponding to the target perspective, so as to generate the target image under the target perspective.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes an image rendering device based on Gaussian splashing, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the image rendering method based on Gaussian splashing as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image rendering method based on Gaussian splattering as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the image rendering method based on Gaussian splattering as described above are implemented.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] The present application proposes a Gaussian splash image rendering method, device, equipment, storage medium and program product, including: obtaining a training image, selecting its reference perspective, obtaining the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model; filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud; inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize the position parameters of the three-dimensional Gaussian splash model; forward mapping and loss calculation of the training image and the multi-view stereo depth to obtain a mixed loss of the training image; transmitting the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model; given a target perspective and the camera parameters corresponding to the target perspective, the trained three-dimensional Gaussian splash model is used to render the image under the target perspective to generate a target image under the target perspective. The present application can alleviate the problem of overfitting of the three-dimensional Gaussian splash model under sparse perspectives during training through forward mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart diagram of the first embodiment of the image rendering method based on Gaussian splatting provided in the present application;

[0048] Figure 2 This is a schematic diagram of the module structure of an image rendering device based on Gaussian splashing in an embodiment of the present application;

[0049] Figure 3 Schematic diagram of the device structure of the hardware operating environment involved in the image rendering method based on Gaussian splattering in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: obtain a training image, select its reference perspective, and obtain the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model; filter and fuse the multi-view stereo depth to generate a multi-view stereo point cloud; input the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize the position parameters of the three-dimensional Gaussian splash model; obtain the mixed loss of the training image through forward mapping and loss calculation of the training image and the multi-view stereo depth; pass the mixed loss back to the initialized three-dimensional Gaussian splash model to train and update the three-dimensional Gaussian splash model; given the target perspective and the camera parameters corresponding to the target perspective, perform image rendering under the target perspective through the trained three-dimensional Gaussian splash model to generate a target image under the target perspective.

[0054] Novel View Synthesis (NVS) is one of the hot topics in computer vision and computer graphics. Given a set of photos with camera parameters, the goal of the Novel View Synthesis task is to use this known information to reconstruct the scene and synthesize photos of the scene from a new perspective.

[0055] At present, Neural Radiance Fields (NeRF) has made significant progress in the task of synthesizing new viewpoints. Neural Radiance Fields is an implicit representation for 3D reconstruction, which aims to use a multi-layer perceptron to fit a set of input images to represent the geometric appearance information of the scene, and finally synthesize the target new viewpoint through volume rendering. Although Neural Radiance Fields has achieved good results in this field, it requires a large number of dense input perspectives for training. In practical application scenarios such as AR / VR, robots, etc., the perspectives we obtain are often sparse. In the case of sparse input, Neural Radiance Fields is prone to overfitting on the training perspectives, and the quality of the synthesis of new perspectives is greatly reduced. In addition, due to the implicit representation and volume rendering of Neural Radiance Fields, its training speed and rendering speed are very slow. Recently, 3D Gaussian Splatting (3DGS) has achieved good results in the task of synthesizing new viewpoints. 3D Gaussian Splatting technology is a point-based display scene representation. The geometric and appearance information in the scene is described by a set of points parameterized as a 3D Gaussian distribution, and the photos are rendered by splatting. Thanks to this displayed scene representation and efficient splash rendering, the training of 3D Gaussian splash technology is very fast and efficient, and can achieve fast real-time rendering with high resolution. However, similar to the neural radiation field, 3D Gaussian splash technology also requires dense perspectives as input. When the number of input perspectives is small, 3D Gaussian splash is also prone to overfitting on the training perspectives, resulting in a significant drop in its performance. Therefore, it is urgent to propose a fast real-time rendering solution that can achieve high resolution.

[0056] Through forward mapping, the present application can alleviate the problem of overfitting of the three-dimensional Gaussian splash model under sparse perspective during training; by generating a multi-view stereo point cloud through a multi-view stereo model, the quality of the point cloud can be improved and the accuracy of the model image rendering can be improved; and by inputting the multi-view stereo point cloud into the three-dimensional Gaussian splash model, the image rendering rate can be improved.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an image rendering device based on Gaussian splashing, etc. The following takes the image rendering device based on Gaussian splashing as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the embodiment of the present application provides an image rendering method based on Gaussian splashing, referring to Figure 1 , Figure 1 It is a flowchart diagram of the first embodiment of the image rendering method based on Gaussian splatting of the present application.

[0059] In this embodiment, the image rendering method based on Gaussian splashing includes steps S10 to S60:

[0060] Step S10: acquiring a training image, selecting a reference viewing angle of the training image, and obtaining a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model.

[0061] It should be noted that the training images refer to the data sets used to train the 3D Gaussian splatter model, which contain a large number of images and their corresponding labels or annotations. During the training process, the model will learn the features of these images and optimize its own parameters based on the labels or annotations to improve the ability to recognize, classify, detect or segment the image content.

[0062] In addition, it should be noted that the reference perspective refers to the perspective that is currently being processed or used as a reference for depth estimation. In the multi-view stereo model MVS imaging process, in order to reconstruct a three-dimensional scene, it is necessary to select a perspective as a starting point, which is the reference perspective; the image of the reference perspective is used as a benchmark, and is compared and matched with images of other perspectives to estimate the depth value of each pixel at this perspective.

[0063] In addition, it should be noted that the multi-view stereo model refers to the MVSFormer model, which is a new multi-view stereo imaging method based on deep learning technology. It enhances the feature extraction capability by introducing the pre-trained ViT model, and combines the efficient depth prediction network and the temperature-based depth estimation strategy to achieve high-precision and high-robustness depth estimation. This model has broad application prospects and important research value in fields such as three-dimensional reconstruction. The multi-view stereo model enhances the feature extraction capability of multi-view stereo MVS (Multi-View Stereo) by introducing pre-trained ViT, thereby learning more reliable feature representations. The core of the model lies in its unique architecture design, including three main parts: feature extraction, correlation body construction and depth estimation; the multi-view stereo model introduces a depth prediction network. Commonly used depth prediction networks are CNN and Transformer, which learn and predict the depth value of each pixel from the correlation body.

[0064] In addition, it should be noted that the multi-view stereo depth refers to the MVS depth of a pixel point in an image estimated by a multi-view stereo model.

[0065] Specifically, firstly, a training image is obtained, then a reference view angle of the training image is selected, and then the multi-view stereo depth of the target viewpoint under the reference view angle of the multi-view stereo model is obtained. The specific steps of obtaining the multi-view stereo depth of the target viewpoint under the reference view angle of the multi-view stereo model include:

[0066] The first step is to extract features from the training images: first, the input training images are downsampled through a multi-view stereo model to save computational overhead; then the downsampled images are sent to the pre-trained visual transformation model ViT (Visual Transformer) for feature extraction. Different types of models can be selected, such as Twins (with efficient attention mechanism and robust position encoding of different scales) or DINO (self-supervised pre-trained ViT model); finally, the extracted multi-scale features, such as 1 / 8, 1 / 16, 1 / 32, 1 / 64 resolution, are then used for subsequent processing. The second step is to construct a correlation body: first, according to the preset initial inverse depth value, the feature points are transformed from the reference view to other viewpoints to construct feature matching pairs; then the similarity or correlation of feature points between different viewpoints is calculated to construct a correlation body. This correlation body contains the matching cost information under different depth assumptions. The third step is to perform depth estimation: The multi-view stereo model introduces a depth prediction network. Commonly used depth prediction networks include convolutional neural networks (CNN) and transformers. It learns and predicts the multi-view stereo depth of each pixel from the relevant body. In order to improve the accuracy and robustness of depth estimation, the multi-view stereo model may adopt a temperature-based depth prediction strategy to balance the accuracy and smoothness of the prediction by adjusting the temperature parameters.

[0067] Step S20: filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud.

[0068] It should be noted that the multi-view stereo point cloud refers to the point cloud obtained by the multi-view stereo model MVS. The point cloud obtained by the multi-view stereo model is of higher quality than the original COLMAP point cloud of the three-dimensional Gaussian splash model, and can promote the optimization of parameters in the subsequent three-dimensional Gaussian splash model. Since the point cloud generated by the multi-view stereo model has a higher density and precision, it can provide richer input data for the three-dimensional Gaussian splash model. These data can not only improve the accuracy of surface reconstruction, but also help optimize the parameter settings in the three-dimensional Gaussian splash model, such as the size, shape and position of the Gaussian distribution. In the three-dimensional Gaussian splash model, the optimization of parameters usually depends on the accuracy and richness of the input data. Using the point cloud generated by the multi-view stereo model as input can make the parameter optimization process more stable, reliable and efficient. By continuously optimizing these parameters, the quality and fidelity of the three-dimensional surface model can be further improved.

[0069] Specifically, the multi-view stereo depth of the target viewpoint at each reference viewpoint is calculated through a multi-view stereo model; the multi-view stereo depths at all viewpoints are filtered through geometric consistency to generate a depth consistency mask; and based on the depth consistency mask, a multi-view stereo point cloud of the target viewpoint is fused and generated.

[0070] Step S30: input the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model.

[0071] Specifically, the multi-view stereo point cloud is first preprocessed such as denoising and smoothing, and the point cloud is downsampled or upsampled according to specific circumstances to meet the accuracy requirements of the three-dimensional Gaussian splash model; then the processed point cloud is used as input to initialize the position parameters in the three-dimensional Gaussian splash model, and the mean (i.e., position) of the Gaussian body in each three-dimensional Gaussian splash model can be set according to the points in the point cloud.

[0072] Step S40 , forward mapping the training image and the multi-view stereo depth and performing loss calculation to obtain a mixed loss of the training image.

[0073] Specifically, the training image is first rendered through an initialized three-dimensional Gaussian splatter model to generate a rendered image. Then, given the training image and the training perspective of the training image, the projection image of the training image at the training perspective is obtained through forward mapping, and the appearance information of the training image at the training perspective is obtained according to the projection image. During the training process, some virtual perspectives are randomly taken as training perspectives to obtain the corresponding appearance information under the training perspective. Then, based on the appearance information and the rendered image, a mixed loss of the training image is generated through a mixed loss function. The mixed loss can provide additional constraints for the three-dimensional Gaussian splatter model and alleviate the overfitting problem.

[0074] Step S50, transmitting the mixed loss back to the initialized three-dimensional Gaussian splash model, and training and updating the three-dimensional Gaussian splash model.

[0075] It should be noted that the back propagation algorithm refers to updating the network weights by calculating the gradient of the loss function with respect to the network parameters, thereby minimizing the loss function.

[0076] In addition, it should be noted that the three-dimensional Gaussian splatting model refers to the 3DGS model (three-dimensional Gaussian splatting model, 3D Gaussian Splatting), which mainly uses Gaussian functions to represent points or volumes in three-dimensional space, thereby achieving efficient and accurate representation of three-dimensional scenes, and has technical characteristics such as high efficiency, detail retention, flexibility, and real-time performance.

[0077] Specifically, the mixed loss is transmitted back to the three-dimensional Gaussian splash model whose position parameters are initialized by the multi-view stereo point cloud through the back-propagation algorithm, and the three-dimensional Gaussian splash model is iteratively trained and updated.

[0078] Step S60 , given a target viewing angle and camera parameters corresponding to the target viewing angle, image rendering is performed under the target viewing angle through the trained three-dimensional Gaussian splash model to generate a target image under the target viewing angle.

[0079] Specifically, during training, we calculate the loss between the image rendered from the corresponding perspective of the 3D Gaussian splat model and the supervisory signal at the training perspective and the virtual perspective of the forward mapping, thereby optimizing the various parameters in the Gaussian model; after training, images of any perspective can be rendered. Therefore, given the target perspective and the camera parameters corresponding to the target perspective, the trained 3D Gaussian splat model is used to render the image at the target perspective, generating the target image at the target perspective. More specifically, the scene input is first used to generate Gaussian points. The input scene is usually a set of images of a static scene, and the corresponding camera parameters calibrated by the motion recovery structure technology. A sparse three-dimensional point cloud is generated from these images and camera parameters. Then, these point clouds are converted into a set of Gaussian points. Each Gaussian point is defined by parameters such as position (mean), covariance matrix (representing shape and direction), opacity α, and color (represented by spherical harmonic function SH); then the position parameters in the three-dimensional Gaussian splash model are initialized by the multi-view stereo point cloud, and the Gaussian points are optimized; then the optimized Gaussian points are projected onto the image plane, and the position of each Gaussian point on the image is calculated according to the camera's perspective and projection matrix. The attributes such as color and opacity of each Gaussian point are accumulated to the corresponding pixel. After the projection and attribute accumulation of all Gaussian points, the final rendered image, i.e., the target image, is generated. The image contains the three-dimensional information of the scene and is presented in the form of a two-dimensional image.

[0080] Through the above scheme, this embodiment can alleviate the problem of overfitting of the three-dimensional Gaussian splash model under sparse perspective during training through forward mapping; by generating a multi-view stereo point cloud through a multi-perspective stereo model, the quality of the point cloud can be improved, and the accuracy and richness of the three-dimensional Gaussian splash model can be improved.

[0081] Based on the above implementation scheme, in a feasible implementation manner, selecting the reference viewing angle of the training image in the above step S10, and obtaining the multi-view stereo depth of the training image under the reference viewing angle through the multi-view stereo model may include steps S11 to S12:

[0082] Step S11, selecting a viewing angle of the training image as a reference viewing angle, and selecting viewing angles of the training image other than the reference viewing angle as source viewing angles.

[0083] It should be noted that the reference perspective refers to the perspective that is currently being processed or used as a reference for depth estimation. In the multi-view stereo model MVS imaging process, in order to reconstruct a three-dimensional scene, it is necessary to select a perspective as a starting point, which is the reference perspective; the image of the reference perspective is used as a benchmark to compare and match with images of other perspectives to estimate the depth value of each pixel at that perspective.

[0084] In addition, it should be noted that the source perspective refers to other perspectives other than the reference perspective that are used to compare and match with the reference perspective image to provide depth information. In multi-perspective stereo model imaging, there are usually multiple source perspective images, which together provide the information required for depth estimation for the reference perspective; the source perspective image provides the necessary depth information for three-dimensional reconstruction by performing feature matching and depth estimation with the reference perspective image. This information is then used to construct a three-dimensional point cloud or mesh to represent the geometric structure of the three-dimensional scene.

[0085] Specifically, there are many perspectives of the target viewpoint in the training image. One of the perspectives is selected as the reference perspective, and the remaining perspectives are used as source perspectives. The multi-view stereo depth under the reference perspective is calculated through the multi-view stereo model; then another perspective is selected as the reference perspective, and the remaining perspectives are used as source perspectives to calculate the multi-view stereo depth under this perspective; all perspectives are selected as reference angles, and the multi-view stereo depth under this perspective is calculated.

[0086] Step S12: extracting features of the training image at a reference perspective and a source perspective through a multi-view stereo model, and generating a multi-view stereo depth of the training image at the reference perspective based on the features at the reference perspective and the source perspective.

[0087] Specifically, first, data preparation is performed on the training images to determine the training images and the camera parameters of the training images; then, the training images are downsampled, and the downsampled images are passed through the Vision Transformer (ViT) to extract features from the training images, and then the extracted features are reduced in dimension and upsampled; then, the above features and related values ​​are used to construct a cost volume, which contains the similarity or difference measurement between the projected images of all source views under different depth assumptions and the reference view; finally, a deep neural network is used to regularize the cost volume to estimate the optimal depth of each pixel. The network finds the most likely depth value by minimizing the loss function (such as a loss function based on photometric consistency) to obtain the multi-view stereo depth (MVS depth) under the determined reference viewing angle.

[0088] Based on the above implementation scheme, in a feasible implementation manner, the above step S20 filters and fuses the multi-view stereo depth to generate a multi-view stereo point cloud, which may include steps S21 to S23:

[0089] Step S21, calculating the multi-view stereo depth of the training image at each viewing angle.

[0090] Specifically, the training image has many perspectives, one of which is selected as the reference perspective, and the remaining perspectives are selected as source perspectives, and the multi-view stereo depth under the reference perspective is calculated through the multi-view stereo model; then another perspective is selected as the reference perspective, and the remaining perspectives are selected as source perspectives, and the multi-view stereo depth under this perspective is calculated; all perspectives are selected as reference angles, and the multi-view stereo depth under this perspective is calculated.

[0091] Step S22: The multi-view stereo depths under all viewing angles are filtered through geometric consistency to generate a depth consistency mask.

[0092] It should be noted that the depth consistency mask is a binary image (or matrix) used to identify geometrically consistent areas in a depth map. In this mask, the value of each pixel (usually 0 or 1) represents the geometric consistency state of the pixel in the corresponding depth map. Specifically, if the depth value of a pixel in the depth map is consistent with its projection under multiple perspectives, the corresponding value of the pixel in the consistency mask will be set to 1 (indicating consistency), otherwise it will be set to 0 (indicating inconsistency). The generation of the mask usually includes three steps: depth estimation, geometric consistency check, and mask generation.

[0093] Specifically, the multi-view stereo depth under all viewing angles is first calculated. Then a geometric consistency check is performed: based on the multi-view stereo depth of all pixels in the training image under all viewing angles, a depth map of the training image is generated. Through the depth map and camera parameters, the pixels in the two-dimensional image are back-projected into three-dimensional space to construct a three-dimensional point cloud of the scene. For each point in the three-dimensional point cloud, check whether its projection is consistent under different viewing angles. This usually involves comparing the depth values ​​of pixels at the same spatial position under different viewing angles. Considering factors such as occlusion, noise, and error, a certain threshold can be set to judge the consistency of the depth value. If the difference between the depth values ​​under two viewing angles is within the threshold range, the depth estimates under the two viewing angles are considered to be consistent. During the inspection process, all inconsistent pixels or areas are recorded. These areas may be caused by occlusion, illumination changes, camera calibration errors, etc. Finally, generate a depth consistency mask: create a binary image (mask) of the same size as the MVS depth map, with the initial value of all pixels set to 1 (indicating that all pixels are assumed to be consistent); according to the results of the geometric consistency check, set the pixel values ​​of the identified inconsistent areas in the mask to 0 (indicating inconsistency); finally output the depth consistency mask. This mask can be used in subsequent 3D reconstruction, scene understanding and other tasks to indicate which areas have reliable depth values ​​and which areas may need further processing or be ignored.

[0094] Step S23: generating a multi-view stereo point cloud of the target viewpoint by fusing the depth consistency mask.

[0095] Specifically, based on the depth consistency mask, the multi-view stereo depth after consistency filtering can be obtained. Using the depth map and camera parameters, the pixels in the training image are back-projected into the three-dimensional space to generate a three-dimensional point cloud at each viewing angle. During the back-projection process, only the pixels marked as consistent in the depth consistency mask should be considered, and the inconsistent pixels should be ignored. The point cloud at each viewing angle is preprocessed, including denoising, filtering and other operations to improve the quality of the point cloud. Then the preprocessed point clouds of each viewing angle are fused to generate a multi-view stereo point cloud.

[0096] Based on the above implementation scheme, in a feasible implementation manner, the above step S40 forward maps the training image and the multi-view stereo depth and calculates the loss to obtain the mixed loss of the training image, including steps S41 to S44:

[0097] Step S41, rendering the training image using the initialized three-dimensional Gaussian splash model to generate a rendered image.

[0098] Specifically, the three-dimensional Gaussian splash model parameterizes the Gaussian points in the scene into a three-dimensional Gaussian distribution. The mean of the three-dimensional Gaussian distribution is the position x of the point in the world coordinate system. Its shape and size are represented by the three-dimensional covariance matrix ∑. The three-dimensional Gaussian distribution is represented as follows:

[0099]

[0100] Among them, G(X) is a three-dimensional Gaussian distribution; x is the position of the Gaussian point in the world coordinate system; E is the three-dimensional covariance matrix of position x.

[0101] Among them, the three-dimensional covariance matrix can be decomposed into rotation and scaling to represent the size and direction of the three-dimensional Gaussian distribution, expressed as:

[0102] E=RSS T R T

[0103] Where E is the three-dimensional covariance matrix of position x; R is the rotation matrix; R T is the transpose of the rotation matrix R; S is the scaling matrix, S T is the transpose of the scaling matrix S.

[0104] If there are n Gaussian points in the scene, the complete three-dimensional Gaussian parameters are expressed as:

[0105]

[0106] Among them, i represents the i-th Gaussian point; α represents the opacity of the Gaussian point; c represents the color of the Gaussian point; g is the three-dimensional Gaussian parameter of the Gaussian point; x is the position of the pixel in the world coordinate system; S is the scaling matrix; and R is the rotation matrix.

[0107] When rendering, the three-dimensional Gaussian distribution is projected to two dimensions. Specifically, the expression for projecting the three-dimensional covariance matrix ∑ to two dimensions is:

[0108] E'=JWEW T J T

[0109] Where J is the Jacobian matrix of the projection transformation; J T is the transpose of the Jacobian matrix; W is an element in the Jacobian matrix, which is used to adjust the value of the covariance matrix during the projection process; W T is the transpose of W; E is the three-dimensional covariance matrix; E' is the covariance matrix of the three-dimensional covariance matrix projected to two dimensions.

[0110] After projection, the color of the current perspective is rendered using opacity blending. The expression for rendering the color of the current perspective is:

[0111]

[0112] Among them, T i is the cumulative opacity, α represents the opacity of the Gaussian point, c represents the color of the Gaussian point, and C is the rendered color.

[0113] T i The calculation formula is:

[0114]

[0115] Through the above steps, a rendering image I' is rendered under a viewing angle.

[0116] The following L1 and SSIM mixed losses are calculated for the rendered image I' and the real photo I at the viewing angle to train the three-dimensional Gaussian splash model. The calculation formula for the mixed loss is:

[0117] L photo =λ1L1(I',I)+(1-λ1)(1-SSIM(I',I))

[0118] Among them, L photo is a hybrid loss; L1 is the average of the sum of the absolute values ​​of the pixel value differences between the two images; SSIM is used to measure the visual impact and structural similarity of the two images; λ1 is a weight parameter between 0 and 1, which is used to balance the L1 loss and SSIM.

[0119] Step S42: forward mapping the training image to obtain a projection image of the training image under a virtual viewing angle.

[0120] Specifically, the device obtains a training image, a training perspective, a camera pose, a multi-view stereo depth at the training perspective, and a camera pose at the target perspective, and obtains a projection image I after the training image is projected at the target perspective by forward mapping. tgt :

[0121]

[0122] Among them, I tgt is the projected image after projection, I src For training images, is the multi-view stereo depth under the training perspective, P src is the camera pose, P tgt The fwd() function usually refers to a perspective transformation operation that transforms an image from one perspective to another.

[0123] Specifically, the coordinate relationship between the projection image and the training image is:

[0124]

[0125] Among them, K is the camera intrinsic parameter matrix; p tgt refers to the pixel coordinates in the projected image after forward mapping; p src refers to the pixel coordinates in the training image.

[0126] Step S43, acquiring appearance information of the training image under the virtual viewing angle according to the projection image under the virtual viewing angle.

[0127] Specifically, some virtual perspectives are randomly adopted, and the obtained multi-view stereo depth is used to obtain the appearance information under the virtual perspective through forward mapping.

[0128] Step S44: generating a hybrid loss through a hybrid loss function based on the appearance information and the rendered image.

[0129] Specifically, based on the appearance information and the rendered image, a mixed loss is generated by a mixed loss function, wherein the calculation formula of the mixed loss function is:

[0130] L fwd =λ2L1(I' f ,I f )+(1-λ2)(1-SSIM(I' f ,I f ))

[0131] Among them, L fwd is a hybrid loss, λ2 is a weight parameter between 0 and 1, which is used to balance L1 loss and SSIM; f is the appearance information of the training image under the virtual perspective; I' f is the rendered image; L1 is the average of the sum of the absolute values ​​of the pixel value differences between the two images; SSIM is used to measure the visual impact and structural similarity of two images.

[0132] This embodiment uses the above scheme. Since the three-dimensional Gaussian splash model also requires dense perspectives as input, when the number of input perspectives is small, the three-dimensional Gaussian splash model is prone to overfitting on the training perspectives, resulting in a significant decrease in its performance. Therefore, this application generates a mixed loss through a mixed loss function based on the appearance information and rendered image under a virtual perspective, which can alleviate the overfitting problem of the three-dimensional Gaussian splash model on the training perspective and improve the rendering quality of the model.

[0133] Based on the above implementation scheme, in a feasible implementation manner, after forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image, the method further includes steps S70 to S90:

[0134] Step S70, obtaining the three-dimensional Gaussian depth of the training image through a three-dimensional Gaussian splash model.

[0135] Specifically, the three-dimensional Gaussian depth of a training image under one viewing angle is obtained by rendering the three-dimensional Gaussian splash model. The calculation formula is:

[0136]

[0137] Where d is the three-dimensional Gaussian depth of the Gaussian point, is the opacity of the Gaussian point.

[0138] Step S80: Calculate the geometric consistency loss of the three-dimensional Gaussian depth based on the multi-view stereo depth.

[0139] Specifically, a geometric consistency loss is applied to the rendered 3D Gaussian depth. The calculation formula of the geometric consistency loss is:

[0140] L CS =∑|D r -D mvs |⊙M

[0141] Among them, L CS is the geometric consistency loss; D r is the three-dimensional Gaussian depth; D mvs is the multi-view stereo depth; M is the consistency mask in the 3D Gaussian splash model; ⊙ is the XOR operator.

[0142] Step S90, transmitting the geometric consistency loss back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and training and updating the three-dimensional Gaussian splash model.

[0143] Specifically, the geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is iteratively trained and updated.

[0144] This embodiment uses the above scheme to calculate the geometric consistency loss, transmit the loss back to the three-dimensional Gaussian splash model, and iteratively train and update the model, so that the collective information of the model can maintain consistent perspective during the training process.

[0145] Based on the above implementation scheme, in a feasible implementation scheme, the above step S70 further includes steps S100 to S110:

[0146] Step S100, generating a monocular predicted depth of a training image through a pre-trained model, and calculating a sorting loss of the three-dimensional Gaussian depth based on the monocular predicted depth.

[0147] It should be noted that the monocular predicted depth refers to predicting the depth information of each pixel in the scene from a two-dimensional image captured by a single camera.

[0148] Specifically, the device generates the monocular predicted depth of each training view of the training image through the pre-trained model DPT, and then calculates the sorting loss. For example, each time the sorting loss is calculated, two batches of non-overlapping pixel point sets S1 and S2 with the same number (512) are randomly sampled, and the calculation formula for calculating the sorting loss is:

[0149]

[0150] Among them, L mono Predict depth for a single camera; D r (S1) is the three-dimensional Gaussian depth of pixel S1; D r (S2) is the three-dimensional Gaussian depth of pixel S2; M'=∏D mono (S1)<D mono (S2).

[0151] Step S110, transmitting the sorting loss back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and training and updating the three-dimensional Gaussian splash model.

[0152] Specifically, the sorting loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is iteratively trained and updated.

[0153] Through the above scheme, this embodiment calculates the sorting loss, transmits the loss back to the three-dimensional Gaussian splash model, and iteratively trains and updates the model. The monocular predicted depth can be used as prior information as supplementary supervision to ensure that the depth map of the missing area after filtering can still remain complete.

[0154] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image rendering method based on Gaussian splatting of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0155] This application also provides an image rendering device based on Gaussian splashing, please refer to Figure 2 , the image rendering device based on Gaussian splashing includes:

[0156] The depth calculation module 201 is used to obtain a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model;

[0157] A point cloud generation module 202, configured to filter and fuse the multi-view stereo depth to generate a multi-view stereo point cloud;

[0158] An initialization module 203 is used to input the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model;

[0159] A mixed loss module 204, configured to perform forward mapping and loss calculation on the training image and the multi-view stereo depth to obtain a mixed loss of the training image;

[0160] A training and updating module 205 is used to transmit the mixed loss back to the initialized three-dimensional Gaussian splash model to perform training and updating on the three-dimensional Gaussian splash model;

[0161] The image rendering module 206 is used to perform image rendering at the target perspective using a trained three-dimensional Gaussian splash model given a target perspective and camera parameters corresponding to the target perspective, thereby generating a target image at the target perspective.

[0162] The Gaussian splash-based image rendering device provided by the present application adopts the Gaussian splash-based image rendering method in the above-mentioned embodiment, which can solve the technical problem of overfitting of the three-dimensional Gaussian splash model during training. Compared with the prior art, the beneficial effects of the Gaussian splash-based image rendering device provided by the present application are the same as the beneficial effects of the Gaussian splash-based image rendering method provided by the above-mentioned embodiment, and the other technical features in the Gaussian splash-based image rendering device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0163] The present application provides an image rendering device based on Gaussian splashing, and the image rendering device based on Gaussian splashing includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image rendering method based on Gaussian splashing in the above-mentioned embodiment one.

[0164] Reference below Figure 3 , which shows a schematic diagram of the structure of an image rendering device based on Gaussian splashing suitable for implementing the embodiment of the present application. The image rendering device based on Gaussian splashing in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3The Gaussian splatter-based image rendering device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0165] like Figure 3 As shown, the image rendering device based on Gaussian splashing may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the image rendering device based on Gaussian splashing are also stored in RAM1004. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the Gaussian splatter-based image rendering device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a Gaussian splatter-based image rendering device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0166] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0167] The Gaussian splash-based image rendering device provided by the present application adopts the Gaussian splash-based image rendering method in the above-mentioned embodiment, which can solve the technical problem of overfitting of the three-dimensional Gaussian splash model during training. Compared with the prior art, the beneficial effects of the Gaussian splash-based image rendering device provided by the present application are the same as the beneficial effects of the Gaussian splash-based image rendering method provided by the above-mentioned embodiment, and the other technical features in the Gaussian splash-based image rendering device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0168] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0169] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0170] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the image rendering method based on Gaussian splattering in the above-mentioned embodiment.

[0171] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0172] The computer-readable storage medium may be included in the image rendering device based on Gaussian splatting; or may exist independently without being assembled into the image rendering device based on Gaussian splatting.

[0173] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an image rendering device based on Gaussian splashing, the image rendering device based on Gaussian splashing: obtains a training image, selects its reference perspective, and obtains the multi-view stereo depth of the training image under the reference perspective through a multi-view stereo model; filters and fuses the multi-view stereo depth to generate a multi-view stereo point cloud; inputs the multi-view stereo point cloud into a three-dimensional Gaussian splashing model to initialize the position parameters of the three-dimensional Gaussian splashing model; obtains a mixed loss of the training image through forward mapping and loss calculation of the training image and the multi-view stereo depth; transmits the mixed loss back to the initialized three-dimensional Gaussian splashing model to train and update the three-dimensional Gaussian splashing model; given a target perspective and a camera parameter corresponding to the target perspective, performs image rendering under the target perspective through the trained three-dimensional Gaussian splashing model to generate a target image under the target perspective.

[0174] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0177] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image rendering method based on Gaussian splattering, and can solve the technical problems of image rendering based on Gaussian splattering. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the image rendering method based on Gaussian splattering provided in the above-mentioned embodiment, and will not be repeated here.

[0178] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned Gaussian splatter-based image rendering method when executed by a processor.

[0179] The computer program product provided by the present application can solve the technical problem of overfitting of the three-dimensional Gaussian splatter model during training. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the image rendering method based on Gaussian splatter provided by the above embodiment, which will not be repeated here.

[0180] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image rendering method based on Gaussian splashing, characterized in that: The method comprises: Acquire a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model; Filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud; Inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model; The training image and the multi-view stereo depth are forward mapped and loss calculated to obtain a mixed loss of the training image; Passing the mixed loss back to the initialized three-dimensional Gaussian splash model, and training and updating the three-dimensional Gaussian splash model; Given a target perspective and camera parameters corresponding to the target perspective, the trained three-dimensional Gaussian splash model is used to render the image under the target perspective to generate a target image under the target perspective.

2. The method according to claim 1, characterized in that The step of selecting a reference viewing angle of the training image and obtaining a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model comprises: Selecting a viewing angle of the training image as a reference viewing angle, and selecting viewing angles of the training image other than the reference viewing angle as source viewing angles; The features of the training image at a reference perspective and a source perspective are extracted through a multi-view stereo model, and the multi-view stereo depth of the training image at the reference perspective is generated based on the features at the reference perspective and the source perspective.

3. The method according to claim 1, characterized in that The step of filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud comprises: Calculating the multi-view stereo depth of the training image at each viewing angle; Multi-view stereo depths from all perspectives are filtered through geometric consistency to generate depth-consistent masks; A multi-view stereo point cloud of the target viewpoint is generated by fusion according to the depth consistency mask.

4. The method according to claim 1, characterized in that The step of forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image comprises: The training image is rendered through the initialized three-dimensional Gaussian splash model to generate a rendered image; The training image is forward mapped to obtain a projection image of the training image under a virtual viewing angle; Acquiring appearance information of the training image under the virtual perspective according to the projection image under the virtual perspective; Based on the appearance information and the rendered image, a hybrid loss is generated by a hybrid loss function.

5. The method according to claim 1, characterized in that After forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain the mixed loss of the training image, the method further includes: Obtain the three-dimensional Gaussian depth of the training image through the three-dimensional Gaussian splash model; Calculating a geometrically consistent loss of the three-dimensional Gaussian depth based on the multi-view stereo depth; The geometric consistency loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

6. The method according to claim 5, characterized in that After the step of obtaining the three-dimensional Gaussian depth of the training image by using the three-dimensional Gaussian splash model, the step further includes: Generate a monocular predicted depth of a training image through a pre-trained model, and calculate a sorting loss of the three-dimensional Gaussian depth based on the monocular predicted depth; The sorting loss is transmitted back to the three-dimensional Gaussian splash model through a back-propagation algorithm, and the three-dimensional Gaussian splash model is trained and updated.

7. An image rendering device based on Gaussian splashing, characterized in that: The device comprises: A depth calculation module, used to obtain a training image, select a reference viewing angle of the training image, and obtain a multi-view stereo depth of the training image under the reference viewing angle through a multi-view stereo model; A point cloud generation module, used for filtering and fusing the multi-view stereo depth to generate a multi-view stereo point cloud; An initialization module, used for inputting the multi-view stereo point cloud into a three-dimensional Gaussian splash model to initialize position parameters of the three-dimensional Gaussian splash model; A mixed loss module, used for forward mapping the training image and the multi-view stereo depth and calculating the loss to obtain a mixed loss of the training image; A training update module, used for transmitting the mixed loss back to the initialized three-dimensional Gaussian splash model, and performing training update on the three-dimensional Gaussian splash model; The image rendering module is used to render the image under the target perspective by using the trained three-dimensional Gaussian splash model given the target perspective and the camera parameters corresponding to the target perspective, so as to generate the target image under the target perspective.

8. An image rendering device based on Gaussian splashing, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Gaussian splatter-based image rendering method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image rendering method based on Gaussian splatting according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the image rendering method based on Gaussian splatting according to any one of claims 1 to 6 are implemented.

Images