Gaussian Image Rendering Method and Apparatus

Abstract

This application provides a Gaussian image rendering method and apparatus. Starting with multi-view image rendering based on an initial Gaussian loss function, color and depth images from different viewpoints are gradually acquired. Multiple loss functions are then used to specifically optimize the properties of the initial Gaussian loss function, resulting in a first Gaussian loss function. Based on the first Gaussian loss function, rendering continues from more viewpoints, and a third loss function, formed by comparing with the real image, is used to further optimize the Gaussian loss function, ultimately yielding a second Gaussian loss function for rendering images from new viewpoints. This entire process ensures that the rendered fifth color image achieves good results in terms of accuracy, realism, and detail rendering, more realistically restoring the visual appearance of the target scene from new viewpoints and reducing problems such as blurring, distortion, and unreasonable object structures in the image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a Gaussian image rendering method and apparatus. Background Technology

[0002] In the field of image processing, novel view synthesis aims to accurately reconstruct objects and render realistic images to meet the needs of numerous applications, such as virtual reality, augmented reality, and 3D modeling. Early view synthesis techniques, due to their limitations, often failed to achieve satisfactory results, especially when the number of known viewpoints decreased. Neural network-based methods were prone to overfitting, leading to floating objects and artifacts in the synthesized images, severely impacting image quality and realism.

[0003] A series of methods based on Neural Radiance Fields (NeRF) attempt to address the aforementioned problems. For example, DietNeRF introduces semantic constraints using a Contrastive Language-Image Pretraining Vision Transformer (CLIPVision Transformer), InfoNeRF applies cross-entropy loss to the density within each ray, and FreeNeRF employs progressive frequency encoding. These methods, by introducing additional information, pre-training models, and using regularization, have improved overfitting to some extent when the known number of viewpoints is reduced. However, the training process of NeRF methods demands enormous computational resources, which undoubtedly limits their wider application and promotion, necessitating further optimization and improvement.

[0004] The recently emerging 3D Gaussian splashing technique uses a set of ellipses as an explicit representation and leverages splatting-based rasterization to render 2D images, successfully achieving accelerated training and real-time rendering, bringing a new direction to the field of view synthesis. However, this technique still has significant shortcomings when facing few-viewpoint view synthesis tasks. Due to the limited image information available for fitting under few-viewpoint conditions, the neural network struggles to learn complete object information, leading to severe overfitting problems, producing a large number of floating objects and artifacts, which greatly degrade the visual effect. Summary of the Invention

[0005] In view of this, embodiments of this application provide a Gaussian image rendering method and apparatus, which divides the few-view synthesis task into two sub-tasks: dense generation and augmented synthesis. In dense generation, dense views are estimated from the few views. In augmented synthesis, both the estimated dense views and the original few views participate in Gaussian training to obtain the final Gaussian representation of the scene, and the optimized new view image is synthesized in real time.

[0006] A first aspect provides a Gaussian image rendering method, comprising: rendering a first color image of a target scene from a first known viewpoint based on an initial Gaussian; rendering a second color image and a depth image of the target scene from a virtual viewpoint based on the initial Gaussian; optimizing various attributes of the initial Gaussian based on a first loss function obtained from the first color image, a second loss function obtained from the second color image and the depth image, to obtain a first Gaussian; rendering a third color image of the target scene from multiple randomly sampled viewpoints based on the first Gaussian; rendering a fourth color image of the target scene from a second known viewpoint based on the first Gaussian, wherein the second known viewpoint includes the first known viewpoint and the multiple randomly sampled viewpoints; optimizing various attributes of the first Gaussian within the target scene based on a third loss function between the fourth color image and the real image, to obtain a second Gaussian, wherein the real image includes an image captured by a camera at a camera pose corresponding to the first known viewpoint and the third color image; and rendering a fifth color image of the target scene from a new viewpoint based on the second Gaussian.

[0007] In one possible implementation, optimizing the attributes of the initial Gaussian based on the first loss function obtained from the first color image, the second loss function obtained from the second color image and the depth image to obtain the first Gaussian includes: optimizing the attributes of the initial Gaussian based on the average value L1 of the absolute differences between the pixel values ​​of the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the structural similarity loss function Lssim between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the smoothing loss function Lsmooth between the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian; and / or, optimizing the attributes of the first Gaussian in the target scene based on the third loss function between the fourth color image and the real image to obtain the second Gaussian includes: optimizing the attributes of the first Gaussian based on the absolute differences L1 between the pixel values ​​of the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image.

[0008] In one possible implementation, the Gaussian property includes position, and the method further includes: before rendering the third color image, determining whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, moving Gaussians in the first Gaussian whose size is smaller than a second threshold in the direction of the position gradient change, or reducing the size of the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copying them; and / or before rendering the fifth color image, determining whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, moving Gaussians in the second Gaussian whose size is smaller than the second threshold in the direction of the position gradient change, or reducing the size of the Gaussians in the second Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copying them.

[0009] In one possible implementation, the Gaussian property includes transparency, and the method further includes: removing Gaussians in the first Gaussian whose transparency is less than a third threshold before rendering the third color image; and / or removing Gaussians in the second Gaussian whose transparency is less than the third threshold before rendering the fifth color image.

[0010] In one possible implementation, the method further includes: before rendering the third color image, calculating the average and standard values ​​of the distances from each Gaussian in the first Gaussian to its nearest K neighboring Gaussians, and comparing the calculated values ​​based on the average and standard values ​​of each Gaussian with a fourth threshold; and deleting Gaussians in the first Gaussian whose calculated values ​​exceed the fourth threshold.

[0011] Secondly, a Gaussian image rendering apparatus is provided, comprising: a first rendering unit for rendering a first color image of a target scene from a first known viewpoint based on an initial Gaussian; a second rendering unit for rendering a second color image and a depth image of the target scene from a virtual viewpoint based on the initial Gaussian; a first optimization unit for optimizing various attributes of the initial Gaussian based on a first loss function obtained from the first color image and a second loss function obtained from the second color image and the depth image, to obtain a first Gaussian; a third rendering unit for rendering a third color image of the target scene from multiple randomly sampled viewpoints based on the first Gaussian; a fourth rendering unit for rendering a fourth color image of the target scene from a second known viewpoint based on the first Gaussian, wherein the second known viewpoint includes the first known viewpoint and the multiple randomly sampled viewpoints; a second optimization unit for optimizing various attributes of the first Gaussian within the target scene based on a third loss function between the fourth color image and a real image, to obtain a second Gaussian, wherein the real image includes an image captured by a camera at a camera pose corresponding to the first known viewpoint and the third color image; and a fifth rendering unit for rendering a fifth color image of the target scene from a new viewpoint based on the second Gaussian.

[0012] In one possible implementation, the first optimization unit is specifically used to: optimize the various attributes of the initial Gaussian based on the average value L1 of the absolute differences between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the structural similarity loss function Lssim between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the smoothing loss function Lsmooth between the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian, to obtain the first Gaussian; and / or the second optimization unit is specifically used to: optimize the various attributes of the first Gaussian based on the absolute differences L1 between the fourth color image and the real image, and the structural similarity loss function Lssim between the fourth color image and the real image, to obtain the second Gaussian.

[0013] In one possible implementation, the Gaussian property includes position, and the Gaussian image rendering apparatus further includes: a Gaussian processing unit, configured to, before rendering the third color image, determine whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians in the first Gaussian whose size is smaller than the second threshold in the direction of the position gradient change, or, reduce the size of the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copy them; and / or, before rendering the fifth color image, determine whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, move the Gaussians in the second Gaussian whose size is smaller than the second threshold in the direction of the position gradient change, or, reduce the size of the Gaussians in the second Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copy them.

[0014] In one possible implementation, the Gaussian property includes transparency, and the Gaussian image rendering apparatus further includes: a Gaussian processing unit, configured to delete Gaussians in the first Gaussian whose transparency is less than a third threshold before rendering the third color image; and / or to delete Gaussians in the second Gaussian whose transparency is less than the third threshold before rendering the fifth color image.

[0015] In one possible implementation, the Gaussian image rendering apparatus further includes: a Gaussian processing unit, configured to calculate the average and standard values ​​of the distances from each Gaussian in the first Gaussian to its nearest K neighboring Gaussians before rendering the third color image, compare the calculated values ​​obtained based on the average and standard values ​​of each Gaussian with a fourth threshold, and delete Gaussians in the first Gaussian whose calculated values ​​exceed the fourth threshold.

[0016] Thirdly, a Gaussian image rendering apparatus is provided, comprising: a processor, and a memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute the Gaussian image rendering method of the first aspect and any possible implementation thereof by calling the program instructions.

[0017] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform a Gaussian image rendering method as described in the first aspect and any possible implementation thereof.

[0018] Based on the above technical solution, the process begins with multi-view image rendering (first known viewpoint and virtual viewpoint) based on an initial Gaussian image. Color and depth images from different viewpoints are gradually acquired, and multiple loss functions (first loss function, second loss function) are constructed to specifically optimize the properties of the initial Gaussian image, resulting in the first Gaussian image. Based on the first Gaussian image, rendering continues in more viewpoints (first known viewpoint, including the first viewpoint and multiple randomly acquired viewpoints), and the Gaussian properties are further optimized using a third loss function formed by comparing with the real image, ultimately yielding the second Gaussian image used to render the image from the new viewpoint. This entire process ensures that the rendered fifth-color image achieves good results in terms of accuracy, realism, and detail rendering, more realistically restoring the visual appearance of the target scene from the new viewpoint and reducing problems such as blurring, distortion, and unreasonable object structure in the image. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0020] Figure 1 A schematic block diagram of a Gaussian image rendering method according to an embodiment of this application is shown.

[0021] Figure 2 Another schematic block diagram of the Gaussian image rendering method according to an embodiment of this application is shown.

[0022] Figure 3 A schematic flowchart of a Gaussian image rendering method according to an embodiment of this application is shown.

[0023] Figure 4 A schematic block diagram of a Gaussian image rendering apparatus according to an embodiment of this application is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the fields of image rendering and computer vision, new view compositing technologies aim to achieve accurate object reconstruction and render highly realistic images, which has crucial application value in many fields such as virtual reality, augmented reality, digital entertainment, and industrial design. Early view compositing technologies, limited by the algorithm design and computing power at the time, often failed to achieve satisfactory results when faced with complex scenes and diverse needs. In particular, when the number of known viewpoints decreased, neural network-based training models were prone to overfitting, leading to frequent artifacts and other defects in the rendered images, severely damaging the visual quality and realism of the images and greatly limiting their practical application.

[0026] To address these challenges, a series of NeRF-based methods have emerged. These methods have made some progress by innovatively introducing additional information, utilizing pre-trained models, and employing regularization techniques to improve overfitting. For example, DietNeRF cleverly uses CLIPVisionTransformer to introduce semantic constraints, InfoNeRF optimizes model performance by applying cross-entropy loss to the density within each ray, and FreeNeRF uses progressive frequency encoding to improve performance. However, although these methods alleviate the problem caused by the reduction in known viewpoints to some extent, the training process of NeRF-like methods requires extremely large computational resources. This not only increases cost and time overhead but also limits their application in resource-constrained environments, urgently requiring further optimization and improvement to enhance their efficiency and practicality.

[0027] Meanwhile, the recently developed 3D Gaussian splatter technique uses a set of ellipses as an explicit representation and efficiently renders 2D images using a splatting-based rasterization method. This has successfully accelerated training and provides real-time rendering capabilities, opening up new directions for view synthesis technology. However, this technique reveals serious shortcomings when handling few-viewpoint view synthesis tasks. Because neural networks struggle to learn complete object information when fitting a small number of input images, severe overfitting occurs, resulting in renderings riddled with floating objects and artifacts. This significantly degrades the visual effect and reduces the usability and credibility of the images.

[0028] In contrast, while NeRF-based methods focus on solving the overfitting problem, in order to infer the missing viewpoints, they have to sacrifice the geometric details and appearance colors of the input viewpoints. This makes it extremely difficult to complete the few-view synthesis task in a single stage, and it is difficult to achieve accurate inference of the missing viewpoints and complete scene reconstruction while ensuring image quality.

[0029] In summary, although existing technologies have achieved some progress in the field of new view compositing, both the 3D Gaussian splashing method and the NeRF-based method have obvious shortcomings when facing the key challenge of few-view view compositing. These shortcomings include overfitting, visual artifacts, computational resource consumption, and the balance between geometric and color details. These problems seriously restrict the further development and widespread application of view compositing technology, and there is an urgent need for an innovative, efficient technical solution that can overcome the above-mentioned defects to promote progress in this field.

[0030] In view of this, embodiments of this application provide a Gaussian image rendering method that divides the few-view synthesis task into two sub-tasks: dense generation and augmented synthesis. In dense generation, dense views are estimated from the few views. In augmented synthesis, both the estimated dense views and the original few views participate in Gaussian training to obtain the final Gaussian representation of the scene, and the optimized new view image is synthesized in real time.

[0031] Figure 1 A schematic block diagram of a Gaussian image rendering method 100 according to an embodiment of this application is shown. Optionally, the Gaussian image rendering method 100 can be executed by a computer, specifically by a computer processor, such as a central processing unit (CPU). Another example is a graphics processing unit (GPU). This embodiment of the application does not limit the executing entity of the method. Figure 1 As shown, the Gaussian image rendering method 100 includes some or all of the following.

[0032] S110, based on the initial Gaussian, renders the first color image of the target scene from the first known viewpoint.

[0033] S120, based on the initial Gaussian, render the second color image and depth image of the target scene from a virtual perspective.

[0034] S130, based on the first loss function obtained from the first color image, the second loss function obtained from the second color image and the depth image, the various attributes of the initial Gaussian are optimized to obtain the first Gaussian.

[0035] S140, based on the first Gaussian, render a third color image of the target scene from multiple randomly sampled viewpoints.

[0036] S150, based on the first Gaussian, render a fourth color image of the target scene under a second known viewpoint, the second known viewpoint including the first known viewpoint and the plurality of randomly sampled viewpoints.

[0037] S160, based on the third loss function between the fourth color image and the real image, optimize each attribute of the first Gaussian in the target scene to obtain the second Gaussian. The real image includes the image captured by the camera in the camera pose corresponding to the first known viewpoint and the third color image.

[0038] S170, based on the second Gaussian, render the fifth color image of the target scene from a new perspective.

[0039] It should be noted that the color image in the embodiments of this application can be in RGB format, that is, the color image is an RGB image. Optionally, the color image can also be in other formats, such as YUV format. This application does not limit this.

[0040] In 3D reconstruction and rendering methods, a set of 3D Gaussian squares is typically used to sparsely describe the geometry and appearance information of a target scene. Each Gaussian square can represent a region with certain attributes such as position, size, orientation, and color. The entire target scene is approximated by combining multiple Gaussians. Target scenes often have complex geometries, rich material properties, and diverse lighting conditions. By optimizing the Gaussian squares, these scene features can be better fitted, more accurately describing the distribution of objects, surface details, and lighting variations in the scene, thus providing a more precise foundation for high-quality image rendering.

[0041] Before executing the Gaussian image rendering method 100, the target scene is first divided into multiple non-overlapping blocks, and then the intersection of these blocks with the initial Gaussian is determined. For the initial Gaussian that intersects with multiple blocks, it is copied, and an associated ID is assigned to each copy. Then, the IDs of all Gaussians in the target scene and all copied initial Gaussians are sorted according to depth, and the sorted IDs are used for rendering.

[0042] The initial Gaussian is the starting point for the entire rendering operation. It contains an initial description of the target scene. For example, it may roughly outline the distribution and appearance attributes of objects in the scene through specific parameter settings (such as position, scale, shape, etc.). It is the basic information source for rendering images from the corresponding viewpoint.

[0043] In S110, the processor first obtains the known camera pose from the camera. Based on the known camera pose, it determines the first known viewpoint. Then, given the initial Gaussian distribution, it renders the first color image of the target scene from the first known viewpoint.

[0044] Unlike a known viewpoint, a virtual viewpoint may not correspond to the actual viewing angle during image acquisition. Instead, it may be an angle artificially set to enrich scene presentation, perform specific analyses, or meet rendering requirements. In one embodiment, the virtual viewpoint may be determined based on randomly sampled camera poses. In another embodiment, the virtual viewpoint may be determined based on a first known viewpoint.

[0045] In S120, after acquiring the virtual viewpoint, the processor renders the second color image and depth image of the target scene under the virtual viewpoint, given the initial Gaussian parameters. For example, a camera pose is randomly sampled within a specified distance in the upper hemisphere of the target scene, and then an RGB image is rendered under that pose using the splatting method, with the depth image calculated from the Gaussian parameters.

[0046] Loss functions are typically used to measure the degree of difference between the rendered image and the desired image. For example, common types of loss functions built based on the first color image include Mean Squared Error (MSE) loss, Structural Similarity Index Measure (SSIM) loss, and perceptual loss. As another example, for a second loss function built based on the second color image and the depth image, Lsmooth, a loss function that measures the smoothness of the image, can be used.

[0047] Optionally, the optimization of various attributes of the initial Gaussian based on the first loss function obtained from the first color image, the second loss function obtained from the second color image and the depth image to obtain the first Gaussian includes: optimizing various attributes of the initial Gaussian based on the average value L1 of the absolute differences of each pixel value between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the structural similarity loss function Lssim between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the smoothing loss function Lsmooth between the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian to obtain the first Gaussian.

[0048] To optimize the properties of the initial Gaussian using the constructed first and second loss functions, a suitable optimization algorithm needs to be selected. Common optimization algorithms include gradient descent, stochastic gradient descent (SGD), and the Adam optimization algorithm. Regardless of the chosen algorithm, the gradient of the loss function with respect to the parameters of the initial Gaussian needs to be calculated. Then, in each iteration, the gradient of the loss function with respect to each parameter is calculated based on the current parameter values. Next, using the update rules of each optimization algorithm, combined with the gradient and previous moment estimates, the parameters of the initial Gaussian, such as the center position and elements of the covariance matrix, are updated, gradually adjusting them in a direction that reduces the loss function value. This iterative update process continues for multiple rounds, for example, by setting a set number of iterations or by using a certain stopping condition (such as the change in the loss function value being less than a certain threshold, indicating convergence to a better solution). After multiple rounds of iterative optimization, convergence is determined by observing the changes in the loss function value. If the loss function value changes very little in several consecutive iterations, the optimization process is considered to have converged. The Gaussian distribution determined by the attribute parameter values ​​of the initial Gaussian at this point (updated center position, covariance matrix, etc.) is the optimized first Gaussian. Compared to the initial Gaussian, this first Gaussian is more reasonable and accurate in terms of its distribution pattern in the target scene and its fit to the actual scene. It can be used to generate higher-quality images that better reflect the true appearance of the target scene in subsequent rendering steps.

[0049] In S140, after obtaining the first Gaussian, multiple viewpoints can be randomly sampled, and based on the first Gaussian, a third color image of the target scene under these randomly sampled viewpoints can be rendered. For example, within a specified distance in the upper hemisphere of the target scene, camera poses are randomly sampled, RGB images are rendered, and the camera poses and corresponding image combinations are stored.

[0050] At this point, the above steps have completed the entire process of estimating dense views using fewer views.

[0051] In S150, the first Gaussian is obtained through optimization in the previous steps. The second known viewpoint includes the first known viewpoint and multiple randomly sampled viewpoints. The first known viewpoint was obtained in previous steps, while the multiple randomly sampled viewpoints were determined in S140 using randomly sampled camera poses. Having obtained the first Gaussian and the second known viewpoint, a fourth color image of the target scene under the second known viewpoint can be rendered based on the first Gaussian.

[0052] Similar to S110, before executing S150, the target scene is first divided into multiple non-overlapping blocks, and then the intersection of these blocks with the first Gaussian is determined. For the first Gaussian that intersects with multiple blocks, it is copied, and an associated ID is assigned to each copy. Then, the IDs of all the first Gaussians in the target scene and all the copied first Gaussians are sorted according to depth, and the sorted IDs are used for rendering.

[0053] Similar to S130, in S160, a third loss function is calculated between the fourth color image and the real image, and the various attributes of the first Gaussian in the target scene are optimized to obtain the second Gaussian. The real image includes the image captured by the camera in S110 with a known pose and the third color image rendered from multiple randomly acquired viewpoints in S140. In other words, the image captured by the camera in S110 with a known pose and the third color image rendered from multiple randomly acquired viewpoints in S140 are combined as the real image to supplement the input image in the Gaussian training.

[0054] Optionally, optimizing the attributes of the first Gaussian in the target scene based on the third loss function between the fourth color image and the real image to obtain the second Gaussian includes: optimizing the attributes of the first Gaussian based on the absolute difference L1 between the pixel values ​​of the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image to obtain the second Gaussian.

[0055] For a detailed explanation of S160, please refer to S130. For the sake of brevity, it will not be described in detail here.

[0056] Finally, in S170, after obtaining the optimized second Gaussian, the camera pose of the image to be synthesized is input to obtain a new viewpoint, and splatting is used to render the RGB image under the new viewpoint from the second Gaussian.

[0057] In some embodiments, such as Figure 2As shown, the Gaussian property includes position, and the method further includes: S180, before rendering the third color image, determining whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, moving the Gaussian slabs with a size smaller than the second threshold in the direction of the position gradient change, or reducing the Gaussian slabs with a size greater than or equal to the second threshold by half of their initial size and copying them; and / or, before rendering the fifth color image, determining whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, moving the Gaussian slabs with a size smaller than the second threshold in the direction of the position gradient change, or reducing the Gaussian slabs with a size greater than or equal to the second threshold by half of their initial size and copying them.

[0058] Specifically, S180 can be understood as a Gaussian adaptive densification process. For example, if the position gradient of the Gaussian (first Gaussian or second Gaussian) exceeds a threshold, the following operations are performed: For a relatively small Gaussian (under-reconstructed), a copy is made and moved in the direction of the position gradient change. For a relatively large Gaussian (over-reconstructed), its scaling scale is reduced, and a copy is made.

[0059] In some embodiments, such as Figure 2 As shown, the Gaussian property includes transparency, and the method further includes: S190, before rendering the third color image, deleting Gaussians in the first Gaussian whose transparency is less than a third threshold; and / or, before rendering the fifth color image, deleting Gaussians in the second Gaussian whose transparency is less than the third threshold.

[0060] Similarly, S190 can also be understood as a Gaussian adaptive densification process. For example, Gaussians with transparency less than a threshold (either the first or second Gaussian) can be removed.

[0061] In some other embodiments, such as Figure 2 As shown, the method further includes: S195, before rendering the third color image, calculating the average and standard values ​​of the distances from each Gaussian in the first Gaussian to its nearest K neighboring Gaussians, comparing the calculated values ​​obtained based on the average and standard values ​​of each Gaussian with a fourth threshold, and deleting Gaussians in the first Gaussian whose calculated values ​​exceed the fourth threshold.

[0062] Step S195 can be understood as the Gaussian cloud denoising process. The key point in this process is to perform a distance-related statistical filtering operation on the first Gaussian cloud before rendering the third color image. By analyzing the distances between each Gaussian cloud in the first Gaussian cloud and its K nearest neighbors, the corresponding statistics (mean and standard values) are calculated, resulting in a comprehensive calculated value. This calculated value is then compared with a preset fourth threshold to determine whether certain Gaussians should be deleted. The aim is to optimize the distribution of the first Gaussian cloud in the target scene, removing those relatively "isolated" or non-distributive Gaussians, in the hope of achieving better results when rendering the third color image subsequently.

[0063] It should be noted that the thresholds involved in the above embodiments, such as the first threshold, the second threshold, the third threshold, and the fourth threshold, are all preset values. Similarly, the K value can also be a preset value. This application embodiment does not limit the magnitude of these thresholds or the K value.

[0064] It should be understood that the first known viewpoint and the virtual viewpoint in the embodiments of this application can be one or more viewpoints. Correspondingly, the first color image rendered under the first known viewpoint and the second color image and depth image rendered under the virtual viewpoint can also be one or more images. The embodiments of this application do not limit the number of the first known viewpoint and the virtual viewpoint.

[0065] It should also be understood that although the embodiments of this application are described as decomposing one task into two tasks, in order to achieve better rendering effects, one task can also be decomposed into more than two tasks. For example, after obtaining the second Gaussian in the embodiments of this application, steps S140 to S160 can be executed repeatedly to finally obtain a better Gaussian for the target scene and synthesize an image from a new perspective in real time.

[0066] The following is combined Figure 3 A specific embodiment of the Gaussian image rendering method of this application is described in detail.

[0067] like Figure 3 As shown, the Gaussian image rendering method includes:

[0068] Step 1: Render the target scene as an RGB image from a known viewpoint.

[0069] Step 2: Render the target scene's RGB and depth images from a virtual perspective.

[0070] Step 3: Gaussian Property Optimization. L1 and Lssim are obtained from the real image from the known viewpoint and the RGB image rendered in Step 1. Lsmooth is obtained from the RGB image and depth image from the virtual viewpoint rendered in Step 2. Lscale is obtained from the scaling factor of the Gaussian. Gradient backpropagation is used to optimize the various properties of the Gaussian.

[0071] Step 4: Gaussian adaptive densification, see the description of S180 above.

[0072] Step 5: Gaussian cloud denoising. See the description in S195 above.

[0073] Step 6: Render dense views. For example, within a specified distance in the upper hemisphere, randomly sample camera poses, render RGB images, and store the combination of camera poses and images.

[0074] Steps one through six above constitute the dense generation process.

[0075] Step 7: Render the known viewpoint RGB image, including the known viewpoint involved in Step 1 and the dense view rendered in Step 6.

[0076] Step 8: Gaussian Property Optimization. Obtain L1 and Lssim from the real and rendered images from the known viewpoints, and optimize the various properties of the Gaussian using gradient backpropagation.

[0077] Step 9: Gaussian adaptive densification, see the description of S180 above.

[0078] Step 10: Render the new view. Input the camera pose for the image you want to composite, and use splatting to render the corresponding RGB image from Gaussian.

[0079] Steps seven through ten above enhance the synthesis process.

[0080] This application also provides a Gaussian image rendering apparatus. For example... Figure 4 As shown, the Gaussian image rendering apparatus 200 includes some or all of the following components.

[0081] The first rendering unit 210 is used to render a first color image of the target scene from a first known viewpoint based on an initial Gaussian.

[0082] The second rendering unit 220 is used to render a second color image and a depth image of the target scene from a virtual perspective based on the initial Gaussian.

[0083] The first optimization unit 230 is used to optimize the various attributes of the initial Gaussian based on the first loss function obtained from the first color image, the second loss function obtained from the second color image and the depth image, to obtain the first Gaussian.

[0084] The third rendering unit 240 is used to render a third color image of the target scene from multiple randomly sampled viewpoints based on the first Gaussian.

[0085] The fourth rendering unit 250 is used to render a fourth color image of the target scene under a second known viewpoint based on the first Gaussian viewpoint, wherein the second known viewpoint includes the first known viewpoint and the plurality of randomly sampled viewpoints;

[0086] The second optimization unit 260 is used to optimize the various attributes of the first Gaussian in the target scene based on the third loss function between the fourth color image and the real image to obtain the second Gaussian. The real image includes the image captured by the camera in the camera pose corresponding to the first known viewpoint and the third color image.

[0087] The fifth rendering unit 270 is used to render a fifth color image of the target scene from a new perspective based on the second Gaussian.

[0088] In some embodiments, the first optimization unit 230 is specifically used for:

[0089] Based on the average value L1 of the absolute differences of each pixel value between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the structural similarity loss function Lssim between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, the smoothing loss function Lsmooth between the second color image and the depth image, and the loss function Lscale obtained based on the scaling of the Gaussian, the various attributes of the initial Gaussian are optimized to obtain the first Gaussian; and / or

[0090] The second optimization unit 260 is specifically used for:

[0091] Based on the absolute difference L1 between each pixel value between the fourth color image and the real image, and the structural similarity loss function Lssim between the fourth color image and the real image, the attributes of the first Gaussian are optimized to obtain the second Gaussian.

[0092] In some embodiments, the Gaussian property includes position, and the Gaussian image rendering apparatus further includes:

[0093] The Gaussian processing unit is configured to, before rendering the third color image, determine whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians in the first Gaussian whose size is smaller than the second threshold in the direction of the position gradient change, or reduce the size of the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copy them; and / or, before rendering the fifth color image, determine whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, move the Gaussians in the second Gaussian whose size is smaller than the second threshold in the direction of the position gradient change, or reduce the size of the Gaussians in the second Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copy them.

[0094] In some embodiments, the Gaussian property includes transparency, and the Gaussian image rendering apparatus further includes:

[0095] A Gaussian processing unit is configured to remove Gaussians with transparency less than a third threshold from the first Gaussian before rendering the third color image; and / or to remove Gaussians with transparency less than the third threshold from the second Gaussian before rendering the fifth color image.

[0096] In some embodiments, the Gaussian image rendering apparatus further includes:

[0097] A Gaussian processing unit is configured to calculate the average and standard values ​​of the distances from each Gaussian in the first Gaussian to its nearest K neighboring Gaussians before rendering the third color image, compare the calculated values ​​obtained based on the average and standard values ​​of each Gaussian with a fourth threshold, and delete Gaussians in the first Gaussian whose calculated values ​​exceed the fourth threshold.

[0098] Based on the same idea, this application also provides another Gaussian image rendering apparatus. This Gaussian image rendering apparatus includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the Gaussian image rendering apparatus performs the various method embodiments described above.

[0099] It should be noted that the details of the device-side embodiment can be found in the method-side embodiment, and for the sake of brevity, they will not be repeated here.

[0100] Based on the same idea, embodiments of this application also provide a computer-readable storage medium, which includes a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute the various method embodiments described above.

[0101] Although this application has been described with reference to preferred embodiments, various modifications can be made thereto and components can be replaced with equivalents without departing from the scope of this application. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no structural conflict. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A Gaussian image rendering method, characterized in that, include: Based on the initial Gaussian, render the first color image of the target scene from the first known viewpoint; Based on the initial Gaussian, render the second color image and depth image of the target scene from a virtual perspective; Based on the first loss function obtained from the first color image, the second loss function obtained from the second color image and the depth image, the various attributes of the initial Gaussian are optimized to obtain the first Gaussian. The second loss function includes the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian. Based on the first Gaussian, render a third color image of the target scene from multiple randomly sampled viewpoints; Based on the first Gaussian, render a fourth color image of the target scene from a second known perspective, where the second known perspective includes the first known perspective and the plurality of randomly sampled perspectives; Based on the third loss function between the fourth color image and the real image, the various attributes of the first Gaussian in the target scene are optimized to obtain the second Gaussian. The real image includes the image captured by the camera in the camera pose corresponding to the first known viewpoint and the third color image. Based on the second Gaussian, render the fifth color image of the target scene from the new perspective. The method further includes: Before rendering the third color image, the average and standard values ​​of the distances from each Gaussian in the first Gaussian to its nearest K neighboring Gaussians are calculated, and the calculated values ​​based on the average and standard values ​​of each Gaussian are compared with a fourth threshold. Delete Gaussians whose calculated values ​​exceed the fourth threshold from the first Gaussian.

2. The Gaussian image rendering method according to claim 1, characterized in that, The first loss function includes the average of the absolute differences L1 between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, and the structural similarity loss function Lssim between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint; and / or The second Gaussian image is obtained by optimizing various attributes of the first Gaussian image within the target scene based on the third loss function between the fourth color image and the real image, including: Based on the absolute difference L1 between each pixel value between the fourth color image and the real image, and the structural similarity loss function Lssim between the fourth color image and the real image, the attributes of the first Gaussian are optimized to obtain the second Gaussian.

3. The Gaussian image rendering method according to claim 1, characterized in that, The Gaussian property includes position, and the method further includes: Before rendering the third color image, it is determined whether the position gradient of the first Gaussian exceeds a first threshold. If the position gradient of the first Gaussian exceeds the first threshold, the Gaussians in the first Gaussian whose size is smaller than the second threshold are moved in the direction of the position gradient change; or, the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold are reduced to half their initial size and copied; and / or Before rendering the fifth color image, it is determined whether the position gradient of the second Gaussian exceeds the first threshold. If the position gradient of the second Gaussian exceeds the first threshold, the Gaussian spheroids with a size smaller than the second threshold are moved in the direction of the position gradient change, or the Gaussian spheroids with a size greater than or equal to the second threshold are reduced to half their initial size and copied.

4. The Gaussian image rendering method according to claim 1, characterized in that, The Gaussian property includes transparency, and the method further includes: Before rendering the third color image, remove Gaussians in the first Gaussian source whose transparency is less than a third threshold; and / or Before rendering the fifth color image, remove Gaussians in the second Gaussian whose transparency is less than the third threshold.

5. A Gaussian image rendering apparatus, characterized in that, include: The first rendering unit is used to render the first color image of the target scene from the first known viewpoint, based on the initial Gaussian. The second rendering unit is used to render a second color image and a depth image of the target scene from a virtual perspective based on the initial Gaussian. The first optimization unit is used to optimize the various attributes of the initial Gaussian based on the first loss function obtained from the first color image, the second loss function obtained from the second color image and the depth image, to obtain the first Gaussian. The second loss function includes the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian. The third rendering unit is used to render a third color image of the target scene from multiple randomly sampled perspectives based on the first Gaussian. The fourth rendering unit is used to render a fourth color image of the target scene under a second known viewpoint based on the first Gaussian viewpoint, wherein the second known viewpoint includes the first known viewpoint and the plurality of randomly sampled viewpoints; The second optimization unit is used to optimize the various attributes of the first Gaussian in the target scene based on the third loss function between the fourth color image and the real image to obtain the second Gaussian. The real image includes the image captured by the camera in the camera pose corresponding to the first known viewpoint and the third color image. The fifth rendering unit is used to render a fifth color image of the target scene from a new perspective, based on the second Gaussian. The Gaussian image rendering device also includes: A Gaussian processing unit is configured to calculate the average and standard values ​​of the distances from each Gaussian in the first Gaussian to its nearest K neighboring Gaussians before rendering the third color image, compare the calculated values ​​obtained based on the average and standard values ​​of each Gaussian with a fourth threshold, and delete Gaussians in the first Gaussian whose calculated values ​​exceed the fourth threshold.

6. The Gaussian image rendering apparatus according to claim 5, characterized in that, The first loss function includes the average of the absolute differences L1 between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint, and the structural similarity loss function Lssim between the first color image and the image acquired by the camera in the camera pose corresponding to the first known viewpoint; and / or The second optimization unit is specifically used for: Based on the absolute difference L1 between each pixel value between the fourth color image and the real image, and the structural similarity loss function Lssim between the fourth color image and the real image, the attributes of the first Gaussian are optimized to obtain the second Gaussian.

7. The Gaussian image rendering apparatus according to claim 5, characterized in that, The Gaussian property includes position, and the Gaussian image rendering apparatus further includes: A Gaussian processing unit is configured to, before rendering the third color image, determine whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians in the first Gaussian whose size is smaller than a second threshold in the direction of the position gradient change, or, reduce the size of the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold by half of their initial size and copy them; and / or Before rendering the fifth color image, it is determined whether the position gradient of the second Gaussian exceeds the first threshold. If the position gradient of the second Gaussian exceeds the first threshold, the Gaussian spheroids with a size smaller than the second threshold are moved in the direction of the position gradient change, or the Gaussian spheroids with a size greater than or equal to the second threshold are reduced to half their initial size and copied.

8. The Gaussian image rendering apparatus according to claim 5, characterized in that, The Gaussian property includes transparency, and the Gaussian image rendering apparatus further includes: A Gaussian processing unit is configured to remove Gaussians with transparency less than a third threshold from the first Gaussian before rendering the third color image; and / or Before rendering the fifth color image, remove Gaussians in the second Gaussian whose transparency is less than the third threshold.

Images