Gaussian image rendering method and device

Abstract

The embodiment of the invention provides a Gaussian image rendering method and device. The method comprises the following steps: starting from multi-view image rendering based on initial Gaussian, gradually obtaining color images and depth images under different views, so as to construct a plurality of loss functions to carry out targeted optimization on attributes of the initial Gaussian, thereby obtaining a first Gaussian; and rendering is continued under more visual angles based on the first Gaussian, and Gaussian attributes are further optimized by using a third loss function formed by comparison with a real image, and finally a second Gaussian is obtained for rendering an image under a new visual angle. According to the whole set of process, the rendered fifth color image can achieve better effects in the aspects of accuracy, authenticity, detail presentation and the like, the visual appearance of the target scene under a new visual angle can be restored more vividly, and the problems of blurring, distortion, unreasonable object structure and the like in the image are reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a Gaussian image rendering method and device. Background Art

[0002] In the field of image technology, new view synthesis aims to accurately reconstruct objects and render realistic images to meet the needs of many application scenarios, such as virtual reality, augmented reality, 3D modeling, etc. Early view synthesis technology solutions, due to their technical limitations, often failed to achieve satisfactory results, especially when the number of known view angles decreased. Neural network-based methods were prone to overfitting, which led to floating objects and artifacts in the synthesized images, seriously affecting the quality and authenticity of the images.

[0003] A series of methods based on Neural Radiance Fields (NeRF) attempt to solve the above problems, such as DietNeRF using Contrastive Language-Image Pretraining Vision Transformer (CLIPVision Transformer) to introduce semantic constraints, InfoNeRF applying cross entropy loss to the density within each set of rays, and FreeNeRF using progressive frequency encoding. These methods improve the overfitting situation when the known view angle is reduced to a certain extent by introducing additional information, pre-training models, and using regularization methods, but the training process of the NeRF method requires extremely large computing resources, which undoubtedly limits its wider application and promotion, and it is urgently needed to be further optimized and improved.

[0004] The recently emerged 3D Gaussian splatting technology uses a set of ellipses as explicit expressions and renders 2D images with the help of splatting-based rasterization, which successfully achieves the effects of training acceleration and real-time rendering, bringing a new development direction to the field of view synthesis. However, this technology still has obvious defects when facing the task of few-view view synthesis. Because the image information available for fitting is limited under the condition of few views, it is difficult for the neural network to learn the complete object information, which leads to serious overfitting problems, resulting in a large number of floating objects and artifacts, which greatly damages the visual effect. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a Gaussian image rendering method and device, which divides the few-view view synthesis task into two subtasks, dense generation and enhanced synthesis. In dense generation, dense views are estimated from few views, and in enhanced synthesis, the estimated dense views and the original few views are both involved in Gaussian training to obtain the final Gaussian representation of the scene, and to synthesize optimized new-view images in real time.

[0006] In a first aspect, a Gaussian image rendering method is provided, comprising: rendering a first color image of a target scene at a first known viewing angle based on an initial Gaussian; rendering a second color image and a depth image of the target scene at a virtual viewing angle based on the initial Gaussian; optimizing various properties of the initial Gaussian based on a first loss function obtained from the first color image and a second loss function obtained from the second color image and the depth image to obtain a first Gaussian; rendering a third color image of the target scene at a plurality of randomly sampled viewing angles based on the first Gaussian; rendering a fourth color image of the target scene at a second known viewing angle based on the first Gaussian, the second known viewing angle including the first known viewing angle and the plurality of randomly sampled viewing angles; optimizing various properties of the first Gaussian in the target scene based on a third loss function between the fourth color image and a real image to obtain a second Gaussian, the real image including an image captured by a camera at a camera posture corresponding to the first known viewing angle and the third color image; rendering a fifth color image of the target scene at a new viewing angle based on the second Gaussian.

[0007] In a possible implementation, the first loss function obtained based on the first color image, the second loss function obtained based on the second color image and the depth image, optimizes various properties of the initial Gaussian to obtain a first Gaussian, including: based on the average value L1 of the absolute difference of each pixel value between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, and the structural similarity loss function Lssim between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian, optimizes various properties of the initial Gaussian to obtain the first Gaussian; and / or, based on the third loss function between the fourth color image and the real image, optimizes various properties of the first Gaussian in the target scene to obtain a second Gaussian, including: based on the absolute difference L1 of each pixel value between the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image, optimizes various properties of the first Gaussian to obtain the second Gaussian.

[0008] In one possible implementation, the attributes of the Gaussian include position, and the method further includes: before rendering the third color image, determining whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, moving the Gaussians in the first Gaussian whose sizes are smaller than the second threshold toward the direction of the position gradient change, or reducing the Gaussians in the first Gaussian whose sizes are greater than or equal to the second threshold by half of the initial size and copying them; and / or before rendering the fifth color image, determining whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, moving the Gaussians in the second Gaussian whose sizes are smaller than the second threshold toward the direction of the position gradient change, or reducing the Gaussians in the second Gaussian whose sizes are greater than or equal to the second threshold by half of the initial size and copying them.

[0009] In one possible implementation, the properties of the Gaussian include transparency, and the method further includes: before rendering the third color image, deleting the Gaussians whose transparency is less than a third threshold in the first Gaussian; and / or before rendering the fifth color image, deleting the Gaussians whose transparency is less than the third threshold in the second Gaussian.

[0010] In a possible implementation, the method further includes: before rendering the third color image, calculating the average value and standard value of the distances of each Gaussian in the first Gaussian to the nearest K neighboring Gaussians, and comparing the calculated value obtained based on the average value and the standard value of each Gaussian with a fourth threshold; and deleting the Gaussians in the first Gaussian whose calculated value exceeds the fourth threshold.

[0011] In a second aspect, a Gaussian image rendering device is provided, including: a first rendering unit, for rendering a first color image of a target scene at a first known viewing angle based on an initial Gaussian; a second rendering unit, for rendering a second color image and a depth image of the target scene at a virtual viewing angle based on the initial Gaussian; a first optimization unit, for optimizing various properties of the initial Gaussian based on a first loss function obtained from the first color image, a second loss function obtained from the second color image and the depth image, to obtain a first Gaussian; a third rendering unit, for rendering a third color image of the target scene at multiple randomly sampled viewing angles based on the first Gaussian; a fourth rendering unit, for rendering a fourth color image of the target scene at a second known viewing angle based on the first Gaussian, the second known viewing angle including the first known viewing angle and the multiple randomly sampled viewing angles; a second optimization unit, for optimizing various properties of the first Gaussian in the target scene based on a third loss function between the fourth color image and a real image, to obtain a second Gaussian, the real image including an image captured by a camera at a camera posture corresponding to the first known viewing angle and the third color image; a fifth rendering unit, for rendering a fifth color image of the target scene at a new viewing angle based on the second Gaussian.

[0012] In a possible implementation, the first optimization unit is specifically configured to optimize various properties of the initial Gaussian to obtain the first Gaussian based on the average L1 of the absolute difference values ​​of each pixel value between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the structural similarity loss function Lssim between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian; and / or the second optimization unit is specifically configured to optimize various properties of the first Gaussian to obtain the second Gaussian based on the absolute difference value L1 of each pixel value between the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image. .

[0013] In a possible implementation, the attribute of the Gaussian includes position, and the Gaussian image rendering device further includes: a Gaussian processing unit, which is used to determine whether the position gradient of the first Gaussian exceeds a first threshold before rendering the third color image, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians in the first Gaussian whose size is smaller than the second threshold toward the direction of the position gradient change, or reduce the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold by half of the initial size and copy them; and / or before rendering the fifth color image, determine whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, move the Gaussians in the second Gaussian whose size is smaller than the second threshold toward the direction of the position gradient change, or reduce the Gaussians in the second Gaussian whose size is greater than or equal to the second threshold by half of the initial size and copy them.

[0014] In a possible implementation, the properties of the Gaussian include transparency, and the Gaussian image rendering device further includes: a Gaussian processing unit, used to delete Gaussians with transparency less than a third threshold in the first Gaussian before rendering the third color image; and / or to delete Gaussians with transparency less than the third threshold in the second Gaussian before rendering the fifth color image.

[0015] In a possible implementation, the Gaussian image rendering device also includes: a Gaussian processing unit, which is used to calculate the average value and standard value of the distances of each Gaussian in the first Gaussian to the nearest K neighboring Gaussians before rendering the third color image, and compare the calculated value obtained based on the average value and the standard value of each Gaussian with a fourth threshold, and delete the Gaussians in the first Gaussian whose calculated value exceeds the fourth threshold.

[0016] In a third aspect, a Gaussian image rendering device is provided, comprising: a processor, and a memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the Gaussian image rendering method of the first aspect and any possible implementation method of the first aspect.

[0017] In a fourth aspect, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to perform a Gaussian image rendering method such as the first aspect and any possible implementation method of the first aspect.

[0018] Based on the above technical solution, we start with multi-perspective image rendering (first known perspective, virtual perspective) based on the initial Gaussian, gradually obtain color images and depth images under different perspectives, and construct multiple loss functions (first loss function, second loss function) to optimize the properties of the initial Gaussian in a targeted manner to obtain the first Gaussian. Based on the first Gaussian, continue to render at more perspectives (first known perspective, including the first perspective and multiple perspectives collected randomly) and use the third loss function formed by comparison with the real image to further optimize the Gaussian properties, and finally obtain the second Gaussian for rendering the image under the new perspective. This whole set of processes enables the rendered fifth color image to achieve better results in terms of accuracy, authenticity, and detail presentation, and can more realistically restore the visual appearance of the target scene under the new perspective, reducing the occurrence of problems such as blur, distortion, and unreasonable object structure in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative work.

[0020] Figure 1 A schematic block diagram of a Gaussian image rendering method according to an embodiment of the present application is shown.

[0021] Figure 2 Another schematic block diagram of the Gaussian image rendering method according to an embodiment of the present application is shown.

[0022] Figure 3 A schematic flowchart of a Gaussian image rendering method according to an embodiment of the present application is shown.

[0023] Figure 4 A schematic block diagram of a Gaussian image rendering device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In the field of image rendering and computer vision, new view synthesis technology is committed to achieving accurate reconstruction of objects and rendering highly realistic images, which has vital application value in many fields such as virtual reality, augmented reality, digital entertainment, and industrial design. Early view synthesis technology solutions were limited by the algorithm design and computing power at the time. When faced with complex scenes and diverse needs, it was often difficult to achieve satisfactory ideal results. In particular, when the number of known perspectives decreased, the training model based on the neural network was prone to overfitting, which led to frequent defects such as floating objects and artifacts in the rendered images, which seriously damaged the visual quality and realism of the image and greatly limited its practical application scope.

[0026] To address these challenges, a series of methods based on NeRF have emerged. These methods have made some progress by innovatively introducing additional information, using pre-trained models, and adopting regularization methods to improve the overfitting problem. For example, DietNeRF cleverly introduces semantic constraints with the help of CLIPVisionTransformer, InfoNeRF optimizes model performance by applying cross-entropy loss to the density within each set of rays, and FreeNeRF uses progressive frequency encoding to improve the effect. However, although these methods have alleviated the problems caused by the reduction of known viewing angles to a certain extent, the training process of NeRF-like methods has extremely large requirements for computing resources, which not only increases the cost and time overhead, but also limits its application in resource-constrained environments. Further optimization and improvement are urgently needed to improve its efficiency and practicality.

[0027] At the same time, the recently developed 3D Gaussian splatting technology uses a set of ellipses as an explicit expression form and uses a splatting-based rasterization method to efficiently render 2D images. It successfully achieves training acceleration and has the ability to render in real time, opening up a new direction for the development of view synthesis technology. However, this technology exposes serious defects when dealing with small-view view synthesis tasks. Since it is difficult for neural networks to learn complete object information when fitting a small number of input images, it leads to serious overfitting, making the rendering results full of a large number of floating objects and artifacts, which greatly damages the visual effect and reduces the usability and credibility of the image.

[0028] In comparison, although the NeRF-based method focuses on solving the overfitting problem, in this process, in order to infer the missing perspective, the geometric details and appearance color of the input viewpoint have to be sacrificed, which makes it extremely difficult to complete the few-view synthesis task in a single stage. It is difficult to accurately infer the missing perspective and completely restore the scene while ensuring image quality.

[0029] In summary, although the existing technology has achieved phased results in the field of new view synthesis, when facing the key challenge of few-view view synthesis, both the 3D Gaussian splashing method and the NeRF-based method have obvious shortcomings, including overfitting, visual defects, computing resource consumption, and the balance between geometry and color details. These problems seriously restrict the further development and wide application of view synthesis technology. An innovative and efficient technical solution that can overcome the above-mentioned defects is urgently needed to promote progress in this field.

[0030] In view of this, an embodiment of the present application provides a Gaussian image rendering method, which divides the few-view view synthesis task into two subtasks, dense generation and enhanced synthesis. In dense generation, dense views are estimated from few views, and in enhanced synthesis, the estimated dense views and the original few views are both involved in Gaussian training to obtain the final Gaussian representation of the scene, and to synthesize optimized new-view images in real time.

[0031] Figure 1 A schematic block diagram of a Gaussian image rendering method 100 according to an embodiment of the present application is shown. Optionally, the Gaussian image rendering method 100 may be executed by a computer, specifically, by a processor of a computer, such as a central processing unit (CPU). Another example is a graphics processing unit (GPU). The present application embodiment does not limit the execution subject of the method. Figure 1 As shown, the Gaussian image rendering method 100 includes part or all of the following contents.

[0032] S110 , rendering a first color image of the target scene at a first known viewing angle based on the initial Gaussian.

[0033] S120: Rendering a second color image and a depth image of the target scene in a virtual perspective based on the initial Gaussian.

[0034] S130, optimizing various properties of the initial Gaussian based on a first loss function obtained from the first color image and a second loss function obtained from the second color image and the depth image to obtain a first Gaussian.

[0035] S140: Render a third color image of the target scene at a plurality of randomly sampled perspectives based on the first Gaussian.

[0036] S150: Render a fourth color image of the target scene at a second known viewing angle based on the first Gaussian, where the second known viewing angle includes the first known viewing angle and the multiple randomly sampled viewing angles.

[0037] S160, based on a third loss function between the fourth color image and the real image, optimizing various attributes of the first Gaussian in the target scene to obtain a second Gaussian, wherein the real image includes an image captured by the camera at a camera posture corresponding to the first known viewing angle and the third color image.

[0038] S170: Render a fifth color image of the target scene at a new viewing angle based on the second Gaussian.

[0039] It should be explained that the color image in the embodiment of the present application may be in RGB format, that is, the color image is an RGB image. Optionally, the color image may also be in other formats, such as YUV format. The embodiment of the present application does not limit this.

[0040] In 3D reconstruction and rendering methods, a set of 3D Gaussians is usually used to sparsely describe the geometry and appearance information of the target scene. Each Gaussian can represent a region with certain attributes such as position, size, direction, and color, and the entire target scene is approximated by a combination of multiple Gaussians. Usually, the target scene has complex geometry, rich material properties, and diverse lighting conditions. By optimizing the Gaussian, it can better fit these scene features and more accurately describe the object distribution, surface details, and lighting changes in the scene, thereby providing a more accurate basis for high-quality image rendering.

[0041] Before executing the Gaussian image rendering method 100, the target scene is first divided into multiple non-overlapping blocks, and then the intersection of these blocks with the initial Gaussian is determined. For the initial Gaussian intersecting with multiple blocks, it is copied, and a relevant ID is assigned to each copy. Then, the IDs of all Gaussians in the target scene and all copied initial Gaussians are sorted according to depth, and the sorted IDs are used for rendering.

[0042] The initial Gaussian is the starting basis for the entire rendering operation. It contains an initial description of the target scene. For example, it may roughly outline the distribution of objects in the scene and the initial state of their appearance attributes through specific parameter settings (such as position, scale, shape, and other parameters). It is the basic information source for subsequent rendering of images from the corresponding perspective.

[0043] In S110, the processor first obtains a known camera pose from the camera, determines the first known viewing angle based on the known camera pose, and then renders a first color image of the target scene at the first known viewing angle when the initial Gaussian is known.

[0044] Unlike the known viewing angle, the virtual viewing angle may not correspond to the actual viewing angle when the image is actually captured, but is an angle artificially set to enrich the scene presentation, perform specific analysis, or meet rendering requirements. In one embodiment, the virtual viewing angle may be determined based on a randomly sampled camera pose. In another embodiment, the virtual viewing angle may also be determined based on the first known viewing angle.

[0045] In S120, after acquiring the virtual perspective, the processor renders the second color image and the depth image of the target scene under the virtual perspective when the initial Gaussian is known. For example, a camera pose is randomly sampled within a specified distance of the upper hemisphere of the target scene, and then an RGB image is rendered under the pose using a splatting method, and a depth value image is calculated from the Gaussian parameters.

[0046] The loss function is usually used to measure the difference between the rendered image and the expected image. For example, for the first loss function constructed based on the first color image, common types include mean squared error (MSE) loss, structural similarity index measure (SSIM) loss, perceptual loss, etc. For another example, for the second loss function constructed based on the second color image and the depth image, a loss function Lsmooth that measures the smoothness of the image can be used.

[0047] Optionally, the first loss function obtained based on the first color image and the second loss function obtained based on the second color image and the depth image optimize various properties of the initial Gaussian to obtain a first Gaussian, including: based on the average value L1 of the absolute difference of each pixel value between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, and the structural similarity loss function Lssim between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian, optimizing various properties of the initial Gaussian to obtain the first Gaussian.

[0048] In order to optimize the various attributes of the initial Gaussian using the constructed first loss function and the second loss function, it is necessary to select a suitable optimization algorithm. Common optimization algorithms include gradient descent, stochastic gradient descent (SGD), Adam optimization algorithm, etc. No matter which optimization algorithm is selected, it is necessary to calculate the gradient of the loss function with respect to the attribute parameters of the initial Gaussian. Then, in each iteration, the gradient of the loss function with respect to each parameter is calculated according to the current parameter value, and then the update rules of each optimization algorithm are used to combine the gradient and the previous moment estimate to update the various attribute parameters of the initial Gaussian, such as the center position and the elements of the covariance matrix, so that it is gradually adjusted in the direction of reducing the loss function value. This iterative update process will continue for multiple rounds, for example, the number of iterations is set to or the optimization process is ended according to a certain stop condition (such as the change in the loss function value is less than a certain threshold, indicating that it has converged to a better solution). After multiple rounds of iterative optimization, whether convergence is determined by observing the change in the loss function value. If the loss function value changes very little in several consecutive iterations, the optimization process is considered to have converged, and the Gaussian distribution determined by the attribute parameter values ​​of the initial Gaussian (updated center position, covariance matrix, etc.) is the optimized first Gaussian. Compared with the initial Gaussian, this first Gaussian is more reasonable and accurate in terms of distribution form in the target scene and degree of fit with the actual scene, and can generate higher quality images that are more in line with the real appearance of the target scene in the subsequent rendering steps.

[0049] In S140, after obtaining the first Gaussian, multiple perspectives may be randomly sampled, and based on the first Gaussian, a third color image of the target scene under the randomly sampled multiple perspectives may be rendered. For example, within a specified distance of the upper hemisphere of the target scene, the camera pose is randomly sampled, an RGB image is rendered, and the camera pose and the corresponding image combination are stored.

[0050] So far, the above steps have completed the entire process of estimating dense views using few views.

[0051] In S150, the first Gaussian is obtained through optimization in the previous step, and the second known perspective includes the first known perspective and a plurality of randomly sampled perspectives, wherein the first known perspective is obtained in the previous step, and the plurality of randomly sampled perspectives are determined by randomly sampled camera poses in S140. When the first Gaussian and the second known perspective are obtained, the fourth color image of the target scene at the second known perspective can be rendered based on the first Gaussian.

[0052] Similar to S110, before executing S150, the target scene is first divided into multiple non-overlapping blocks, and then the intersection of these blocks with the first Gaussian is determined. For the first Gaussian that intersects with multiple blocks, it is copied, and a relevant ID is assigned to each copy. Then, the IDs of all the first Gaussians in the target scene and all the copied first Gaussians are sorted according to depth, and the sorted first Gaussians are used for rendering.

[0053] Similar to S130, next, in S160, a third loss function between the fourth color image and the real image is calculated, and various attributes of the first Gaussian in the target scene are optimized to obtain a second Gaussian. The real image includes an image captured by the camera used in S110 at a known position and a third color image rendered at multiple randomly captured perspectives in S140. That is, the image captured by the camera used in S110 at a known position and the third color image rendered at multiple randomly captured perspectives in S140 are used together as the real image to increase the input image in Gaussian training.

[0054] Optionally, the optimizing the various properties of the first Gaussian in the target scene based on the third loss function between the fourth color image and the real image to obtain the second Gaussian includes: optimizing the various properties of the first Gaussian based on the absolute difference L1 of each pixel value between the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image to obtain the second Gaussian.

[0055] The detailed process of S160 can refer to S130, and for the sake of brevity, no detailed description is given here.

[0056] Finally, in S170, after the optimized second Gaussian is obtained, the camera pose of the desired synthesized image is input, that is, a new viewing angle is obtained, and splatting is used to render an RGB image under the new viewing angle from the second Gaussian.

[0057] In some embodiments, Figure 2As shown, the attributes of the Gaussian include position, and the method further includes: S180, before rendering the third color image, determining whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, moving the Gaussians in the first Gaussian whose sizes are smaller than the second threshold toward the direction of the position gradient change, or reducing the Gaussians in the first Gaussian whose sizes are greater than or equal to the second threshold by half of the initial size and copying them; and / or, before rendering the fifth color image, determining whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, moving the Gaussians in the second Gaussian whose sizes are smaller than the second threshold toward the direction of the position gradient change, or reducing the Gaussians in the second Gaussian whose sizes are greater than or equal to the second threshold by half of the initial size and copying them.

[0058] Specifically, S180 can be understood as a Gaussian adaptive densification process. For example, if the position gradient of a Gaussian (the first Gaussian or the second Gaussian) exceeds a threshold, the following operations are performed: for a relatively small Gaussian (under-reconstruction), a copy is made and moved in the direction of position gradient change. For a relatively large Gaussian (over-reconstruction), its scaling is reduced and a copy is made.

[0059] In some embodiments, Figure 2 As shown, the properties of the Gaussian include transparency, and the method further includes: S190, before rendering the third color image, deleting the Gaussians in the first Gaussian whose transparency is less than a third threshold; and / or, before rendering the fifth color image, deleting the Gaussians in the second Gaussian whose transparency is less than the third threshold.

[0060] Similarly, S190 can also be understood as a Gaussian adaptive densification process. For example, the Gaussian (the first Gaussian or the second Gaussian) with a transparency less than a threshold value can be deleted.

[0061] In some other embodiments, such as Figure 2 As shown, the method also includes: S195, before rendering the third color image, calculating the average value and standard value of the distances of each Gaussian in the first Gaussian to the nearest K neighboring Gaussians, and comparing the calculated value obtained based on the average value and the standard value of each Gaussian with a fourth threshold, and deleting the Gaussians in the first Gaussian whose calculated value exceeds the fourth threshold.

[0062] Step S195 can be understood as a Gaussian cloud denoising process. In this process, the focus is on performing a screening operation based on distance-related statistics on the first Gaussian before rendering the third color image. By analyzing the distance between each Gaussian in the first Gaussian and its nearest K neighboring Gaussians, the corresponding statistics (average value and standard value) are calculated, and a comprehensive calculated value is obtained, and then this calculated value is compared with the preset fourth threshold value to determine whether to delete certain Gaussians. The purpose is to optimize the distribution of the first Gaussian in the target scene and remove those Gaussians that are relatively "isolated" or do not conform to a certain distribution law, in the hope of obtaining better results when rendering the third color image later.

[0063] It should be noted that the thresholds involved in the above embodiments, for example, the first threshold, the second threshold, the third threshold, and the fourth threshold are all pre-set values. Similarly, the K value can also be a pre-set value. The embodiment of the present application does not limit the size of these thresholds and the K value.

[0064] It should be understood that the first known perspective and the virtual perspective in the embodiments of the present application can be one or more perspectives. Accordingly, the first color image rendered at the first known perspective and the second color image and depth image rendered at the virtual perspective can also be one or more images. The embodiments of the present application do not limit the number of first known perspectives and virtual perspectives.

[0065] It should also be understood that although the embodiment of the present application is described as decomposing one task into two tasks, in order to achieve a better rendering effect, one task can also be decomposed into more than two tasks. For example, after obtaining the second Gaussian in the embodiment of the present application, steps S140 to S160 can be repeatedly executed to finally obtain a better Gaussian of the target scene and synthesize an image of a new perspective in real time.

[0066] Combine the following Figure 3 A specific embodiment of the Gaussian image rendering method of the present application is described in detail.

[0067] like Figure 3 As shown, the Gaussian image rendering method includes:

[0068] Step 1: Render the RGB image of the target scene at a known viewing angle.

[0069] Step 2: Render the RGB image and depth image of the target scene under the virtual perspective.

[0070] Step 3: Gaussian attribute optimization. Obtain L1 and Lssim from the real image of known viewpoint and the RGB image rendered in step 1, obtain Lsmooth from the RGB image and depth image of the virtual viewpoint rendered in step 2, and obtain Lscale from the scaling scale of Gaussian. Gradient back propagation optimizes various attributes of Gaussian.

[0071] Step 4: Gaussian adaptive densification, see the description of S180 above.

[0072] Step 5: Gaussian cloud denoising. See the description of S195 above.

[0073] Step 6: Render a dense view. For example, within a specified distance in the upper hemisphere, randomly sample the camera pose, render the RGB image, and store the camera pose and image combination.

[0074] The above steps 1 to 6 are the process of intensive generation.

[0075] Step 7: Rendering a known-view RGB image, including the known view involved in step 1 and the dense view rendered in step 6.

[0076] Step 8: Gaussian attribute optimization: Obtain L1 and Lssim from real images and rendered images with known viewpoints, and optimize the various attributes of Gaussian using gradient backpropagation.

[0077] Step 9: Gaussian adaptive densification, see the description of S180 above.

[0078] Step 10: Render the new view. Enter the camera pose of the image you want to synthesize, and use splatting to render the corresponding RGB image from the Gaussian.

[0079] The above steps 7 to 10 enhance the synthesis process.

[0080] The present application also provides a Gaussian image rendering device. Figure 4 As shown, the Gaussian image rendering device 200 includes part or all of the following contents.

[0081] A first rendering unit 210, configured to render a first color image of a target scene at a first known viewing angle based on an initial Gaussian;

[0082] A second rendering unit 220, configured to render a second color image and a depth image of the target scene in a virtual perspective based on the initial Gaussian;

[0083] A first optimization unit 230, configured to optimize various properties of the initial Gaussian based on a first loss function obtained from the first color image, and a second loss function obtained from the second color image and the depth image, to obtain a first Gaussian;

[0084] A third rendering unit 240, configured to render a third color image of the target scene at a plurality of randomly sampled perspectives based on the first Gaussian;

[0085] A fourth rendering unit 250, configured to render a fourth color image of the target scene at a second known viewing angle based on the first Gaussian, where the second known viewing angle includes the first known viewing angle and the plurality of randomly sampled viewing angles;

[0086] A second optimization unit 260 is used to optimize various attributes of the first Gaussian in the target scene based on a third loss function between the fourth color image and a real image to obtain a second Gaussian, wherein the real image includes an image captured by the camera at a camera posture corresponding to the first known viewing angle and the third color image;

[0087] The fifth rendering unit 270 is configured to render a fifth color image of the target scene at a new viewing angle based on the second Gaussian.

[0088] In some embodiments, the first optimization unit 230 is specifically configured to:

[0089] Based on the average L1 of the absolute difference of each pixel value between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the structural similarity loss function Lssim between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian, various properties of the initial Gaussian are optimized to obtain the first Gaussian; and / or

[0090] The second optimization unit 260 is specifically used for:

[0091] Based on the absolute difference L1 of each pixel value between the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image, each attribute of the first Gaussian is optimized to obtain the second Gaussian.

[0092] In some embodiments, the attribute of the Gaussian includes a position, and the Gaussian image rendering apparatus further includes:

[0093] A Gaussian processing unit is used to determine whether the position gradient of the first Gaussian exceeds a first threshold before rendering the third color image, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians in the first Gaussian whose size is smaller than a second threshold toward the direction of the position gradient change, or reduce the Gaussians in the first Gaussian whose size is greater than or equal to the second threshold by half of the initial size and copy them; and / or, before rendering the fifth color image, determine whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, move the Gaussians in the second Gaussian whose size is smaller than the second threshold toward the direction of the position gradient change, or reduce the Gaussians in the second Gaussian whose size is greater than or equal to the second threshold by half of the initial size and copy them.

[0094] In some embodiments, the Gaussian attribute includes transparency, and the Gaussian image rendering device further includes:

[0095] The Gaussian processing unit is used to delete Gaussians whose transparency is less than a third threshold in the first Gaussian before rendering the third color image; and / or, before rendering the fifth color image, delete Gaussians whose transparency is less than the third threshold in the second Gaussian.

[0096] In some embodiments, the Gaussian image rendering device further includes:

[0097] A Gaussian processing unit is used to calculate the average value and standard value of the distances of each Gaussian in the first Gaussian to the nearest K neighboring Gaussians before rendering the third color image, and compare the calculated value obtained based on the average value and the standard value of each Gaussian with a fourth threshold, and delete the Gaussians in the first Gaussian whose calculated value exceeds the fourth threshold.

[0098] Based on the same idea, an embodiment of the present application also provides another Gaussian image rendering device. The Gaussian image rendering device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the Gaussian image rendering device executes the above-mentioned various method embodiments.

[0099] It should be noted that the detailed contents of the device side embodiment can be found in the method side embodiment, and for the sake of brevity, they will not be repeated here.

[0100] Based on the same idea, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned various method embodiments.

[0101] Although the present application has been described with reference to preferred embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, the various technical features mentioned in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A Gaussian image rendering method, characterized in that: include: Rendering a first color image of the target scene at a first known viewing angle based on the initial Gaussian; Based on the initial Gaussian, rendering a second color image and a depth image of the target scene in a virtual perspective; Based on a first loss function obtained from the first color image, and a second loss function obtained from the second color image and the depth image, optimizing various properties of the initial Gaussian to obtain a first Gaussian; Based on the first Gaussian, rendering a third color image of the target scene at a plurality of randomly sampled perspectives; Based on the first Gaussian, rendering a fourth color image of the target scene at a second known viewing angle, where the second known viewing angle includes the first known viewing angle and the plurality of randomly sampled viewing angles; Optimizing various attributes of the first Gaussian in the target scene based on a third loss function between the fourth color image and a real image to obtain a second Gaussian, wherein the real image includes an image captured by a camera at a camera posture corresponding to the first known viewing angle and the third color image; Based on the second Gaussian, a fifth color image of the target scene at a new viewing angle is rendered.

2. The Gaussian image rendering method according to claim 1, characterized in that: The first loss function obtained based on the first color image, the second loss function obtained based on the second color image and the depth image, optimizing various attributes of the initial Gaussian to obtain a first Gaussian, includes: Based on the average L1 of the absolute difference of each pixel value between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the structural similarity loss function Lssim between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian, various properties of the initial Gaussian are optimized to obtain the first Gaussian; and / or The optimizing each property of the first Gaussian in the target scene based on the third loss function between the fourth color image and the real image to obtain the second Gaussian includes: Based on the absolute difference L1 of each pixel value between the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image, each attribute of the first Gaussian is optimized to obtain the second Gaussian.

3. The Gaussian image rendering method according to claim 1, characterized in that: The properties of the Gaussian include position, and the method further includes: Before rendering the third color image, determine whether the position gradient of the first Gaussian exceeds a first threshold, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians whose sizes are smaller than the second threshold in the first Gaussian toward the direction of the position gradient change, or reduce the Gaussians whose sizes are greater than or equal to the second threshold in the first Gaussian to half of the initial size and copy them; and / or Before rendering the fifth color image, determine whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, move the Gaussians in the second Gaussian whose sizes are smaller than the second threshold toward the direction of the position gradient change, or reduce the Gaussians in the second Gaussian whose sizes are greater than or equal to the second threshold to half of the initial size and copy them.

4. The Gaussian image rendering method according to claim 1, characterized in that: The Gaussian property includes transparency, and the method further includes: Before rendering the third color image, deleting Gaussians whose transparency is less than a third threshold value in the first Gaussians; and / or Before rendering the fifth color image, delete Gaussians whose transparency is less than the third threshold in the second Gaussians.

5. The Gaussian image rendering method according to claim 1, characterized in that: The method further comprises: Before rendering the third color image, calculating an average value and a standard value of distances of each Gaussian in the first Gaussian to the nearest K neighboring Gaussians, and comparing a calculated value obtained based on the average value and the standard value of each Gaussian with a fourth threshold; The Gaussians whose calculated values ​​in the first Gaussians exceed the fourth threshold are deleted.

6. A Gaussian rendering device, characterized in that: include: A first rendering unit, configured to render a first color image of a target scene at a first known viewing angle based on an initial Gaussian; A second rendering unit, configured to render a second color image and a depth image of the target scene under a virtual perspective based on the initial Gaussian; A first optimization unit, configured to optimize various properties of the initial Gaussian based on a first loss function obtained from the first color image, and a second loss function obtained from the second color image and the depth image, to obtain a first Gaussian; A third rendering unit, configured to render a third color image of the target scene at a plurality of randomly sampled viewing angles based on the first Gaussian; a fourth rendering unit, configured to render a fourth color image of the target scene at a second known viewing angle based on the first Gaussian, wherein the second known viewing angle includes the first known viewing angle and the plurality of randomly sampled viewing angles; A second optimization unit is used to optimize various attributes of the first Gaussian in the target scene based on a third loss function between the fourth color image and a real image to obtain a second Gaussian, wherein the real image includes an image captured by the camera at a camera posture corresponding to the first known viewing angle and the third color image; A fifth rendering unit is used to render a fifth color image of the target scene at a new viewing angle based on the second Gaussian.

7. The Gaussian rendering device according to claim 1, characterized in that: The first optimization unit is specifically used for: Based on the average L1 of the absolute difference of each pixel value between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the structural similarity loss function Lssim between the first color image and the image collected by the camera at the camera posture corresponding to the first known viewing angle, the smoothing loss function Lsmooth of the second color image and the depth image, and the loss function Lscale obtained based on the scaling size of the Gaussian, various properties of the initial Gaussian are optimized to obtain the first Gaussian; and / or The second optimization unit is specifically used for: Based on the absolute difference L1 of each pixel value between the fourth color image and the real image and the structural similarity loss function Lssim between the fourth color image and the real image, each attribute of the first Gaussian is optimized to obtain the second Gaussian.

8. The Gaussian rendering device according to claim 1, characterized in that: The attribute of the Gaussian image includes a position, and the Gaussian image rendering device further includes: a Gaussian processing unit, configured to determine whether the position gradient of the first Gaussian exceeds a first threshold before rendering the third color image, and if the position gradient of the first Gaussian exceeds the first threshold, move the Gaussians whose sizes are smaller than a second threshold in the first Gaussian toward the direction of the position gradient change, or reduce the Gaussians whose sizes are greater than or equal to the second threshold in the first Gaussian by half of the initial size and copy them; and / or Before rendering the fifth color image, determine whether the position gradient of the second Gaussian exceeds the first threshold, and if the position gradient of the second Gaussian exceeds the first threshold, move the Gaussians in the second Gaussian whose sizes are smaller than the second threshold toward the direction of the position gradient change, or reduce the Gaussians in the second Gaussian whose sizes are greater than or equal to the second threshold to half of the initial size and copy them.

9. The Gaussian rendering device according to claim 1, characterized in that: The Gaussian attribute includes transparency, and the Gaussian image rendering device further includes: a Gaussian processing unit, configured to delete Gaussians whose transparency is less than a third threshold value in the first Gaussian before rendering the third color image; and / or Before rendering the fifth color image, delete Gaussians whose transparency is less than the third threshold in the second Gaussians.

10. The Gaussian rendering device according to claim 1, characterized in that: The Gaussian image rendering device also includes: A Gaussian processing unit is used to calculate the average value and standard value of the distances of each Gaussian in the first Gaussian to the nearest K neighboring Gaussians before rendering the third color image, and compare the calculated value obtained based on the average value and the standard value of each Gaussian with a fourth threshold, and delete the Gaussians in the first Gaussian whose calculated value exceeds the fourth threshold.

Images