A method and system for removing video floating artifacts based on neural radiation field

Abstract

The present invention discloses a method and system for removing floating artifacts from a video based on a neural radiation field, the method comprising: obtaining a training view, and then performing preliminary training on a fully connected neural network according to the training view to obtain a neural radiation field model; projecting each pixel point in the training view to the neural radiation field model to obtain a spatial radiation distribution, and obtaining a distribution map according to the spatial radiation distribution; calculating the weight of each pixel point according to the distribution map, and then performing weighted training on the neural radiation field model according to the weight and a preset coordinate truncation function to obtain a floating artifact removal model; obtaining a video image to be rendered, and inputting the video image to be rendered into the floating artifact removal model to generate a rendered image. The present invention can effectively prevent the occurrence of floating artifacts, and enhance the quality of the rendered image and the geometric fidelity of the reconstruction, and can be widely used in the field of three-dimensional reconstruction technology.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a method and system for removing video floating artifacts based on neural radiation fields. Background Art

[0002] NeRF technology can be applied to new perspective synthesis, scene expansion, and special effects production of videos. Through NeRF technology, producers can generate new perspectives and scenes based on existing video materials, bringing a richer visual experience to the audience. At the same time, NeRF technology can also be used to create complex special effects in videos, such as light and shadow effects, rendering of transparent and translucent objects, etc., to improve the overall quality of the video. Existing NeRF technology can synthesize realistic new perspective views in small-scale scenes, but when the task is extended to large-scale unbounded scenes, artifacts called floaters may appear. Such artifacts usually appear as translucent floating objects that are incorrectly modeled near the camera, causing the generated video to be blurred, reducing the rendering quality of the video and the geometric fidelity of the reconstruction. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a method and system for removing video floating artifacts based on neural radiation fields, which can improve the rendering quality of the video and the geometric fidelity of the reconstruction.

[0004] To achieve the above object, one aspect of an embodiment of the present application proposes a method for removing video floating artifacts based on a neural radiation field, comprising the following steps:

[0005] Acquire a training view, and then perform preliminary training on a fully connected neural network according to the training view to obtain a neural radiation field model;

[0006] Projecting each pixel point in the training view onto the neural radiation field model to obtain a spatial radiation distribution, and obtaining a distribution map according to the spatial radiation distribution;

[0007] Calculating the weight of each pixel point according to the distribution map, and then performing weighted training on the neural radiation field model according to the weight and a preset coordinate truncation function to obtain a floating artifact removal model;

[0008] The video image to be rendered is acquired, and the video image to be rendered is input into the floating artifact removal model to generate a rendered image.

[0009] In some embodiments, the step of projecting each pixel point in the training view onto the neural radiation field model to obtain the spatial radiation distribution specifically includes:

[0010] Inputting the training view and the camera pose corresponding to the training view into the neural radiation field model;

[0011] Projecting each pixel point in the training view onto the neural radiation field model to obtain three-dimensional spatial information corresponding to each pixel point;

[0012] Obtaining a depth map of each pixel at the camera position by volume rendering calculation according to the three-dimensional space information;

[0013] Obtaining the spatial position of each pixel point according to the depth map and orthogonal integral calculation;

[0014] Discretize the spatial position corresponding to each pixel point to obtain a voxel grid containing each pixel point;

[0015] The number of pixels of each voxel grid is calculated to obtain the spatial radiation distribution.

[0016] In some embodiments, the step of obtaining a distribution map according to the spatial radiation distribution is specifically:

[0017] A preset blank hash table is initialized to obtain a hash table, and then each of the voxel grids and the number of pixel points is stored in the hash table to obtain the distribution map.

[0018] In some embodiments, the step of calculating the weight of each pixel point according to the distribution map is specifically as follows:

[0019] The multiplicative inverse of the distribution map is used as the weight of each pixel point.

[0020] In some embodiments, the step of performing weighted training on the neural radiation field model according to the weight and the preset coordinate truncation function to obtain the floating artifact removal model specifically includes:

[0021] Using the weight as a coefficient of a regularization loss function to obtain a first loss, and using the weight as a coefficient of a reconstruction loss function to obtain a second loss;

[0022] A loss function is obtained according to the first loss and the second loss, and then weighted training is performed on the neural radiation field model according to the loss function and the coordinate truncation function to obtain the floating artifact removal model.

[0023] In some embodiments, the first loss is:

[0024]

[0025]

[0026] in, represents the first loss, represents the regularized loss function, W represents the weight, N represents the set of all rays r, s represents a set of regularized ray lengths, α represents the ray volume density distribution, u and v represent the spatial coordinates between any two pixel points on the ray r, α s (u) represents the interpolation of the step function at the spatial coordinate u, α s (v) represents the interpolated value of the step function at the spatial coordinate v.

[0027] In some embodiments, the coordinate truncation function is:

[0028]

[0029] in, represents the three-dimensional space coordinates of the pixel point, trancate(·) represents truncation of the three-dimensional space coordinates outside the central area, k represents a hyperparameter controlling the number of coordinate truncation bits, ||·|| ∞ Indicates L ∞ Norm.

[0030] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a video floating artifact removal system based on neural radiation field, comprising:

[0031] A model preliminary training module, used to obtain a training view, and then perform preliminary training on a fully connected neural network according to the training view to obtain a neural radiation field model;

[0032] A spatial radiation distribution acquisition module, used for projecting each pixel point in the training view onto the neural radiation field model to obtain spatial radiation distribution, and obtaining a distribution map according to the spatial radiation distribution;

[0033] A model weighted training module, used to calculate the weight of each pixel point according to the distribution map, and then perform weighted training on the neural radiation field model according to the weight and a preset coordinate truncation function to obtain a floating artifact removal model;

[0034] The rendering image generation module is used to obtain the video image to be rendered, input the video image to be rendered into the floating artifact removal model, and generate a rendering image.

[0035] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method for removing video floating artifacts based on neural radiation fields as described above is realized.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the video floating artifact removal method based on neural radiation field as described above.

[0037] The beneficial effects of the present invention are as follows: the method and system for removing floating artifacts from videos based on neural radiation fields of the present invention, on the one hand, mine the potential spatial radiation distribution in the input training view through a pre-trained neural radiation field model, and then obtain the weight corresponding to each pixel point according to the obtained spatial radiation distribution, and redistribute the gradient according to the weight, so as to ensure that the sparse sampling area has a stronger gradient in the subsequent optimization stage, thereby effectively preventing the occurrence of floating artifacts; on the other hand, the coordinate truncation function is used to gradually truncate the spatial coordinates according to the distance of the pixel point from the center, so that the voxel scale of the distant area is gradually increased, effectively enhancing the quality of the rendered image and the geometric fidelity of the reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A flowchart of the steps of a method for removing floating artifacts in a video based on a neural radiation field provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of comparison between a reference image and a distribution map provided in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram comparing the effects of different hyperparameters on coordinate truncation provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of qualitative results of a background refinement and ablation experiment provided by an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of qualitative results of a weighted regularization ablation experiment provided by an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the structure of a video floating artifact removal system based on neural radiation field provided by an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0047] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0048] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0049] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0050] NeRF (Neural Radiance Fields) uses multi-layer perceptrons (MLPs) as implicit equations to represent continuous space. By minimizing the color loss between the rendered image and the captured image at a given camera pose, 3D scenes can be directly optimized using only RGB image input. After NeRF converges, the color and volume density of any point in space can be evaluated to achieve new perspective rendering.

[0051] NeRF technology can be applied to new perspective synthesis, scene extension, and special effects production of videos. Through NeRF technology, producers can generate new perspectives and scenes based on existing video materials, bringing a richer visual experience to the audience. At the same time, NeRF technology can also be used to create complex special effects in videos, such as light and shadow effects, rendering of transparent and translucent objects, etc., to improve the overall quality of the video. Although NeRF can synthesize realistic new views in small-scale scenes, artifacts may appear when the task is extended to large-scale indoor or outdoor environments. This type of artifact is called "floating artifact" or "background collapse". One of the main reasons for their occurrence is that the radiation sampling of distant surfaces in the training images is too sparse. Given a limited set of images, the reconstruction problem of NeRF is ill-posed and there are degenerate solutions. Due to the ambiguity between shape and radiation (captured images), the model may generate incorrect geometry for a set of training images. However, as the number of captured viewpoints increases, NeRF is more likely to converge towards the correct surface close to the truth during the optimization process. On the other hand, when dealing with larger-scale scenes, due to the imbalance of radiation sampling, the increased number of training views can only guarantee partially accurate geometry. And since most scenes are usually captured with a central focus, distant objects may be undersampled. Therefore, for more distant objects, surface radiosity sampling becomes more sparse, increasing the likelihood of reconstructing incorrect geometry.

[0052] To this end, an embodiment of the present invention proposes a method for removing floating artifacts in a video based on a neural radiation field. On the one hand, the potential spatial radiation distribution in the input training view is mined through a pre-trained neural radiation field model, and then the weight corresponding to each pixel point is obtained according to the obtained spatial radiation distribution, and the gradient is redistributed according to the weight, ensuring that the sparse sampling area has a stronger gradient in the subsequent optimization stage, thereby effectively preventing the appearance of floating artifacts; on the other hand, the coordinate truncation function is used to gradually truncate the spatial coordinates according to the distance of the pixel point from the center, so that the voxel scale of the distant area gradually increases, effectively enhancing the quality of the rendered image and the geometric fidelity of the reconstruction. The floating artifact removal method can be applied to scenes such as distant view rendering, character and object rendering, and special effect scene synthesis, but is not limited to this.

[0053] Reference Figure 1 , Figure 1 The present invention provides a flowchart of a method for removing floating artifacts in a video based on a neural radiation field according to an embodiment of the present invention. The present invention provides a method for removing floating artifacts in a video based on a neural radiation field, and the method includes steps S101 to S104:

[0054] S101, obtaining a training view, and then performing preliminary training on a fully connected neural network according to the training view to obtain a neural radiation field model;

[0055] Specifically, the training views can be obtained using RGB images taken by mobile devices or consumer cameras. These images need to be from different perspectives and angles to cover the entire scene. For each image, the internal and external parameters of the camera, such as camera position, orientation, focal length, etc., need to be extracted. These parameters describe the position and orientation of the camera, as well as the perspective projection of the image. Usually, the camera parameters corresponding to the image are obtained by preprocessing the image using COLMAP software during the preprocessing process. COLMAP is an open source computer vision software package for reconstructing three-dimensional scenes from image collections. It combines SfM (Structure-from-Motion) and MVS (Multi-View Stereo) technologies to estimate the camera position, posture, and three-dimensional structure in the scene from a camera image sequence. In addition, public datasets used for NeRF usually include camera parameter data.

[0056] S102, projecting each pixel point in the training view onto the neural radiation field model to obtain spatial radiation distribution, and obtaining a distribution map according to the spatial radiation distribution;

[0057] Specifically, the latent spatial radiation distribution in the input pixel space is mined through a pre-trained neural radiation field model. The obtained distribution is then used for weighted regularization and reconstruction loss. Unlike previous methods that require deep priors from pre-trained models or sensors, the embodiments of the present invention reuse the neural radiation field pipeline to obtain a "free" spatial radiation distribution for subsequent scene fine-tuning.

[0058] As an optional implementation, the step of projecting each pixel point in the training view onto the neural radiation field model to obtain the spatial radiation distribution can be specifically divided into the following steps S1021 to S1026:

[0059] S1021, inputting the training view and the camera pose corresponding to the training view into the neural radiation field model;

[0060] S1022, projecting each pixel point in the training view onto the neural radiation field model to obtain three-dimensional spatial information corresponding to each pixel point;

[0061] S1023, obtaining a depth map of each pixel under the camera posture through volume rendering calculation according to the three-dimensional space information;

[0062] S1024, obtaining the spatial position of each pixel point according to the depth map and orthogonal integral calculation;

[0063] Specifically, an embodiment of the present invention regards the pixel points of the training view as samples of the neural radiation field model to model the spatial distribution of captured radiation, wherein the initial neural radiation field model is optimized without any modification until convergence. In order to obtain the spatial distribution of surface radiation, each pixel point is reprojected to its corresponding surface position. The neural radiation field is a three-dimensional spatial model that can capture the geometry and color information of the scene. The reprojection process maps two-dimensional pixels to corresponding positions in three-dimensional space in order to utilize the three-dimensional information in the neural radiation field model. For each training view, the volume rendering integral is performed on the depth of all points along the ray by the following formula to obtain the corresponding depth map

[0064]

[0065] in, Represents the depth map corresponding to ray r, x i represents the coordinates (spatial position) of the i-th pixel, T i represents the transmittance of the i-th pixel, α i =1-exp(-α i δ i ), δ i represents the sampling step of the i-th pixel, N represents the set of all pixels, o represents the camera origin, ox i Indicates the depth of the i-th pixel.

[0066] The spatial position x of each pixel i Obtained by the following formula:

[0067]

[0068] Where d represents the viewing direction.

[0069] S1025, discretizing the spatial position corresponding to each pixel point to obtain a voxel grid containing each pixel point;

[0070] S1026. Calculate the number of pixels in each voxel grid to obtain spatial radiation distribution.

[0071] Specifically, in order to obtain the spatial radiation distribution of these surface points (pixel points), the continuous neural radiation field is discretized into a voxel grid to represent the distribution of surface points within the voxel grid. That is, the space is discretized into a resolution of h 3 The number of surface points (number of pixels) in each voxel grid is calculated and approximated as the spatial radiation distribution of the surface radiation.

[0072] As an optional implementation, the step of obtaining a distribution map according to the spatial radiation distribution may specifically be the following step S1027:

[0073] S1027, initializing a preset blank hash table to obtain a hash table, and then storing the number of each voxel grid and pixel point in the hash table to obtain a distribution map.

[0074] Specifically, since there are many voxels in the discrete space, but the voxels with pixels are relatively sparse, a hash table structure is used to store the voxel grid and the number of pixels inside it for the sake of space and time efficiency. i , using the hash value h(y) of the corresponding voxel grid i ) queries the hash table and increases the number of pixels in the voxel grid by 1, then traverses the next pixel until all pixels are traversed. Finally, traverse all pixels again. i , query the number of pixels in the voxel grid as the adjacent pixel n i , the more pixels there are, the more the surface of the object it represents is captured, and a distribution map is obtained. Figure 2 The figure shows the comparison between the reference image and the distribution map. Figure 2 Reference images in the top row, visualization of distribution maps in the bottom row, where brightness represents the density of sampled radiance on the corresponding surface. Since the "garden" scene shown (top row) is shot around a table, the surface of the table is densely sampled in all training images, resulting in a prominent area in the distribution map. On the other hand, trees and buildings in the distance are only captured in a few images, causing them to appear dim or even almost invisible in the distribution map.

[0075] S103, calculating the weight of each pixel point according to the distribution map, and then performing weighted training on the neural radiation field model according to the weight and a preset coordinate truncation function to obtain a floating artifact removal model;

[0076] As an optional implementation, the step of calculating the weight of each pixel point according to the distribution map can be specifically divided into the following steps S1031:

[0077] S1031, using the multiplication inverse of the distribution map as the weight of each pixel;

[0078] Specifically, the neighboring pixel n i The reciprocal of is taken as the weight W of each pixel i ,Right now In the embodiment of the present invention, the floating artifact removal process is guided by the weight of each pixel point. Those areas with lower radiation intensity (i.e., areas with more obvious floating artifacts) will be given greater weights, so that they will receive more attention and processing during the reconstruction process, thereby specifically reducing or eliminating floating artifacts in the video.

[0079] As an optional implementation, the neural radiation field model is weighted trained according to the weight and the preset coordinate truncation function to obtain the floating artifact removal model. This step can be specifically divided into the following steps S1032 and S1033:

[0080] S1032, using the weight as a coefficient of a regularization loss function to obtain a first loss, and using the weight as a coefficient of a reconstruction loss function to obtain a second loss;

[0081] S1033. Obtain a loss function according to the first loss and the second loss, and then perform weighted training on the neural radiation field model according to the loss function and the coordinate truncation function to obtain a floating artifact removal model.

[0082] As a further optional implementation, the first loss is:

[0083]

[0084]

[0085] in, represents the first loss, represents the regularized loss function, W represents the weight, N represents the set of all rays r, s represents a set of regularized ray lengths, α represents the ray volume density distribution, u and v represent the spatial coordinates between any two pixel points on the ray r, α s (u) represents the interpolation of the step function at the spatial coordinate u, α s (v) represents the interpolated value of the step function at the spatial coordinate v.

[0086] Specifically, when NeRF is extended to large-scale scenes, floating artifacts may occur. The existing technology introduces a regularized loss function to alleviate these artifacts, encouraging a single-peak volume density along each ray. Let s represent a set of regularized ray distances, and the regularized loss function can be expressed as follows:

[0087]

[0088] Although a regularization loss function is introduced to constrain geometric shapes and prevent the generation of floating objects, directly applying the regularization term globally in other models similar to NeRF may not effectively prevent the appearance of floating artifacts. When the regularization strength is too low, the appearance of floating artifacts cannot be prevented, and too high a strength will cause the model to fail to converge. Floating artifacts only appear in sparsely sampled areas, so stronger constraints on geometric shapes need to be imposed in these areas. Therefore, in the optimization process, an embodiment of the present invention assigns higher regularization weights to relevant rays (pixels), that is, uses the weight W obtained above as the coefficient of the regularization loss function, and obtains the first loss function of the following formula:

[0089]

[0090] If NeRF samples pixels uniformly from all images, the gradient will be dominated by the main objects in the scene, which are more likely to be sampled. This unbalanced gradient distribution may cause the details of distant objects to be lost or blurred. In order to compensate for the sampling imbalance, the embodiment of the present invention proposes a weighted reconstruction loss to scale the gradient of the sparsely sampled area, that is, to incorporate the weight W into the reconstruction loss, amplify the gradient of the distant surface, and improve the surface quality, and obtain the second loss of the following formula:

[0091]

[0092] Among them, W r represents the weight of ray r, represents the predicted color value of ray r, C(r) represents the actual color value of ray r, and N represents the set of all rays r.

[0093] The overall loss function As shown below, it includes distortion regularization loss (first loss) and weighted reconstruction loss (second loss):

[0094]

[0095] Among them, λ represents the loss weight.

[0096] As a further optional implementation, the coordinate truncation function is:

[0097]

[0098] in, represents the three-dimensional spatial coordinates of the pixel point, trancate(·) represents the truncation of the three-dimensional spatial coordinates outside the central area, k represents the hyperparameter controlling the number of coordinate truncation bits, ||·|| ∞ Indicates L ∞ Norm.

[0099] Specifically, the original NeRF projects along rays and samples infinitesimal voxels along them, optimizing voxels that are close to the ground truth surface. This method works well for densely sampled surface points because there are a sufficient number of optimized voxels to represent the surface of the object. However, as the rays go deeper into the scene, the distribution of surface points (pixels) becomes increasingly sparse. When voxels are sampled at a location far from the center of the scene, it is difficult for infinitesimal and sparse voxels to effectively represent the details of distant surfaces. Therefore, an embodiment of the present invention proposes a novel coordinate reparameterization method, which enlarges the voxels according to the distance of the voxels from the central area, that is, gradually truncates the spatial coordinates according to the distance of the pixel from the center, so that the voxel scale in the distant area gradually increases. The spatial coordinates outside the central area are truncated by the coordinate truncation function, so that the resolution outside the central area of ​​the scene is reduced, so that the low-resolution radiation field can better represent the rough surface captured farther away. As Figure 3 The following is a comparison diagram of the effects of different hyperparameters on coordinate truncation. Figure 3 It can be seen that a smaller hyperparameter k will lead to better details of distant surfaces (black boundaries). In the embodiment of the present invention, the hyperparameter k is minimized according to the distance of the pixel from the center.

[0100] S104 , obtaining a video image to be rendered, and inputting the video image to be rendered into a floating artifact removal model to generate a rendered image.

[0101] Specifically, by inputting the video image to be rendered into the trained neural radiation field model, a clearer and higher-quality rendered image can be obtained, bringing a richer visual experience to the audience.

[0102] The above is an explanation of the video floating artifact method based on the neural radiation field of the embodiment of the present invention. It can be recognized that, compared with the prior art, the embodiment of the present invention, on the one hand, mines the potential spatial radiation distribution in the input training view through the pre-trained neural radiation field model, and then obtains the weight corresponding to each pixel point according to the obtained spatial radiation distribution, and redistributes the gradient according to the weight, ensuring that the sparse sampling area has a stronger gradient in the subsequent optimization stage, thereby effectively preventing the occurrence of floating artifacts; on the other hand, the coordinate truncation function is used to gradually truncate the spatial coordinates according to the distance of the pixel point from the center, so that the voxel scale of the distant area gradually increases, effectively enhancing the quality of the rendered image and the geometric fidelity of the reconstruction, and removing floating artifacts in the video.

[0103] In order to further verify the accuracy of the embodiment of the present invention, the effect of the embodiment of the present invention is further described below in combination with experiments.

[0104] The experiment of the embodiment of the present invention is implemented based on Instant-NGP, which is a hash-based NeRF variant. Since Instant-NGP does not solve the problem of floating artifacts, it is selected as a benchmark model to demonstrate the effectiveness of the floating artifact method proposed in the embodiment of the present invention. The training process includes two stages: in the first stage, Instant-NGP is trained without any modification until the radiation field converges; in the second stage, the radiation field is fine-tuned based on the floating artifact method proposed in the embodiment of the present invention to eliminate floating artifacts and improve the rendering quality. In order to be fair in the experiment, Instant-NGP and the floating artifact method proposed in the embodiment of the present invention perform the same number of iterations. Since the floating artifact method proposed in the embodiment of the present invention utilizes the regularization term proposed by Mip-NeRF 360, Instant-NGP will also incorporate the same term. Specifically, in the first stage, the model is trained for 20,000 iterations until convergence. Afterwards, the floating artifact method proposed in the embodiment of the present invention is used for additional fine-tuning for 30,000 iterations. Similarly, the baseline model Instant-NGP is additionally trained for 30,000 iterations without any modification. Following the setting of Mip-NeRF 360, the views are split into training and testing views in a ratio of 8:1.

[0105] Experiments are conducted on the large-scale unbounded dataset introduced in Mip-NeRF 360, which includes three outdoor scenes and four indoor scenes (two unpublished outdoor scenes are excluded). The dataset contains a series of RGB images taken from different angles within the scene. However, it may still contain sparse or unobserved angles, which represents what may happen in real-world applications. Objects within the scene are static and are always under consistent exposure. In addition, the evaluation focuses on the rendering quality of new view synthesis, using evaluation metrics from previous NeRF methods: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Perceptual Similarity Index (LPIPS).

[0106] Experiment 1: Qualitative comparison

[0107] In order to evaluate the effectiveness of the floating artifact method proposed in the embodiment of the present invention, it is compared with previous works, including Instant-NGP, DVGO and Plenoxels. Among them, Instant-NGP uses multiple hash tables to encode spatial features at different resolutions and uses a small MLP for decoding; DVGO uses a dense voxel grid to represent the scene geometry, and uses a feature voxel grid and a shallow MLP to represent the view-dependent appearance; Plenoxels avoids the use of neural networks, but only relies on sparse grids to model geometry and view-dependent appearance. In the more traditional NeRF framework, the floating artifact method proposed in the embodiment of the present invention is compared with the original NeRF, NeRF++ and Mip-NeRF 360. These methods only use neural networks as implicit representations of the scene, resulting in relatively slow training and rendering speeds. Among them, NeRF++ and Mip-NeRF360 reparameterize the space to extend NeRF to unbounded scenes with larger scales. For DVGO, Plenoxels, NeRF and NeRF++, the embodiment of the present invention uses nerf-factory implementation and adopts the default iteration settings. Since continuing iterations after the radiation field converges may result in more artifacts and reduced rendering quality, the results obtained by using default settings at convergence of these methods are selected. For Instant-NGP, alignment with the floating artifact method proposed in the embodiment of the present invention was performed by using 20,000 and 50,000 iterations.

[0108] The quantitative evaluation results of the comparative experiment are shown in Table 1 below. Considering the strong correlation between the reconstruction quality and the scene scale, the results are divided into two parts: indoor and outdoor for comparison. It can be seen from Table 1 that the floating artifact method proposed in the embodiment of the present invention can effectively surpass the original NeRF, NeRF++, Instant-NGP, DVGO and Plenoxels. There are two reasons why the floating artifact method proposed in the embodiment of the present invention can surpass the baseline of fast NeRF: first, the weighted regularization strategy enables the model to apply stronger regularization terms without compromising the overall rendering quality, significantly reducing floating artifacts; second, the gradient weights of distant surfaces are amplified in the embodiment of the present invention, so that the network can better model the global rather than the central area.

[0109]

[0110] Table 1

[0111] Experiment 2: Ablation experiment

[0112] Ablation experiments are conducted on the distortion regularization loss, weighted reconstruction loss, and coordinate reparameterization method proposed in the embodiment of the present invention. The quantitative evaluation results of the ablation experiment are shown in Table 2 below. It can be seen from Table 2 that each proposed module contributes to the final rendering quality.

[0113]

[0114] Table 2

[0115] Background refinement: We performed ablation experiments by fine-tuning without using the weighted reconstruction loss, which means training the model for more iterations. Figure 4 The following is a schematic diagram of the qualitative results of the background refinement and ablation experiment. Figure 4 It can be seen that after background refinement, the texture on the distant wall is restored. The embodiment of the present invention also conducts experiments on outdoor scenes, and the quantitative evaluation results of the background refinement ablation experiment are shown in Table 3 below. It can be seen that the embodiment of the present invention can slightly improve PSNR and SSIM and slightly reduce LPIPS by fine-tuning the model using weighted reconstruction loss.

[0116]

[0117] Table 3

[0118] Weighted regularization: In the weighted regularization ablation study, the rendering quality of strategy regularization (i.e., the use of weights as coefficients of the weighted regularization function proposed in the embodiment of the present invention), global regularization, and no regularization are compared. The loss weight value λ in the loss function is set to 0.001 and 0.004, respectively, to conduct experiments on global regularization and strategy regularization, respectively. The quantitative evaluation results of the weighted regularization ablation experiment are shown in Table 4 below. It can be seen from Table 4 that the use of global regularization leads to a decrease in rendering quality, and the extreme global regularization term (when the λ value is 0.004) will also cause a sharp drop in rendering quality; in contrast, the use of strategy regularization can ensure rendering quality while eliminating artifacts, and the extreme strategy regularization term (when the λ value is 0.004) does not significantly reduce the rendering quality, indicating the robustness of the floating artifact removal method proposed in the embodiment of the present invention to the regularization weight. As Figure 5 The figure shows the qualitative results of the weighted regularization ablation experiment. Figure 5From left to right in each row: NGP 20K iteration rendering depth map, NGP 50K iteration rendering depth map, NGP 20K iteration and 30K fine-tuning (embodiment of the present invention) depth map and the image rendered by the embodiment of the present invention. Continuing training after NGP converges (first column) may not improve the reconstruction quality, but may introduce more artifacts (second column), while further training using the method proposed in the embodiment of the present invention (third column) is proven to be effective, which can eliminate artifacts and generate clearer and higher quality geometric shapes, indicating that the floating artifact removal method proposed in the embodiment of the present invention can effectively eliminate artifacts.

[0119]

[0120] Table 3

[0121] Coordinate reparameterization: As shown in Table 2, if the coordinate reparameterization method is not used in model training, its PSNR and SSIM will be slightly reduced. Coordinate reparameterization mainly enhances the details of the distant surface of the image, and the distant surface only constitutes a small part of the image, so its contribution to the overall index is not very significant. To further illustrate the effectiveness of the coordinate reparameterization method, use Eq. k is used instead of the coordinate truncation function to perform coordinate truncation, and k is set to 1000, 500, and 250 respectively to perform coordinate truncation on the entire scene. Figure 3 , the rendering quality of the surface at different distances under different coordinate cutoffs is shown, and it can be seen that the coordinate reparameterization method proposed in the embodiment of the present invention is effective in improving the rendering quality.

[0122] Reference Figure 6 The embodiment of the present invention further provides a video floating artifact system based on neural radiation field, including:

[0123] A model preliminary training module is used to obtain a training view, and then perform preliminary training on the fully connected neural network according to the training view to obtain a neural radiation field model;

[0124] A spatial radiation distribution acquisition module is used to project each pixel point in the training view onto the neural radiation field model to obtain the spatial radiation distribution, and obtain a distribution map according to the spatial radiation distribution;

[0125] The model weighted training module is used to calculate the weight of each pixel point according to the distribution map, and then perform weighted training on the neural radiation field model according to the weight and the preset coordinate truncation function to obtain a floating artifact removal model;

[0126] The rendering image generation module is used to obtain the video image to be rendered, input the video image to be rendered into the floating artifact removal model, and generate a rendering image.

[0127] The contents of the above-mentioned video floating artifact method embodiment based on neural radiation field are all applicable to the present video floating artifact system embodiment based on neural radiation field. The functions specifically implemented by the present video floating artifact system embodiment based on neural radiation field are the same as those of the above-mentioned video floating artifact method embodiment based on neural radiation field, and the beneficial effects achieved are also the same as those achieved by the above-mentioned video floating artifact method embodiment based on neural radiation field.

[0128] The embodiment of the present invention also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned video floating artifact method based on neural radiation field. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0129] like Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention, referring to FIG. Figure 7 , an embodiment of the present invention provides an electronic device, including:

[0130] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0131] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the video floating artifact method based on the neural radiation field of the embodiment of the present invention;

[0132] Input / output interface 1003, used to implement information input and output;

[0133] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0134] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0135] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0136] An embodiment of the present invention also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned video floating artifact method based on neural radiation field.

[0137] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.

[0139] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0140] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0141] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0143] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0144] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0145] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for removing floating artifacts in a video based on neural radiation field, characterized in that: The following steps are involved: Acquire a training view, and then perform preliminary training on a fully connected neural network according to the training view to obtain a neural radiation field model; Projecting each pixel point in the training view onto the neural radiation field model to obtain a spatial radiation distribution, and obtaining a distribution map according to the spatial radiation distribution; Calculating the weight of each pixel point according to the distribution map, and then performing weighted training on the neural radiation field model according to the weight and a preset truncated signed distance function to obtain a floating artifact removal model; Acquire a video image to be rendered, input the video image to be rendered into the floating artifact removal model, and generate a rendered image; The step of performing weighted training on the neural radiation field model according to the weight and the preset truncated signed distance function to obtain the floating artifact removal model specifically includes: Using the weight as a coefficient of a regularization loss function to obtain a first loss, and using the weight as a coefficient of a reconstruction loss function to obtain a second loss; A loss function is obtained according to the first loss and the second loss, and then weighted training is performed on the neural radiation field model according to the loss function and the truncated signed distance function to obtain the floating artifact removal model.

2. According to claim 1, a method for removing video floating artifacts based on neural radiation field is characterized in that: The step of projecting each pixel point in the training view onto the neural radiation field model to obtain the spatial radiation distribution specifically includes: Inputting the training view and the camera pose corresponding to the training view into the neural radiation field model; Projecting each pixel point in the training view onto the neural radiation field model to obtain three-dimensional spatial information corresponding to each pixel point; Obtaining a depth map of each pixel at the camera position by volume rendering calculation according to the three-dimensional space information; Obtaining the spatial position of each pixel point according to the depth map and orthogonal integral calculation; Discretize the spatial position corresponding to each pixel point to obtain a voxel grid containing each pixel point; The number of pixels of each voxel grid is calculated to obtain the spatial radiation distribution.

3. According to claim 2, a method for removing video floating artifacts based on neural radiation field is characterized in that: The step of obtaining a distribution map according to the spatial radiation distribution is specifically as follows: A preset blank hash table is initialized to obtain a hash table, and then each of the voxel grids and the number of pixel points is stored in the hash table to obtain the distribution map.

4. According to claim 1, a method for removing video floating artifacts based on neural radiation field is characterized in that: The step of calculating the weight of each pixel point according to the distribution map is specifically as follows: The multiplicative inverse of the distribution map is used as the weight of each pixel point.

5. The method for removing floating artifacts in a video based on neural radiation field according to claim 1, characterized in that: The first loss is: ； ； in, represents the first loss, represents the regularized loss function, represents the weight, Represents all rays A collection of represents a set of regularized ray lengths, represents the ray volume density distribution, and Represents the ray The spatial coordinates between any two pixels on the Represented in space coordinates The interpolation of the step function at , Represented in space coordinates Interpolation of the step function at .

6. The method for removing video floating artifacts based on neural radiation field according to claim 1 is characterized in that: The truncated signed distance function is: ； in, represents the three-dimensional space coordinates of the pixel point, Indicates that the three-dimensional space coordinates outside the central area are truncated, represents the hyperparameter controlling the number of coordinate cutoffs, express Norm.

7. A video floating artifact removal system based on neural radiation field, characterized in that: include: A model preliminary training module, used to obtain a training view, and then perform preliminary training on a fully connected neural network according to the training view to obtain a neural radiation field model; A spatial radiation distribution acquisition module, used for projecting each pixel point in the training view onto the neural radiation field model to obtain spatial radiation distribution, and obtaining a distribution map according to the spatial radiation distribution; A model weighted training module, used for calculating the weight of each pixel point according to the distribution map, and then performing weighted training on the neural radiation field model according to the weight and a preset truncated signed distance function to obtain a floating artifact removal model; A rendering image generation module, used for acquiring a video image to be rendered, inputting the video image to be rendered into the floating artifact removal model, and generating a rendering image; The step of performing weighted training on the neural radiation field model according to the weight and the preset truncated signed distance function to obtain the floating artifact removal model specifically includes: Using the weight as a coefficient of a regularization loss function to obtain a first loss, and using the weight as a coefficient of a reconstruction loss function to obtain a second loss; A loss function is obtained according to the first loss and the second loss, and then weighted training is performed on the neural radiation field model according to the loss function and the truncated signed distance function to obtain the floating artifact removal model.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method for removing video floating artifacts based on neural radiation fields as described in any one of claims 1 to 6 are realized.

9. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the method for removing video floating artifacts based on neural radiation fields as described in any one of claims 1 to 6.

Images