A fast radiated field reconstruction method under sparse view input

Abstract

The present application relates to a kind of sparse view input under the fast radiation field reconstruction method of view, the position of the edge pixel of profile map in space is calculated, and the axis-aligned bounding box of the target to be reconstructed is calculated;The voxelization is carried out to axis-aligned bounding box, and the voxel confidence to be optimized and the learnable voxel feature vector of each voxel vertex are simultaneously allocated as the three-dimensional representation of this local region;The voxel confidence of each voxel vertex, by projecting the voxel vertex to the profile map of each input image, the calculation of visible view number is carried out, and the voxel confidence is initialized;Volume rendering is carried out to input RGB image, and the voxel confidence and voxel feature vector of voxel vertex and multilayer perception are optimized by reconstructing luminosity error loss and total differential error loss, and the radiation field representation of scene is obtained;For optimization process, the geometry estimation of radiation field is constantly refined by periodically carrying out voxel pruning, and more refined radiation field representation is obtained.

Description

Technical Field

[0001] This invention relates to a fast radiation field reconstruction method under sparse viewpoint input conditions, applicable to the field of novel viewpoint synthesis under sparse viewpoint input conditions. Background Technology

[0002] Novel perspective synthesis is a hot topic of common interest in computer vision and computer graphics. Specifically, the novel perspective synthesis task can be summarized as capturing an object from a known perspective (including the captured images and their corresponding intrinsic and extrinsic parameters), reconstructing the object in 3D from the captured images (including the object's geometry, surface material, lighting conditions, etc.), and then synthesizing an image of the object from an unknown perspective. Unlike traditional 3D reconstruction, the goal of novel perspective synthesis is to synthesize realistic images from unknown perspectives rather than explicit 3D reconstruction results. In recent years, the emergence of radiation fields has attracted significant attention to implicit scene representation, leading to a series of studies analyzing, optimizing, and extending radiation field representation methods, such as improving radiation field reconstruction and rendering efficiency, studying scene generalization, and reconstructing radiation fields on larger scales.

[0003] While radiation fields can synthesize high-quality images from new perspectives, they suffer from two main problems. First, training a radiation field requires a large number of images from different perspectives (approximately 50 different perspectives for a forward-facing camera layout and approximately 100 different perspectives for an inward-facing camera layout). However, in practical 3D reconstruction tasks, acquiring and calibrating a large number of images from different perspectives is typically very laborious. Furthermore, when the input perspectives are sparse (e.g., 4 different perspectives), the radiation field often overfits to the training perspectives, leading to a significant decrease in the accuracy of geometric reconstruction. This problem arises from the ambiguity between geometry and color introduced by the perspective-dependent radiation field modeling method used to account for the change in object appearance color with viewing angle. Specifically, for a scene or object to be optimized, even with a completely incorrect geometric estimate, there always exists a suitable radiation field that perfectly fits the incorrect geometry to the image at the known perspective. This incorrect geometric estimate prevents image synthesis at new perspectives. In addition, training the radiation field and rendering images from new perspectives using it are both time-consuming. This is mainly due to two reasons: First, the radiance field representation uses a multilayer perceptron with eight 256-dimensional hidden layers as the implicit radiance field representation, meaning that a single radiance field query is time-consuming. Second, during the training and rendering of new viewpoint images, hundreds of radiance field queries are required for the light emitted by each pixel in the image. A large number of these sampling points are located in empty areas of space and do not contribute to the final color value. Typically, training the radiance field for a single scene on a single GPU takes about 10 hours, and using a pre-trained radiance field to render a 400-pixel image... A new perspective image at 400 dpi would take several minutes to render, making real-time rendering difficult.

[0004] The two main limitations of radiation fields mentioned above make them difficult to apply in practical scenarios such as augmented reality and robot navigation. Therefore, many researchers have made significant efforts to extend radiation fields to sparse viewpoint inputs and to improve their training and rendering efficiency. On the one hand, in sparse viewpoint inputs, researchers mainly improve the reconstruction accuracy of radiation fields by introducing additional pre-trained networks based on cross-scenes and adding regularization strategies. Pre-trained networks based on cross-scenes require several days of training on large datasets and several hours of scene-by-scene optimization to achieve satisfactory radiation field reconstruction results. Furthermore, methods that introduce regularization strategies often significantly increase convergence time due to the introduction of additional ray sampling at unknown viewpoints and the computation of additional loss functions. On the other hand, to improve the efficiency of radiation field reconstruction and training and rendering, researchers typically use a combination of traditional explicit and implicit representations to reduce the number of network queries and simplify the complex multilayer perceptron structure. However, like traditional radiation field reconstruction methods, these methods also fail to perform well in sparse viewpoints.

[0005] Therefore, the present invention aims to overcome the two major limitations of radiation field representation at the same time and achieve fast radiation field reconstruction under sparse viewpoint input. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a fast radiation field reconstruction method under sparse viewpoint input, which addresses the problems of poor accuracy and slow reconstruction and rendering speed of radiation field reconstruction. The method provides an initial geometric estimate for the explicit-implicit radiation field representation by using a visual shell calculated from the image contour, thereby reducing the understanding space and improving the reconstruction accuracy and speed of the radiation field under sparse viewpoint input.

[0007] The technical solution of this invention is: a fast radiation field reconstruction method under sparse viewpoint input, the implementation steps of which are as follows:

[0008] (1) The RGB image input from the sparse viewpoint is segmented into foreground and background to obtain the contour map of each input image. By calculating the position of the edge pixels of the contour map in space, an axis-aligned bounding box of the target to be reconstructed is calculated. Compared with the traditional radiation field reconstruction algorithm that performs global modeling of the entire Euclidean space, this invention narrows the optimization range by taking the axis-aligned bounding box of the reconstruction target as a new region of interest, thereby improving the reconstruction speed.

[0009] (2) The axis-aligned bounding box in step (1) is voxelized, dividing the bounding box into voxels of equal size. Each voxel vertex is assigned a voxel confidence score to be optimized and a voxel feature vector as the local 3D scene attribute representation of the voxel. The voxel confidence score is used to calculate the density at any point in space to obtain the geometric representation of the scene, while the voxel feature vector and a multilayer perceptron with shared weights are used to calculate the color radiation at any point in space, thus forming the appearance representation of the scene. Compared with radiation field representation methods that only include a multilayer perceptron and cannot incorporate geometric constraints, the innovation of this method lies in the addition of voxel representation, which allows subsequent initialization of scene geometry using the number of visible viewpoints, improving the speed and accuracy of reconstruction.

[0010] (3) Initialize the voxel confidence in step (2). Calculate the number of visible viewpoints of the voxel vertex, i.e., the number of voxel vertexes observed by the input viewpoint, by projecting the voxel vertex onto the contour map of each input image. Then initialize the voxel confidence corresponding to the voxel based on the number of visible viewpoints to obtain the initial value of the voxel confidence of the voxel vertex. By introducing voxel confidence initialization, the reconstruction of the radiation field is accelerated, and the generation of floating clouds in the blank area is avoided, thereby improving the reconstruction accuracy of the radiation field.

[0011] (4) Perform volume rendering pixel by pixel for the RGB image of each sparse viewpoint. In the volume rendering, the volume density required for each query point on the ray corresponding to each pixel is calculated by interpolation using the initial value of the confidence of the adjacent voxels. The color radiation of the query point is first obtained by interpolation using the feature vectors of the adjacent voxels to obtain the feature vector corresponding to the query point. Then, the feature vector corresponding to the query point is input into the multilayer perceptron for decoding to obtain the color radiation required for the query point. The above process is performed for each pixel of the sparse input viewpoint to complete the rendering of the RGB image of the input viewpoint. The voxel confidence and voxel feature vector of each voxel vertex in the iterative optimization step (2) and the multilayer perceptron with shared weights are minimized by minimizing the reconstruction photometric error loss and total differential error loss between the rendered RGB image and the sparse viewpoint input image.

[0012] (5) For the optimization process in step (4), the geometric estimation of the radiation field is continuously refined by periodically performing voxel clipping, while avoiding unnecessary radiation field queries.

[0013] Compared to traditional radiation field representation methods, the explicit-implicit combined representation method in this invention is more conducive to geometric initialization using contour messages, effectively reducing the problem of floating clouds in blank areas under sparse viewpoint input, while improving the convergence speed of radiation field reconstruction and the rendering efficiency of new viewpoint images.

[0014] In step (1), the RGB image input from the sparse viewpoint is segmented into foreground and background to obtain a contour map of each input image. By calculating the position of the edge pixels in space, an axis-aligned bounding box of the target to be reconstructed can be calculated, as follows:

[0015] RGB images with sparse viewpoint input Thresholding segmentation is used to segment the foreground and background, resulting in the corresponding binary image. First, initialize an empty set. This serves as a set of projection points on the object's edges. Then, by projecting the edge pixels of the contour map into space and culling other contours, the spatial positions of the object's edge points are estimated. Finally, the object's axis-aligned bounding box is calculated. The specific steps are as follows:

[0016] For each contour map Extracting edges edge pixels For each edge pixel, through this contour The corresponding camera optical center emits a line through the edge pixel from the camera's perspective. light In the light Sampling is performed uniformly on the surface to obtain... sampling points For each sampling point Project it onto the remaining In the contour map, if the sampling point can be located within all the remaining contour maps, then the sampling point is retained as a projection point of the object edge, and all the retained spatial points are added to it. The spatial position estimate of the object's edge is obtained. The extent of the bounding box is determined by calculating the maxima and minima of the coordinates of all spatial points in each direction.

[0017] Thus, the initial region of interest to be optimized is obtained and represented by an axis-aligned bounding box. Radiation field optimization will then be performed within this region. Through this region of interest search, most of the blank areas in space are skipped. At the same time, since the axis-aligned bounding box tightly surrounds the object to be reconstructed, high-resolution voxelization can be performed within such a small area. Compared with many existing voxel-based radiation field representation methods, this invention can achieve high-precision modeling of the radiation field without consuming too many computational resources.

[0018] In step (2), the axis-aligned bounding box obtained in step (1) is voxelized, and each voxel vertex is assigned an optimization confidence level and a learnable voxel feature vector as a 3D representation of the local region. The feature vector of each queried sampling point and the viewing angle are input into a multilayer perceptron to be optimized to model the radiation field with variable viewing angle, as follows:

[0019] The axis-aligned bounding box obtained in step (1) is divided into Given voxels of equal size, for each voxel vertex, assign a voxel confidence γ to be optimized and a learnable voxel feature vector. Although the spatial modeling is discrete, the volume density and eigenvector representation of any point in the voxel grid can be obtained through trilinear interpolation.

[0020]

[0021]

[0022] in, and These represent the voxel confidence and voxel feature vector stored in the eight nearest neighbor voxels of the sampling point to be queried, respectively. This represents the activation function. Let represent the trilinear interpolation function. For the color of the sampled points, we use a 64-dimensional multilayer perceptron with four hidden layers for regression:

[0023]

[0024]

[0025]

[0026] in, This indicates the observed direction of the sampling point. This represents the feature vector of the sampling points obtained through interpolation. The location encoding function is used to map the input to a high-dimensional space, thereby enhancing the network's ability to capture high-frequency details. MLP stands for Multilayer Perceptron. At this point, we have completed the modeling of the density and color radiation of the query points to be sampled at any location within the entire region of interest. The radiation field representation of this invention mainly consists of two parts: one is the voxel confidence and voxel feature vector representation to be optimized stored in voxel points, and the other is a color radiation multilayer perceptron used to regress the query points to be sampled.

[0027] In step (3), the voxelized radiation field representation obtained in step (2) is used to calculate the viewing angle by projecting the voxel vertices onto the contour map of each input image, and the voxel confidence is initialized based on the number of viewing angles, as follows:

[0028] For the radiation field representation in step (2), geometric initialization is performed using the object contour image from step (1). The confidence level is initialized by determining whether a voxel contains scene content through the calculation of the visible quantity. This is because trilinear interpolation of density is applied... The activation function can therefore be used by initializing the voxel confidence to a negative value, i.e. The activation function's cutoff region is used to skip optimization in that region. First, each voxel vertex is denoted as... For each voxel vertex, an initial voxel confidence score is assigned. , each voxel vertex Project the image onto all input viewpoint imaging planes, calculate the number of viewpoints that lie within the contour of the input image, and if the number of viewpoints is visible... Equal to the number of input viewpoints Then the voxel confidence level Initialize to Otherwise The specific calculation method is as follows:

[0029]

[0030] In step (4), volume rendering is performed on the input RGB image. The voxel confidence and voxel feature vector of the voxel vertex are optimized by reconstructing photometric error and total differential loss, as well as the multilayer perceptron. The method is as follows:

[0031] Similar to traditional radiation field optimization methods, volume rendering is performed on the RGB images at each sparse viewpoint, minimizing the reconstructed photometric error loss between the rendered image and the input image at the sparse viewpoint. The radiation field representation will be optimized, and the optimization objective is:

[0032]

[0033] in, This represents a set of rays randomly sampled within the image contour. This represents a randomly sampled ray of light. Indicates light rays The color value of the corresponding pixel, This represents the predicted pixel color value after volume rendering using a radiation field representation. We render the pixel's color by uniformly sampling points along a ray and accumulating the density and radiance values ​​of all samples:

[0034]

[0035]

[0036] in, This indicates the number of uniformly sampled points along a ray of light. Indicates the distance from the nearest sampling point to the sampling point. Cumulative opacity, The sampling points were measured. The degree of contribution to the final accumulated color values. Indicates the sampling step size. Indicates sampling point The volume density, in the radiation field representation, is calculated by interpolating the voxel confidence of the vertices of the 8 nearest neighbor voxels. Indicates sampling point The radiation value is obtained by inputting the feature vector corresponding to the point and the observed direction into a multilayer perceptron for regression. Since the volume rendering process is completely differentiable, it can be directly incorporated into the calculation of the loss function to achieve supervised optimization.

[0037] Furthermore, it was found that the optimization of the object's geometric representation easily leads to discontinuous volume density distributions due to the addition of explicit voxel representation; therefore, a total differential error loss is introduced. We use this loss function to regularize the gradient of the voxel confidence. By adding such a loss function, we can obtain a smoother geometric distribution.

[0038]

[0039] in, Represents a set of randomly sampled voxels. Voxel representation In direction Differential on, Voxel representation In direction Differential on, Voxel representation In direction The differential on.

[0040] Therefore, the final loss function of this invention for:

[0041]

[0042] in, This represents the weight of the total differential loss function.

[0043] In step (5), for the optimization process in step (4), the geometric estimation of the radiation field is continuously refined by periodically performing voxel clipping, while avoiding unnecessary radiation field queries.

[0044] Although the voxel confidence initialization strategy has removed most voxel grids that do not contain actual content, some blank voxels are still not removed. This is because there are some errors in the acquisition of the contour image, and because shape reconstruction under sparse viewpoint input often slightly exceeds the actual shape of the object. Therefore, periodic voxel clipping is used to effectively remove blank voxels, continuously refining the geometric representation of the radiation field, while also avoiding unnecessary radiation field queries and improving reconstruction efficiency. The specific implementation method is as follows:

[0045] With a period of 1000 iterations, the voxel confidence of the entire voxel space is queried, and a threshold of 0.1 is used for pruning, removing voxels with a confidence of less than 0.1.

[0046] The advantages of this invention compared to the prior art are:

[0047] (1) The present invention provides an initial geometric estimate of an object by utilizing the contour information that is readily available in practical applications. Compared with the traditional radiation field reconstruction method based on sparse view input, the reconstruction accuracy of the present invention is higher and the ability to capture details is stronger, effectively improving the reconstruction accuracy of the radiation field under sparse view input.

[0048] (2) The present invention shortens the convergence time of the existing radiation field reconstruction method by 30 times, effectively improving the reconstruction speed of the radiation field under sparse view input, making the application of radiation field representation in virtual reality and other scenarios more convenient.

[0049] In summary, this invention uses only image contour information as an additional aid to achieve the goal of rapid reconstruction of the radiation field under sparse viewpoint input. Attached Figure Description

[0050] Figure 1 This is a flowchart of the fast radiation field reconstruction method under sparse viewpoint input according to the present invention. Detailed Implementation

[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0052] like Figure 1 As shown, the specific implementation steps of the present invention are as follows:

[0053] Step 1: Use an adaptive thresholding method for each RGB image input from a sparse viewpoint. Contour extraction is performed to obtain the corresponding contour map. To calculate the axis-aligned bounding box of the target to be reconstructed, we project the edge pixels in the contour map into space and obtain the spatial position of the contour edges by culling points that do not meet the contour consistency constraints.

[0054] To store these spatial points for subsequent statistical calculations, an empty set P is initialized as the set of coordinates of the edge pixels of the contour map in space. For each RGB image, the extracted contour map... Extracting edges edge pixels For this outline diagram Each edge pixel in the contour The corresponding camera optical center emits a line through the edge pixel from the camera's perspective. light In the light Sampling is performed uniformly on the surface to obtain... sampling points For each sampling point Project it onto the remaining In a contour map, if a sampling point can be located within all remaining contour maps, then the sampling point is added to set P.

[0055] By statistically analyzing the maxima and minima of the coordinates of all sampling points in the set P in each direction, the coordinates of the eight vertices of the axis-aligned bounding box are calculated. This yields the initial region of interest to be optimized. Subsequent radiation field optimization will be performed within this region. This region of interest search skips most of the blank areas in space. Furthermore, since the axis-aligned bounding box tightly surrounds the object to be reconstructed, high-resolution voxelization can be performed within such a small area. Compared with many existing voxel-based radiation field representation methods, this invention can achieve high-precision modeling of the radiation field without consuming excessive computational resources.

[0056] Step 2: Voxelize the axis-aligned bounding box from step (1), dividing the bounding box into K voxels of equal size. At the same time, assign a voxel confidence γ to be optimized and a voxel feature vector to each voxel vertex. As a local 3D scene attribute representation of voxels, voxel confidence is used to calculate the volume density at any point in space to obtain the geometric representation of the scene, while voxel feature vectors and a multilayer perceptron with shared weights are used to calculate the color radiation at any point in space to form the appearance representation of the scene.

[0057] To enable the querying of volume density and color radiation at any point in space, a trilinear interpolation method was used to achieve a continuous radiation field representation. The calculation method is as follows:

[0058]

[0059]

[0060] in, This represents the volume density of the sampling points to be queried. This represents the feature vector of the sampling point to be queried. and These represent the voxel confidence and voxel feature vector stored in the eight nearest neighbor voxels of the sampling point to be queried, respectively. This represents the activation function. The trilinear interpolation function is used to perform regression on the color radiation of the sampling points using a 64-dimensional multilayer perceptron with four hidden layers.

[0061]

[0062]

[0063]

[0064] in, This represents the color radiation of the sampling point to be queried. This represents a multilayer perceptron. Indicates the observation direction of the sampling point. This represents the positional encoding function, used to map the input to a high-dimensional space, thereby enhancing the multilayer perceptron's ability to capture high-frequency details. Position encoding function The required hyperparameters. Due to the addition of optimizable voxel feature vectors, a 64-dimensional multilayer perceptron with four hidden layers is used for modeling view-variable radiation. Compared with the original 256-dimensional multilayer perceptron with 16 hidden layers, the time for a single radiation field query in this invention is shorter. At the same time, by pre-calculating the voxel confidence, the radiation field query in blank voxels can be avoided, further shortening the rendering time of a single ray.

[0065] Step 3: For each voxel vertex obtained in step (2) Project it onto the contour map obtained in step (1). In the middle, count the number of items located inside the outline map. Then the initial value of the voxel confidence of the voxel vertex. The calculation method is as follows:

[0066]

[0067] in, This indicates the number of sparse viewpoint inputs.

[0068] Using the initialization method described above, we remove voxels that do not meet the contour consistency requirement. Compared with the random initialization radiation field reconstruction method, this invention provides a compact initial shape for reconstruction, avoiding the generation of clouds in blank areas, thereby avoiding radiation field queries in most blank areas and accelerating radiation field reconstruction.

[0069] Step 4: Query the voxel confidence γ and voxel feature vector of each voxel vertex in step (2). The volume density and color radiation of each ray sampled point are obtained, and the RGB image at each sparse viewpoint is volume-rendered. The voxel confidence, voxel feature vector and multilayer perceptron are iteratively optimized by minimizing the reconstruction photometric error and total differential loss.

[0070] Among them, the reconstruction photometric error loss The optimization objective is:

[0071]

[0072] In the above formula, This represents a set of rays randomly sampled within the image contour. This represents a randomly sampled ray; specifically, in each iteration, we randomly sample 8192 rays from all sparse viewpoint inputs. Indicates light rays The color value of the corresponding pixel, The radiation field represents the predicted pixel color value after volume rendering. Specifically, we render the pixel color by uniformly sampling points along a ray and accumulating the color radiation of all sampling points with the volume density of each sampling point as the weight. The calculation method is as follows:

[0073]

[0074]

[0075] in, This indicates the number of uniformly sampled points along a ray of light. Indicates the distance from the nearest sampling point to the sampling point. Cumulative opacity, The sampling points were measured. The degree of contribution to the accumulated color values ​​in the final drawing; Indicates the sampling step size; Indicates sampling point The volume density, in the radiation field representation, is calculated by interpolating the initial values ​​of the voxel confidence scores of the eight nearest-neighbor voxel vertices. Indicates sampling point The radiation value is obtained by inputting the feature vector corresponding to the point and the observed direction into a multilayer perceptron for regression calculation;

[0076] In addition to calculating the reconstructed photometric error, the optimization objective also includes a total differential error loss. The gradient used for regularizing voxel confidence is calculated as follows:

[0077]

[0078] in, Represents a set of randomly sampled voxels. Voxel representation In direction Differential on, Voxel representation In direction Differential on, Voxel representation In direction The derivative of the expression. Specifically, 2018 voxels are randomly sampled in each iteration.

[0079] Combining the two loss functions above, the final loss function of this invention is... for:

[0080]

[0081] in, This represents the weight assigned to the total differential loss function. In this invention, it will... The value is set to 0.1. By introducing total differential loss, this invention enables the reconstructed radiation field to have a smoother geometric distribution, which is closer to the true radiation field distribution.

[0082] Step 5: For the optimization process in step (4), the present invention introduces a periodic voxel clipping process to continuously refine the geometric estimation of the radiation field.

[0083] Although the voxel confidence initialization strategy has eliminated most voxel grids that do not contain actual content, some blank voxels are still not eliminated. This is because there are some errors in the acquisition of the contour image, and because shape reconstruction under sparse viewpoint input often slightly exceeds the actual shape of the object. Therefore, we use periodic voxel clipping to effectively remove blank voxels, continuously refining the geometric representation of the radiation field, while also avoiding unnecessary radiation field queries and improving reconstruction efficiency. Specifically, this invention uses 1000 iterations as the optimization period to query the voxel confidence of the entire voxel space. A threshold of 0.1 is used for clipping: voxels with a confidence of less than 0.1 are removed, ultimately obtaining a refined explicit-implicit radiation field representation.

[0084] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A fast radiation field reconstruction method under sparse viewpoint input, characterized in that, Includes the following steps: (1) Perform foreground and background segmentation on the RGB image input from the sparse viewpoint to obtain the contour map of each input image. By calculating the position of the edge pixels of the contour map in space, an axis-aligned bounding box of the target to be reconstructed is obtained. (2) Voxelize the axis-aligned bounding box in step (1), divide the bounding box into voxels of equal size, and assign a voxel confidence and a voxel feature vector to each voxel vertex as the local three-dimensional scene attribute representation of the voxel. The voxel confidence is used to calculate the volume density of the sample point to be queried in any point in space to obtain the geometric representation of the scene. The voxel feature vector and a multilayer perceptron with shared weights are used to calculate the color radiation of the sample point to be queried in any point in space to form the appearance representation of the scene. (3) Initialize the voxel confidence in step (2). Calculate the number of visible viewpoints of the voxel vertex, i.e., the number of voxel vertexes observed by the input viewpoint, by projecting the voxel vertex onto the contour map of each input image. Then initialize the voxel confidence corresponding to the voxel based on the number of visible viewpoints to obtain the initial value of the voxel confidence of the voxel vertex. By introducing voxel confidence initialization, the reconstruction of the radiation field is accelerated, and the generation of floating clouds in the blank area is avoided, thereby improving the reconstruction accuracy of the radiation field. (4) Perform volume rendering pixel by pixel on the input RGB image. The volume density required for each query point on the ray corresponding to each pixel in the volume rendering is calculated by interpolation using the initial value of the voxel confidence of the adjacent voxel vertices. The color radiation of each query point on the ray is first obtained by interpolation using the voxel feature vectors of the adjacent voxel vertices. Then, the feature vector corresponding to the query point is input into the multilayer perceptron to decode and obtain the color radiation required for the query point. Then, the color radiation of all query points on the ray corresponding to the pixel is accumulated with volume density as the weight to obtain the color rendering of the pixel. Perform the above rendering process for each pixel in the image to obtain the rendered RGB image. Iteratively optimize the voxel confidence and voxel feature vector of each voxel vertex in step (2) and the multilayer perceptron with shared weights by minimizing the reconstruction photometric error loss and total differential loss between the rendered RGB image and the sparse viewpoint input image. (5) In the iterative optimization process of step (4), the geometric estimation of the radiation field is continuously refined by periodically performing voxel clipping, while avoiding redundant radiation field queries, thereby improving the reconstruction efficiency of the radiation field. Finally, the voxel confidence estimate representing the scene geometry and the voxel feature vector and multilayer perceptron representing the scene appearance are obtained, thus completing the reconstruction of the radiation field under sparse view input.

2. The fast radiation field reconstruction method under sparse viewpoint input according to claim 1, characterized in that: The specific implementation of step (1) is as follows: (11) For the input RGB image Thresholding segmentation is used to segment the foreground and background, resulting in corresponding contour maps. At the same time, an empty set P is initialized as the set of coordinate points of the edge pixels of the contour map in space; (12) The contour map extracted for each RGB image Extracting edges edge pixels For each edge pixel, through this contour The corresponding camera optical center emits a line through the edge pixel from the camera's perspective. light In the light Uniform sampling was performed to obtain sampling points For each sampling point Project to the remaining In the contour map, if the sampling point can be located within all the remaining contour maps, then the sampling point is added to the set P; (13) The maximum and minimum values ​​of the coordinates of all sampling points in the statistical set P in each direction are used as the coordinate values ​​of the eight vertices of the axis-aligned bounding box to obtain the axis-aligned bounding box of the target to be reconstructed.

3. The fast radiation field reconstruction method under sparse viewpoint input according to claim 1, characterized in that: Step (2) is implemented as follows: To enable the querying of volume density and color radiance for any sampling point in space, a continuous radiance field representation was achieved using trilinear interpolation. The calculation method is as follows: in, This represents the volume density of the sampling points to be queried. This represents the feature vector corresponding to the sampling point to be queried. and These represent the voxel confidence and voxel feature vector stored in the eight nearest neighbor voxels of the sampling point to be queried, respectively. This represents the activation function. Let represent the trilinear interpolation function; for the color radiation of the sample points to be queried, a 64-dimensional multilayer perceptron with four hidden layers is used for regression: in, This represents the color radiation of the sampling point to be queried. This represents a multilayer perceptron. This indicates the observed direction of the sampling point to be queried. This represents the positional encoding function, used to map the input to a high-dimensional space, thereby enhancing the multilayer perceptron's ability to capture high-frequency details. Position encoding function The required hyperparameters.

4. The fast radiation field reconstruction method under sparse viewpoint input according to claim 1, characterized in that: The specific implementation of step (3) is as follows: For each voxel vertex Projecting it onto the number of inputs obtained in step (1) is Outline drawing In the middle, count the number of items located inside the outline map. Then the initial value of the voxel confidence of the voxel vertex. The calculation method is as follows: The above initialization removes voxels that do not meet the contour consistency requirement, providing a compact initial shape for reconstruction. This avoids radiation field queries in most blank areas and accelerates the reconstruction of the radiation field.

5. The fast radiation field reconstruction method under sparse viewpoint input according to claim 1, characterized in that: In step (4), volume rendering is performed on the input RGB image. The voxel confidence, voxel feature vector, and multilayer perceptron are optimized by minimizing the reconstruction photometric error loss and total differential error loss between the rendered RGB image and the sparse viewpoint input image. The specific implementation is as follows: Volume rendering is performed on the input RGB image, first by minimizing the reconstructed photometric error loss between the rendered image and the input image with sparse viewpoints. Optimize the representation of the radiation field ， The optimization objective is: in, This represents a set of rays randomly sampled within the image contour. This represents a randomly sampled ray of light. Indicates light rays The color value of the corresponding pixel, This represents the predicted pixel color value after volume rendering. The pixel color is drawn by uniformly sampling points along a ray and accumulating the color radiation of all sampling points with the volume density of each sampling point as the weight. in, This indicates the number of uniformly sampled points along a ray of light. Indicates the distance from the nearest sampling point to the sampling point. Cumulative opacity, The sampling points were measured. The degree of contribution to the accumulated color values ​​in the final drawing; Indicates the sampling step size; Indicates sampling point The volume density, in the radiation field representation, is calculated by interpolating the initial voxel confidence values ​​of the vertices of the eight nearest-neighbor voxels. Indicates sampling point The color radiation is obtained by inputting the feature vector corresponding to the point and the observed direction into a multilayer perceptron for regression calculation; In addition, a total differential error loss is introduced. Gradients of regularized voxel confidence in all directions: in, Represents a set of randomly sampled voxels. Voxel representation In direction Differential on, Voxel representation In direction Differential on, Voxel representation In direction Differential on; Final loss function for: in, This represents the weight of the total differential loss function, and its value ranges from 0 to 1.

6. The fast radiation field reconstruction method under sparse viewpoint input according to claim 1, characterized in that: In step (5), for the optimization process in step (4), the geometric estimation of the radiation field is continuously refined by periodically performing voxel clipping, resulting in a more refined representation of the radiation field.

Images