Tree nerve rendering and reconstruction method based on sparse view

Abstract

The invention belongs to the technical field of tree three-dimensional reconstruction, and particularly discloses a tree neural rendering and reconstruction method based on a sparse view, and the method comprises the steps: collecting an RGB image of a tree through a camera, obtaining camera parameters, inputting the camera parameters into a pre-trained depth estimation model, generating a rough depth map, and inputting the rough depth map into a neural radiation field network; introducing a cone sampling strategy, integrating a position coding method, a coding annealing strategy, depth sorting and depth consistency constraint, and extracting point cloud and picture features in the picture; carrying out implicit surface reconstruction based on volume rendering; introducing color constraint and geometric constraint based on color deviation and geometric deviation of implicit surface reconstruction; and a three-dimensional model of the tree is reconstructed from the implicit surface by using a Marking Cubes algorithm. By adopting the technical scheme, tree rendering is carried out based on the neural radiation field, tree reconstruction is carried out based on body rendering, and the reconstructed tree model can better retain and reproduce original geometric features and real textures of the tree.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional reconstruction of trees, and relates to a tree neural rendering and reconstruction method based on sparse views. Background Art

[0002] With the rapid development of information technology, the digital industry is booming. People have put forward higher and higher requirements for the reconstruction and simulation of three-dimensional scenes. Especially with the emergence of the concept of digital twins, highly accurate digital reconstruction of real-world objects has become an urgent need, among which the three-dimensional reconstruction of trees is particularly prominent.

[0003] At present, although researchers around the world have accumulated remarkable results in tree 3D reconstruction technology, the modeling strategies adopted include a variety of methods such as program-based modeling methods, reconstruction based on point cloud data, and reverse engineering based on image information. However, even so, these methods still expose their limitations when dealing with challenges such as the complex geometric structure of trees, micro-texture details, changes in lighting conditions, and incomplete information caused by lack of perspective. Tree 3D modeling methods generally have problems such as high degree of manual participation, limited ability to reconstruct specific trees in real environments, lack of model realism, and a long time required to achieve high-precision models.

[0004] Although the program-based modeling method can simulate the growth process and different forms of trees, its process is complex and requires a high degree of human intervention. In addition, due to the universality of the rules, it is difficult to accurately reproduce the real forms and details of different types of trees in the real world. Although point cloud-based modeling technology can provide high-precision three-dimensional models, it relies on high-performance scanning equipment, is costly, and has limitations in scanning complex environments and large areas of trees. Data noise and occlusion problems also restrict the quality of the model. Image-based modeling methods have improved accuracy and efficiency to a certain extent, but they have high requirements for image data acquisition conditions, requiring specific environments, multi-angle shooting, and sufficient number of images. At the same time, due to the limitations of the camera's viewing angle and the occlusion effect under shooting conditions, it is difficult to fully restore the complete three-dimensional information of trees from a limited two-dimensional image sequence. When dealing with visual misleading caused by occluded parts and changes in lighting conditions, it is often difficult to achieve an ideal reconstruction effect by relying solely on image information.

[0005] Therefore, finding a new modeling technology that can effectively overcome the above limitations, adapt to complex environments and limited view conditions, accurately capture the complex structure and texture details of trees, and be efficient and economical, has important research significance and broad application prospects. Summary of the invention

[0006] The purpose of the present invention is to provide a tree neural rendering and reconstruction method based on sparse views, which can restore the morphology, texture details and overall volume of real-world trees, and reconstruct tree models with higher quality.

[0007] In order to achieve the above object, the basic scheme of the present invention is: a tree neural rendering and reconstruction method based on sparse view, comprising the following steps:

[0008] Use the camera to collect RGB pictures of trees and obtain camera parameters;

[0009] The RGB image of the tree and the camera parameters are fed into a pre-trained depth estimation model to generate a rough depth map that is fed into the neural radiance field network.

[0010] A cone sampling strategy is introduced to perform ray sampling on the rough depth map to obtain a Gaussian model;

[0011] The integrated position encoding method is used to position encode the Gaussian model;

[0012] Based on the encoding annealing strategy, sparse view learning is performed on the coarse depth map;

[0013] Introducing depth sorting and depth consistency constraints to regularize and optimize the neural radiation field;

[0014] Extract point clouds from images based on the neural radiance field network, and input the RGB images of trees and camera parameters into the pre-trained feature extraction model to obtain image features;

[0015] Based on volume rendering, implicit surface reconstruction is performed;

[0016] Based on the color deviation and geometric deviation of implicit surface reconstruction, color constraints and geometric constraints are introduced;

[0017] The Marching Cubes algorithm is used to reconstruct the 3D tree model from the implicit surface.

[0018] The working principle and beneficial effects of this basic solution are as follows: This technical solution introduces a coding annealing strategy to quickly learn low-frequency information in the early stage of network training, and gradually turns on the high-frequency part of the coding as the training progresses, reducing the dependence on dense view data. At the same time, a pre-trained depth estimation model is used to generate a rough depth map, and depth sorting and depth consistency constraints are proposed to integrate more robust depth prior information into the training process of the neural radiation field, thereby improving the rendering quality in sparse view scenes. By introducing a cone sampling strategy and integrated position coding, the inherent aliasing problem of the neural radiation field is effectively solved.

[0019] Color constraints and geometric constraints are introduced to reduce color deviations through linear interpolation, and color weight loss is introduced to eliminate abnormal weight distribution and encourage the weight distribution function to converge to the tree surface. Point clouds are extracted using a rendering network based on neural radiation fields, providing explicit geometric priors to optimize the geometry of trees, and feature extraction is performed through a pre-trained model to optimize geometric feature consistency during network training. In the case of relatively sparse view input, the rendering effect of real-world tree scenes is improved, and the morphology, texture details and overall volume of real-world trees can be restored, and the reconstructed tree model is of higher quality.

[0020] Furthermore, a cone sampling strategy is introduced to perform light sampling on the rough depth map, and the method to obtain the Gaussian model is as follows:

[0021] For any pixel point, a cone is projected from the camera's projection origin o along the direction d passing through the pixel center. The vertex of the cone is located at o, and the radius parameter of the cone surface at the image plane o+d is set to ,Will Set to the width of pixels in world coordinates, scaled by 2 / √12, to produce a cone;

[0022] The set of positions x that lie within the frustum of the cone between two t values ​​[t0, t1] It is expressed as:

[0023]

[0024] Among them, t represents a parameterized representation of the distance from the camera projection origin o to a certain point x of the frustum, t0, t1 define the near plane and far plane of the frustum, that is, the distance from the camera origin to the boundaries of the two ends of the frustum, which is used to determine the depth range of sampling; is an indicator function. The first half of the formula ensures that the projection distance of x along the d direction is between t0 and t1. The second half of the formula checks whether x satisfies the shape constraint of the cone. If x is between In the cone defined by

[0025] The position encoding of this pixel is re-expressed as the expectation of all coordinate position encodings within the cone

[0026]

[0027] Among them, γ(x) represents the position encoding of the original neural radiation field, γ * The representation is defined as integrated position encoding IPE;

[0028] Use multivariate Gaussian functions to approximate the frustum, and calculate The mean and covariance of each truncated cone are assumed to be circular and symmetric around the axis of the cone, so in addition to o and d, the multivariate Gaussian function model is represented by 3 values: the average distance μ along the ray t , the variance along the ray direction Variance along the ray perpendicular direction

[0029]

[0030] Among them, t μ represents the midpoint, which is the point on the central axis of the cone that divides the height of the cone into two equal parts. μ =(t0+t1); t δ The half-width is half the width of the cone truncated cone in the direction perpendicular to the central axis, which is proportional to the radius of the cone. Related, t δ =(t1-t0) / 2;

[0031] Convert the Gaussian model from the cone coordinate system to the world coordinate system to obtain the final multivariate Gaussian model:

[0032]

[0033] Among them, μ is the mean vector of the multivariate Gaussian distribution, which represents the central tendency of the point position in the cone truncated cone; ∑ is the covariance matrix of the multivariate Gaussian distribution, which describes the discrete degree of the distribution of the point position in the cone truncated cone and the correlation between the dimensions; I represents the identity matrix, which is a square matrix whose elements on the diagonal are all 1 and the elements on the off-diagonal are all 0.

[0034] The cone sampling strategy is adopted to replace the traditional ray sampling method to effectively reduce the aliasing effect.

[0035] Furthermore, the steps of position encoding the Gaussian model using the integrated position encoding method are as follows:

[0036] IPE is the expectation of the position encoding coordinates of the Gaussian distribution, and the position encoding is rewritten in matrix form:

[0037]

[0038] Where P is a matrix with each row representing a frequency component and each column corresponding to a spatial dimension (x, y, z), which is used to generate the input of the sine and cosine functions. L is an integer representing the number of frequency levels used in the position encoding. γ(x) is the integrated position encoding of position x, which contains the values ​​of all sine and cosine functions calculated by the P matrix and position x. It is a vector. Px represents the product of the matrix P and the position vector x.

[0039] Using the covariance property, the Gaussian model is positionally encoded to obtain the encoded multivariate Gaussian representation: μ γ =Pμ,Σ γ =PΣP T ; The encoded expectation is expressed as:

[0040]

[0041] Among them, μ γ is the mean of the encoded Gaussian distribution, indicating the center position of the data; Σ γ is the covariance matrix of the encoded Gaussian distribution, which indicates the degree of discreteness of the data; Represents mathematical expectation, the random variable x has mean μ and variance σ 2 Normal distribution of

[0042] The final IPE encoding γ(μ,∑) is expressed as:

[0043]

[0044] in, represents element-wise multiplication; the covariance matrix Σ γ The computational cost is relatively high, and only the diagonal values ​​need to be considered, so in practical implementation it is expressed as:

[0045] diag(∑ γ )=[diag(∑),4diag(∑),…,4 L-1 diag(∑)] T

[0046]

[0047] Among them, diag(∑) represents the vector composed of the diagonal elements of the covariance matrix Σ.

[0048] A cone sampling strategy is adopted to replace the traditional ray sampling method, and integrated position coding is introduced to replace the original position coding, thereby effectively reducing the aliasing effect.

[0049] Furthermore, based on the encoding annealing strategy, the steps of sparse view learning for the coarse depth map are:

[0050] Given a positional encoding of length L, the frequency mask m is used to adjust the visible frequencies based on the training time step:

[0051] γ′ L (t,T;x)=γ L (x)⊙m(t,T,L)

[0052] Among them, γ′ L(t,T; x) represents the output of the ith position after frequency mask adjustment, γ L (x) is a function transformation, which indicates the effect of position encoding on input x, and the output length is L. ⊙ is a multiplication operator, which indicates the multiplication of corresponding elements in two vectors or matrices of the same dimension; m(t,T,L) is:

[0053]

[0054] m i (t,T,L) represents the value of the ith position of the mask m(t,T,L), t and T represent the current training iteration number and the final iteration number of coding annealing, respectively, and L represents the hyperparameter that controls the maximum coding frequency; the low-frequency coding learns the geometric shape of the scene, and as the training progresses, the sliding truncated Hanning window function is used to gradually open the high-frequency part of the coding.

[0055] For the problem of sparse view learning, the coding annealing strategy is used to prioritize low-frequency features in the early stage of training, and gradually introduce high-frequency features as training progresses. This coarse-to-fine training method helps the model better learn the three-dimensional shape of trees under limited view conditions and reduces dependence on dense view data.

[0056] Furthermore, we introduce depth sorting and depth consistency constraints to regularize and optimize the neural radiation field. The specific steps are as follows:

[0057] Relax the depth constraint and extract a robust local depth ranking from the coarse depth map, so that the depth ranking of the neural radiance field is consistent with the depth ranking of the coarse depth map, and use relative depth instead of absolute depth to supervise the neural radiance field network;

[0058] Given a pair of pixels in a single image, the depth ordering prior is extracted from the coarse depth map to the neural radiance field. On the one hand, given a local region P of an RGB image with pose, the depth d of the ray traced from P is calculated as:

[0059]

[0060] Where N is the number of sampling points, i is the serial number of the sampling point, and w i represents the weight of the i-th sampling point, t i Represents the parameterized representation of the distance from the camera projection origin o to the i-th sampling point;

[0061] Get the depth of the network rendering view;

[0062] On the other hand, we use the depth d′ estimated by the depth estimation model and crop the local patch with the same spatial position as d to transfer the depth sorting knowledge from d′ to d for depth sorting. Let a and b be the indices of the pixel coordinates of d′ and d. The loss of depth sorting is Defined as:

[0063]

[0064] Among them, d a ,d b are the depth values ​​of the pixels indexed as a and b from d, respectively; m is a hyperparameter, i.e., the local depth sorting error; two depth pixels of d′ are randomly sampled, denoted as d′ a and d′ b , and d′ a ≤d′ b ; If the order of d′ and d is inconsistent, that is, d′ a ≤d′ b , d a >d a , and within the minimum error range, the neural radiation field is penalized by the loss function;

[0065] Depth-invariant geometric consistency regularization is used to measure the depth relationship between pixels in the same local area. For a local area P of the RGB image, the depth map d rendered by the neural radiance field and the predicted depth d′ of the depth estimation model are compared. The scale-invariant mean square depth error is used to measure the depth relationship between pixels in the same block. The depth consistency loss function Defined as:

[0066]

[0067] Where P is a matrix with each row representing a frequency component and each column corresponding to a spatial dimension (x, y, z), which is used to generate the input of the sine and cosine functions; l, j are the coordinate indices of the pixels in the local area P of the RGB image; d lj , d′ lj Represent the true depth value and predicted depth value at position (l, j) respectively.

[0068] A rough depth map is generated using a pre-trained depth estimation model, and depth sorting and depth consistency constraints are introduced to integrate more robust depth prior information into the training process of the neural radiance field. Under the condition of relatively sparse view input, the rendering effect of real-world tree scenes is improved, and the overall morphology and texture details of the trees are restored.

[0069] Furthermore, the losses of the neural radiance field network include color loss, depth sorting loss, depth consistency loss, color loss Consistent with the original loss function of the neural radiance field, the mean square error between the color obtained by volume rendering and the color of the real image is obtained:

[0070]

[0071] in, represents the color value predicted by the model for light r, and C(r) represents the color value corresponding to light r in the real image;

[0072] The pre-trained depth estimation model is used to generate a rough depth map, and depth sorting and depth consistency constraints are proposed. The depth sorting loss and depth consistency loss are respectively and

[0073] Total loss function of the neural radiance field network It is expressed as:

[0074]

[0075] Among them, λ1 and λ2 are hyperparameters used to balance the relative importance between different loss functions.

[0076] Set the loss function and optimize the network.

[0077] Furthermore, implicit surface reconstruction is performed based on volume rendering. The specific method is as follows:

[0078] The neural implicit function is expressed as f(x)=0, where Represents the three-dimensional spatial position of the mapping; reconstructs the surface of the geometric shape by sampling points in the input space and determining their location based on the output of the neural implicit function The zero level set is the set of points that satisfy f(x) = 0; the surface It is expressed as:

[0079]

[0080] Where f(x) is the spatial position The signed distance function SDF, f(x) = 0 represents the point on the surface of the reconstructed object;

[0081] Color mapping function c: Color coding and spatial location and viewing direction The input image is used as supervision to minimize the difference between the rendered image and the input image to learn the weights of the neural network. Finally, the zero level set of the neural SDF represents the reconstructed surface.

[0082] For a pixel on the image and the camera position, a ray is emitted from the camera center and passes through the pixel as:

[0083] p(t)=o+tv|t≥0

[0084] Where o is the camera center, v is the unit direction vector of the ray, and t is the distance between x and o;

[0085] The volume rendering C(o,v) of the original neural radiation field is expressed as:

[0086]

[0087] Where T(t) represents the cumulative transmittance from the camera to point p(t) on the light, which represents the probability that the light is not absorbed or scattered by any medium before reaching point p(t); t f , t n is the upper and lower limits of the integral, t n Usually set to a value very close to the camera, t f It is usually set to a value large enough to include all important objects in the scene; p(t) is a parameterized representation of light, which is represented as a ray extending from the camera position o along the direction d, and the parameterized form is p(t) = o + td, where t is the distance from the camera to a point on the light; σ(p(t)) is the volume density or scattering coefficient at point p(t), which represents the intensity of the interaction between the light and the medium at this point; c(p(t), v) is the radiance at point p(t) for the line of sight direction v, that is, the color, which describes the color information of the light after being scattered at this point;

[0088] The original neural surface reconstruction method NeuS introduces a probability density function S-density: φ s (f(x)), f(x) is the signed distance function, where S-density is the Logistic probability distribution function Φ s Derivative of (x):

[0089] φ s (x) = se -sx / (1+e -sx ) 2

[0090] Φ s (x) = (1 + e -sx ) -1 ,φ s (x) = Φ′ s (x)

[0091] Among them, Φ s (x) is the logistic function, φ s(x) is the probability density function S-density introduced by NeuS. S is a parameter used to control the shape of the Logistic distribution. The larger the value of s, the steeper the curve; the smaller the value of s, the flatter the curve; Φ′ s (x) is Φ s The first derivative of (x), the base of the e exponential function;

[0092] Using S-density for volume rendering, training the SDF network, the volume rendering formula is redefined as:

[0093]

[0094] In order to achieve unbiased estimation, NeuS redefines the volume rendering formula and designs an unbiased weight function:

[0095]

[0096] Among them, the cumulative transmittance T(t) along the ray is expressed as Φ through S-density conversion s (f(p(t))), the volume density σ(t) is replaced by the redefined ρ(t):

[0097]

[0098] The integral is approximated by discretized sampling, and the rendered color of the pixel is accumulated along a ray with N discrete sampling points:

[0099]

[0100] Among them, α i Represents discrete opacity, T i represents the cumulative transmittance along the ray, where α i Further expressed as:

[0101]

[0102] Point clouds are extracted using a neural radiance field based rendering network, providing explicit geometric priors to optimize the geometry of trees.

[0103] Furthermore, based on the color deviation of implicit surface reconstruction, color constraints are introduced, specifically:

[0104] c v Defined as the color for volume rendering, Among them, t *It represents the parametric representation of the best point where the ray intersects the geometric surface during ray tracing; argmin is a mathematical term used to represent the parameter value at which a function achieves the minimum value in its domain; t represents the distance between x and o, v represents the unit direction vector of the ray, and tv represents the distance between x and o in the direction of the ray; p is the parameter representation of the ray; Represents a geometric surface;

[0105] Use MLP to train an SDF network F Θ Get its SDF value and pass it through another MLP network Get the color value, mathematically expressed as:

[0106] sdf(p)=F Θ (p)

[0107]

[0108] Among them, sdf(p) represents the output value of the SDF network when p is used as input; c(o,v,t) represents the output value of the SDF network at the camera position o, the light direction v and the distance t through the MLP network The calculated color value;

[0109] The estimated color c of the image at camera position o v It is expressed as:

[0110]

[0111] Where o is the camera center, v is the unit direction vector of the ray, t is the distance between x and o, and w(t) is the weight of point t; the volume rendering color of a pixel is written in discrete form as:

[0112]

[0113] Among them, w(t i ) is t i The weight of the point discretization, c(t i ) is t i The color value of the point discretization; n is the number of sampling points of discrete sampling, and i represents the i-th segment of the integral interval;

[0114] Assume that the initial intersection point of the ray and the surface is denoted by p(t * ) and sdf(t * )=0; along the v direction is p(t * ) surface color c s :

[0115] c s =c(t * )

[0116] Among them, c(t * ) represents the color value at the initial intersection point p(t*), that is, the color of the object surface; t * The point is the intersection of the ray and the geometric surface, and this point satisfies sdf(t * )=0, indicating that it is exactly on the surface of the geometric surface; p(t * ) is a parameterized representation of the intersection ray;

[0117] Separate the tree surface from other parts, when sdf(t * )=0, use t j Represents the distance from the surface point t * The nearest sampling point, the formula is rewritten as:

[0118]

[0119] The color deviation between the volume rendering color and the surface color is expressed as:

[0120]

[0121] in, Denoted as the deviation ε caused by the volume rendering weight function weight , c(t j )-c(t * ) represents the deviation ε caused by discrete sampling sample ;

[0122] The color deviation can be expressed as:

[0123] Δc=ε weight +ε sample

[0124] There is a significant deviation between the surface color of the object and the color estimated by the network. The deviation consists of two parts: the deviation caused by discrete sampling ε sample and the deviation ε caused by the volume rendering weight function weight ;

[0125] Using the SDF value of the sampling point calculated by volume rendering, linear interpolation is used to extract the zero crossing point and reduce the sampling deviation ε sample , the sampled point along the ray is represented as p(t k ), k = 1, ..., N, N is the number of sampling points for stratified sampling;

[0126] Search along the direction of the ray for the first point that satisfies the condition: the SDF value of this point is positive, and the SDF value of the next sampling point is negative, expressed as:

[0127] f(p(t k ))·f(p(t k+1 ))<0

[0128] Where f is the geometric network of the output SDF value, p(t k ) is the first sampling point before the surface of the object; use differentiable linear interpolation to approximate the surface point where the SDF is zero

[0129]

[0130] Using the sampling point set p(t k ) and approximate surface points Color C is obtained by volume rendering total :

[0131]

[0132] The color deviation is rewritten as:

[0133]

[0134] make The color deviation is further rewritten as:

[0135] Δc=ε' weight +ε interp

[0136] Among them, ε interp represents the deviation caused by linear interpolation, ε' weight Expressed as a new weight function bias;

[0137] In order to reduce the updated weight bias, a color weight loss function is introduced

[0138]

[0139] l weight Penalize abnormal weight values ​​far away from the object surface, especially weight distribution far away from the surface but with large weight values, and encourage the weight distribution function to converge to the object surface.

[0140] Through linear interpolation, the color deviation is reduced, and at the same time, color weight constraints are introduced to eliminate abnormal weights and encourage the weight distribution function to converge to the object surface.

[0141] Furthermore, based on the geometric deviation of implicit surface reconstruction, geometric constraints are introduced as follows:

[0142] Explicit geometric constraints:

[0143] Extract point clouds to approximate the surface points of the object, provide explicit geometric constraints through the projection features of the supervised surface points, and optimize the geometric shape of the object;

[0144] The neural radiation field represents the geometry and appearance of the scene as a continuous volumetric radiation field parameterized by a neural network. It infers the spatial geometry by observing the changes in color and density along the camera light and extracts point cloud data. The cumulative transmittance T(t) of the neural radiation field represents the cumulative value of the volume density σ of all previous objects that have passed through the ray when it reaches an object; when T(t) drops sharply after a certain point, it indicates that this point may be on the surface of the object. Mathematically, the point cloud is represented by solving the minimum value of the first-order derivative of T(t) with respect to t:

[0145]

[0146] Among them, argmin is a mathematical term used to represent the parameter value at which a function takes its minimum value in its domain; q represents the point at which the first-order derivative of the cumulative transmittance takes its minimum value;

[0147] From the neural radiance field, we get the point cloud {P|x p ∈P}, there may be outliers in the recovered point cloud. The point cloud distribution around the implicit surface is different, resulting in different levels of noise in different areas. The filter is used to exclude outliers and produce a relatively accurate and clean point cloud;

[0148] If the points in the point cloud are located on the surface of the object, it can be approximately expressed as: G(p i )=0,p i ∈P, where G represents the signed distance function and P represents the point cloud set; the SDF network is supervised by supervising the surface points:

[0149]

[0150] in, represents the geometric loss function;

[0151] Geometric feature consistency:

[0152] Use the Vis-MVSNet pre-trained model to extract features from RGB images, and then use the geometric feature consistency loss to constrain the neural implicit surface reconstruction. The Vis-MVSNet pre-trained model uses a standard U-Net to extract image feature maps;

[0153] The U-Net network includes a bottom layer, an encoder, a decoder, and a head. The bottom layer consists of multiple blocks, each of which consists of a series of convolutional layers and batch normalization layers. The encoder and the head are both composed of multiple blocks. In addition to the convolutional layer and batch normalization layer, the encoder also includes a pooling layer to gradually reduce the spatial size of the feature map. The decoder consists of a deconvolution layer, a convolutional layer, and possible residual connections to gradually restore the spatial size of the feature map.

[0154] Assume that the point where the ray through view 0 intersects the surface is x, and the projection of view 0 and its adjacent views are represented as The feature loss is defined as:

[0155]

[0156] Among them, N c Represents the number of feature channels, N v represents the number of adjacent source views, F represents the extracted feature map, and p i =K i (R i x+t i ) represents the projection of x in view i, [K i ,R i ,t i ] is the corresponding camera parameter, pixel p i The image feature at is F i (p i ).

[0157] Point clouds are extracted using a neural radiance field based rendering network that provides explicit geometric priors to optimize the geometry of trees, and feature extraction is performed through a pre-trained model to optimize geometric feature consistency during network training.

[0158] Furthermore, the loss functions of the implicit surface reconstruction network based on volume rendering include color loss, Eikonal loss, color weight loss, geometric loss, and feature loss;

[0159] Color Loss is the difference between the RGB colors obtained from the real input image and the RGB colors obtained from the volume rendering:

[0160]

[0161] Where m is the number of pixels, C i Indicates the RGB color value obtained by the image. RGB color values ​​obtained from volume rendering;

[0162] The Eikonal loss constrains the gradient of the SDF to 1 so that it satisfies the properties of the distance function, thereby normalizing the SDF and ensuring that it provides the correct distance field representation. The Eikonal loss is added at the sampling point to constrain the signed distance field gradient of the SDF network:

[0163]

[0164] in, is the set of sampling points, ∥·∥2 is the L2 norm;

[0165] For the color deviation of volume rendering, the color deviation is reduced by linear interpolation and color weight loss is introduced

[0166] For the geometric deviation of volume rendering, the geometric loss is And the feature loss function is

[0167] Total loss function It is expressed as:

[0168]

[0169] Among them, λ3, λ4, λ5, λ6 are hyperparameters used to balance the relative importance between different loss functions.

[0170] Set the loss function and optimize the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0171] Figure 1 It is a structural schematic diagram of a tree neural rendering and reconstruction method based on sparse views of the present invention;

[0172] Figure 2 It is a flowchart of tree rendering of the tree neural rendering and reconstruction method based on sparse views of the present invention;

[0173] Figure 3 It is a flowchart of tree reconstruction of the tree neural rendering and reconstruction method based on sparse views of the present invention;

[0174] Figure 4 It is a structural schematic diagram of the U-Net network of the tree neural rendering and reconstruction method based on sparse views of the present invention. DETAILED DESCRIPTION

[0175] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0176] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0177] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0178] The present invention discloses a tree neural rendering and reconstruction method based on sparse views. Aiming at the problem of rendering and reconstructing an editable tree surface mesh model from a new perspective of trees, the present invention combines neural radiation fields to perform tree rendering and tree reconstruction in a relatively sparse view scene. Compared with traditional methods based on program modeling and multi-view reconstruction, the method of the present invention can restore the morphology, texture details and overall volume of real-world trees, and the reconstructed tree model is of higher quality.

[0179] like Figure 1 As shown, the tree neural rendering and reconstruction method based on sparse views includes the following steps:

[0180] like Figure 2 As shown, the camera is used to collect RGB pictures of trees and obtain camera parameters;

[0181] The RGB image of the tree and the camera parameters are fed into a pre-trained depth estimation model (ZoeDepth model) to generate a rough depth map and fed into the Neural Radiance Fields (NeRF) network.

[0182] A cone sampling strategy is introduced to perform ray sampling on the rough depth map to obtain a Gaussian model;

[0183] The integrated position encoding method is used to perform position encoding on the Gaussian model.

[0184] Based on the encoding annealing strategy, sparse view learning is performed on the coarse depth map;

[0185] Introducing depth sorting and depth consistency constraints to regularize and optimize the neural radiation field;

[0186] like Figure 3 As shown, the point cloud in the picture is extracted based on the neural radiation field network, and the RGB picture of the tree and the camera parameters are input into the pre-trained feature extraction model (such as the Vis-MVSNet model) to obtain the picture features;

[0187] Based on volume rendering, implicit surface reconstruction is performed;

[0188] Based on the color deviation and geometric deviation of implicit surface reconstruction, color constraints and geometric constraints are introduced;

[0189] The Marching Cubes algorithm is used to reconstruct the 3D model of trees from the implicit surface. The Marching Cubes algorithm is a classic isosurface extraction algorithm used to generate an explicit triangular patch mesh model from an implicit surface representation. The algorithm traverses each voxel in the 3D space and determines the topological structure and geometric shape of the isosurface according to the SDF value of the voxel vertex, specifically:

[0190] The 3D space is divided into a regular cubic grid, and then for each voxel, the SDF values ​​of its 8 vertices are obtained. According to the size relationship between the SDF value of the vertex and zero, the intersection of the voxel and the isosurface is determined, and the topological structure of the isosurface within the voxel is determined. For each voxel edge that intersects with the isosurface, the position of the intersection is calculated by linear interpolation. Finally, based on the determined topological structure and the calculated intersection position, the triangular facets within the voxel are generated and added to the final mesh model. By traversing all voxels and performing the above steps, the Marching Cubes algorithm can extract a complete isosurface mesh model from the SDF, realizing the transformation from implicit surface representation to explicit geometric structure.

[0191] For each extracted surface vertex, use its coordinates as input and calculate the color value at that point through the color field function. Assign the calculated color value to the corresponding vertex, and combine the extracted colored surface triangles into a complete mesh model. In order to obtain a higher quality texture representation, the vertex color can be mapped to a texture map. Through UV unwrapping, the surface of the mesh model is mapped to a 2D texture space. In order to minimize distortion, the LSCM algorithm is used to obtain a parametric mapping on a two-dimensional plane. According to the parametric mapping, each vertex on the three-dimensional mesh model is mapped to the corresponding parameter coordinates (u, v) on the two-dimensional plane, and the vertex color is transferred to the corresponding texture coordinates to generate a texture map. When rendering, the generated texture map is used to represent the color information of the model.

[0192] This embodiment performs tree rendering based on the neural radiation field. To address the aliasing problem of NeRF when the distance to the target object is different and the resolution is different, a cone sampling strategy is adopted to replace the traditional ray sampling method, and an integrated position coding is introduced to replace the original position coding, thereby effectively reducing the aliasing effect.

[0193] For the problem of sparse view learning, this paper proposes a coding annealing strategy, which gives priority to low-frequency features in the early stage of training, and gradually introduces high-frequency features as the training progresses. This coarse-to-fine training method helps the model better learn the three-dimensional shape of trees under limited view conditions and reduce dependence on dense view data. To address the problem of insufficient sparse view information, this paper uses a pre-trained depth estimation model to generate a coarse depth map, and introduces depth sorting and depth consistency constraints to integrate more robust depth prior information into the training process of the neural radiation field. Under the condition of relatively sparse view input, the rendering effect of real-world tree scenes is improved, and the overall morphology and texture details of the trees are restored.

[0194] Tree reconstruction is performed based on volume rendering, combining neural implicit surface representation with volume rendering. For complex scenes such as trees, its accurate reconstruction capability is still affected by the color deviation and geometric deviation inherent in volume rendering. For color deviation, linear interpolation is used to reduce color deviation, and color weight constraints are introduced to eliminate abnormal weights and encourage the weight distribution function to converge to the surface of the object. For geometric deviation, point clouds are extracted from the rendering network based on neural radiation fields, explicit geometric constraints are provided, the geometric shape of the object is optimized, and a pre-trained model is used for feature extraction to optimize the consistency of geometric features during network training. By introducing color constraints and geometric constraints, the method of the present invention successfully overcomes the problems faced by volume rendering methods in the reconstruction of complex tree scenes, ensuring that the reconstructed tree model can better retain and reproduce the original geometric features and true texture of the tree.

[0195] In a preferred embodiment of the present invention, a cone sampling strategy is introduced to perform light sampling on a rough depth map, and a method for obtaining a Gaussian model is as follows:

[0196] For any pixel point, a cone is projected from the camera's projection origin o along the direction d passing through the pixel center. The vertex of the cone is located at o, and the radius parameter of the cone surface at the image plane o+d is set to Will Set to the width of pixels in world coordinates, scaled by 2 / √12, to produce a cone;

[0197] The set of positions x that lie within the frustum of the cone between two t values ​​[t0, t1] It is expressed as:

[0198]

[0199] Among them, t represents a parameterized representation of the distance from the camera projection origin o to a certain point x of the frustum, t0, t1 define the near plane and far plane of the frustum, that is, the distance from the camera origin to the boundaries of the two ends of the frustum, which is used to determine the depth range of sampling; is an indicator function. The first half of the formula ensures that the projection distance of x along the d direction is between t0 and t1. The second half of the formula checks whether x satisfies the shape constraint of the cone. If x is between In the cone defined by

[0200] The position encoding of this pixel is re-expressed as the expectation of all coordinate position encodings within the cone

[0201]

[0202] Among them, γ(x) represents the position encoding of the original neural radiation field, γ * The representation is defined as integrated position encoding IPE;

[0203] For the cone sampling method, the integral of the original position encoding expression has no closed form solution and cannot be calculated effectively. Using a multivariate Gaussian function to approximate the frustum can effectively approximate the required features. In order to approximate the frustum with a multivariate Gaussian function, it is necessary to calculate The mean and covariance of each truncated cone are assumed to be circular and symmetric around the axis of the cone, so in addition to o and d, the multivariate Gaussian function model is represented by 3 values: the average distance μ along the ray t , the variance along the ray direction Variance along the ray perpendicular direction

[0204]

[0205] Among them, t μ represents the midpoint, which is the point on the central axis of the cone (i.e. the direction of the light) that divides the height of the cone into two equal parts. μ =(t0+t1); t δ The half-width is half the width of the cone truncated cone in the direction perpendicular to the central axis, which is proportional to the radius of the cone. Related, t δ =(t1-t0) / 2;

[0206] Convert the Gaussian model from the cone coordinate system to the world coordinate system to obtain the final multivariate Gaussian model:

[0207]

[0208] Among them, μ is the mean vector of the multivariate Gaussian distribution, which represents the central tendency of the point position in the cone truncated cone; Σ is the covariance matrix of the multivariate Gaussian distribution, which describes the discrete degree of the distribution of the point position in the cone truncated cone and the correlation between the dimensions; I represents the identity matrix, which is a square matrix whose elements on the diagonal are all 1 and the elements on the off-diagonal are all 0.

[0209] In a preferred embodiment of the present invention, the steps of position encoding the Gaussian model using an integrated position encoding method are as follows:

[0210] IPE is the expectation of the position encoding coordinates of the Gaussian distribution, and the position encoding is rewritten in matrix form:

[0211]

[0212] Where P is a matrix with each row representing a frequency component and each column corresponding to a spatial dimension (x, y, z), which is used to generate the input of the sine and cosine functions. L is an integer representing the number of frequency levels used in the position encoding. γ(x) is the integrated position encoding of position x, which contains the values ​​of all sine and cosine functions calculated by the P matrix and position x. It is a vector. Px represents the product of the matrix P and the position vector x.

[0213] Using the covariance property, the Gaussian model is positionally encoded to obtain the encoded multivariate Gaussian representation: μ γ =Pμ,Σ γ =PΣP T ; The encoded expectation is expressed as:

[0214]

[0215] Among them, μ γ is the mean of the encoded Gaussian distribution, indicating the center position of the data; Σ γ is the covariance matrix of the encoded Gaussian distribution, which indicates the degree of discreteness of the data; Represents mathematical expectation, the random variable x has mean μ and variance σ 2 Normal distribution of

[0216] The final IPE encoding γ(μ,Σ) is expressed as:

[0217]

[0218] in, represents element-wise multiplication; the covariance matrix Σ γ The computational cost is relatively high, and only the diagonal values ​​need to be considered, so in practical implementation it is expressed as:

[0219] diag(Σ γ)=[diag(Σ),4diag(Σ),…,4 L-1 diag(Σ)] T

[0220]

[0221] Where diag(Σ) represents the vector consisting of the diagonal elements of the covariance matrix Σ.

[0222] Since the neural radiation field samples points along each ray and encodes all frequencies equally, high-frequency PE features will be aliased, resulting in rendering artifacts. IPE, on the other hand, integrates the features of position encoding by calculating expectations, only considering the marginal distribution of γ(x). When there is a frequency period that is less than the integrated frequency period, the high-frequency dimension of IPE will shrink toward zero. And thanks to the properties of IPE, there is no need to use a coarse-to-fine strategy of two MLPs like the original neural radiation field. This method can be implemented using only one MLP, which reduces the parameters of the original neural radiation field model by half and has better anti-aliasing performance.

[0223] In a preferred embodiment of the present invention, based on the coding annealing strategy, the steps of performing sparse view learning on the coarse depth map are:

[0224] The proposed coding annealing strategy uses low-frequency coding to learn the geometry of the scene at the beginning of training, and then gradually adds high-frequency coding to obtain details. Given a position code of length L, a frequency mask m is used to adjust the visible frequency based on the training time step:

[0225] γ′ L (t,T;x)=γ L (x)⊙m(t,T,L)

[0226] Among them, γ′ L (t,T; x) represents the output of the ith position after frequency mask adjustment, γ L (x) is a function transformation, which indicates the effect of position encoding on input x, and the output length is L. ⊙ is a multiplication operator, which indicates the multiplication of corresponding elements in two vectors or matrices of the same dimension; m(t,T,L) is:

[0227]

[0228] m i(t,T,L) represents the value of the ith position of the mask m(t,T,L), t and T represent the current training iteration number and the final iteration number of the coding annealing, respectively, and L represents the hyperparameter that controls the maximum coding frequency; the low-frequency coding learns the geometry of the scene, and as the training progresses, the high-frequency part of the coding is gradually opened using a sliding truncated Hanning window function. Avoid unstable and susceptible high-frequency signals at the beginning of training, and gradually provide high-frequency information to avoid over-smoothing.

[0229] Through the coding annealing strategy, low-frequency information is quickly learned in the early stage of network training. As the training progresses, the high-frequency part of the coding is gradually enabled, allowing the network to focus on learning high-frequency details, achieving a balance between high and low frequencies, and improving the performance of the network under sparse view conditions.

[0230] In a preferred embodiment of the present invention, depth sorting and depth consistency constraints are introduced to regularize and optimize the neural radiation field. The specific steps are as follows:

[0231] The present invention uses the pre-trained depth estimation model ZoeDepth to predict a rough depth map, introduces depth information into the NeRF network, proposes depth sorting and depth consistency constraints, and regularizes and optimizes the training of the NeRF (neural radiance field) network, thus making up for the defects of insufficient viewing angle and insufficient constraints of NeRF under sparse views.

[0232] Depth ranking: Instead of directly supervising NeRF with coarse depth prior, the present invention relaxes the depth constraint and extracts robust local depth ranking from the coarse depth map, so that the depth ranking of the neural radiance field is consistent with the depth ranking of the coarse depth map, and uses relative depth rather than absolute depth to supervise the neural radiance field network;

[0233] Given a pair of pixels in a single image, the depth ordering prior is extracted from the coarse depth map to the neural radiance field. On the one hand, given a local region P of an RGB image with pose, the depth d of the ray traced from P is calculated as:

[0234]

[0235] Where N is the number of sampling points, i is the serial number of the sampling point, and w i represents the weight of the i-th sampling point, t i Represents the parameterized representation of the distance from the camera projection origin o to the i-th sampling point;

[0236] Get the depth of the network rendering view;

[0237] On the other hand, we use the depth d′ estimated by the depth estimation model and crop the local patch with the same spatial position as d to transfer the depth sorting knowledge from d′ to d for depth sorting. Let a and b be the indices of the pixel coordinates of d′ and d. The loss of depth sorting is Defined as:

[0238]

[0239] Among them, d a ,d b are the depth values ​​of the pixels indexed as a and b from d, respectively; m is a hyperparameter, i.e., the local depth sorting error; two depth pixels of d′ are randomly sampled, denoted as d′ a and d′ b , and d′ a ≤d′ b ; If the order of d′ and d is inconsistent, that is, d′ a ≤d′ b , d a >d a , and within the minimum error range, the neural radiance field is penalized by the loss function; by penalizing NeRF through the loss function, even if the depth estimation is not completely accurate, the depth sorting loss can guide NeRF to learn the correct depth order, which helps to improve the geometric reconstruction quality of the scene under sparse views.

[0240] Depth-invariant geometric consistency regularization is used to measure the depth relationship between pixels in the same local area. For a local area P of the RGB image, the depth map d rendered by the neural radiance field and the predicted depth d′ of the depth estimation model are compared. The scale-invariant mean square depth error is used to measure the depth relationship between pixels in the same block. The depth consistency loss function Defined as:

[0241]

[0242] Where is a matrix, each row represents a frequency component, each column corresponds to a spatial dimension (x, y, z), used to generate the input of the sine and cosine functions; l, j are the coordinate indices of the pixels in the local area P of the RGB image; d lj , d′ lj They represent the true depth value and predicted depth value at position (l, j) respectively.

[0243] For the depth map d rendered by NeRF, adjust its scale to achieve the best matching scale with the depth d′ predicted by the pre-trained model. Although the scale of the predicted depth provided by the pre-trained model may be inaccurate, the predicted depth can usually maintain a relatively accurate local structure (i.e., relative relationship) in a small local area. This 3D consistency relationship is beneficial to the optimization process of NeRF. By utilizing these local structures, even if the absolute scale is unknown, NeRF can learn the relative depth information of the scene through optimization, thereby improving the geometric consistency of reconstruction.

[0244] In a preferred embodiment of the present invention, the loss of the neural radiation field network includes color loss, depth sorting loss, depth consistency loss, color loss Consistent with the original loss function of the neural radiance field, the mean square error between the color obtained by volume rendering and the color of the real image is obtained:

[0245]

[0246] in, represents the color value predicted by the model for light r, and C(r) represents the color value corresponding to light r in the real image;

[0247] The pre-trained depth estimation model is used to generate a rough depth map, and depth sorting and depth consistency constraints are proposed. The depth sorting loss and depth consistency loss are respectively and

[0248] Total loss function of the neural radiance field network It is expressed as:

[0249]

[0250] Among them, λ1 and λ2 are hyperparameters used to balance the relative importance between different loss functions.

[0251] In a preferred embodiment of the present invention, implicit surface reconstruction is performed based on volume rendering, and the specific method is as follows:

[0252] In the neural implicit function, the input is a coordinate point in three-dimensional space, and the output is a real number, which represents the function value at the coordinate point. By training the neural network, it can learn the characteristics and structure of geometric shapes, and then generate a geometric surface composed of points with zero function values.

[0253] The neural implicit function is expressed as f(x)=0, where Represents the three-dimensional spatial position of the mapping; reconstructs the surface of the geometric shape by sampling points in the input space and determining their location based on the output of the neural implicit function The zero level set is the set of points that satisfy f(x) = 0; the surface It is expressed as:

[0254]

[0255] Where f(x) is the spatial position The signed distance function SDF, f(x) = 0 represents the point on the surface of the reconstructed object;

[0256] Color mapping function c: Color coding and spatial location and viewing direction The input image is used as supervision to minimize the difference between the rendered image and the input image to learn the weights of the neural network. Finally, the zero level set of the neural SDF represents the reconstructed surface.

[0257] For a pixel on the image and the camera position, a ray is emitted from the camera center and passes through the pixel as:

[0258] p(t)=o+tv|t≥0

[0259] Where o is the camera center, v is the unit direction vector of the ray, and t is the distance between x and o;

[0260] The volume rendering C(o,v) of the original neural radiation field is expressed as:

[0261]

[0262] Where T(t) represents the cumulative transmittance from the camera to point p(t) on the light, which represents the probability that the light is not absorbed or scattered by any medium before reaching point p(t); t f , t n is the upper and lower limits of the integral, t n Usually set to a value very close to the camera, t f It is usually set to a value large enough to include all important objects in the scene; p(t) is a parameterized representation of light, which is represented as a ray extending from the camera position o along the direction d, and the parameterized form is p(t) = o + td, where t is the distance from the camera to a point on the light; σ(p(t)) is the volume density or scattering coefficient at point p(t), which represents the intensity of the interaction between the light and the medium at this point; c(p(t), v) is the radiance at point p(t) for the line of sight direction v, that is, the color, which describes the color information of the light after being scattered at this point;

[0263] In order to accurately describe the volume density and apply the volume rendering method to the training of the SDF network, a bridge must be built to realize the conversion from the signed distance field to the volume rendering. The original neural surface reconstruction method NeuS introduces a probability density function S-density: φ s (f(x)), f(x) is the signed distance function, where S-density is the Logistic probability distribution function Φ s Derivative of (x):

[0264] φ s (x) = se -sx / (1+e -sx ) 2

[0265] Φ s (x) = (1 + e -sx ) -1 ,φ s (x) = Φ′ s (x)

[0266] Among them, Φ s (x) is the logistic function, φ s (x) is the probability density function S-density introduced by NeuS. S is a parameter used to control the shape of the Logistic distribution. The larger the value of s, the steeper the curve; the smaller the value of s, the flatter the curve; Φ′ s (x) is Φ s The first derivative of (x), the base of the e exponential function;

[0267] Using S-density for volume rendering, training the SDF network, the volume rendering formula is redefined as:

[0268]

[0269] Directly using S-density for volume rendering does not guarantee that the maximum value of the probability distribution function is exactly the surface of the object, that is, it cannot guarantee that the zero level set represented by the SDF network is the real surface. There is a certain deviation between them. In order to achieve unbiased estimation, NeuS redefines the volume rendering formula and designs an unbiased weight function:

[0270]

[0271] Among them, the cumulative transmittance T(t) along the ray is expressed as Φ through S-density conversion s (f(p(t))), the volume density σ(t) is replaced by the redefined ρ(t):

[0272]

[0273] For the convenience of calculation, the integral is approximated by discretized sampling, and the rendered color of the pixel is accumulated along the ray with N discrete sampling points:

[0274]

[0275] Among them, α i Represents discrete opacity, T i represents the cumulative transmittance along the ray, where α i Further expressed as:

[0276]

[0277] In a preferred embodiment of the present invention, a color constraint is introduced based on the color deviation of implicit surface reconstruction, specifically:

[0278] c v Defined as the color for volume rendering, Among them, t * It represents the parametric representation of the best point where the ray intersects the geometric surface during ray tracing; argmin is a mathematical term used to represent the parameter value (the value of the independent variable) at which a function achieves the minimum value in its domain; t represents the distance between x and o, v represents the unit direction vector of the ray, and tv represents the distance between x and o in the direction of the ray; p is the parameter representation of the ray; Represents a geometric surface;

[0279] Use MLP (Multilayer Perceptron) to train an SDF network F Θ Get its SDF value and pass it through another MLP network Get the color value, mathematically expressed as:

[0280] sdf(p)=F Θ (p)

[0281]

[0282] Among them, sdf(p) represents the output value of the SDF network when p is used as input; c(o,v,t) represents the output value of the SDF network at the camera position o, the light direction v and the distance t through the MLP network The calculated color value;

[0283] The estimated color c of the image at camera position o v It is expressed as:

[0284]

[0285] Where o is the camera center, v is the unit direction vector of the ray, t is the distance between x and o, and w(t) is the weight of point t; the volume rendering color of a pixel is written in discrete form as:

[0286]

[0287] Among them, w(t i ) is t i The weight of the point discretization, c(t i ) is t i The color value of the point discretization; n is the number of sampling points of discrete sampling, and i represents the i-th segment of the integral interval;

[0288] Assume that the initial intersection point of the ray and the surface is denoted by p(t * ) and sdf(t * )=0; along the v direction is p(t * ) surface color c s :

[0289] c s =c(t * )

[0290] Among them, c(t * ) represents the color value at the initial intersection point p(t*), that is, the color of the object surface; t * The point is the intersection of the ray and the geometric surface, and this point satisfies sdf(t * )=0, indicating that it is exactly on the surface of the geometric surface; p(t * ) is a parameterized representation of the intersection ray;

[0291] In the surface reconstruction task, the tree surface and other parts are separated. When sdf(t * )=0, use t j Represents the distance from the surface point t * The nearest sampling point, the formula is rewritten as:

[0292]

[0293] The color deviation between the volume rendering color and the surface color is expressed as:

[0294]

[0295] in, Denoted as the deviation ε caused by the volume rendering weight function weight , c(t j )-c(t * ) represents the deviation ε caused by discrete sampling sample ;

[0296] The color deviation can be expressed as:

[0297] Δc=ε weight +ε sample

[0298] There is a significant deviation between the surface color of the object and the color estimated by the network. The deviation consists of two parts: the deviation caused by discrete sampling ε sample and the deviation ε caused by the volume rendering weight function weight ;

[0299] Using the SDF value of the sampling point calculated by volume rendering, linear interpolation is used to extract the zero crossing point and reduce the sampling deviation ε sample , the sampled point along the ray is represented as p(t k ), k = 1, ..., N, N is the number of sampling points for stratified sampling;

[0300] Search along the direction of the ray for the first point that satisfies the condition: the SDF value of this point is positive, and the SDF value of the next sampling point is negative, expressed as:

[0301] f(p(t k ))·f(p(t k+1 ))<0

[0302] Where f is the geometric network of the output SDF value, p(t k ) is the first sampling point before the object surface; only the first ray intersecting the surface is considered, because the intersection of other rays with the surface is blocked. Since the layered sampling strategy attaches great importance to sampling near the surface point, p(t k ) and p(t k+1 ) should be small. Use differentiable linear interpolation to approximate surface points where the SDF is zero.

[0303]

[0304] Using the sampling point set p(t k ) and approximate surface points Color C is obtained by volume rendering total :

[0305]

[0306] The color deviation is rewritten as:

[0307]

[0308] make The color deviation is further rewritten as:

[0309] Δc=ε' weight +εinterp

[0310] Among them, ε interp represents the deviation caused by linear interpolation, ε' weight Expressed as the new weight function bias; the original discrete sampling bias ε sample Further reduced to the linear interpolation deviation ε interp Compared with the original sampling deviation, the linear interpolation deviation is an order of magnitude smaller than the discrete sampling deviation.

[0311] In order to reduce the updated weight bias, a color weight loss function is introduced

[0312]

[0313] l weight Penalize abnormal weight values ​​far away from the object surface, especially weight distribution far away from the surface but with large weight values, and encourage the weight distribution function to converge to the object surface.

[0314] In a preferred embodiment of the present invention, geometric constraints are introduced based on the geometric deviation of implicit surface reconstruction, as follows:

[0315] Explicit geometric constraints:

[0316] Extract point clouds to approximate the surface points of the object, provide explicit geometric constraints through the projection features of the supervised surface points, and optimize the geometric shape of the object;

[0317] The neural radiation field represents the geometry and appearance of the scene as a continuous volumetric radiation field parameterized by a neural network. It infers the spatial geometry by observing the changes in color and density along the camera light and extracts point cloud data. The cumulative transmittance T(t) of the neural radiation field represents the cumulative value of the volume density σ of all previous objects that have passed through the ray when it reaches an object; when T(t) drops sharply after a certain point, it indicates that this point may be on the surface of the object. Mathematically, the point cloud is represented by solving the minimum value of the first-order derivative of T(t) with respect to t:

[0318]

[0319] Among them, argmin is a mathematical term used to represent the parameter value at which a function takes its minimum value in its domain; q represents the point at which the first-order derivative of the cumulative transmittance takes its minimum value;

[0320] From the neural radiance field, we get the point cloud {P|x p ∈P}, there may be outliers in the recovered point cloud. The point cloud distribution around the implicit surface is different, resulting in different levels of noise in different areas. The filter is used to exclude outliers and produce a relatively accurate and clean point cloud;

[0321] If the points in the point cloud are located on the surface of the object, it can be approximately expressed as: G(p i )=0,p i ∈P, where G represents the signed distance function and P represents the point cloud set; the SDF network is supervised by supervising the surface points:

[0322]

[0323] in, represents the geometric loss function;

[0324] Geometric feature consistency:

[0325] Since point clouds are usually noisy, additional constraints are needed to improve this situation. The present invention optimizes the consistency of geometric features during network training.

[0326] Use the Vis-MVSNet pre-trained model to extract features from RGB images, and then use the geometric feature consistency loss to constrain the neural implicit surface reconstruction. The Vis-MVSNet pre-trained model uses a standard U-Net to extract image feature maps;

[0327] like Figure 4 As shown in the figure, the U-Net network includes a bottom layer, an encoder, a decoder and a head part. The bottom layer consists of multiple blocks, each block consists of a series of convolutional layers and batch normalization layers. The encoder and the head part are both composed of multiple blocks. In addition to the convolutional layer and the batch normalization layer, the encoder also includes a pooling layer for gradually reducing the spatial size of the feature map. The decoder consists of a deconvolution layer, a convolutional layer and possible residual connections for gradually restoring the spatial size of the feature map.

[0328] Assume that the point where the ray through view 0 intersects the surface is x, and the projection of view 0 and its adjacent views are represented as These projections refer to the same 3D point in space, so their image features should be consistent, and the feature loss is defined as:

[0329]

[0330] Among them, N c Represents the number of feature channels, N v represents the number of adjacent source views, F represents the extracted feature map, and p i =K i (R i x+t i ) represents the projection of x in view i, [K i ,R i ,t i] is the corresponding camera parameter, pixel p i The image feature at is F i (p i ).

[0331] In a preferred embodiment of the present invention, the loss function of the implicit surface reconstruction network based on volume rendering includes color loss, Eikonal loss, color weight loss, geometric loss, and feature loss;

[0332] Color Loss is the difference between the RGB colors obtained from the real input image and the RGB colors obtained from the volume rendering:

[0333]

[0334] Where m is the number of pixels, C i Indicates the RGB color value obtained by the image. RGB color values ​​obtained from volume rendering;

[0335] The Eikonal loss constrains the gradient of the SDF to 1 so that it satisfies the properties of the distance function, thereby normalizing the SDF and ensuring that it provides the correct distance field representation. The Eikonal loss is added at the sampling point to constrain the signed distance field gradient of the SDF network:

[0336]

[0337] in, is the set of sampling points, ||·||2 is the L2 norm;

[0338] For the color deviation of volume rendering, the color deviation is reduced by linear interpolation and color weight loss is introduced

[0339] For the geometric deviation of volume rendering, the geometric loss is And the feature loss function is

[0340] Total loss function It is expressed as:

[0341]

[0342] Among them, λ3, λ4, λ5, λ6 are hyperparameters used to balance the relative importance between different loss functions.

[0343] The present invention also provides a tree neural rendering and reconstruction system based on sparse views, including a data acquisition module and a processing module, wherein the data processing module includes a camera and other equipment for collecting RGB images of trees. The input end of the processing module is connected to the output end of the data processing module, and the processing module executes the method of the present invention to realize tree neural rendering and reconstruction.

[0344] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0345] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A sparse view based tree neural rendering and reconstruction method, characterized in that: The steps include: Use the camera to collect RGB pictures of trees and obtain camera parameters; The RGB image of the tree and the camera parameters are fed into a pre-trained depth estimation model to generate a rough depth map that is fed into the neural radiance field network. A cone sampling strategy is introduced to perform ray sampling on the rough depth map to obtain a Gaussian model; The integrated position encoding method is used to position encode the Gaussian model; Based on the encoding annealing strategy, sparse view learning is performed on the coarse depth map; Introducing depth sorting and depth consistency constraints to regularize and optimize the neural radiation field; Extract point clouds from images based on the neural radiance field network, and input the RGB images of trees and camera parameters into the pre-trained feature extraction model to obtain image features; Based on volume rendering, implicit surface reconstruction is performed; Based on the color deviation and geometric deviation of implicit surface reconstruction, color constraints and geometric constraints are introduced; The Marching Cubes algorithm is used to reconstruct the 3D tree model from the implicit surface.

2. The sparse view-based tree neural rendering and reconstruction method according to claim 1, characterized in that: The cone sampling strategy is introduced to perform light sampling on the rough depth map. The method to obtain the Gaussian model is as follows: For any pixel point, a cone is projected from the camera's projection origin o along the direction d passing through the pixel center. The vertex of the cone is located at o, and the radius parameter of the cone surface at the image plane o+d is set to Will Set to the width in pixels in world coordinates, according to Scale to create a cone; The set of positions x that lie within the frustum of the cone between two t values ​​[t0, t1] It is expressed as: Among them, t represents a parameterized representation of the distance from the camera projection origin o to a certain point x of the frustum, t0, t1 define the near plane and far plane of the frustum, that is, the distance from the camera origin to the boundaries of the two ends of the frustum, which is used to determine the depth range of sampling; is an indicator function. The first half of the formula ensures that the projection distance of x along the d direction is between t0 and t1. The second half of the formula checks whether x satisfies the shape constraint of the cone. If x is between In the cone defined by The position encoding of this pixel is re-expressed as the expectation of all coordinate position encodings within the cone Among them, γ(x) represents the position encoding of the original neural radiation field, and γ* represents the integrated position encoding IPE defined as; Use multivariate Gaussian functions to approximate the frustum, and calculate The mean and covariance of each truncated cone are assumed to be circular and symmetric around the axis of the cone, so in addition to o and d, the multivariate Gaussian function model is represented by 3 values: the average distance μ along the ray t , the variance along the ray direction Variance along the ray perpendicular direction Among them, t μ represents the midpoint, which is the point on the central axis of the cone that divides the height of the cone into two equal parts. μ =(t0+t1); t δ The half-width is half the width of the cone truncated cone in the direction perpendicular to the central axis, which is proportional to the radius of the cone. Related, t δ =(t1-t0) / 2; Convert the Gaussian model from the cone coordinate system to the world coordinate system to obtain the final multivariate Gaussian model: Among them, μ is the mean vector of the multivariate Gaussian distribution, which represents the central tendency of the point position in the cone truncated cone; Σ is the covariance matrix of the multivariate Gaussian distribution, which describes the discrete degree of the distribution of the point position in the cone truncated cone and the correlation between the dimensions; I represents the identity matrix, which is a square matrix whose elements on the diagonal are all 1 and the elements on the off-diagonal are all 0.

3. The sparse view-based tree neural rendering and reconstruction method according to claim 2, characterized in that: Using the integrated position encoding method, the steps for position encoding of the Gaussian model are as follows: IPE is the expectation of the position encoding coordinates of the Gaussian distribution, and the position encoding is rewritten in matrix form: Where P is a matrix with each row representing a frequency component and each column corresponding to a spatial dimension (x, y, z), which is used to generate the input of the sine and cosine functions. L is an integer representing the number of frequency levels used in the position encoding. γ(x) is the integrated position encoding of position x, which contains the values ​​of all sine and cosine functions calculated by the P matrix and position x. It is a vector. Px represents the product of the matrix P and the position vector x. Using the covariance property, the Gaussian model is positionally encoded to obtain the encoded multivariate Gaussian representation: μ γ =P μ ,Σ γ =PΣP T ; The encoded expectation is expressed as: Among them, μ γ is the mean of the encoded Gaussian distribution, indicating the center position of the data; Σ γ is the covariance matrix of the encoded Gaussian distribution, which indicates the degree of discreteness of the data; Represents mathematical expectation, the random variable x has mean μ and variance σ 2 Normal distribution of The final IPE encoding γ(μ,Σ) is expressed as: Where ° represents element-wise multiplication; the covariance matrix Σ γ The computational cost is relatively high, and only the diagonal values ​​need to be considered, so in practical implementation it is expressed as: diag(Σ γ )=[diag(Σ),4diag(Σ),…,4 L-1 diag(Σ)] T Wherein, diag(∑) represents the vector consisting of the diagonal elements of the covariance matrix Σ.

4. The sparse view-based tree neural rendering and reconstruction method according to claim 1, characterized in that: Based on the encoding annealing strategy, the steps of sparse view learning for the coarse depth map are: Given a positional encoding of length L, the frequency mask m is used to adjust the visible frequencies based on the training time step: γ L ′ (t,T;x)=γ L (x)⊙m(t,T,L) Among them, γ L ′ (t,T; x) represents the output of the ith position after frequency mask adjustment, γ L (x) is a function transformation, which indicates the effect of position encoding on input x, and the output length is L. ⊙ is a multiplication operator, which indicates the multiplication of corresponding elements in two vectors or matrices of the same dimension; m(t,T,L) is: m i (t,T,L) represents the value of the ith position of the mask m(t,T,L), t and T represent the current training iteration number and the final iteration number of coding annealing, respectively, and L represents the hyperparameter that controls the maximum coding frequency; the low-frequency coding learns the geometric shape of the scene, and as the training progresses, the sliding truncated Hanning window function is used to gradually open the high-frequency part of the coding.

5. The sparse view-based tree neural rendering and reconstruction method according to claim 4, characterized in that: Introduce depth sorting and depth consistency constraints to regularize and optimize the neural radiation field. The specific steps are as follows: Relax the depth constraint and extract a robust local depth ranking from the coarse depth map, so that the depth ranking of the neural radiance field is consistent with the depth ranking of the coarse depth map, and use relative depth instead of absolute depth to supervise the neural radiance field network; Given a pair of pixels in a single image, the depth ordering prior is extracted from the coarse depth map to the neural radiance field. On the one hand, given a local region P of an RGB image with pose, the depth d of the ray traced from P is calculated as: Where N is the number of sampling points, i is the serial number of the sampling point, and w i represents the weight of the i-th sampling point, t i Represents the parameterized representation of the distance from the camera projection origin o to the i-th sampling point; Get the depth of the network rendering view; On the other hand, the depth d estimated by the depth estimation model ′ , and cut out the local patch with the same spatial position as d, and transfer the depth sorting knowledge from d ′ Move to d to perform depth sorting, set a and b to d ′ and the index of the pixel coordinates of d, the loss of depth sorting Defined as: Among them, d a ,d b are the depth values ​​of pixels indexed as a and b from d, respectively, and m is a hyperparameter, i.e., the local depth sorting error; randomly sampling d ′ Two depth pixels, denoted as d ′ a and d ′ b , and d ′ a ≤d ′ b ; if d ′ The order of d is inconsistent, that is, d ′ a ≤d ′ b , d a >d a , and within the minimum error range, the neural radiation field is penalized by the loss function; Depth-invariant geometric consistency regularization is used to measure the depth relationship between pixels in the same local area. For a local area P of the RGB image, the depth d rendered by the neural radiance field is compared with the predicted depth d of the depth estimation model. ′ , using the scale-invariant mean square depth error to measure the depth relationship between pixels in the same block, the depth consistency loss function Defined as: Where P is a matrix with each row representing a frequency component and each column corresponding to a spatial dimension (x, y, z), which is used to generate the input of the sine and cosine functions; l, j are the coordinate indices of the pixels in the local area P of the RGB image; d lj , d l ′ j Represent the true depth value and predicted depth value at position (l, j) respectively.

6. The sparse view-based tree neural rendering and reconstruction method according to claim 5, characterized in that: The losses of the neural radiance field network include color loss, depth sorting loss, depth consistency loss, color loss Consistent with the original loss function of the neural radiance field, the mean square error between the color obtained by volume rendering and the color of the real image is obtained: in, represents the color value predicted by the model for light r, and C(r) represents the color value corresponding to light r in the real image; The pre-trained depth estimation model is used to generate a rough depth map, and depth sorting and depth consistency constraints are proposed. The depth sorting loss and depth consistency loss are respectively and Total loss function of the neural radiance field network It is expressed as: Among them, λ1 and λ2 are hyperparameters used to balance the relative importance between different loss functions.

7. The sparse view-based tree neural rendering and reconstruction method according to claim 1, characterized in that: Based on volume rendering, implicit surface reconstruction is performed. The specific method is as follows: The neural implicit function is expressed as f(x)=0, where Represents the three-dimensional spatial position of the mapping; reconstructs the surface of the geometric shape by sampling points in the input space and determining their location based on the output of the neural implicit function The zero level set is the set of points that satisfy f(x) = 0; the surface It is expressed as: Where f(x) is the spatial position The signed distance function SDF, f(x) = 0 represents the point on the surface of the reconstructed object; Color Mapping Function Color coding and spatial location and viewing direction The input image is used as supervision to minimize the difference between the rendered image and the input image to learn the weights of the neural network. Finally, the zero level set of the neural SDF represents the reconstructed surface. For a pixel on the image and the camera position, a ray is emitted from the camera center and passes through the pixel as: Where o is the camera center, v is the unit direction vector of the ray, and t is the distance between x and o; The volume rendering C(o,v) of the original neural radiation field is expressed as: Where T(t) represents the cumulative transmittance from the camera to point p(t) on the light, which represents the probability that the light is not absorbed or scattered by any medium before reaching point p(t); t f , t n is the upper and lower limits of the integral, t n Usually set to a value very close to the camera, t f It is usually set to a value large enough to include all important objects in the scene; p(t) is a parameterized representation of light, which is represented as a ray extending from the camera position o along the direction d, and the parameterized form is p(t) = o + td, where t is the distance from the camera to a point on the light; σ(p(t)) is the volume density or scattering coefficient at point p(t), which represents the intensity of the interaction between the light and the medium at this point; c(p(t), v) is the radiance at point p(t) for the line of sight direction v, that is, the color, which describes the color information of the light after being scattered at this point; The original neural surface reconstruction method NeuS introduces a probability density function S-density: φ s (f(x)), f(x) is the signed distance function, where S-density is the Logistic probability distribution function Φ s Derivative of (x): φ s (x)=se -sx / (1+e -sx ) 2 F s (x)=(1+e -sx ) -1 ,f s (x)=Φ ′ s (x) Among them, Φ s (x) is the logistic function, φ s (x) is the probability density function S-density introduced by NeuS. S is a parameter used to control the shape of the Logistic distribution. The larger the value of s, the steeper the curve; the smaller the value of s, the flatter the curve; Φ ′ s (x) is Φ s The first derivative of (x), the base of the e exponential function; Using S-density for volume rendering, training the SDF network, the volume rendering formula is redefined as: In order to achieve unbiased estimation, NeuS redefines the volume rendering formula and designs an unbiased weight function: Among them, the cumulative transmittance T(t) along the ray is expressed as Φ through S-density conversion s (f(p(t))), the volume density σ(t) is replaced by the redefined ρ(t): The integral is approximated by discretized sampling, and the rendered color of the pixel is accumulated along a ray with N discrete sampling points: Among them, α i Represents discrete opacity, T i represents the cumulative transmittance along the ray, where α i Further expressed as:

8. The method for tree neural rendering and reconstruction based on sparse views as claimed in claim 7, characterized in that: Based on the color deviation of implicit surface reconstruction, color constraints are introduced, specifically: c v Defined as the color for volume rendering, Among them, t * It represents the parametric representation of the best point where the ray intersects the geometric surface during ray tracing; argmin is a mathematical term used to represent the parameter value at which a function achieves the minimum value in its domain; t represents the distance between x and o, v represents the unit direction vector of the ray, and tv represents the distance between x and o in the direction of the ray; p is the parameter representation of the ray; Represents a geometric surface; Use MLP to train an SDF network F Θ Get its SDF value and pass it through another MLP network Get the color value, mathematically expressed as: sdf(p)=F Θ (p) Among them, sdf(p) represents the output value of the SDF network when p is used as input; c(o,v,t) represents the output value of the SDF network at the camera position o, the light direction v and the distance t through the MLP network The calculated color value; The estimated color c of the image at camera position o v It is expressed as: Where o is the camera center, v is the unit direction vector of the ray, t is the distance between x and o, and w(t) is the weight of point t; the volume rendering color of a pixel is written in discrete form as: Among them, w(t i ) is t i The weight of the point discretization, c(t i ) is t i The color value of the point discretization; n is the number of sampling points of discrete sampling, and i represents the i-th segment of the integral interval; Assume that the initial intersection point of the ray and the surface is denoted by p(t * ) and sdf(t * )=0; along the v direction is p(t * ) surface color c s : c s =c(t * ) Among them, c(t * ) represents the color value at the initial intersection point p(t*), that is, the color of the object surface; t * The point is the intersection of the ray and the geometric surface, and this point satisfies sdf(t * )=0, indicating that it is exactly on the surface of the geometric surface; p(t * ) is a parameterized representation of the intersection ray; Separate the tree surface from other parts, when sdf(t * )=0, use t j Represents the distance from the surface point t * The nearest sampling point, the formula is rewritten as: The color deviation between the volume rendering color and the surface color is expressed as: in, Denoted as the deviation ε caused by the volume rendering weight function weight , c(t j )-c(t * ) represents the deviation ε caused by discrete sampling sample ; The color deviation can be expressed as: Δc=ε weight +e sample There is a significant deviation between the surface color of the object and the color estimated by the network. The deviation consists of two parts: the deviation caused by discrete sampling ε sample and the deviation ε caused by the volume rendering weight function weight ; Using the SDF value of the sampling point calculated by volume rendering, linear interpolation is used to extract the zero crossing point and reduce the sampling deviation ε sample , the sampled point along the ray is represented as p(t k ), k = 1, ..., N, N is the number of sampling points for stratified sampling; Search along the direction of the ray for the first point that satisfies the condition: the SDF value of this point is positive, and the SDF value of the next sampling point is negative, expressed as: f(p(t k ))·f(p(t k+1 ))<0 Where f is the geometric network of the output SDF value, p(t k ) is the first sampling point before the surface of the object; use differentiable linear interpolation to approximate the surface point where the SDF is zero Using the sampling point set p(t k ) and approximate surface points Color C is obtained by volume rendering total : The color deviation is rewritten as: make The color deviation is further rewritten as: Δc=ε' weight +e interp Among them, ε interp represents the deviation caused by linear interpolation, ε' weight Expressed as a new weight function bias; In order to reduce the updated weight bias, a color weight loss function is introduced l weight Penalize abnormal weight values ​​far away from the object surface, especially weight distribution far away from the surface but with large weight values, and encourage the weight distribution function to converge to the object surface.

9. The method for tree neural rendering and reconstruction based on sparse views as claimed in claim 8, characterized in that: Based on the geometric deviation of implicit surface reconstruction, geometric constraints are introduced as follows: Explicit geometric constraints: Extract point clouds to approximate the surface points of the object, provide explicit geometric constraints through the projection features of the supervised surface points, and optimize the geometric shape of the object; The neural radiation field represents the geometry and appearance of the scene as a continuous volumetric radiation field parameterized by a neural network. It infers the spatial geometry by observing the changes in color and density along the camera light and extracts point cloud data. The cumulative transmittance T(t) of the neural radiation field represents the cumulative value of the volume density σ of all previous objects that have passed through the ray when it reaches an object; when T(t) drops sharply after a certain point, it indicates that this point may be on the surface of the object. Mathematically, the point cloud is represented by solving the minimum value of the first-order derivative of T(t) with respect to t: Among them, argmin is a mathematical term used to represent the parameter value at which a function takes its minimum value in its domain; q represents the point at which the first-order derivative of the cumulative transmittance takes its minimum value; From the neural radiance field, we get the point cloud {P|x p ∈P}, there may be outliers in the recovered point cloud. The point cloud distribution around the implicit surface is different, resulting in different levels of noise in different areas. The filter is used to exclude outliers and produce a relatively accurate and clean point cloud; If the points in the point cloud are located on the surface of the object, it can be approximately expressed as: G(p i )=0,p i ∈P, where G represents the signed distance function and P represents the point cloud set; the SDF network is supervised by supervising the surface points: in, represents the geometric loss function; Geometric feature consistency: Use the Vis-MVSNet pre-trained model to extract features from RGB images, and then use the geometric feature consistency loss to constrain the neural implicit surface reconstruction. The Vis-MVSNet pre-trained model uses a standard U-Net to extract image feature maps; The U-Net network includes a bottom layer, an encoder, a decoder, and a head. The bottom layer consists of multiple blocks, each of which consists of a series of convolutional layers and batch normalization layers. The encoder and the head are both composed of multiple blocks. In addition to the convolutional layer and batch normalization layer, the encoder also includes a pooling layer to gradually reduce the spatial size of the feature map. The decoder consists of a deconvolution layer, a convolutional layer, and possible residual connections to gradually restore the spatial size of the feature map. Assume that the point where the ray through view 0 intersects the surface is x, and the projection of view 0 and its adjacent views are represented as The feature loss is defined as: Among them, N c Represents the number of feature channels, N v represents the number of adjacent source views, F represents the extracted feature map, and p i =K i (R i x+t i ) represents the projection of x in view i, [K i ,R i ,t i ] is the corresponding camera parameter, pixel p i The image feature at is F i (p i ).

10. The method for tree neural rendering and reconstruction based on sparse views as claimed in claim 9, characterized in that: The loss functions of the implicit surface reconstruction network based on volume rendering include color loss, Eikonal loss, color weight loss, geometric loss, and feature loss; Color Loss is the difference between the RGB colors obtained from the real input image and the RGB colors obtained from the volume rendering: Where m is the number of pixels, C i Indicates the RGB color value obtained by the image. RGB color values ​​obtained from volume rendering; The Eikonal loss constrains the gradient of the SDF to 1 so that it satisfies the properties of the distance function, thereby normalizing the SDF and ensuring that it provides the correct distance field representation. The Eikonal loss is added at the sampling point to constrain the signed distance field gradient of the SDF network: in, is the set of sampling points, ∥·∥2 is the L2 norm; For the color deviation of volume rendering, the color deviation is reduced by linear interpolation and color weight loss is introduced For the geometric deviation of volume rendering, the geometric loss is And the feature loss function is Total loss function It is expressed as: Among them, λ3, λ4, λ5, λ6 are hyperparameters used to balance the relative importance between different loss functions.

Images