A sparse image view synthesis method based on temporal filtered neural radiance fields

Abstract

The present invention proposes a sparse image view synthesis method based on time-domain filtered neural radiation fields. This method innovatively introduces a time-domain filtered neural radiation field model to solve the problem that the neural radiation field represented by the multi-layer perceptron has high rendering overhead and cannot reduce the aliasing noise of the synthesized image by oversampling. By applying filtering technology in the time domain, the computational burden required for oversampling is effectively shared over consecutive frames, thereby significantly improving the quality of the synthesized image and effectively reducing noise and aliasing without increasing the number of additional network inferences. In addition, in order to improve the geometric extraction capability of the neural radiation field, the method introduces a deep prior network and designs an inter-frame geometric consistency loss to enhance the expression capability of the neural radiation field in multiple views and improve the generalization performance of the network.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and three-dimensional scene understanding, and specifically is a sparse image view synthesis method based on time-domain filtered neural radiance fields. Background Art

[0002] Sparse view synthesis is an important task in the field of 3D vision. It aims to synthesize a new view or image from a given set of images. The main challenge of sparse view synthesis lies in how to reasonably infer and fill the blank areas between views using limited data.

[0003] The existing technologies are distinguished as follows:

[0004] Compared with the patent CN114241113A "An efficient neural radiation field rendering method based on depth-guided sampling"

[0005] 1. Patent CN114241113A designs a depth-guided feature. In front of the target view, several depth planes are sampled, and the extracted image features are used to construct a cost volume with variance as the cost on each depth plane. The cost volume is processed using a 3D convolution-based neural network to obtain a probability volume and the depth of each pixel.

[0006] 2. In this patent, a deep prior feature is designed, and additional constraints are introduced by pre-training the deep model before training, which enhances the generalization of the network when processing sparse input images, thereby significantly improving the quality of synthesized views under new perspectives and reducing the generation of artifacts.

[0007] 3. Patent CN114241113A uses the depth error formula During the training process, the neural network is trained. In the interval where its depth may be [D(u,v)-D(u,v),D(u,v)+D(u,v)], sample N k {x k |k=1,…,N k}, K ranges from 2 to 8, where xk is the coordinate of the target view in the normalized device coordinate space.

[0008] 4. In this patent, a sorting-based depth comparison formula is used Let \(m1\) represent the margin for depth error. In the comparison formula, two depth pixels on the depth map \(d2\) are randomly selected and denoted as \(dk12\) and \(dk22\), where \(dk12 < dk22\). If the depth orderings of \(d1\) and \(d2\) are inconsistent, a penalty is imposed on the neural radiance field. In this way, the relative depth ordering estimated by the neural radiance field is constrained to be consistent with the depth ordering result of the pre-trained model, thereby distilling the depth prior of the pre-trained model into the neural radiance field.

[0009] In the technical comparison with the patent CN116957931A, "A Method for Improving the Image Quality of Camera Images Based on Neural Radiance Fields"

[0010] 1. In the patent CN116957931A, high-definition image prior data is used for pre-training the super-resolution reconstruction network. The high-definition images required for pre-training are not limited to the images captured by good cameras and can also be obtained from other sources. Both the input and output are high-resolution images. Feature extraction and encoding are performed through the Transformer encoder to obtain a sparse feature codebook, and then the codebook is reconstructed into a high-definition image through the Transformer decoder.

[0011] 2. In this patent, a depth prior feature is designed, and additional constraint conditions are introduced through a pre-trained depth model before training, enhancing the generalization ability of the network when processing sparse input images, thereby significantly improving the quality of the synthesized views from new perspectives and reducing the generation of artifacts.

[0012] 3. In the patent CN116957931A, the sparse features obtained based on the pre-training in the previous step and the network weights of the decoder part will be frozen. The encoder structure of the Transformer needs to be trained in this part. The training data required in step S4 comes from step S1 and step S2. The input is a low-definition image, and the output is the corresponding high-definition image, making the sparse codebook obtained from the low-definition image have the smallest error with the sparse codebook obtained from the high-definition image, thus achieving the improvement of the image quality from the low-definition image to the high-definition image.

[0013] 4. In this patent, a depth comparison formula based on sorting is used. Let \(m1\) represent the margin for depth error. In the comparison formula, two depth pixels on the depth map \(d2\) are randomly selected and denoted as \(dk12\) and \(dk22\), where \(dk12 < dk22\). If the depth orderings of \(d1\) and \(d2\) are inconsistent, a penalty is imposed on the neural radiance field. In this way, the relative depth ordering estimated by the neural radiance field is constrained to be consistent with the depth ordering result of the pre-trained model, thereby distilling the depth prior of the pre-trained model into the neural radiance field. Summary of the Invention

[0014] To solve the above technical problems, the present invention proposes a sparse image view synthesis method based on time-domain filtered neural radiance field. This method uses a denoising scheme based on time-domain filtering to reduce the noise of synthesizing new views under sparse input and improve rendering quality. In order to improve the geometric extraction ability of the neural radiance field, this method introduces a deep prior network and increases the inter-frame geometric consistency loss.

[0015] To achieve the above object, the technical solution adopted by the present invention is:

[0016] A sparse image view synthesis method based on time-domain filtered neural radiance fields includes a preprocessing stage, a neural radiance field training stage, and a rendering stage. The stages are characterized by:

[0017] 1) Preprocessing stage:

[0018] A series of processing is performed on the input related RGB image to obtain the appearance information and depth geometry information of the scene, and then the motion recovery structure algorithm is used to obtain the camera pose corresponding to the image;

[0019] 2) Neural radiation field training phase:

[0020] The constraint-based neural radiance field model solves the problem of insufficient network constraints caused by sparse input data. By introducing additional constraints, the generalization of the network is enhanced when processing sparse input images.

[0021] 3) Rendering stage:

[0022] A neural radiance field model based on time-domain filtering is used to solve the problem that the aliasing noise of the synthesized image cannot be reduced by oversampling due to the high rendering overhead of the neural radiance field represented by the multi-layer perceptron. By applying filtering technology in the time domain, the computational burden required for supersampling is effectively distributed to consecutive frames, thereby significantly improving the quality of the synthesized image and effectively reducing noise and aliasing without increasing the number of additional network inference times.

[0023] Furthermore, step 1) the pre-processing stage specifically includes the following steps:

[0024] Step 1.1: Depth prior acquisition: input the RGB image into the existing depth estimation network Depth-anything to obtain the relative depth estimation image D;

[0025] Step 1.2: Correspondence search, identify the same point projections in the scene overlap in the input image I = {Ii|i = 1...NI}, and output a set of geometrically verified image pairs C and the image projection map of each point;

[0026] Step 1.3: Incremental reconstruction, process the input scene graph of step 1.2 into a camera pose estimate P = {Pc∈SE(3)|c=1...NP} for the corresponding image and a set of sparse scene reconstruction point clouds X = {Xk∈R3|k=1...Nx}.

[0027] Furthermore, in step 1.2 of the pre-processing stage:

[0028] A feature extraction method is implemented: for each image Ii, a set of local features Fi = {(xj, fj) | j = 1...NFi} located at xj∈R2 is detected by the SfM algorithm, represented by the appearance descriptor fi. These features should be invariant under radiometric and geometric changes so that SfM can uniquely identify them in multiple images;

[0029] The feature correspondence is determined by comparing the appearance description fj of the features, and the best matching feature is found in image Ia for each feature in image Ib. This is then confirmed by the set C of potentially overlapping image pairs. Since the matching is based only on appearance and cannot guarantee structural consistency, the mapping relationship between feature points is estimated by projective geometry as a verification annotation. Different mappings describe their geometric relationships based on the spatial configuration of image pairs. The homography transformation matrix H describes the position mapping relationship between the object in the world coordinate system and the pixel coordinate system when the camera captures the scene; epipolar geometry uses a calibrated essential matrix E or an uncalibrated fundamental matrix F to describe the camera relationship under different perspectives under movement. If a transformation maps a sufficient number of feature points between different images, these feature points are considered to be geometrically verified.

[0030] Furthermore, in step 1.3 of the pre-processing stage:

[0031] The initialization model is constructed by two specially selected images. The image with the most triangulated points is selected to minimize the uncertainty in the camera reconstruction. When a new image captures the scene from another angle, triangulation is used to confirm the new scene point Xk and integrate it into the scene point set X.

[0032] Furthermore, step 2) the neural radiation field training phase specifically includes the following steps:

[0033] Step 2.1: Use the monocular depth prior as the sorting-based depth prior under the input viewpoint, calculate the sorting loss, and distill the geometric extraction ability of the pre-trained model. For an RGB image I with camera pose P, first use the neural radiance field to render the depth map dnerf from the camera pose P and randomly select a local area d1. At the same time, feed the image I into depth anything to generate the estimated depth map dda and select the same local area d2. Let k1 and k2 be two 2D coordinates on the depth image.

[0034] Step 2.2: If adjacent depth pixels are continuous in the depth map provided by depth anything, then this continuity constraint is applied to the depth pixels corresponding to NeRF to ensure their continuity. KNN(·) is the nearest neighbor function. For the current pixel k1, it is used to return the k nearest depth values ​​of a small area centered on k1 in the depth map of depth anything. The formula is as follows;

[0035]

[0036] Step 2.3: Generate unknown viewpoint camera pose. Considering that the known viewpoint camera pose space restricts the unknown viewpoint camera space, the unknown viewpoint camera pose space is inferred from the known viewpoint camera pose space, and sampled in this space to generate the unknown viewpoint camera pose. The formula is as follows;

[0037] P unknown ={[R|t]|R~R unknown |t,t~t unknown}

[0038] Step 2.4: Importance sampling, using multi-dimensional Gaussian distribution sampling, that is, the camera frustum of each input image only needs to be calculated once before training, and the central area with the most overlaps is calculated. This area is considered to be the key area for reconstructing the object. The sampling formula is as follows;

[0039]

[0040] Step 2.5: Local geometric smoothness constraint of unknown viewpoint. Calculate the depth map dv at the unknown viewpoint v∈Punknown, and then calculate the local geometric smoothness constraint loss, where Rv represents the set of projections from the camera at the viewpoint to the scene, and Patch is the local size. The formula is as follows.

[0041]

[0042] Step 2.6: Local color regularization for unknown viewpoints. For an unknown viewpoint v′∈Punknown, calculate the likelihood estimate of the image patch generated by the neural radiance field at that viewpoint and maximize the likelihood estimate during the training phase. The formula is as follows:

[0043]

[0044] Furthermore, step 3) the rendering stage specifically includes the following steps:

[0045] Step 3.1: Generate a low-discrepancy sequence. In the neural radiance field, for each frame, each pixel can only have one ray for sampling and shading. In order to generate different ray samples for all pixels in each frame, it is necessary to add a viewport sub-pixel offset to the camera projection matrix. The offset of each frame is usually generated from a uniformly distributed sampling sequence, so that each pixel can be evenly covered by samples generated in multiple frames. In rendering, in order to make the temporal filtering accumulation result converge as quickly as possible, the sequence must have the following property: subsequences of any length in the sequence must be evenly distributed on the pixels;

[0046] Step 3.2: Pixel coordinate reprojection. In neural radiance field rendering, since the camera position changes between frames, the 3D point corresponding to the current pixel will have different pixel coordinates in the previous frame. Therefore, it is necessary to calculate the corresponding position of each pixel in the previous frame to obtain historical data.

[0047] Step 3.3: Color resampling, resampling by bilinear interpolation;

[0048] Step 3.4: Color mixing, store the weighted average of the accumulated samples of each pixel as a single color, which is both the output of the current frame and the historical input result of the next frame. The formula is as follows.

[0049]

[0050] Furthermore, in step 3.2 of the rendering phase:

[0051] The three-dimensional coordinates of each pixel in the clip space are reconstructed from the depth buffer and calculated by the camera projection matrices Mn and Mn-1 of the current frame and the previous frame. The formula is as follows.

[0052]

[0053] For neural radiation field rendering, consider the sampling ray ri(p) of pixel point p in the i-th frame. After the MLP network calculates a series of sampling points, their three-dimensional space coordinates are:

[0054]

[0055] And the corresponding weights are:

[0056]

[0057] The neural radiation field depth z of the pixel value is calculated and the three-dimensional coordinates of the clipping space are reconstructed using the depth:

[0058]

[0059] Then calculate and obtain the coordinates of the corresponding pixel points in the previous frame.

[0060] Compared with the prior art, the present invention has the following significant advantages:

[0061] The present invention discloses a sparse image view synthesis method based on time-domain filtered neural radiation field. This method innovatively introduces a time-domain filtered neural radiation field model, aiming to solve the problem that the neural radiation field represented by the multi-layer perceptron has a high rendering overhead and cannot reduce the aliasing noise of the synthesized image by oversampling. By applying filtering technology in the time domain, the computational burden required for oversampling is effectively shared over continuous frames, thereby significantly improving the quality of the synthesized image and effectively reducing noise and aliasing without increasing the number of additional network inferences. In addition, in order to improve the geometric extraction capability of the neural radiation field, the method introduces a deep prior network and designs an inter-frame geometric consistency loss to enhance the expression capability of the neural radiation field in multiple views and improve the generalization performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a diagram of the neural radiation field network structure according to an embodiment of the present invention;

[0063] Figure 2 This is a training flow chart of an embodiment of the present invention;

[0064] Figure 3 This is a flow chart of reasoning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0066] Example: This embodiment provides a sparse image view synthesis method based on time-domain filtered neural radiation field. This method uses a denoising scheme based on time-domain filtering to reduce the noise of synthesizing new views under sparse input and improve rendering quality. In order to improve the geometric extraction capability of the neural radiation field, this method introduces a deep prior network and increases the inter-frame geometric consistency loss.

[0067] The method of this embodiment mainly includes a preprocessing stage, a neural radiation field training stage, and a rendering stage. The stages are as follows:

[0068] 1) Preprocessing stage:

[0069] A series of processing is performed on the input related RGB images to obtain the appearance information and depth geometry information of the scene. Then the motion recovery structure algorithm is used to obtain the camera pose corresponding to the image. The overall structure is as follows Figure 1 As shown in the training flow chart, Figure 2 As shown, the inference flow chart is as follows Figure 3 shown.

[0070] Step 1) The pre-processing stage specifically includes the following steps:

[0071] Step 1.1: Depth prior acquisition, input the RGB image into the existing depth estimation network Depth-anything to obtain the relative depth estimation image D.

[0072] Step 1.2: Correspondence search, identify the same point projections in the scene overlap in the input image I = {Ii|i = 1...NI}, and output a set of geometrically verified image pairs C and the image projection map of each point

[0073] In step 1.2 of the pre-processing stage:

[0074] A feature extraction method is implemented: for each image Ii, a set of local features Fi = {(xj, fj) | j = 1...NFi} located at xj∈R2 is detected by the SfM algorithm, represented by the appearance descriptor fi. These features should be invariant under radiometric and geometric changes so that SfM can uniquely identify them in multiple images.

[0075] The feature correspondence is determined by comparing the appearance description fj of the features, and the best matching feature is found in image Ia for each feature in image Ib. This is then confirmed by a set C of potentially overlapping image pairs. Since the matching is based only on appearance and cannot guarantee structural consistency, the mapping relationship between feature points is estimated through projective geometry as a verification annotation. Different mappings describe the geometric relationship between image pairs based on their spatial configuration. The homography transformation matrix H describes the position mapping relationship between the object in the world coordinate system and the pixel coordinate system when the camera captures the scene; epipolar geometry uses a calibrated essential matrix E or an uncalibrated fundamental matrix F to describe the camera relationship under different perspectives under movement. If a transformation maps a sufficient number of feature points between different images, these feature points are considered to be geometrically verified.

[0076] Step 1.3: Incremental reconstruction, process the input scene graph of step 1.2 into a camera pose estimate P = {Pc∈SE(3)|c=1...NP} for the corresponding image and a set of sparse scene reconstruction point clouds X = {Xk∈R3|k=1...Nx}.

[0077] In the pre-processing step 1.3:

[0078] This method constructs an initialization model using two carefully selected images. A suitable pair of initialization images is crucial because incorrect initialization can cause reconstruction to become stuck in a local optimum, preventing it from reaching the global optimum and resulting in suboptimal reconstruction results. Initializing from a relatively densely populated viewpoint in the scene graph generally results in more robust and accurate reconstructions due to greater overlap of camera frustums. In contrast, initializing from a sparser viewpoint results in shorter runtimes. The selection of the next best view is crucial; an incorrect choice can lead to a cascade of camera misalignment and triangulation errors. Furthermore, the selection of the next best view significantly impacts the quality of the pose estimate and the completeness and accuracy of the triangulation. Accurate pose estimates are crucial for a robust method, as triangulation may fail if the pose is inaccurate. The method's strategy is to select the image that sees the most triangulated points to minimize uncertainty in the camera reconstruction. A newly added image should observe known scene points and, in addition, partially extend the set of points X by triangulating the scene. When a new image captures the scene from another perspective, triangulation can be used to confirm the new scene point Xk and integrate it into the set of scene points X. In this method, triangulation is crucial for improving model stability and fault tolerance, as it enhances model robustness through redundancy and facilitates the integration of new images, thanks to the ability to establish more 2D-to-3D correspondences. To reduce cumulative error, this method performs bundle adjustment (BA) after image augmentation, registration, and triangulation.

[0079] 2) Neural radiation field training phase:

[0080] A constraint-based neural radiance field model aims to address the problem of insufficient network constraints caused by sparse input data. In traditional neural radiance field models, sparse input images often cause the network to overfit to the input viewpoint, thereby compromising the generalization ability of the neural radiance field. This overfitting phenomenon is particularly evident when generating synthetic images from new perspectives, manifested as insufficient clarity of the generated images and the appearance of artifacts. By introducing additional constraints, this model enhances the generalization of the network when processing sparse input images, thereby significantly improving the quality of synthetic views from new perspectives and reducing the generation of artifacts.

[0081] Step 2) The neural radiation field training phase specifically includes the following steps:

[0082] Step 2.1: Use the monocular depth prior as a sorting-based depth prior for the input viewpoint, calculate the sorting loss, and distill the geometric extraction capabilities of the pre-trained model. Specifically, for an RGB image I at camera pose P, first use the neural radiance field to render a depth map dnerf from camera pose P and randomly select a local region d1. Simultaneously, feed image I into depth anything to generate an estimated depth map dda and select the same local region d2. Let k1 and k2 be two 2D coordinates on the depth image.

[0083] Step 2.2: If adjacent depth pixels are continuous in the depth map provided by depth_anything, then this continuity constraint is applied to the corresponding depth pixels in NeRF to ensure continuity. KNN(·) is a nearest neighbor function. For the current pixel k1, it is used to return the k nearest depth values ​​of a small area (e.g., a 7×7 area) centered on k1 in the depth map of depth_anything. The formula is as follows.

[0084]

[0085] Step 2.3: Generate unknown viewpoint camera pose. Considering that the known viewpoint camera pose space constrains the unknown viewpoint camera pose space, the unknown viewpoint camera pose space is inferred from the known viewpoint camera pose space, and sampled in this space to generate the unknown viewpoint camera pose. The formula is as follows.

[0086] P unknown ={[R|t]|R~R unknown |t,t~t unknown}

[0087] Step 2.4: Importance sampling, using multi-dimensional Gaussian distribution sampling. Specifically, the camera frustum of each input image only needs to be calculated once before training, and the central area with the largest number of overlaps is calculated (this area can be considered as the key area of ​​the object to be reconstructed). The sampling formula is as follows.

[0088]

[0089] Step 2.5: Local geometric smoothness constraint of unknown viewpoint. Calculate the depth map dv at the unknown viewpoint v∈Punknown, and then calculate the local geometric smoothness constraint loss, where Rv represents the set of projections from the camera at the viewpoint to the scene, and Patch is the local size. The formula is as follows.

[0090]

[0091] Step 2.6: Local color regularization for unknown viewpoints: For an unknown viewpoint v′∈Punknown, calculate the likelihood estimate of the image patch generated by the neural radiance field at that viewpoint and maximize the likelihood estimate during the training phase. The formula is as follows.

[0092]

[0093] 3) Rendering stage:

[0094] A neural radiance field model based on time-domain filtering is used to address the problems of high rendering overhead of neural radiance fields represented by multi-layer perceptrons and the inability to reduce aliasing noise in synthesized images through supersampling. By applying filtering technology in the time domain, the computational burden required for supersampling is effectively distributed to consecutive frames, thereby significantly improving the quality of synthesized images and effectively reducing noise and aliasing without increasing the number of additional network inference times.

[0095] Step 3) The rendering phase specifically includes the following steps:

[0096] Step 3.1: Generate a low-discrepancy sequence. In the neural radiance field, for each frame, each pixel can only have one ray for sampling and shading. In order to generate different ray samples for all pixels in each frame, a viewport sub-pixel offset needs to be added to the camera projection matrix. The offset of each frame is usually generated from a uniformly distributed sampling sequence, so that each pixel can be evenly covered by samples generated in multiple frames. In rendering, as the camera perspective changes, a certain position in the scene may be rendered in the image or occluded at any time (manifested by the weight of the point on the ray being almost 0). Therefore, the first sample that the pixel begins to accumulate must be able to start from any index in the sampling sequence, and in order for the temporal filtering accumulation result to converge as quickly as possible, the sequence must have the following properties: subsequences of any length in the sequence must be evenly distributed on the pixels.

[0097] Step 3.2: Pixel coordinate reprojection. In neural radiation field rendering, since the camera position changes between frames, the 3D point corresponding to the current pixel will have different pixel coordinates in the previous frame. Therefore, it is necessary to calculate the corresponding position of each pixel in the previous frame to obtain historical data.

[0098] In step 3.2 of the rendering phase:

[0099] The three-dimensional coordinates of each pixel in the clip space are reconstructed from the depth buffer and calculated by the camera projection matrices Mn and Mn-1 of the current frame and the previous frame. The formula is as follows.

[0100]

[0101] For neural radiation field rendering, consider the sampling ray ri(p) of pixel point p in the i-th frame. After the MLP network calculates a series of sampling points, their three-dimensional space coordinates are:

[0102]

[0103] And the corresponding weights are:

[0104]

[0105] The neural radiation field depth z of the pixel value is calculated and the three-dimensional coordinates of the clipping space are reconstructed using the depth:

[0106]

[0107] Then calculate and obtain the coordinates of the corresponding pixel points in the previous frame.

[0108] Step 3.3: Color resampling. Most of the time, the reprojected coordinates will not exactly reach the pixel center of the previous frame. In this case, resampling is necessary. Common sampling methods include nearest neighbor interpolation and bilinear interpolation. This method uses bilinear interpolation for resampling.

[0109] Step 3.4: Color Mixing. It is impractical to store the temporal cumulative samples of each pixel in the previous frame. This paper stores the weighted average of the cumulative samples of each pixel as a single color, which is both the output of the current frame and the historical input result of the next frame. The formula is as follows.

[0110]

[0111] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A sparse image view synthesis method based on time-domain filtered neural radiance field, comprising a preprocessing stage, a neural radiance field training stage, and a rendering stage, characterized in that: The stages are as follows: 1) Preprocessing stage: A series of processing is performed on the input related RGB image to obtain the appearance information and depth geometry information of the scene, and then the motion recovery structure algorithm is used to obtain the camera pose corresponding to the image; 2) Neural radiation field training phase: Through constraint-based neural radiation field models; Step 2) The neural radiation field training phase specifically includes the following steps: Step 2.1: Using the monocular depth prior as the depth prior based on sorting under the input viewpoint, calculate the sorting loss and distill the geometric extraction ability of the pre-trained model. For an RGB image I with camera pose P, first render the depth map from the camera pose P through the neural radiance field. And randomly select a local area , and the image Feed depth anything to generate an estimated depth map And select the same local area ,set up are two 2D coordinates on the depth image; Step 2.2: If adjacent depth pixels are continuous in the depth map provided by depth anything, then this continuity constraint is applied to the depth pixels corresponding to NeRF to ensure their continuity. KNN(·) is a nearest neighbor function. For the current pixel k1, it is used to return the k nearest depth values ​​of a small area centered on k1 in the depth map of depth anything. The formula is as follows; ； Step 2.3: Generate unknown viewpoint camera pose. Considering that the known viewpoint camera pose space restricts the unknown viewpoint camera space, the unknown viewpoint camera pose space is inferred from the known viewpoint camera pose space, and sampled in this space to generate the unknown viewpoint camera pose. The formula is as follows; ； Step 2.4: Importance sampling, using multi-dimensional Gaussian distribution sampling, that is, the camera frustum of each input image only needs to be calculated once before training, and the central area with the most overlaps is calculated. This area is considered to be the key area for reconstructing the object. The sampling formula is as follows; ； Step 2.5: Local geometric smoothness constraints at unknown viewpoints Calculate the depth map under this viewpoint , and then calculate the local geometric smoothness constraint loss, where It represents the set of projections from the camera at that viewpoint to the scene, and Patch is the local size. The formula is as follows: ； Step 2.6: Local color regularization for unknown viewpoints , calculate the likelihood estimate of the image patch generated by the neural radiation field under this viewpoint, and maximize the likelihood estimate during the training phase. The formula is as follows; ； 3) Rendering stage: Use a neural radiation field model based on time-domain filtering; Step 3) The rendering phase includes the following steps: Step 3.1: Generate a low-discrepancy sequence. In the neural radiance field, for each frame, each pixel can only have one ray sampled and shaded. In order to generate different ray samples for all pixels in each frame, it is necessary to add a viewport sub-pixel offset to the camera projection matrix. The offset for each frame is generated from a uniformly distributed sampling sequence, so that each pixel is evenly covered by samples generated in multiple frames. In rendering, in order to make the temporal filtering accumulation result converge as quickly as possible, the sequence must have the following property: subsequences of any length in the sequence must be evenly distributed on the pixels; Step 3.2: Pixel coordinate reprojection. In neural radiance field rendering, since the camera position changes between frames, the 3D point corresponding to the current pixel will have different pixel coordinates in the previous frame. Therefore, it is necessary to calculate the corresponding position of each pixel in the previous frame to obtain historical data. Step 3.3: Color resampling, resampling by bilinear interpolation; Step 3.4: Color mixing, the weighted average of the accumulated samples of each pixel is stored as a single color, which is both the output of the current frame and the historical input result of the next frame. The formula is as follows; 。 2. The method for sparse image view synthesis based on temporal filtered neural radiance fields according to claim 1, characterized in that: Step 1) The preprocessing stage specifically includes the following steps: Step 1.1: Depth prior acquisition: input the RGB image into the existing depth estimation network Depth-anything to obtain the relative depth estimation image D; Step 1.2: Corresponding search, in the input image Identify the same point projections in scene overlaps and output a set of geometrically verified image pairs and the image projection of each point; Step 1.3: Incremental reconstruction, process the input scene graph of step 1.2 into the camera pose estimate of the corresponding image and a set of sparse scene reconstruction point clouds .

3. The method for sparse image view synthesis based on temporal filtered neural radiance fields according to claim 2, characterized in that: In step 1.2 of the pre-processing stage: A feature extraction method was implemented: for each image , detect a group of The local feature set , through the appearance descriptor It indicates that these features should be invariant under radiometric and geometric changes so that SfM can uniquely identify them across multiple images; By comparing the appearance description of the features To determine the feature correspondence, for the image Each feature in the image Find the best matching features in the set of potentially overlapping image pairs Confirmation is performed. Since the matching is based only on appearance and cannot guarantee structural consistency, the mapping relationship between feature points is estimated through projective geometry as a verification annotation. Different mappings describe their geometric relationships according to the spatial configuration of the image pairs. The homography transformation matrix H describes the position mapping relationship between the object in the world coordinate system and the pixel coordinate system when the camera captures the scene; the epipolar geometry uses the calibrated essential matrix E or the uncalibrated basic matrix F to describe the camera relationship under different perspectives under movement. If a transformation maps a sufficient number of feature points between different images, these feature points are considered to be geometrically verified.

4. The method for sparse image view synthesis based on temporal filtered neural radiance fields according to claim 2, characterized in that: In the pre-processing step 1.3: The initialization model is constructed by two specially selected images. The image with the most triangulated points is selected to minimize the uncertainty in the camera reconstruction. When a new image captures the scene from another angle, triangulation is used to confirm the new scene points. , and integrate it into the scene point set .

5. The method for sparse image view synthesis based on temporal filtered neural radiance fields according to claim 1, characterized in that: In step 3.2 of the rendering phase: By reconstructing the 3D coordinates of each pixel in the clip space from the depth buffer and using the camera projection matrix of the current frame and the previous frame and To calculate, the formula is as follows; ； For neural radiation field rendering, consider the sampling ray ri(p) of pixel point p in the i-th frame. After the MLP network calculates a series of sampling points, their three-dimensional space coordinates are: ； And the corresponding weights are: ； The neural radiation field depth z of the pixel value is calculated and the three-dimensional coordinates of the clipping space are reconstructed using the depth: ； Then calculate and obtain the coordinates of the corresponding pixel points in the previous frame.

Images