A neural radiance field novel view synthesis method for blurred scenes

Abstract

The application provides a neural radiance field new view synthesis method for a blurred scene, and is based on a traditional blurring process; the application uses image depth feature information to design a rigid blurring kernel, guides uniform modeling of light in a blurred image, and calculates contribution of light color to a final synthesized blurred pixel value. In addition, the application uses an adversarial generation network discriminator to evaluate a blurred degree of a generated image, and optimizes network parameters in combination with a reconstruction loss, so that a high-quality clear view image is finally generated.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and novel perspective synthesis, specifically a novel perspective synthesis method for neural radiation fields in blurred scenes. Background Technology

[0002] Novel perspective synthesis is an important task in computer vision and computer graphics. Specifically, it requires generating a new image in a given pose from an image corresponding to that pose. Currently, neural radiation field manipulation is the mainstream method in this field. This method can generate realistic images by inputting 3D coordinates and orientation.

[0003] Given a clear image, neural radiation fields can generate images with a sharp viewpoint. However, in the real world, images are often blurred due to various internal and external factors. This severely impacts the results of neural radiation field reconstruction, producing various artifacts. Some existing methods based on neural radiation fields only use image-independent two-dimensional coordinates to calculate the blur kernel, lacking strong feature information guidance.

[0004] The differences between the existing technologies are as follows:

[0005] In a technical comparison with patent CN117745924A, "Neural Rendering Method, System and Device Based on Depth Unbiased Estimation";

[0006] 1. Patent CN117745924A discloses a depth-unbiased estimation rendering network. This network optimizes the initially estimated depth map by successively inputting the original image into the depth estimation network. It utilizes a unique depth estimation network loss and unbiased depth loss to precisely adjust the depth differences between adjacent frames, enabling the generation of unbiased depth maps that maintain consistency across multiple viewpoints. This patent applies depth estimation to reconstruct the color volume density of the depth map under unbiased estimation.

[0007] 2. This patent relates to a depth-feature-based blur kernel. Based on the assumption that the image sampling process uses the same camera and that blurring occurs simultaneously in each image, monocular depth features are used as input to a multilayer perceptron to model a rigid blur kernel. The influence of depth on blurring is utilized to characterize the blurring process of each image. Specifically, for the input of the original light rays, the rigid blur kernel of the corresponding frame is calculated, and the rotation and translation of each sampling point are calculated to obtain the rigidly transformed light rays and the contribution weight of the color rendered for each ray to the final blurred pixel. Finally, the final blurred pixel value is obtained by weighted summation of color and weight. This patent uses depth map feature extraction as input to the network.

[0008] 3. Patent CN117745924A employs a multiple loss function for training. Specific calculations include... It consists of five parts: reconstruction loss, depth estimation network loss, unbiased depth loss, point cloud loss, and photometric loss. To achieve depth consistency among multiple initial depth views, an unbiased depth loss term is specifically introduced. This loss term estimates the depth of sampling points through a specially designed network branch, thereby optimizing the consistency of the depth map.

[0009] 4. This patent involves a composite loss, specifically comprising two parts: reconstruction loss and discrimination loss. The reconstruction loss calculates pixel-level loss using weighted blurred pixel values ​​and the real input blurred image; the discrimination loss pre-trains an adversarial generative network for blurred images, freezes the discriminator weights, and uses it to evaluate the sharpness of the generated images, thus assessing the quality of the generated images from a new perspective.

[0010] In comparison with the technology of patent CN 116524115 A "Multi-view Anti-motion Blur 3D Reconstruction System and Method Based on Neural Radiation Field";

[0011] 1. Patent CN 116524115 A discloses a variable sparse kernel generation unit that generates corresponding blur kernels for images captured from different viewpoints to ensure consistent motion blur of the target across all viewpoints. By introducing viewpoint embedding, the blur kernels are further constrained, thereby enhancing noise resistance during reconstruction. These blur kernels are used to blur real, clear images, and the accuracy of the anti-blur generator is improved through iterative optimization using these clear images as a benchmark.

[0012] 2. This patent relates to a depth-feature-based blur kernel. Based on the assumption that the image sampling process uses the same camera and that blurring occurs simultaneously in each image, monocular depth features are used as input to a multilayer perceptron to model a rigid blur kernel. The influence of depth on blurring is utilized to characterize the blurring process of each image. Specifically, for the input of the original light rays, the rigid blur kernel of the corresponding frame is calculated, and the rotation and translation of each sampling point are calculated to obtain the rigidly transformed light rays and the contribution weight of the color rendered for each ray to the final blurred pixel. Finally, the final blurred pixel value is obtained by weighted summation of color and weight. This patent uses depth map feature extraction as input to the network.

[0013] 3. In patent CN 116524115 A, the set of clear images reconstructed by the anti-blur generator is input into the anti-blur decision unit. Simultaneously, a set of real, multi-view clear images is also used as ground truth input. The generator's performance is evaluated using a loss function. If it is not optimal, the parameters are further trained and optimized. When optimality is achieved, the generator's network model and parameters are saved. This process aims to optimize generator performance and ensure the highest quality output images.

[0014] 4. This patent involves a composite loss, specifically comprising two parts: reconstruction loss and discrimination loss. The reconstruction loss calculates pixel-level loss using weighted blurred pixel values ​​and the real input blurred image; the discrimination loss pre-trains an adversarial generative network for blurred images, freezes the discriminator weights, and uses it to evaluate the sharpness of the generated images, thus assessing the quality of the generated images from a new perspective. Summary of the Invention

[0015] To address the aforementioned technical problems, this invention proposes a novel perspective synthesis method for neural radiation fields in blurred scenes. This method fully considers the relationship between depth and camera motion blur, using depth features to guide the rigid deformation of light rays to simulate the blurring process. Simultaneously, it generates color combination weights based on depth features to synthesize the final blurred image. This invention can generate realistic and clear images from a set of image inputs affected by camera motion blur.

[0016] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0017] A novel perspective synthesis method for neural radiation fields in blurred scenes includes a blur kernel optimization stage and a neural radiation field reconstruction stage, with each stage as follows, characterized in that:

[0018] 1) Fuzzy kernel optimization stage;

[0019] Depth features are introduced as prior guidance to the blur kernel, which learns the distribution of blurred rays and the contribution of the color projected by each blurred ray onto the 2D plane to the final calculated blurred pixels. The same camera is used during shooting, and the blur in each frame is generated simultaneously, unifying the blur modeling method. Frame depth features are used to understand the structural information in the scene, guiding the blur kernel to model the rigid deformation of rays and the weights of color composition.

[0020] 2) The neural radiation field reconstruction stage;

[0021] In the neural radiation field reconstruction stage, the discriminator of the generative adversarial network is used to evaluate the quality of the generated image, and indirectly to evaluate the degree of blurring. The task of this stage is to use the original light rays. The transformed light is then sent into the neural radiation field for reconstruction.

[0022] Furthermore, the specific steps of step 1) fuzzy kernel optimization stage are as follows;

[0023] Step 1: Project light rays from the virtual camera position onto the image to obtain the light information corresponding to each pixel, and transform it from the camera coordinate system to the world coordinate system using the camera transformation matrix;

[0024] Step 2: Extract depth maps using a monocular depth estimation model and extract depth features d using a residual neural network. s .

[0025] Step 3: Fuzzy kernel optimization based on deep features;

[0026] Step 4: Feed the depth features extracted in Step 2 into a multilayer perceptron F2 to obtain m. s；0,...,N This represents the percentage of each ray's pixel value in the final blurred pixel value.

[0027] Furthermore, the specific steps for coordinate transformation in step 1 are as follows:

[0028] Step 1.1: Obtain the ray originating from the camera origin and passing through each pixel. Assume the image width H and height W, the camera intrinsic matrix K, the camera-to-world coordinate system transformation matrix c2w, the focal length f, the 2D image coordinates (i, j), and the optical center coordinates (c...). x ,c y );

[0029] Step 1.2: Calculate the direction of the ray. The direction of the entire ray can be calculated as (ic). x jc y ,f)-(0,0,0)=(ic x,j -c y,f Normalizing the entire vector to the z-coordinate yields the direction of the ray. Then, the c2w matrix is ​​used to transform the ray direction from the camera coordinate system to the world coordinate system;

[0030] Step 1.3: Calculate the origin of the ray and calculate the initial position r in the world coordinate system based on the c2w matrix. o =c2w[:3,-1].

[0031] Furthermore, step 3, the fuzzy kernel optimization based on deep features, includes the following steps:

[0032] Step 3.1: Encode the rotation axis as Input deep features d s The rotation and translation of the light sampling points are predicted using a multilayer perceptron F1.

[0033] Step 3.2: Calculate the rotation matrix Where [x] × The outer product matrix represents the vector x;

[0034] Step 3.3: Calculate the sampling point translation p = Gv, where

[0035] Step 3.4: Calculate the result after rigid deformation for the sampling points on each ray.

[0036]

[0037] Where i∈{1,...,K} represents the K rays obtained after each ray in the s-th frame undergoes rigid deformation, meaning the final blurred result is obtained by mixing these K rays. j∈{1,...,N c} represents the number of sampling points on a ray.

[0038] Furthermore, the specific steps of step 2) neural radiation field reconstruction stage are as follows;

[0039] Step 1: Use a multilayer perceptron F3 to calculate the corresponding voxel density σ and color value for the input sampling points. For the training phase, it is necessary to include... and All the light rays are fed into the model; for the inference stage, only the raw light rays need to be fed into the multilayer perceptron to obtain a result representing a clear scene.

[0040] Step 2: The sampling points in the previous step used a uniform sampling transformation. In order to better calculate the distribution of sampling points along the light rays, an inverse sampling transformation is used to obtain the sampling point positions that conform to the volume density distribution.

[0041] Step 3: The sampling points obtained by the inverse sampling transformation are also fed into the multilayer perceptron F3 to obtain the fine sampling volume density σ. f and color c f .

[0042] Step 4: Combine the results of fine sampling and coarse sampling, and use the volume rendering formula to render the color value of each ray on the 2D image.

[0043] Step 5: Use the Gamma correction function Map scene radiance to image intensity.

[0044] Step 6: Use the color combination weight m obtained in Step 4 s；0,...,N Combined with the calculated color values, a blurred image is generated.

[0045] Step 7: Calculate the blurred image of the original input using reconstruction loss. Blurry images generated by the model The interpolation between the two images is optimized to make the generated image closer to the real image:

[0046]

[0047] Step 8: Train a generative adversarial network based on the fuzzy dataset and optimize it using adversarial loss and content loss functions.

[0048]

[0049] Step 9: Freeze the discriminator parameters of the generative adversarial network. This discriminator, optimized by the loss function in Step 8, can evaluate the quality of a generated blurred image, and therefore can be indirectly used to evaluate the sharpness or blurriness of an image. The discriminator is then used to evaluate the generated image to obtain the discriminant loss:

[0050]

[0051] Step 10: Optimize the network training by combining reconstruction loss and discrimination loss;

[0052]

[0053] Beneficial effects:

[0054] The method of this invention utilizes depth prior to guide the generation of the blur kernel, overcoming the limitation of existing methods that do not fully consider scene structure information, and better simulates the distribution of blurred light rays; it introduces a discriminator of an adversarial generative network to evaluate the quality of the generated image from a new perspective. Attached Figure Description

[0055] Figure 1 This is a model structure diagram of the fuzzy scene new perspective synthesis method according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart of the training process of the fuzzy scene new perspective synthesis method according to an embodiment of the present invention;

[0057] Figure 3 This is a flowchart of the reasoning stage of the fuzzy scene new perspective synthesis method according to an embodiment of the present invention. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0059] The model structure of this invention is as follows Figure 1 The flowchart of the training process is shown below. Figure 2 The flowchart of the reasoning stage is shown below. Figure 3 The method for synthesizing novel perspectives of neural radiation fields in blurred scenes is shown below. It includes a blur kernel optimization stage and a neural radiation field reconstruction stage, with each stage as follows:

[0060] 1) Fuzzy kernel optimization stage:

[0061] The blur kernel optimization stage simulates the physical process of camera motion blur generation. For each ray, a rigid transformation of the ray is used to generate N nearby rays. The blurred pixel values ​​seen in the 2D image are generated by combining these N surrounding rays with the original ray. The goal of the blur kernel optimization stage is to input depth features and the original ray to obtain N rays that can represent the blurring process and their corresponding color combination weights. These weights are then used in the next stage, where the neural radiation field is used to calculate and render each ray.

[0062] Step 1: Project light rays from the virtual camera position onto the image to obtain the light information corresponding to each pixel, and transform it from the camera coordinate system to the world coordinate system using the camera-to-world matrix.

[0063] Step 1.1: Obtain the ray originating from the camera origin and passing through each pixel. Assume the image width H and height W, the camera intrinsic matrix K, the camera-to-world transformation matrix c2w, the focal length f, the 2D image coordinates (i, j), and the optical center coordinates (c...). x ,c y ).

[0064] Step 1.2: Calculate the direction of the ray. The direction of the entire ray can be calculated as (ic). x,j -c y,f )-(0,0,0)=(ic x,j -c y,f Normalizing the entire vector to the z-coordinate yields the direction of the ray. Then, the c2w matrix is ​​used to transform the ray direction from the camera coordinate system to the world coordinate system.

[0065] Step 1.3: Calculate the origin of the ray and calculate the initial position r in the world coordinate system based on the c2w matrix. o =c2w[:3,-1].

[0066] Step 2: Extract depth maps using a monocular depth estimation model and extract depth features d using a residual neural network. s .

[0067] Step 3: Fuzzy kernel optimization based on deep features includes the following steps:

[0068] Step 3.1: Encode the rotation axis as Input deep features d s The rotation and translation of the light sampling points are predicted using a multilayer perceptron F1.

[0069] Step 3.2: Calculate the rotation matrix Where [x] × The outer product matrix represents the vector x.

[0070] Step 3.3: Calculate the sampling point translation p = Gv, where

[0071] Step 3.4: Calculate the result after rigid deformation for the sampling points on each ray.

[0072]

[0073] Where i∈{1,...,K} represents the K rays obtained after each ray in the s-th frame undergoes rigid deformation, meaning the final blurred result is obtained by mixing these K rays. j∈{1,...,N c} represents the number of sampling points on a ray.

[0074] Step 4: Feed the depth features extracted in Step 2 into a multilayer perceptron F2 to obtain m. s；0,...,N This represents the percentage of each ray's pixel value in the final blurred pixel value.

[0075] 2) Neural radiation field reconstruction stage:

[0076] Step 1: Use a multilayer perceptron F3 to calculate the corresponding voxel density σ and color value for the input sampling points. For the training phase, it is necessary to include... and All the light rays are fed into the model; for the inference stage, only the raw light rays need to be fed into the multilayer perceptron to obtain a result representing a clear scene.

[0077] Step 2: The sampling points in the previous step used a uniform sampling transformation. In order to better calculate the distribution of sampling points along the light rays, an inverse sampling transformation is used to obtain the sampling point positions that conform to the volume density distribution.

[0078] Step 3: The sampling points obtained by the inverse sampling transformation are also fed into the multilayer perceptron F3 to obtain the fine sampling volume density σ. f and color c f .

[0079] Step 4: Combine the results of fine sampling and coarse sampling, and use the volume rendering formula to render the color value of each ray on the 2D image.

[0080] Step 5: Use the Gamma correction function Map scene radiance to image intensity.

[0081] Step 6: Use the color combination weights s obtained in Step 4 ；0,...,N Combined with the calculated color values, a blurred image is generated.

[0082] Step 7: Calculate the blurred image of the original input using reconstruction loss. Blurry images generated by the model The interpolation between the two images is optimized to make the generated image closer to the real image:

[0083]

[0084] Step 8: Train a generative adversarial network based on the fuzzy dataset and optimize it using adversarial loss and content loss functions.

[0085]

[0086] Step 9: Freeze the discriminator parameters of the generative adversarial network. This discriminator, optimized by the loss function in Step 8, can evaluate the quality of a generated blurred image, and therefore can be indirectly used to evaluate the sharpness or blurriness of an image. The discriminator is then used to evaluate the generated image to obtain the discriminant loss:

[0087]

[0088] Step 10: Optimize the network by combining reconstruction loss and discrimination loss.

[0089]

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A novel perspective synthesis method for neural radiation fields in blurred scenes, comprising a blur kernel optimization stage and a neural radiation field reconstruction stage, wherein the stages are as follows, characterized in that: 1) Fuzzy kernel optimization stage; Depth features are introduced as prior guidance for the blur kernel to learn the distribution of blurred rays and the contribution of the color of each blurred ray projected onto the two-dimensional plane to the final calculated blurred pixels. The same camera is used during the shooting process and the blur in each frame is generated simultaneously, unifying the blur modeling method. Frame depth features are used to understand the structural information in the scene and guide the blur kernel to model the rigid deformation of rays and the weights of color composition. The specific steps of step 1) fuzzy kernel optimization stage are as follows; Step 1: Project light rays from the virtual camera position onto the image to obtain the light information corresponding to each pixel, and transform it from the camera coordinate system to the world coordinate system using the camera transformation matrix; The specific steps for coordinate transformation in step 1 are as follows: Step 1.1: Obtain the ray originating from the camera origin and passing through each pixel. Assume the image width H and height W, the camera intrinsic matrix K, the camera-to-world coordinate system transformation matrix c2w, the focal length f, the 2D image coordinates (i, j), and the optical center coordinates are... ; Step 1.2: Calculate the direction of the light ray. The direction of the entire light ray can be calculated as follows: Normalizing the entire vector to the z-coordinate yields the direction of the ray. Then, the c2w matrix is ​​used to transform the ray direction from the camera coordinate system to the world coordinate system; Step 1.3: Calculate the origin of the ray and the initial position in the world coordinate system based on the c2w matrix. ; Step 2: Extract depth maps using a monocular depth estimation model and extract depth features using a residual neural network. ; Step 3: Fuzzy kernel optimization based on deep features; Step 3, the fuzzy kernel optimization based on deep features, includes the following steps: Step 3.1: Encode the rotation axis as Input deep features Using a multilayer perceptron Predict the rotation and translation of the light sampling points; Step 3.2: Calculate the rotation matrix ,in The outer product matrix represents the vector x; Step 3.3: Calculate the translation of the sampling points ,in ; Step 3.4: Calculate the result after rigid deformation for the sampling points on each ray. : ； in This represents the K rays obtained after each ray in the s-th frame undergoes rigid deformation; the final blurred result is obtained by mixing these K rays. This represents the number of sampling points on a ray of light; Step 4: Feed the deep features extracted in Step 2 into the multilayer perceptron. get The percentage of each ray's pixel value in the final blurred pixel value; 2) The neural radiation field reconstruction stage; In the neural radiation field reconstruction stage, the discriminator of the generative adversarial network is used to evaluate the quality of the generated image, and indirectly to evaluate the degree of blurring. The task of this stage is to use the original light rays. The transformed light is then sent into the neural radiation field for reconstruction.

2. The method for synthesizing novel perspectives of neural radiation fields in blurred scenes according to claim 1, characterized in that: The specific steps of step 2) neural radiation field reconstruction stage are as follows; Step 1: Using a multilayer perceptron Calculate the corresponding voxel density for the input sampling points. And color values, for the training phase, need to include and All the light rays are fed into the model; for the inference stage, only the raw light rays need to be fed into the multilayer perceptron to obtain a result representing a clear scene; Step 2: The sampling points in the previous step used uniform sampling transformation. In order to better calculate the distribution of sampling points along the light rays, inverse sampling transformation is used to obtain the sampling point positions that conform to the volume density distribution. Step 3: The sampling points obtained by the inverse sampling transform are also fed into the multilayer perceptron. In the process, the fine sample volume density is obtained. and color ; Step 4: Combine the results of fine sampling and coarse sampling, and use the volume rendering formula to render the color value of each ray on the 2D image. ; Step 5: Use the Gamma correction function Map scene radiance to image intensity; Step 6: Use the color combination weights obtained in Step 4 Combined with the calculated color values, a blurred image is generated. ; Step 7: Calculate the blurred image of the original input using reconstruction loss. Blurry images generated by the model The interpolation between the two images is optimized to make the generated image closer to the real image: ； Step 8: Train a generative adversarial network based on the fuzzy dataset and optimize it using adversarial loss and content loss functions; ； Step 9: Freeze the discriminator parameters of the generative adversarial network. This discriminator, optimized by the loss function in step 8, can evaluate the quality of a generated blurred image, and therefore can be indirectly used to evaluate the sharpness or blurriness of an image. Use the discriminator to evaluate the generated image and obtain the discriminant loss: ； Step 10: Optimize and train the network by combining reconstruction loss and discrimination loss; 。

Images