NeRF three-dimensional reconstruction method based on deformable pose fuzzy kernel

Abstract

The invention discloses a NeRF three-dimensional reconstruction method based on a deformable pose fuzzy kernel. The method comprises the following steps: firstly, carrying out information extraction on blurred images shot at different visual angles in the same scene to obtain a pixel matrix and estimated pose information; a blurred kernel and visual angle coding representation is created for each blurred image, and the blurred kernel is deformed by using a multi-layer perceptron. And carrying out parametric estimation on the camera pose at the beginning and ending of exposure, and carrying out linear interpolation on the camera pose according to the number of the epipolar points. Then inputting the blurring kernel and the estimated camera pose sequence into a ray coding layer, modeling other rays in a plurality of adjacent frames by estimating the pose in the camera motion process, adjusting the weight corresponding to the rays according to the target blurring kernel, and generating target pixels to effectively extract pixel information of the adjacent frames; therefore, the reconstructed blurred pixel is closer to a real image. In the scene reconstruction stage, the rays of the new view angle are directly rendered by removing the blurring kernel, and the output of a clear scene is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, relates to a three-dimensional reconstruction method, and specifically relates to a NeRF three-dimensional reconstruction method based on a deformable pose blur kernel. Background Art

[0002] 3D reconstruction is the process of creating or restoring 3D objects and scenes from information obtained from 2D images or other data sources (lidar data, depth sensors, etc.). With the continuous improvement of computing power, 3D reconstruction has become a key technology for many real-world applications, such as virtual and augmented reality, medical image processing, etc.

[0003] Traditional 3D reconstruction methods usually use 3D point clouds, triangle meshes or volume networks to explicitly represent a 3D object or scene. Unlike traditional 3D reconstruction methods, Neural radiance field (NeRF) is a novel and effective implicit scene representation. NeRF uses a multi-layer perceptron (MLP) to represent the color and density of each point in the scene in the form of a continuous volume function. NeRF-based 3D reconstruction is a technology that trains a multi-layer perceptron (MLP) with 3D scene information through 2D images and corresponding camera poses, and then renders the trained implicit scene into an explicit static 3D scene through volume rendering.

[0004] In the process of image acquisition in real-world applications (such as medical imaging and drone photography), it is inevitable that information loss will occur to varying degrees. The most common situation is motion blur, that is, the image produces blur artifacts due to the movement of the camera or object during the exposure process. Using images with motion blur to train NeRF will greatly affect the effect of the final three-dimensional scene reconstruction. The existing technology generally uses blur kernels to simulate motion blur during the training process, and removes the blur kernels in the inference stage, which to a certain extent solves the impact of images with motion blur on NeRF reconstruction. This method requires the real camera pose of the preset image, but in real scenes, the camera pose is generally estimated by COLMAP software. Images with motion blur will cause the camera pose obtained by the software to have a larger error than the clear image, resulting in the poor performance of the existing technology in processing motion blurred images when the camera pose is unknown. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a NeRF 3D reconstruction method based on a deformable pose blur kernel. For the cases where blurred images and camera pose are unknown, a camera pose sequence is generated by interpolating the estimated camera pose, and a deformable blur kernel is used to simulate the generation of motion blur. The loss of generated motion blur and real blur is minimized to optimize the deformable blur kernel and camera pose. The blur kernel is removed in the scene reconstruction stage to solve the problem of blurred image input and unknown camera pose.

[0006] A NeRF 3D reconstruction method based on a deformable pose blur kernel specifically comprises the following steps: Step 1: Fuzzy image data extraction and processing Blurred image data extraction and processing is to extract image information and estimated pose information from multiple input blurred images, so as to facilitate subsequent simulation of motion blur for processing. Specifically, the following steps are included: Step 1.1: Input dataset extraction.

[0007] Shots from different angles of the same scene Extract pixel information from a blurred image and represent it with a vector , Represents the pixel matrix of the input a-th image. Read the estimated pose information corresponding to the image , Represents the camera pose matrix corresponding to the ath image, including a rotation matrix of size 3×3 , and a 3D column vector , where the rotation matrix Describes the rotation information of the camera in the world coordinate system, vector Represents the translation information of the camera in the world coordinate system.

[0008] Step 1.2: Posture information processing.

[0009] The camera pose matrix for each image Add random perturbations to parameterize the estimated camera pose at the start of exposure and the pose at the end of exposure ,in is a Lie algebra, which is a 6-dimensional vector. The obtained camera pose estimation information is spliced ​​to obtain the set and ,in, and They respectively represent the estimated information sets of the starting and ending positions of the camera pose in a camera exposure.

[0010] Step 1.3: Initialize the fuzzy kernel information.

[0011] Create m initial blur kernels for m blurred images And the corresponding view encoding representation .in , is the number of kernel points of a blur kernel, Represents the normalized coordinates of the kernel point in the blur kernel, which are randomly initialized using a normal distribution. The vector values ​​are initialized to 0.

[0012] Step 2: Blur Kernel Deformation After obtaining the blurred image information and camera pose information, Pictures The generation of blurred pixels at coordinates, encoded according to the viewing angle and pixel coordinates Initial blur kernel To transform, the specific steps are as follows: Step 2.1: Select the corresponding initial blur kernel according to the blurred image , first The core point By denormalization, we get , and then the new core point Perform a random perturbation of the normal distribution and obtain , forming a new blur kernel . For blur kernel In The pixel coordinates of the kernel points are sampled and encoded to generate a two-dimensional vector .

[0013] Step 2.2: Select the corresponding viewing angle encoding based on the blurred image , encode the perspective According to the number of blur kernel points n, the copy and expansion operation is performed to obtain a two-dimensional vector , then and Splicing to get a two-dimensional vector representation . Calculate the coordinates of the blurred pixels to be generated One-dimensional vector representation of , and then perform a copy and expansion operation to obtain a two-dimensional vector ,and The concatenation process results in a two-dimensional vector .

[0014] Step 2.3: The vector obtained in step 2.2 Enter the first MLP layer Then the output With vector Concatenate to get vector . Then Input to the second MLP layer middle:

[0015] in is the offset of the core point, It is the weight of kernel points when synthesizing blurred pixels.

[0016] Step 2.4: Use the kernel point offset obtained in step 2.3 Blur Kernel Deformation .

[0017] Step 3: Blurred pixel rendering According to the blur kernel obtained in step 2 , simulate the generation of motion blurred pixels and complete the model training at the same time. The specific steps are as follows: Step 3.1 Before obtaining the ray code, the exposure camera pose needs to be estimated, and the deformation blur kernel is obtained according to step 2 The number of core points n, for the camera pose obtained in step 1 , Perform linear interpolation: Indicates the exposure time. represents the estimated camera pose obtained by interpolation at time t during the exposure process. Splice and get the blur kernel The corresponding estimated camera pose sequence .

[0018] Step 3.2: According to the camera intrinsic parameter matrix And the estimated camera pose sequence obtained in step 3.1 , the blur kernel output from step 2 The corresponding core point information in is converted into ray representation through ray encoding: in The vectors representing the starting point and direction of the ray are respectively. The ray group is obtained by processing the n core points and the estimated camera pose separately. .

[0019] Step 3.3: Input the ray group Rays obtained in step 3.2 into NeRF and render to obtain the pixel RGB set representation. : in , Represents the RGB value obtained by rendering each ray. are the parameters of the NeRF model.

[0020] Step 3.4: Weights obtained from step 2.3 Output of NeRF model Perform weighted summation to obtain the simulated blurred pixel RGB representation : Step 3.5 Structural loss is added to the loss function , construct the loss function : in, represents the RGB value of the real fuzzy pixel, B is the fuzzy pixel set, It is a structural loss The weight parameter, is a hyperparameter. Indicates randomly selecting a pixel set from the predicted pixel set And the corresponding pixel set in the training set The SSIM quality criterion is calculated to take into account structural information when calculating pixels one by one, thereby improving the reconstruction effect.

[0021] Let the loss function Minimize and optimize model parameters , , , , .in represent Parameters, .

[0022] Step 4: Scene reconstruction After completing steps 2 and 3, select A continuous generated perspective, the corresponding camera pose matrix Input NeRF in sequence to obtain a set of pixel groups , pixel group , H and W are the height and width of the frame image, respectively, and c represents the pixel color. Use image processing tools to operate on the pixel groups in S to obtain a continuous generated view representation , Represents a reconstructed 2D view.

[0023] The continuous new views IM are used as frame data of the clear scene output, and the reconstructed clear scene is output.

[0024] The present invention has the following beneficial effects: 1. To address the problem that there is a large deviation between the estimated camera pose of motion blurred images and the actual pose, the camera pose sequence at different times during the exposure process is estimated through interpolation method, the blur kernel is combined with the estimated camera pose sequence, and the blur kernel and the estimated camera pose sequence are continuously optimized during the training stage.

[0025] 2. The initial blur kernel is deformed by the blur kernel to obtain the blur kernel corresponding to the target pixel, and then each kernel point of the blur kernel is bound to an estimated camera pose, and then multiple rays of adjacent frames are calculated based on the bound estimated camera pose. Finally, the weights corresponding to the rays are adjusted by the target blur kernel, so that the blurred pixels can effectively extract the pixel information of adjacent frames, so that the reconstructed blurred pixels are closer to the real blurred pixels.

[0026] 3. Experiments were conducted on 5 static scenes. Compared with the existing methods for fuzzy input NeRF, the LPIPS of this method was improved by 2% to 6.5% in the 5 scenes; PSNR and SSIM were improved by 3.42 and 3.09% on average in the 5 scene experiments, with significant results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of the NeRF 3D reconstruction method based on deformable pose blur kernel; Figure 2 is a diagram of an example of a training set used in the embodiment; Figure 3 is a diagram of an example of a training set used in the embodiment; Figure 4 Schematic diagram of the structure of the multi-layer perceptron MLP used in the embodiment; Figure 5 This is a comparison diagram of the effects of static scene reconstruction in the embodiment. DETAILED DESCRIPTION

[0028] The present invention is further explained below with reference to the accompanying drawings: like Figure 1 As shown in FIG. 1 , a NeRF 3D reconstruction method based on a deformation pose blur kernel includes a training phase and a classification phase. The specific steps can be divided into four parts: blur image data extraction and processing, blur kernel deformation, blur pixel rendering, and scene reconstruction. The blur image data used in this embodiment comes from the motion blur synthetic dataset in the DeblurNeRF public dataset, and the camera pose is estimated and generated by the COLMAP software. Figure 1A NeRF 3D reconstruction method based on a deformation pose blur kernel is shown for reconstruction, including blur image data extraction and processing, blur kernel deformation, blur pixel rendering and scene reconstruction. The specific steps are as follows:

[0029] Step 1: Target data extraction and processing Target data extraction includes extracting input image data and camera pose data. For the input image data, images with motion blur are used as training samples, and clear images are used as corresponding labels. The specific steps are as follows:

[0030] Step 1.1, use the scene COZYROOM generated by render as the static scene to be reconstructed. 29 images with motion blur at different viewing angles are rendered, such as Figure 2 As shown; 5 clear images, such as Figure 3 As shown in Figure 2. The size of all images is 400 × 600. The 34 images obtained are input into COLMAP to estimate the camera pose information corresponding to each image.

[0031] Step 1.2: Normalize the pixel RGB data of the image data obtained in step 1.1 to obtain the image vector, where the image vectors of the training set and the test set are , .

[0032] Step 1.3: Initialize the starting camera pose and the ending camera pose during the camera exposure process according to the camera pose information obtained in step 1.1. The pseudo code is as follows:

[0033] in, Represents the camera pose matrix corresponding to the a-th image. and They respectively represent the estimated information sets of the starting and ending positions of the camera pose in a camera exposure. Indicates that a sum is initialized randomly Matrices of the same shape, Indicates the jitter of the calculated matrix.

[0034] Step 1.4: Set the number of core points , create an initial blur kernel that corresponds to the blurred image one by one according to the normal distribution Then initialize the corresponding perspective encoding set representation according to the number of images 29 , each initial perspective encoding is a vector of length 32.

[0035] Step 2: Blur Kernel Deformation The blur kernel is deformed according to the coordinates of the target pixel, using the second image Taking the pixel point as an example, the specific steps are as follows: Step 2.1: Select the corresponding blur kernel according to the target pixel k 2 =[ −0.0409,−0.0222 ,…,[0.0550,0.0690]] .right Perform anti-normalization processing, add normal distribution random perturbations to the anti-normalized results, and obtain a new blur kernel = [ −0.3790,−0.2102 ,…,[0.6493,0.5643]] .calculate The encoding vector .

[0036] Step 2.2: Select the corresponding viewing angle code according to the target pixel e 2 =[0,…,0] By the number of kernel points of the blur kernel Perform a copy and extend operation to obtain . The obtained in step 2.1 and Perform the concatenation operation and get the new vector shape: . The coordinates of the blurred pixels to be generated , calculate its vector representation , after the dimension of sp is expanded, Splice to get vector .

[0037] Step 2.3, if Figure 4 As shown, the vector Input to the first MLP layer: Output vector y∈ , through the residual operation and Concatenate to get a new input vector . Then Input to the second MLP layer:

[0038] Output vector , the offset of the core point is obtained by cutting operation and a one-dimensional weight of length 5 . Finally use the offset Blur Kernel Deformation k 2 = [ −0.6463,−1.7088 1 ,…, [2.1307,0.6183] 5 ] .

[0039] Step 3: Blurred pixel rendering According to the blur kernel obtained in step 2 , and the camera pose information in step 1, render motion blurred pixels. Taking the pixel point as an example, the specific steps are as follows:

[0040] Step 3.1: Based on blur kernel The number of kernel points, the pose before and after exposure estimated in step 1 Perform linear interpolation, set the exposure time to 1 second, and the interpolation interval to 0.2 seconds to obtain the estimated camera pose during one exposure. .

[0041] Step 3.2: Blur kernel k 2 = [ −0.6463,−1.7088 1 ,…, [2.1307,0.6183] 5 ] With the target pixel coordinates Add and estimate the camera pose Convert to ray representation in 3D space: in is the camera intrinsic parameter matrix, is the coordinate information of the kernel point in the blur kernel, ∈ , respectively represent the starting position and direction vector of the ray.

[0042] Step 3.3: Group the rays Input into NeRF and render to get the corresponding RGB pixel set Step 3.4: Through the weight vector For a set of RGB pixels Perform weighted summation to obtain the target pixel of the second image RGB information of the pixel Finally, get the target pixel of the second image in the training set True RGB information , minimize the corresponding loss function , and continuously optimize the model parameters.

[0043] Step 4: Scene reconstruction For the 29 pose information in the training set, select 120 continuous new perspective poses , the new perspective pose Input NeRF in sequence to obtain the new perspective image generated by rendering ∈ Then use image and video processing tools to process the vector , output the reconstructed clear scene.

[0044] Finally, different methods are used to generate reconstructed images for images with motion blur in 5 different scenes. ,like Figure 5 As shown, (a) is the original NeRF reconstructed image, (b) is the DeblurNeRF reconstructed image, (c) is the reconstructed image by this method, and (d) is the original scene. And calculate the three indicators of the reconstructed image PSNR, SSIM, and LPIPS. PSNR is the peak signal-to-noise ratio (db), which is used to evaluate the quality of the image; SSIM is the structural similarity index, which evaluates the image quality by comparing the brightness, contrast and structural information of the image; LPIPS is an image quality evaluation indicator based on deep learning, which extracts image features through a pre-trained convolutional neural network and calculates the feature differences between the two images. The results are shown in Table 1:

[0045] Table 1

[0046] As can be seen from Table 1, the quality of scene reconstruction is very poor on the unprocessed original NeRF (native NeRF). The methods of pre-deblurring the image, MPR+NeRF and PVD+NeRF, cannot restore the real image well because the information of the blurred image is limited. The evaluation indicators obtained by this method are better than DeblurNeRF and DP-NeRF in most scenarios. PSNR and SSIM are improved to varying degrees in other scenarios except for the "pool" scenario. On average, both have significant improvements. Compared with DeblurNeRF and DP-NeRF, PSNR has increased by 2.43 and 2.28 respectively, and SSIM has increased by 3.09% and 6.34% respectively; LPIPS has been improved in all five scenarios compared with the original method, and has increased by 4.39% and 2.52% on average compared with DeblurNeRF and DP-NeRF.

Claims

1. A NeRF 3D reconstruction method based on deformable pose blur kernel, characterized by: The specific steps include: Step 1: Fuzzy image data extraction and processing Shots from different angles of the same scene Blurred image, extract its pixel matrix And estimated pose information , ; Create a corresponding blur kernel for each blurred image and the corresponding view encoding representation , and initialize. The camera pose matrix Add random perturbations to parameterize the camera pose at the beginning and end of exposure , ; Step 2: Blur Kernel Deformation According to the perspective encoding and the coordinates of the blurred pixels , use the multi-layer perceptron to get the kernel point offset of the blur kernel deformation And the weights when synthesizing blurred pixels , for the initial blur kernel Transform to get , is the number of kernel points of a blur kernel; Step 3: Blurred pixel rendering According to the number of kernel points n in the blur kernel, the camera pose , Perform linear interpolation: ； represents the estimated camera pose obtained by interpolation at time t during the exposure process. Splice and get the blur kernel The corresponding estimated camera pose sequence ; Blur Kernel The core point information in is converted into ray representation through ray coding, and the ray group is obtained. : ； in are the vectors representing the starting point and direction of the ray respectively; represents the camera intrinsic parameter matrix, Blur kernel The coordinates of the t0th kernel point in the ray group Rays are input into NeRF and rendered to obtain the pixel RGB set representation ; Then according to the weight right Perform weighted summation to obtain the simulated blurred pixel RGB representation , construct the loss function , with the goal of minimizing the loss function, optimize the multilayer perceptron parameters 、NeRF parameters , blur kernel, view encoding and estimated pose information; Step 4: Scene reconstruction choose The corresponding camera pose matrix is ​​input into NeRF to obtain the reconstructed clear scene.

2. The NeRF 3D reconstruction method based on a deformable pose blur kernel as claimed in claim 1, characterized in that: The camera pose matrix , including a rotation matrix of size 3×3 , and a 3D column vector , where the rotation matrix Describes the rotation information of the camera in the world coordinate system, vector Represents the translation information of the camera in the world coordinate system.

3. The NeRF 3D reconstruction method based on deformable pose blur kernel as claimed in claim 1, characterized in that: Create m initial blur kernels for m blurred images and the corresponding view encoding representation ;in , Represents the normalized coordinates of the kernel point in the blur kernel, which are randomly initialized using a normal distribution. The vector value is initialized to 0 directly.

4. The NeRF 3D reconstruction method based on deformable pose blur kernel as claimed in claim 1, characterized in that: Initial blur kernel To transform, the specific steps are: Step 2.1: First, The core point By denormalizing , and then for the new core point Perform a random perturbation of the normal distribution and obtain , forming a new blur kernel . For blur kernel In The pixel coordinates of the kernel points are sampled and encoded to generate a two-dimensional vector ; Step 2.2: Select the corresponding viewing angle encoding based on the blurred image , encode the perspective According to the number of blur kernel points n, the copy and expansion operation is performed to obtain a two-dimensional vector , then and Splicing to get a two-dimensional vector representation ; Calculate the coordinates of the blurred pixels to be generated One-dimensional vector representation of , and then perform a copy and expansion operation to obtain a two-dimensional vector ,and The concatenation process results in a two-dimensional vector ; Step 2.3: The vector obtained in step 2.2 Enter the first MLP layer Then the output With vector Concatenate to get vector . Then Input to the second MLP layer middle: ； ； in, ; Step 2.4: Use the kernel point offset obtained in step 2.3 Blur Kernel Deformation .

5. The NeRF 3D reconstruction method based on deformable pose blur kernel as claimed in claim 1, characterized in that: The loss function for: ； ； in, represents the RGB value of the real fuzzy pixel, B is the fuzzy pixel set, It is a structural loss The weight parameter, represents the parameters of the multilayer perceptron, ;in is a hyperparameter. Indicates randomly selecting a pixel set from the predicted pixel set And the corresponding pixel set in the training set Calculate the SSIM quality criterion.

6. The NeRF 3D reconstruction method based on deformable pose blur kernel as claimed in claim 1, characterized in that: choose A continuous generated perspective, the corresponding camera pose matrix Input the trained NeRF in sequence to get the pixel group set , , H and W are the height and width of the frame image, respectively, and c represents the pixel color. Use image processing tools to operate on the pixel groups in S to obtain a continuous generated view representation , Represents a reconstructed two-dimensional view; the continuous new view IM is used as the frame data of the clear scene output to obtain the reconstructed clear scene.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

Images