Dense light field reconstruction method based on three-dimensional multi-direction epipolar plane image
Through a dense light field reconstruction method based on three-dimensional multi-directional optoel plane image, a high-resolution optoel plane diagram is generated using a three-dimensional U-shaped neural network, which solves the problem of light field reconstruction in the existing technology under low signal-to-noise ratio and complex occlusion scenarios. Quality issues achieve high-quality and robust dense light field reconstruction.
Patent Information
- Application Number
- CN202410981800.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
AI Technical Summary
Existing light field reconstruction technology is difficult to achieve high-quality dense light field reconstruction in low signal-to-noise ratio and complex occlusion scenarios, and there are problems of perspective angle consistency and spectrum energy loss.
Using a dense light field reconstruction method based on three-dimensional multi-directional opposing pole plane image, a multi-reference viewing angle focus stack image is constructed by extracting a two-dimensional opposing pole plane view of a low-angle resolution light field, and a three-dimensional U-shaped neural network is used to generate high resolution The opposing plan, combined with error and gradient optimization network, is finally reconstructed into a dense light field.
High-quality light field reconstruction in low signal-to-noise ratio and complex occlusion scenarios is achieved, maintaining perspective consistency, and improving the robustness and texture clarity of the reconstruction results.
Smart Images

Figure CN118967911A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of light field reconstruction, and in particular relates to a dense light field reconstruction method based on three-dimensional multi-directional epipolar plane images. Background Art
[0002] Traditional imaging technology loses the key angular dimension information of three-dimensional objects, which limits the solution of problems in fields such as computer vision. Light field imaging technology can simultaneously record the spatial and angular dimension information of light, providing a data basis for three-dimensional reconstruction and virtual reality. In order to capture the angular information in the four-dimensional light field, researchers have invented a variety of light field data acquisition devices, such as robotic arms, camera arrays, and all-optical cameras based on microlens arrays. However, current light field devices all have the problem of "resolution compromise", that is, the balance between angular resolution and spatial / temporal resolution or system complexity. In order to solve the problem of "resolution compromise", researchers have proposed a large number of light field reconstruction algorithms from low angular resolution to high angular resolution.
[0003] According to different objective functions, light field reconstruction algorithms can be divided into two categories: single-view optimization and multi-view optimization. The former treats each view of the light field as an independent object. After the model is trained, this type of method can flexibly synthesize interpolated and extrapolated viewpoints. However, this method cannot guarantee the consistency of viewpoints. The latter focuses on optimizing a series of viewpoints in a single pass. These methods can maintain the consistency of viewpoints, but this type of method comes at the cost of fixed viewpoints and fixed sampling. Therefore, once the model is optimized, this type of method cannot flexibly select interpolated or extrapolated viewpoints.
[0004] Recently, the light field reconstruction method based on neural radiance field has become increasingly popular. However, this method trains each scene separately, has low generalization, requires a high number of input viewpoints, and still has difficulty in providing reliable reconstruction results under complex occlusion, thin structures, and large parallax conditions. Frequency-domain-based light field Fourier slices and disparity layer expressions are also used for light field reconstruction tasks. Although they can overcome the problems of occlusion and large parallax to a certain extent, such methods suffer from chromatic aberration caused by spectral energy loss during the reconstruction process. Summary of the invention
[0005] The present invention aims to propose a dense light field reconstruction method based on three-dimensional multi-directional epipolar plane images, which can realize dense light field reconstruction in scenes with low signal-to-noise ratio and complex occlusion.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is:
[0007] The dense light field reconstruction method based on three-dimensional multi-directional epipolar plane images includes the following steps:
[0008] Extract the 2D epipolar plane image of the low-angle resolution light field in the dataset and construct a multi-reference view focus stack image;
[0009] Construct a three-dimensional U-shaped neural network;
[0010] Input the multi-reference view focus stack images into a three-dimensional U-shaped neural network to generate a three-dimensional multi-directional epipolar plane image;
[0011] Synthesize high-resolution epipolar plane images from 3D multi-directional epipolar plane images;
[0012] The error and gradient between the synthesized high-resolution epipolar plane image and the true value are used as the loss function to optimize the 3D U-shaped neural network.
[0013] After the network converges, the high-resolution epipolar plane image generated after optimization is reconstructed into a dense light field.
[0014] Furthermore, extracting a two-dimensional epipolar plane image of a low-angle resolution light field in a data set and constructing a multi-reference view focus stack image includes the following steps:
[0015] Fix one dimension of the four-dimensional light field space and angle respectively, and extract the two-dimensional epipolar plane map. The specific formula is:
[0016] E H (u,x)=LF(u,v * ,x,y * ) (1)
[0017] Among them, LF(u,v,x,y) represents the four-dimensional light field, (u,v) represents the angle dimension coordinates of the light field, (x,y) represents the spatial dimension coordinates of the light field, H represents the height of each perspective image of the light field, and v * and * Respectively represent fixed dimensions;
[0018] The epipolar plane image is sheared in different directions and integrated to obtain a multi-reference perspective focus stack image. The specific formula is:
[0019]
[0020] Where s represents the shear parameter, N u represents the number of light field viewpoints, MRFS(s,u,x) represents the multi-reference view focus stack image, u i Represents the angle-dimensional coordinate of the i-th light field viewpoint.
[0021] Furthermore, the three-dimensional U-shaped neural network adopts an encoder-decoder structure, which consists of four levels, each of which includes three convolutional layers. The convolution step size and the number of channels between adjacent layers in the encoder are halved, and the convolution step size and the number of channels between adjacent layers in the decoder are doubled. The last convolution layer of each level of the encoder is concatenated with the first convolution layer of each level of the corresponding decoder. The number of channels of each convolution layer of the encoder-decoder is 64, 128, 256, and 512 from top to bottom, respectively; the size of the three-dimensional convolution kernel in the network is (3×3×3); each layer of the network is activated by the ReLU function, and the last layer is activated by the tan function.
[0022] Furthermore, the three-dimensional multi-directional epipolar plane image is stacked by two-dimensional epipolar plane maps, each of which contains a texture image with a specific slope; the epipolar plane image is represented by three-dimensional volume data of four channels, the first three channels are color values c(s,u,x), and the fourth channel is transparency α(s,u,x).
[0023] Further, synthesizing a high-resolution epipolar plane image specifically includes the following steps:
[0024] Along the reference viewpoint u * Perform shearing operations on c(s,u,x) and α(s,u,x). The specific formula is:
[0025]
[0026] The color values and transparency of all viewpoints are weighted. The specific formula is:
[0027]
[0028] in, represents the weighting coefficients of different viewpoints, and Respectively represent the color value and transparency weight of the viewpoint;
[0029] Synthesize a high-resolution epipolar plane image:
[0030]
[0031] Among them, N s represents the number of shear layers, Denotes the reference viewpoint u * High-resolution epipolar plane image.
[0032] Further, the three-dimensional U-shaped neural network is optimized, specifically including the following steps:
[0033] High-resolution epipolar plane images synthesized by minimization and the epipolar plane image truth value Egt The difference between (u,x) is used as the loss function, and the specific formula is:
[0034]
[0035] in, express and E gt The direct error between (u,x), express and E gt The gradient between (u,x).
[0036] According to one aspect of the present invention, a storage medium is provided, wherein instructions are stored in the storage medium, and when a computer reads the instructions, the computer executes any one of the above-mentioned dense light field reconstruction methods based on three-dimensional multi-directional epipolar plane images.
[0037] According to another aspect of the present invention, there is provided an electronic device, comprising a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. The method of the present invention can simultaneously realize the reconstruction of the internal and external viewpoints of the light field, and can achieve good reconstruction results in scenes with low signal-to-noise ratio, high robustness and complex occlusion.
[0040] 2. The present invention constructs a multi-reference perspective focus stack image, which provides a priori for the generation of three-dimensional multi-directional epipolar plane images.
[0041] 3. The present invention proposes a three-dimensional multi-directional epipolar plane image expression of the light field, based on which high-quality reconstruction of the internal and external perspectives of the light field can be achieved.
[0042] 4. The U-shaped network proposed in the present invention generates a three-dimensional multi-directional epipolar plane image with clear texture and structural information, and the image reconstruction quality is better. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0044] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;
[0045] Figure 2 is a model structure diagram of an embodiment of the present invention;
[0046] Figure 3A schematic diagram of a three-dimensional multi-directional epipolar plane image and an epipolar plane image according to an embodiment of the present invention;
[0047] Figure 4 A comparison diagram of the reconstruction results of the internal viewpoint and epipolar plane images in an embodiment of the present invention;
[0048] Figure 5 It is a comparison diagram of the image reconstruction results at the external viewpoint and the epipolar plane according to an embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0050] like Figure 1 As shown, the dense light field reconstruction method based on three-dimensional multi-directional epipolar plane images includes the following steps:
[0051] Extract 2D epipolar plane images and construct multi-reference view focus stack images;
[0052] Construct a three-dimensional U-shaped neural network;
[0053] The multi-reference perspective focus stack images are input into a 3D U-shaped neural network to generate 3D multi-directional epipolar plane images;
[0054] Synthesize high-resolution epipolar plane images from 3D multi-directional epipolar plane images;
[0055] The error and gradient between the synthesized high-resolution epipolar plane image and the true value are used as the loss function to optimize the 3D U-shaped neural network.
[0056] The high-resolution epipolar plane image generated after optimization is reconstructed into a dense light field.
[0057] Firstly, the epipolar plane map is extracted from the low-resolution light field as the algorithm input, and the epipolar plane map is sheared and integrated in different directions to construct a multi-reference view focus stack image. Subsequently, a three-dimensional U-shaped neural network is constructed with the multi-reference view focus stack image as input to generate three-dimensional multi-directional epipolar plane images. Then, a high-resolution epipolar plane is synthesized from the three-dimensional multi-directional epipolar plane images. At the same time, the error and gradient between the synthesized high-resolution epipolar plane map and the true value are calculated, and this is used as the loss function to optimize the above-mentioned three-dimensional U-shaped neural network.
[0058] The epipolar plane diagram is obtained by fixing one dimension of the four-dimensional light field space and angle respectively:
[0059] E H (u,x)=LF(u,v * ,x,y * ) (1)
[0060] Where LF(u,v,x,y) is a four-dimensional light field, (u,v) is the angle dimension coordinate of the light field, (x,y) is the spatial dimension coordinate of the light field, H is the height of each perspective image of the light field, and v is the height of each perspective image of the light field. * and * Represent fixed dimensions respectively.
[0061] The multi-reference view focus stack image is obtained by selecting different reference viewpoints and performing shearing and integration operations on the epipolar plane. Its expression is:
[0062]
[0063] Where s represents the shear parameter, N u represents the number of light field viewpoints, MRFS(s,u,x) represents the multi-reference view focus stack image, u i Represents the angle-dimensional coordinate of the i-th light field viewpoint.
[0064] like Figure 2 As shown in the figure, the three-dimensional U-shaped neural network adopts an encoder-decoder structure, which consists of four levels, each of which includes three convolutional layers. The convolution step size and the number of channels between adjacent layers in the encoder are halved, and the convolution step size and the number of channels between adjacent layers in the decoder are doubled. The last convolution layer of each level of the encoder is spliced with the first convolution layer of each level of the corresponding decoder. The number of channels of each level of the convolution layer of the encoder-decoder is 64, 128, 256, and 512 from top to bottom, respectively; the size of the three-dimensional convolution kernel in the network is (3×3×3); each layer of the network is activated by the ReLU function, and the last layer is activated by the tan function.
[0065] like Figure 3 As shown, the three-dimensional multi-directional epipolar plane image is formed by stacking multiple two-dimensional epipolar plane images, and each two-dimensional epipolar plane image only contains a texture image with a specific slope. Therefore, the present invention uses four channels of three-dimensional volume data to represent the epipolar plane image, the first three channels are color values c(s,u,x), and the fourth channel is transparency α(s,u,x).
[0066] In order to calculate the loss function, it is necessary to synthesize the epipolar plane image from the 3D multi-directional epipolar plane image generated by the network. The specific synthesis scheme is as follows:
[0067] Since the 3D multi-directional epipolar plane image generated by the network consists of color values c(s,u,x) and transparency α(s,u,x), the scheme firstly * (The reference viewpoint in the present invention is the central viewpoint) Perform shearing operations on c(s,u,x) and α(s,u,x):
[0068]
[0069] Then, the color values c(s,u,x) and transparency α(s,u,x) of all viewpoints are weighted:
[0070]
[0071] in, is the weighting coefficient for different viewpoints.
[0072] Finally, the present invention synthesizes the epipolar plane image by the following formula
[0073]
[0074] Among them, N s is the number of shear layers, represents the reference viewpoint u * High-resolution epipolar plane image.
[0075] The three-dimensional multi-directional epipolar plane image has no true value, and it is difficult to directly optimize the network through the difference between the three-dimensional U-shaped network output and the true direct difference. Therefore, the present invention adopts an indirect solution, that is, by minimizing the synthesized epipolar plane image and the epipolar plane image truth value E gt The difference between (u,x) is used to optimize the network. The specific loss function is as follows:
[0076]
[0077] in, express and E gt The direct error between (u,x), express and E gt The gradient between (u,x).
[0078] The present invention can be applied to viewpoint interpolation and extrapolation of different scenes at the same time. First, the present invention proposes an epipolar plane image perception metric function to evaluate the viewpoint reconstruction structure, and the construction method is as follows:
[0079]
[0080] Where E H and E gt are the reconstructed epipolar plane graph and the corresponding true value, E L is the lth layer output of the perceptual loss network, and L=5 in the present invention. and W l E LThe epipolar plane image perception metric function evaluates the light field reconstruction quality by extracting the linear features of the epipolar plane image. At present, most of the light field reconstruction methods and evaluation indicators proposed only focus on the quality of a single light field viewpoint image. The epipolar plane image perception metric function aims to evaluate the reconstruction quality of the entire light field image sequence, that is, the viewpoint consistency. In the present invention, the above indicators will be used to evaluate the light field reconstruction quality.
[0081] For the task of reconstructing (interpolating) the viewpoints inside the light field, the present invention takes 5 viewpoints as input, namely the 1st, 11th, 21st, 31st, and 41st viewpoints, and reconstructs the remaining 36 viewpoints in the middle. The networks proposed in the present invention are all implemented in Tensorflow, and the Adam optimizer is selected to minimize the loss function. The network is trained for 90 rounds, the batch size of the training data is 10, and the learning rate is set to 1e-4, and it is reduced by 0.9 times every five rounds. Figure 4 The comparison of the internal viewpoint reconstruction and epipolar plane image reconstruction results of different methods is shown. The method of the present invention shows its advantages in the reconstruction of complex occlusion and slender structures. For example, the basket basket parts are mutually occluded and the structure is complex. All the benchmark methods have discontinuous epipolar plane image texture when reconstructing this part, and the reconstruction of the scene itself also has obvious errors. Only the present invention can reconstruct clear details in these areas and maintain the continuity of the epipolar plane image texture, that is, viewpoint consistency.
[0082] For the task of light field external viewpoint reconstruction (extrapolation), the present invention uses 5 central viewpoints as input, namely the 21st, 34th, 41st, 51st, and 61st viewpoints, and reconstructs the remaining 76 viewpoints from the 1st to the 81st viewpoint. The network setting and training method are consistent with viewpoint interpolation. Figure 5 The comparison of external viewpoint reconstruction and epipolar plane image reconstruction results of different methods is shown. The present invention further demonstrates the robustness of the external viewpoint reconstruction structure. In the task of external viewpoint reconstruction (extrapolation), the reconstruction results of the benchmark method are obviously blurred, for example, the tires of a bicycle and the white petals. Only the present invention can reconstruct clear texture and structural information in these areas.
[0083] The examples described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should all fall within the protection scope of the present invention.
Claims
1. A dense light field reconstruction method based on three-dimensional multi-directional epipolar plane images, characterized in that: The steps include: Extract the 2D epipolar plane image of the low-angle resolution light field in the dataset and construct a multi-reference view focus stack image; Construct a three-dimensional U-shaped neural network; Input the multi-reference view focus stack images into a three-dimensional U-shaped neural network to generate a three-dimensional multi-directional epipolar plane image; Synthesize high-resolution epipolar plane images from 3D multi-directional epipolar plane images; The error and gradient between the synthesized high-resolution epipolar plane image and the true value are used as the loss function to optimize the 3D U-shaped neural network. After the network converges, the high-resolution epipolar plane image generated after optimization is reconstructed into a dense light field.
2. The method according to claim 1, characterized in that Extracting a two-dimensional epipolar plane image and constructing a multi-reference view focus stack image includes the following steps: Fix one dimension of the four-dimensional light field space and angle respectively, and extract the two-dimensional epipolar plane map. The specific formula is: E H (u,x)=LF(u,v * ,x,y * ) (1) Among them, LF(u,v,x,y) represents the four-dimensional light field, (u,v) represents the angle dimension coordinates of the light field, (x,y) represents the spatial dimension coordinates of the light field, H represents the height of each perspective image of the light field, and v * and * Respectively represent fixed dimensions; The epipolar plane image is sheared in different directions and integrated to obtain a multi-reference perspective focus stack image. The specific formula is: Where s represents the shear parameter, N u represents the number of light field viewpoints, MRFS(s,u,x) represents the multi-reference view focus stack image, u i Represents the angle-dimensional coordinate of the i-th light field viewpoint.
3. The method according to claim 1, characterized in that: The three-dimensional U-shaped neural network adopts an encoder-decoder structure and consists of four levels. Each level includes three convolutional layers. The convolution step size and the number of channels between adjacent layers in the encoder are halved, and the convolution step size and the number of channels between adjacent layers in the decoder are doubled. The last convolution layer of each level of the encoder is concatenated with the first convolution layer of each level of the corresponding decoder. The number of channels of each convolution layer of the encoder-decoder is 64, 128, 256, and 512 from top to bottom. The size of the three-dimensional convolution kernel in the network is (3×3×3). Each layer of the network is activated by the ReLU function, and the last layer is activated by the tan function.
4. The method according to claim 1, characterized in that: The three-dimensional multi-directional epipolar plane image is formed by stacking two-dimensional epipolar plane images, each of which contains a texture image with a specific slope. The epipolar plane image is represented by three-dimensional volume data of four channels, the first three channels are color values c(s,u,x), and the fourth channel is transparency α(s,u,x).
5. The method according to claim 1, characterized in that Synthesizing a high-resolution epipolar plane image includes the following steps: Along the reference viewpoint u * Perform shearing operations on c(s,u,x) and α(s,u,x). The specific formula is: The color values and transparency of all viewpoints are weighted. The specific formula is: in, represents the weighting coefficients of different viewpoints, and Respectively represent the color value and transparency weight of the viewpoint; Synthesize a high-resolution epipolar plane image: Among them, N s represents the number of shear layers, represents the reference viewpoint u * High-resolution epipolar plane image.
6. The method according to claim 1, characterized in that Optimizing the three-dimensional U-shaped neural network specifically includes the following steps: High-resolution epipolar plane images synthesized by minimization and the epipolar plane image truth value E gt The difference between (u,x) is used as the loss function, and the specific formula is: in, express and E gt The direct error value between (u,x), express and E gt The gradient between (u,x).
7. A storage medium, characterized in that: The storage medium stores instructions, and when a computer reads the instructions, the computer executes the dense light field reconstruction method based on three-dimensional multi-directional epipolar plane images according to any one of claims 1 to 6.
8. An electronic device, characterized in that: The invention comprises a processor and the storage medium as claimed in claim 7, wherein the processor executes instructions in the storage medium.
