Method for generating monocular image to light field image based on voxel optimization

Abstract

The invention provides a method for generating a monocular image to a light field image based on voxel optimization. Compared with the prior art, the method obtains the depth map through deep learning, and avoids the dependence on a depth camera. Aiming at the problem that the resolutions of the texture slices and the voxel plane are not matched, the method provides a multi-scale adaptive resolution matching algorithm, and the algorithm realizes automatic up-sampling or down-sampling processing of the texture slices through an adaptive selection module, so that the resolution matching of the texture slices and the voxel plane is ensured. Furthermore, in order to solve the problem that holes are easy to appear in the existing light field image generation process, the invention provides an edge visual angle information filling algorithm, the filling amount corresponding to the holes is accurately calculated by constructing a geometrical relationship, and the holes are filled in combination with a diffusion model, so that the hole phenomenon of the light field image is solved, and the image quality is improved. And the integrity and quality of the light field image are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated imaging and image processing, in particular to a method for generating a light field image from a monocular image based on voxel optimization. BACKGROUND

[0002] Integrated imaging three-dimensional (3D) display is a display technology that can realize naked-eye 3D visual effect without wearing additional auxiliary equipment. This technology can provide full-parallax information and complete physiological depth cues at the same time, and has the advantage of immersive experience, so it has broad application prospects in the fields of education, military, medical treatment and industrial design.

[0003] The realization of integrated imaging 3D display depends on the acquisition of dense parallax maps. In real scenes, dense parallax maps are usually captured by a multi-camera array. However, large-scale camera array has complex structure, high construction and maintenance cost, and limited camera arrangement density due to physical size limitation, so it is difficult to directly obtain dense view images that meet the requirements, thereby restricting the further popularization and application of the technology.

[0004] Existing researches mainly generate dense light field images from sparse views to replace large-scale camera acquisition, and the methods mainly include two categories: one is based on optical flow estimation, which extracts the optical flow information between sparse views and generates intermediate views by interpolation to realize the expansion from sparse to dense; the other is based on image super-resolution, which improves the overall resolution of the light field image generated from sparse views to obtain a dense light field image. Although the above methods can reduce the number of cameras, they still rely on camera arrays, and the acquisition process is complex and costly. In recent years, some methods have tried to directly generate dense light field images from a single color image and its depth map, which has simplified the acquisition process, but the generated results still have a small number of holes, which cannot meet the requirements of high-quality three-dimensional display. SUMMARY

[0005] The present application provides a method for generating a light field image from a monocular image based on voxel optimization, and the implementation process is shown in the accompanying Figure 1 Compared with the prior art, the method of the present application acquires a depth map through deep learning, avoiding the dependence on a depth camera. In view of the problem that the resolution of the texture slice does not match that of the voxel plane, the present application proposes a multi-scale adaptive resolution matching algorithm, which realizes automatic up-sampling or down-sampling processing of the texture slice through an adaptive selection module, so as to ensure the resolution matching of the texture slice and the voxel plane. Further, in order to solve the problem that holes are prone to occur in the existing light field image generation process, the present application proposes an edge view angle information filling algorithm, which accurately calculates the filling amount corresponding to the holes by constructing a geometric relationship, and fills the holes in combination with a diffusion model, thereby solving the hole phenomenon of the light field image and ensuring the integrity and quality of the light field image.

[0006] The method includes the following steps:

[0007] Step 1: Based on the known structural parameters of the 3D display, establish the voxel space for the 3D display, as shown in the attached figure. Figure 2 As shown;

[0008] Step 2: Perform depth reconstruction on the input monocular image, and use the reconstructed depth information combined with the selected voxel space to divide the 3D scene, obtaining a series of texture slices. The flowchart is attached. Figure 3 As shown;

[0009] Step 3: Texture slicing preprocessing, including resolution matching and edge view information filling;

[0010] Step 4: Pixel mapping to generate a light field image, specifically including:

[0011] Based on the voxel space established in step 1, the mapping relationship between voxels and corresponding pixel points is established, and its expression is shown in equation (1):

[0012] F:V x →HomoPx (1)

[0013] Among them, V x Let represent the set of voxels, HomoPx represent the set of pixels with the same name, and F represent the mapping function. The processed texture slices are placed on the corresponding voxel planes, and the final light field image is generated based on the mapping relationship established by equation (1).

[0014] Further, step 1 includes the following sub-steps:

[0015] Sub-step 1.1: Establish the voxel space of the 3D display and obtain the required voxel space data, specifically including:

[0016] A world coordinate system is established with the geometric center of the microlens array plane as the origin, where the XY plane coincides with the microlens array plane and the Z axis is perpendicular to the microlens array plane. Each image element contains R×R reconstructed rays, where R is the lateral resolution of the image element covered by the lens element.

[0017] A reconstructed ray emitted from a pixel in any given image pixel intersects pairwise with every reconstructed ray from its neighboring image pixels, forming at most (R-1) reconstructed rays. 2 There are voxels, and the voxels are distributed at R-1 different depth locations, so the total number of voxel planes is:

[0018] N p =R-1 (2)

[0019] Let the k-th voxel plane be defined, where 1 ≤ k ≤ N. pThe distance z(k) from the microlens array plane is expressed as:

[0020]

[0021] Where p represents the pitch of the lens element, and g is the distance between the pixel plane and the microlens array plane. d The size of a single pixel on a 2D display screen;

[0022] The distance Δz(k) between adjacent voxel planes is:

[0023]

[0024] Let Δx(k) and Δy(k) represent the horizontal and vertical voxel intervals on the adjacent voxel plane, respectively. Then we have:

[0025]

[0026] The number of voxels N(k) on different voxel planes is:

[0027]

[0028] Where W and H represent the width and height of the display, respectively; Δx(k) and Δy(k) represent the horizontal and vertical spacing of adjacent voxels, respectively; and round(·) represents the rounding operation.

[0029] Sub-step 1.2: Plot the curve of the number of voxels changing with the voxel plane according to equation (6), and obtain the preferred voxel space in which the voxels on each voxel plane are evenly distributed and have high spatial resolution.

[0030] Furthermore, step 2 includes the following sub-steps:

[0031] Sub-step 2.1: Input the monocular image into the monocular prediction depth network. First, perform relative depth estimation on the monocular image, and then further perform absolute depth estimation based on this. By combining the results of relative and absolute depth estimation, the depth information is restored, thereby obtaining the corresponding depth map. The flowchart is shown in the attached figure. Figure 4 As shown;

[0032] Sub-step 2.2: Combine the depth information in the voxel space with the depth map to divide the 3D scene into a series of texture slices {S1,…,S...} n}

[0033] Furthermore, step 3 includes the following sub-steps:

[0034] Sub-step 3.1: Use a multi-scale adaptive resolution matching algorithm to match the resolution of the texture slices so that the resolution of the matched texture slices is the same as the resolution of the voxel plane;

[0035] Specifically, this includes: First, calculating the number of voxels N(k) for each voxel plane according to equation (6), and combining this with the resolution N of the input monocular image, obtaining the sampling factors {α1,…,α} corresponding to each texture slice through equation (7). n}; Subsequently, the texture slices are classified using an adaptive selection module, when α k When α > 1, the texture slice is classified as an upsampled texture slice; when α k When α < 1, the texture slice is classified as a downsampled texture slice; when α k When the resolution is 1, the texture slice is classified as a texture slice that does not require processing. Finally, the upsampled texture slice is upsampled using a deep learning super-resolution reconstruction model, and the downsampled texture slice is processed using a downsampling algorithm, thereby achieving compatibility between the texture slice and the voxel plane resolution. The process is shown in the attached figure. Figure 5 As shown;

[0036]

[0037] Sub-step 3.2: An edge view information filling algorithm is used to supplement the missing view information in the matched texture slices, thereby ensuring that the generated light field image is hole-free. First, the geometric relationship is constructed based on the maximum viewing angle θ and the distance Δz(k) between adjacent voxel surfaces, as shown in the appendix. Figure 6 As shown, the required edge width Δx to be filled is calculated using equation (8). The fill amount Δn corresponding to each texture slice is calculated according to equation (9). k And convert it into a filling mask, as shown in the attached image. Figure 7 As shown. Finally, a diffusion model is used to fill information in the edge regions of the texture slices to obtain a high-quality light field image.

[0038] Δx=Δz(k)*tanθ (8)

[0039]

[0040] This invention proposes a method for generating light field images from monocular images based on voxel optimization, which effectively simplifies the light field image generation process. The invention introduces a monocular predictive depth network to perform depth reconstruction on the input image to obtain a depth map, reducing the complexity of data acquisition. Addressing the issue of resolution mismatch between texture slices and voxel planes, this invention proposes a multi-scale adaptive resolution matching algorithm. This algorithm enables batch processing of texture slices, effectively enhancing high-frequency details during upsampling and retaining high-frequency information well after downsampling, ensuring that image quality does not suffer significant distortion. This invention processes texture slices using an edge view information filling algorithm, resulting in a hole-free final light field image. Using this method, a high-quality light field image can be generated using only a single monocular image, significantly simplifying the light field image generation process and providing an efficient and feasible technical path for the widespread application of 3D light field displays. Attached Figure Description

[0041] Appendix Figure 1 This is a flowchart of the method for generating a light field image from a monocular image based on voxel optimization proposed in this invention.

[0042] Appendix Figure 2 This is a schematic diagram of the preferred voxel space.

[0043] Appendix Figure 3 This is a flowchart for dividing a 3D scene based on a preferred voxel space.

[0044] Appendix Figure 4 This is a schematic diagram of the deep restoration process.

[0045] Appendix Figure 5 This is a flowchart of a multi-scale adaptive resolution matching algorithm.

[0046] Appendix Figure 6 This is a schematic diagram for calculating the fill volume.

[0047] Appendix Figure 7 A schematic diagram for filling in edge view information.

[0048] Appendix Figure 8 A graph showing the number of voxels and depth for different voxel planes.

[0049] Appendix Figure 9 The effect of adaptive resolution matching downsampling.

[0050] Appendix Figure 10 The effect of upsampling for adaptive resolution matching.

[0051] Appendix Figure 11 The effect of filling in edge view information.

[0052] Appendix Figure 12This refers to the depth map and 3D display effect diagram corresponding to the scene to be displayed.

[0053] The figure labels in the above figures are:

[0054] 1 Microlens array, 2 Image element, 3 Lens element, 4 Reconstructed ray, 5 Optimized voxel space, 6 Voxel, 7 Monocular image, 8 Depth image, 9 Texture slice, 10 Zoe-Depth network structure, 11 Texture slice placed on different voxel planes, 12 Fill mask, 13 Filled texture slice, 14 Filled region, 15 Line graph of voxel number changing with voxel plane, 16 Line graph of voxel plane depth change, 17 Texture slice after resolution matching, 18 Light field image generated by texture slice, 19 Display effect of light field image generated by texture slice, 20 Light field image generated without edge view information filling, 21 Display effect of light field image generated without edge view information filling, 22 Light field image generated with edge view information filling, 23 Display effect of light field image generated with edge view information filling, 24 Left view of 3D display effect, 25 Right view of 3D display effect.

[0055] It should be understood that the accompanying drawings are only schematic and are not drawn to scale. Detailed Implementation

[0056] The following detailed description of a typical embodiment of the method for generating a light field image from a monocular image based on voxel optimization according to the present invention provides a further specific description of the invention. It is necessary to point out that the following embodiments are only used for further illustration of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-essential improvements and adjustments made to the present invention by those skilled in the art based on the above description still fall within the scope of protection of the present invention.

[0057] This embodiment includes the following steps:

[0058] Step 1: Based on the known structural parameters of the 3D display as shown in Table 1, establish the voxel space for the 3D display, as shown in the attached table. Figure 2 As shown;

[0059] Table 1 Parameters of the Light Field 3D Display

[0060]

[0061] The 3D display in this example is an integrated imaging 3D display.

[0062] Step 2: Perform depth reconstruction on the input monocular image, and use the reconstructed depth information combined with the selected voxel space to divide the 3D scene, obtaining a series of texture slices. The flowchart is attached. Figure 3 As shown;

[0063] The monocular images in this example are taken by mobile phones or cameras in real-world scenes;

[0064] Step 3: Texture slicing preprocessing, including resolution matching and edge view information filling;

[0065] Step 4: Pixel mapping to generate a light field image, specifically including;

[0066] Based on the voxel space established in step 1, the mapping relationship between voxels and corresponding pixel points is established, and its expression is shown in equation (1):

[0067] F:V x →HomoPx (1)

[0068] Among them, V x Let represent the voxel set, HomoPx represent the set of corresponding pixel points, and F represent the mapping function. The processed texture slices are placed on the corresponding voxel plane, and the final light field image is generated based on the mapping relationship established by equation (1). The generated light field image and its display effect are shown in the attached figure. Figure 12 As shown.

[0069] Further, step 1 includes the following sub-steps:

[0070] Sub-step 1.1: Establish the voxel space of the 3D display and obtain the required voxel space data, specifically including;

[0071] A world coordinate system is established with the geometric center of the microlens array plane as the origin, where the XY plane coincides with the microlens array plane and the Z axis is perpendicular to the microlens array plane. Each image element contains R×R reconstructed rays, where R is the lateral resolution of the image element covered by the lens element.

[0072] A reconstructed ray emitted from a pixel in any given image pixel intersects pairwise with every reconstructed ray from its neighboring image pixels, forming at most (R-1) reconstructed rays. 2 There are voxels, and the voxels are distributed at R-1 different depth locations, so the total number of voxel planes is:

[0073] N p =R-1 (2)

[0074] Let the k-th voxel plane be defined, where 1 ≤ k ≤ N. p The distance z(k) from the microlens array plane is expressed as:

[0075]

[0076] Where p represents the pitch of the lens element, and g is the distance between the pixel plane and the microlens array plane. dThe size of a single pixel on a 2D display screen;

[0077] The distance Δz(k) between adjacent voxel planes is:

[0078]

[0079] Let Δx(k) and Δy(k) represent the horizontal and vertical voxel intervals on the adjacent voxel plane, respectively. Then we have:

[0080]

[0081] The number of voxels N(k) on different voxel planes is:

[0082]

[0083] Where W and H represent the width and height of the display, respectively; Δx(k) and Δy(k) represent the horizontal and vertical spacing of adjacent voxels, respectively; and round(·) represents the rounding operation.

[0084] Equations (2)-(6) describe all the voxel characteristic parameters of the integrated imaging 3D display. It can be seen that the above voxel characteristic parameters are only related to the structural parameters of the integrated imaging 3D display, such as lens element pitch, number of lens units, screen size and pixel size of the 2D display, and the distance between the pixel plane and the microlens array plane. In other words, once the structural parameters of the system are determined, all possible voxel positions and their spatial distribution patterns of the integrated imaging 3D display are uniquely determined.

[0085] Sub-step 1.2: Plot a curve showing the change in the number of voxels with the voxel plane according to equation (6), as shown in the attached figure. Figure 8 As shown in the figure, the number of voxels first increases and then decreases with the increase of the voxel plane. Based on this rule, the interval marked by the dashed box in the figure is selected as the optimal voxel space. In this region, the voxels are evenly distributed and have high spatial resolution.

[0086] Furthermore, step 2 includes the following sub-steps:

[0087] Sub-step 2.1: Input the monocular image into the Zoe-Depth network structure, and first perform relative depth estimation on the monocular image. Then, further perform absolute depth estimation based on this. By combining the results of relative and absolute depth estimation, the depth information is restored, thereby obtaining the corresponding depth map. The flowchart is shown in the attached figure. Figure 4 As shown;

[0088] Sub-step 2.2: Combine the depth information in the voxel space with the depth map to divide the 3D scene into a series of texture slices {S1,…,S...}13}

[0089] Furthermore, step 3 includes the following sub-steps:

[0090] Sub-step 3.1: Use a multi-scale adaptive resolution matching algorithm to match the resolution of the texture slices so that the resolution of the matched texture slices is the same as the resolution of the voxel plane;

[0091] Specifically, this includes: First, calculating the number of voxels N(k) for each voxel plane according to equation (6), and combining this with the resolution N of the input monocular image, obtaining the sampling factors {α1,…,α} corresponding to each texture slice through equation (7). 13 Subsequently, the texture slices are classified using an adaptive selection module, when α k When α > 1, the texture slice is classified as an upsampled texture slice. k When α < 1, the texture slice is classified as a downsampled texture slice. k When the resolution is 1, the texture slice is classified as a texture slice that does not require processing. Finally, the upsampled texture slice is upsampled using the Real-ESRGAN super-resolution reconstruction model, and the downsampled texture slice is processed using the Lanczos2 downsampling algorithm, thus achieving compatibility between the texture slice and the voxel plane resolution. The results are shown in the appendix. Figure 9 As shown in Figure 10;

[0092]

[0093] Sub-step 3.2: An edge view information filling algorithm is used to supplement the missing view information in the matched texture slices, thereby ensuring that the generated light field image is hole-free. First, the geometric relationship is constructed based on the maximum viewing angle θ and the distance Δz(k) between adjacent voxel surfaces, as shown in the appendix. Figure 7 As shown, the required edge width Δx to be filled is calculated using equation (8). The fill amount Δn corresponding to each texture slice is calculated according to equation (9). k And convert it into a filling mask, as shown in the attached image. Figure 7 As shown in the figure. Finally, the Stable Diffusion model is used to fill the edge regions of the texture slices with information to obtain a high-quality light field image. The light field image generated after processing by this algorithm and the display effect are shown in the attached figure. Figure 11 As shown.

[0094] Δx=Δz(k)*tanθ (8)

[0095]

[0096] This invention proposes a method for generating light field images from monocular images based on voxel optimization, which effectively simplifies the light field image generation process. The invention introduces a monocular predictive depth network to perform depth reconstruction on the input image to obtain a depth map, reducing the complexity of data acquisition. Addressing the issue of resolution mismatch between texture slices and voxel planes, this invention proposes a multi-scale adaptive resolution matching algorithm. This algorithm enables batch processing of texture slices, effectively enhancing high-frequency details during upsampling and retaining high-frequency information well after downsampling, ensuring that image quality does not suffer significant distortion. This invention processes texture slices using an edge view information filling algorithm, resulting in a hole-free final light field image. Using this method, a high-quality light field image can be generated using only a single monocular image, significantly simplifying the light field image generation process and providing an efficient and feasible technical path for the widespread application of 3D light field displays.

Claims

1. A method for generating a light field image from a monocular image based on voxel optimization, characterized in that, The method acquires depth maps through deep learning to avoid dependence on depth cameras; it employs a multi-scale adaptive resolution matching algorithm, using an adaptive selection module to automatically upsample or downsample texture slices to ensure resolution matching between texture slices and voxel planes; and it uses an edge view information filling algorithm to calculate hole filling amount based on geometric relationships and combines a diffusion model to fill holes, thereby eliminating holes in the light field image and improving the integrity and quality of the light field image. The method includes the following steps: Step 1: Based on the known structural parameters of the 3D display, establish the voxel space for the 3D display; Step 2: Perform depth reconstruction on the input monocular image, and use the reconstructed depth information combined with the selected voxel space to divide the 3D scene and obtain a series of texture slices; Step 3: Texture slicing preprocessing, including resolution matching and edge view information filling; Step 4: Pixel mapping to generate a light field image, specifically including; Based on the voxel space established in step 1, the mapping relationship between voxels and corresponding pixel points is established, and its expression is as follows: F:V x →HomoPx (1) Among them, V x Let represent the set of voxels, HomoPx represent the set of pixels with the same name, and F represent the mapping function; place the processed texture slice on the corresponding voxel plane, and generate the final light field image based on the mapping relationship established by equation (1).

2. The method for generating a light field image from a monocular image based on voxel optimization according to claim 1, characterized in that, In step 1, based on the known structural parameters of the 3D display, a voxel space is established, which specifically includes the following sub-steps: Sub-step 1.1: Establish the voxel space of the 3D display and obtain the required voxel space data, specifically including: A world coordinate system is established with the geometric center of the microlens array plane as the origin, where the XY plane coincides with the microlens array plane and the Z axis is perpendicular to the microlens array plane. Each image element contains R×R reconstructed rays, where R is the lateral resolution of the image element covered by the lens element. A reconstructed ray emitted from a pixel in any given image pixel intersects pairwise with every reconstructed ray from its neighboring image pixels, forming at most (R-1) reconstructed rays. 2 There are voxels, and the voxels are distributed at R-1 different depth locations, so the total number of voxel planes is: N p =R-1 (2) Let the k-th voxel plane be defined, where 1 ≤ k ≤ N. p The distance z(k) from the microlens array plane is expressed as: Where p represents the pitch of the lens element, g is the distance between the pixel plane and the microlens array plane, p d The size of a single pixel on a 2D display screen; The distance Δz(k) between adjacent voxel planes is: Let Δx(k) and Δy(k) represent the horizontal and vertical voxel intervals on the adjacent voxel plane, respectively. Then we have: The number of voxels N(k) on different voxel planes is: Where W and H represent the width and height of the display, respectively, Δx(k) and Δy(k) represent the horizontal and vertical spacing of adjacent voxels, and round(·) represents the rounding operation; Sub-step 1.2: Plot the curve of the number of voxels changing with the voxel plane according to equation (6), and obtain the preferred voxel space in which the voxels on each voxel plane are evenly distributed and have high spatial resolution.

3. The method for generating a light field image from a monocular image based on voxel optimization according to claim 1, characterized in that, In step 2, depth reconstruction is performed on the input monocular image, and the reconstructed depth information is used in conjunction with the selected voxel space to divide the 3D scene and obtain a series of texture slices. Specifically, this includes the following sub-steps: Sub-step 2.1: Input the monocular image into the monocular prediction depth network. First, perform relative depth estimation on the monocular image, and then perform absolute depth estimation on this basis. By combining the results of relative depth estimation and absolute depth estimation, the depth information is restored, thereby obtaining the corresponding depth map. Sub-step 2.2: Combine the depth information in the voxel space with the depth map to divide the 3D scene into a series of texture slices {S1,…,S...} n } 4. The method for generating a light field image from a monocular image based on voxel optimization according to claim 1, characterized in that, Step 3, texture slicing preprocessing, including resolution matching and edge view information filling, specifically includes the following sub-steps: Sub-step 3.1: Use a multi-scale adaptive resolution matching algorithm to match the resolution of the texture slices so that the resolution of the matched texture slices is the same as the resolution of the voxel plane; Specifically, this includes: First, calculating the number of voxels N(k) for each voxel plane according to equation (6), and combining this with the resolution N of the input monocular image, obtaining the sampling factors {α1,…,α} corresponding to each texture slice through equation (7). n Subsequently, an adaptive selection module is used to classify texture slices, where when α k When α > 1, the texture slice is classified as an upsampled texture slice; when α k When α < 1, the texture slice is classified as a downsampled texture slice; when α k When = 1, the texture slice is classified as a texture slice that does not require processing; finally, the upsampled texture slice is improved by using a deep learning super-resolution reconstruction model, and the downsampled texture slice is processed by a downsampling algorithm, so as to achieve compatibility between texture slice and voxel plane resolution. Sub-step 3.2: An edge view information filling algorithm is used to supplement the missing view information of the matched texture slices, thereby ensuring that the generated light field image is free of holes. First, the geometric relationship is constructed based on the maximum viewing angle θ and the distance Δz(k) between adjacent voxel surfaces, and the required edge width Δx to be filled is calculated by equation (8). Then, the filling amount Δn corresponding to each texture slice is calculated according to equation (9). k This is then converted into a filling mask; finally, a diffusion model is used to fill the edge regions of the texture slices with information to obtain a high-quality light field image. Δx=Δz(k)*tanθ (8)

Images