Method for converting 2D image into 3D image

Abstract

The invention, which relates to the technical field of image processing, discloses a method for converting a 2D image into a 3D image, comprising the following steps: step 1, obtaining a 2D image; step 2, monocular depth estimation is carried out on the 2D image to obtain a relative depth map; 3, determining the longitudinal depth and the zero parallax surface of the scene according to the content of the 2D image; 4, carrying out value domain transformation on the depth map; 5, calculating the parallax of each pixel of the 2D image according to the transformed depth map, and then rendering an image of a virtual viewpoint according to the parallax; and step 6, repairing the cavity. According to the method for converting the 2D image into the 3D image, the watching experience is improved, the stereoscopic impression is enhanced, the ghosting phenomenon is reduced, and the proper proportion scale is kept; the conversion efficiency and the automation degree are improved, and manual intervention is reduced; meanwhile, the generalization ability is enhanced; in addition, the hole repairing effect is optimized, so that the converted 3D image is more natural and perfect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for converting a 2D image into a 3D image. Background Art

[0002] In recent years, the hot sales of VR / AR devices have attracted the attention of audiences and the market for 3D pictures and videos, and also promoted the development of 3D display technology. With the continuous growth of the number of 3D movie screens and the launch of naked-eye 3D display screens or VR glasses by major technology companies, the number of existing 3D picture and video resources has restricted the development of the 3D display industry. On the one hand, the shooting of 3D film and television works relies on expensive professional equipment. On the other hand, there is still a lack of mature and reliable 2D-3D video conversion technology for the large number of existing 2D videos on the Internet. Therefore, the study of 2D picture / video to binocular 3D picture / video conversion technology can not only greatly enrich the existing 3D resource library, but also convert the 2D pictures / videos taken in people's daily life into 3D pictures / videos, providing people with an immersive and three-dimensional appreciation method. 3D pictures / videos are composed of left and right pictures / video screens. Under the action of 3D display technology and human binocular vision system, the audience can feel the 3D stereoscopic effect.

[0003] Physiologically, the human visual system is mainly composed of the left and right eyes, the visual nerves, and the occipital lobe area in the cerebral cortex. Such physiological characteristics lead to slight differences in the imaging of light information in the scene on the retinas of the two eyes due to the differences in angle and distance between the two eyes when a person looks at the same object or area with both eyes. This difference is called binocular disparity. Binocular depth perception is the theoretical basis of 3D stereoscopic vision. A 3D image is composed of two left and right images with a certain horizontal disparity between them. The image and video 2D to 3D conversion includes two core steps: (1) According to the geometric relationship of space and the characteristics of the human eye stereoscopic vision system, the front-back (depth) relationship between each object is extracted from the scene screen of the planar 2D image to obtain a depth map containing the depth information of all pixels; (2) Based on the planar 2D image, combined with the depth information provided by the depth map, the 2D image scene is inversely transformed into a three-dimensional world coordinate space, and then mapped from the three-dimensional world coordinate space to the two-dimensional imaging plane of the virtual viewpoint to obtain a virtual right (or left) viewpoint image. The former is called depth information extraction (or depth estimation), and the latter is called virtual viewpoint synthesis.

[0004] According to a stereoscopic image disparity optimization method with announcement number CN102802015A, the invention is applied to a naked-eye 3D device based on binocular disparity, and proposes a stereoscopic image disparity optimization method, which includes the following steps: first, in the world coordinate system, determine the minimum disparity value Pmin and the maximum disparity value Pmax; then, convert the minimum disparity value Pmin and the maximum disparity value Pmax in the world coordinate system into the minimum disparity value P1min and the maximum disparity value P1max in the screen coordinate system of the stereo display screen; then, calculate the number of offset pixels PNmin corresponding to the disparity value when the virtual object is closest to the viewer and the number of offset pixels PNmax corresponding to the disparity value when the virtual object is farthest from the viewer; finally, calculate the number of pixels PN offset between corresponding points in the left and right views according to the depth map, set the zero depth layer and optimize the disparity. The present invention combines the display performance of the stereoscopic mobile terminal and the physiological function of the human eye to perceive stereo, derives the optimal disparity range, and adjusts and optimizes the disparity of the existing stereoscopic video according to the range, thereby improving the comfort of watching stereoscopic video on the mobile terminal.

[0005] With respect to the above-mentioned related schemes, the above-mentioned method can optimize the existing stereoscopic image parallax, but since the maximum and minimum parallaxes of the display terminal are fixed in advance, and then the maximum and minimum depths of the image are mapped to this parallax range, and since the vertical depth of the actual scene is ignored, the stereoscopic effect is not obvious or the stretching deformation is serious, ghosting and other problems may occur for scenes with different average distances. Summary of the invention

[0006] The purpose of the present invention is to provide a method for converting 2D images into 3D images, so as to solve the technical problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for converting a 2D image to 3D, comprising the following steps:

[0008] Step 1: Get a 2D image;

[0009] Step 2: Perform monocular depth estimation on the 2D image to obtain a relative depth map;

[0010] Step 3: Determine the vertical depth and zero parallax plane of the scene based on the 2D image content;

[0011] Step 4: transform the depth map into a value range;

[0012] Step 5: Calculate the disparity of each pixel of the 2D image according to the transformed depth map, and then render the image of the virtual viewpoint according to the disparity;

[0013] Step 6: Repair the cavity.

[0014] Preferably, the 2D image in step one includes a 2D picture and a 2D video. When processing the video, each frame of the image is still extracted. The image includes various resolutions, such as 720p, 1080p, 2K, and 4K, and can be played on TVs, computer monitors, outdoor advertising screens, tablets and other mobile devices, and virtual image devices such as telescopic screens.

[0015] Preferably, the monocular depth estimation method in step 2 includes the traditional method of linear perspective principle and motion parallax, and also includes the current mainstream machine learning method based on deep learning and large depth data sets.

[0016] Preferably, the relative depth in step 2 is the relative distance between different scene elements in the image, the maximum depth is mapped to a grayscale value of 0, and the minimum depth is mapped to a grayscale value of 255, to obtain a relative depth map in the form of a grayscale map.

[0017] Preferably, in step three, to ensure the display effect, the maximum display depth Zmax and the minimum display depth Zmin of the naked-eye 3D display system are predetermined, and the maximum display depth Zmax is also called the maximum depth of field, and the minimum display depth Zmin is also called the minimum depth of field.

[0018] Preferably, the maximum display depth Z1max and the minimum display depth Z1min of the longitudinal depth in step three are, and Z1max≤Zmax||Z1min≥Zmin, Z1max and Z1min are related to each scene, and can ensure the optimal display effect in different complex scenes. The model obtained in this step uses absolute depth as training data, and does not require high accuracy of Zmax and Zmin. A small amount of data can be used to train the absolute depth estimation model, and this model is applied to generalize to the acquired 2D image to obtain Zmax and Zmin. Similar algorithms in the industry: Metric3D, Arxiv:https: / / arxiv.org / abs / 2307.10984.

[0019] Preferably, the formula for range transformation in step 4 is:

[0020]

[0021] Where S is the original depth, ranging from 0 to 255; S1 is the depth after transformation according to the vertical depth of the scene. Each pixel of the original 2D image corresponds to an original depth S, which corresponds to the new depth S1 after the range transformation.

[0022] Preferably, in step five, when calculating the disparity, the original 2D image can be used as the view for the left eye, and then the right eye view can be rendered according to the disparity value mapping; the original 2D image can also be used as the view for the right eye to render the left eye view; or even the original 2D image can be used as the view between the two eyes, and then the left eye and right eye views can be rendered respectively according to the disparity value.

[0023] Preferably, in step six, the Criminisi algorithm is used to fill the holes, combining the texture synthesis method with the idea of ​​information diffusion. First, a suitable texture sample block is delineated, and then the hole area is filled and diffused in a predetermined order. The relevant information of the Criminisi algorithm is Criminisi A, et al. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.

[0024] Preferably, the specific repair process in step six is:

[0025] 1) First determine the hole area and the non-hole area in the image;

[0026] 2) Calculation priority: Calculation priority is used to determine the order of repairing different locations. In most cases, the edges of the hole areas with more known information and stronger structure have higher priority, which makes the repair area gradually spread from the edge of the hole to the center.

[0027] 3) Searching for the best matching block, that is, searching for the matching block closest to the block to be repaired in the known area of ​​the current image based on the similarity of structure and texture;

[0028] 4) Hole filling: copy the matching block to the block to be repaired and fill the hole area in the block with pixels.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention can generate a 3D image with a strong sense of stereo by using accurate monocular depth estimation and parallax rendering. Whether it is a close-up, mid-range or distant view, the converted 3D image can present a realistic depth effect, making the audience feel as if they are in the scene. Accurate depth estimation and meticulous parallax adjustment effectively reduce the ghosting phenomenon, providing a clearer and smoother 3D viewing experience. In addition, the depth distance is determined according to different scenes of the original 2D image, and the depth map is remapped to this distance range, ensuring that the converted 3D image is consistent with the original image in terms of scale, avoiding visual discomfort caused by disproportionate proportions.

[0031] 2. The present invention adopts a series of automated processing steps, including monocular depth estimation, depth map value range transformation, disparity calculation and virtual viewpoint rendering, etc. These steps are interconnected to form an efficient and smooth conversion process, which greatly improves the conversion efficiency of 2D to 3D. The entire conversion process requires almost no human intervention, which reduces the dependence on the skills of professionals, allowing more people to easily master and apply this method for 2D to 3D conversion.

[0032] 3. The present invention combines the linear perspective principle, motion parallax and machine learning methods based on deep learning and large depth data sets to perform monocular depth estimation, and obtains strong generalization performance, so that more types of 2D images can be processed and high-quality 3D images can be generated. The advanced Criminisi algorithm is used for hole repair, and the hole area is effectively filled through the idea of ​​texture synthesis and information diffusion, making the converted 3D image more natural and flawless.

[0033] In summary, the 2D image conversion method proposed in the present invention not only improves the viewing experience, enhances the stereoscopic effect, reduces the ghosting phenomenon, and maintains a suitable scale; it also improves the conversion efficiency and automation, reduces manual intervention, and enhances the generalization ability; in addition, it also optimizes the hole repair effect, making the converted 3D image more natural and perfect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of converting 2D image to 3D according to the present invention;

[0035] Figure 2 It is a relative depth map in the form of a grayscale map of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 and Figure 2 The present invention provides a technical solution: a method for converting a 2D image to a 3D image, comprising the following steps:

[0038] Step 1: Get 2D images, which include 2D pictures and 2D videos. When processing videos, each frame of the image is still extracted. The images include various resolutions, such as 720p, 1080p, 2K, and 4K, and can be played on TVs, computer monitors, outdoor advertising screens, tablets and other mobile devices, and virtual image devices such as telescopic screens.

[0039] Step 2: Perform monocular depth estimation on the 2D image to obtain a relative depth map. The monocular depth estimation method includes the traditional method of linear perspective principle and motion parallax, and also includes the current mainstream machine learning method based on deep learning and large depth data sets. In order to obtain better estimation effect and stronger generalization performance, the present invention obtains the relative depth, that is, the relative distance between different scene elements in the image. The maximum depth is mapped to the grayscale value of 0, and the minimum depth is mapped to the grayscale value of 255, so as to obtain a relative depth map in the form of a grayscale map;

[0040] Step 3: To ensure the display effect, the maximum display depth Zmax and the minimum display depth Zmin of the naked eye 3D display system are determined in advance. The maximum display depth Zmax is also called the maximum depth of field, and the minimum display depth Zmin is also called the minimum depth of field. According to the 2D image content, the vertical depth and the zero parallax plane of the scene are determined. The maximum display depth Z1max and the minimum display depth Z1min of the vertical depth are determined, and Z1max≤Zmax||Z1min≥Zmin. Z1max and Z1min are related to each scene, which can ensure the best display effect in different complex scenes. This step uses the absolute depth as the training data to obtain the model, and the accuracy requirements for Zmax and Zmin are not high. A small amount of data can be used to train the absolute depth estimation model, and this model is applied to the acquired 2D image to obtain Zmax and Zmin. Similar algorithms in the industry: Metric3D, Arxiv:https: / / arxiv.org / abs / 2307.10984;

[0041] Step 4: Perform a range transformation on the depth map to map the original depth S to a new depth S1 to adapt to the vertical depth range of the scene. The formula for range transformation is:

[0042]

[0043] Where S is the original depth, ranging from 0 to 255; S1 is the depth after transformation according to the vertical depth of the scene. Each pixel of the original 2D image corresponds to an original depth S, which corresponds to the new depth S1 after the range transformation.

[0044] Step 5: Calculate the disparity of each pixel of the 2D image according to the transformed depth map, and then render the image of the virtual viewpoint according to the disparity. When calculating the disparity, the original 2D image can be used as the view of the left eye, and then the right eye view can be rendered according to the disparity value mapping; the original 2D image can also be used as the view of the right eye to render the left eye view; or even the original 2D image can be used as the view between the two eyes, and then the left eye and right eye views can be rendered respectively according to the disparity value;

[0045] Step 6: After the view rendering is completed, the holes are repaired. These holes come from the fact that when the pixels in the original view image are mapped to the right (or left) view, some pixels are "occluded" due to the depth relationship, resulting in the overlap of different pixels. The total number of pixels in the original view image is fixed and limited. Therefore, there must be some pixels that have never been mapped in the new view image, which are holes. The new view image must be repaired to achieve a natural 3D display effect. This method uses the Criminisi algorithm to repair holes, combining the texture synthesis method with the idea of ​​information diffusion. First, a suitable texture sample block is delineated, and then the hole area is filled and diffused in a predetermined order. The relevant information of the Criminisi algorithm is Criminisi A, et al. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.

[0046] Specific repair process of the cavity:

[0047] 1) First determine the hole area and the non-hole area in the image;

[0048] 2) Calculation priority: Calculation priority is used to determine the order of repairing different locations. In most cases, the edges of the hole areas with more known information and stronger structure have higher priority, which makes the repair area gradually spread from the edge of the hole to the center.

[0049] 3) Searching for the best matching block, that is, searching for the matching block closest to the block to be repaired in the known area of ​​the current image based on the similarity of structure and texture;

[0050] 4) Hole filling: copy the matching block to the block to be repaired and fill the hole area in the block with pixels.

[0051] Working principle: First, obtain a 2D image, then perform monocular depth estimation on the 2D image to obtain a relative depth map in the form of a grayscale image. To ensure the display effect, predetermine the maximum display depth Zmax and minimum display depth Zmin of the naked-eye 3D display system, and the maximum display depth Zmax is also called the maximum depth of field, and the minimum display depth Zmin is also called the minimum depth of field. According to the 2D image content, determine the vertical depth of the scene and the zero parallax plane, that is, the maximum display depth Z1max and the minimum display depth Z1min of the vertical depth, and Z1max≤Zmax||Z1min≥Zmin. Next, perform a range transformation on the depth map according to the range transformation formula, map the original depth S to the new depth S1 to adapt to the vertical depth range of the scene, and calculate the depth map of each 2D image according to the transformed depth map. The disparity of pixels is calculated, and then the image of the virtual viewpoint is rendered according to the disparity. When calculating the disparity, the original 2D image can be used as the view of the left eye, and then the right eye view is rendered according to the disparity value mapping; the original 2D image can also be used as the view of the right eye to render the left eye view; the original 2D image can even be used as the view between the two eyes, and then the left eye and right eye views are rendered respectively according to the disparity value. After the view rendering is completed, the Criminisi algorithm is used to fill the holes to eliminate the unmapped pixels caused by the "occlusion" phenomenon in the depth relationship. The texture synthesis method and the idea of ​​information diffusion are combined together. First, a suitable texture sample block is delineated, and then the hole area is filled and diffused in a predetermined order. The content not described in detail in this description belongs to the prior art known to professional and technical personnel in this field.

[0052] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting 2D images to 3D, characterized in that: The steps include: Step 1: Get a 2D image; Step 2: Perform monocular depth estimation on the 2D image to obtain a relative depth map; Step 3: Determine the vertical depth and zero parallax plane of the scene based on the 2D image content; Step 4: transform the depth map into a value range; Step 5: Calculate the disparity of each pixel of the 2D image according to the transformed depth map, and then render the image of the virtual viewpoint according to the disparity; Step 6: Repair the cavity.

2. The method for converting 2D images to 3D images according to claim 1, characterized in that: The 2D images in step 1 include 2D pictures and 2D videos. When processing the video, each frame of the image is still extracted. The images include various resolutions, such as 720p, 1080p, 2K, and 4K, and can be played on TVs, computer monitors, outdoor advertising screens, tablets and other mobile devices, and virtual image devices such as telescopic screens.

3. The method for converting 2D images to 3D images according to claim 1, characterized in that: The monocular depth estimation method in step 2 includes the traditional method of linear perspective principle and motion parallax, and also includes the current mainstream machine learning method based on deep learning and large depth data sets.

4. The method for converting 2D images to 3D images according to claim 3, characterized in that: The relative depth in step 2 is the relative distance between different scene elements in the image. The maximum depth is mapped to a grayscale value of 0, and the minimum depth is mapped to a grayscale value of 255, thereby obtaining a relative depth map in the form of a grayscale map.

5. The method for converting 2D images to 3D images according to claim 1, characterized in that: In step three, to ensure the display effect, the maximum display depth Zmax and the minimum display depth Zmin of the naked-eye 3D display system are predetermined, and the maximum display depth Zmax is also called the maximum depth of field, and the minimum display depth Zmin is also called the minimum depth of field.

6. The method for converting 2D images to 3D images according to claim 5, characterized in that: In the step three, the maximum display depth Z1max and the minimum display depth Z1min of the longitudinal depth are such that Z1max≤Zmax||Z1min≥Zmin. Z1max and Z1min are related to each scene, and can ensure the best display effect in different complex scenes. This step uses the absolute depth as the training data to obtain the model, and does not require high accuracy for Zmax and Zmin. A small amount of data can be used to train the absolute depth estimation model, and this model can be applied to generalize to the acquired 2D image to obtain Zmax and Zmin. Similar algorithms in the industry include: Metric3D, Arxiv: https: / / arxiv.org / abs / 2307.10984.

7. The method for converting 2D images to 3D images according to claim 1, characterized in that: The formula for range transformation in step 4 is: Where S is the original depth, ranging from 0 to 255; S1 is the depth after transformation according to the vertical depth of the scene. Each pixel of the original 2D image corresponds to an original depth S, which corresponds to the new depth S1 after the range transformation.

8. The method for converting 2D images to 3D images according to claim 1, characterized in that: In step five, when calculating the disparity, the original 2D image can be used as the view for the left eye, and then the right eye view is rendered according to the disparity value mapping; the original 2D image can also be used as the view for the right eye to render the left eye view; or even the original 2D image can be used as the view between the two eyes, and then the left eye and right eye views are rendered respectively according to the disparity value.

9. The method for converting 2D images to 3D images according to claim 1, characterized in that: In step six, the Criminisi algorithm is used to fill holes, combining the texture synthesis method with the idea of ​​information diffusion. First, a suitable texture sample block is delineated, and then the hole area is filled and diffused in a predetermined order. The relevant information of the Criminisi algorithm is Criminisi A, et al. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.

10. The method for converting 2D images to 3D images according to claim 9, characterized in that: The specific repair process in step 6 is as follows: 1) First determine the hole area and the non-hole area in the image; 2) Calculation priority: Calculation priority is used to determine the order of repairing different locations. In most cases, the edges of the hole areas with more known information and stronger structure have higher priority, which makes the repair area gradually spread from the edge of the hole to the center. 3) Searching for the best matching block, that is, searching for the matching block closest to the block to be repaired in the known area of ​​the current image based on the similarity of structure and texture; 4) Hole filling: copy the matching block to the block to be repaired and fill the hole area in the block with pixels.

Images