Multichannel high-quality depth estimation system

Abstract

A system and method for providing multi-channel high-quality depth estimation from monocular camera to provide augmented reality (AR) and virtual reality (VR) features to images. The invention also includes a method of enhancing generalization on a deployment-friendly monocular depth inference pipeline with semantic information. Furthermore, lively and complete reconstruction is ensured by inpainting missing depth and background within a single image input.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to monocular depth estimation. More particularly, the present invention relates to multi-channel high-quality depth estimation from a monocular camera to provide augmented reality (AR) and virtual reality (VR) features to images. BACKGROUND

[0002] In monocular depth estimation, depth can be obtained from the monoscopic spatiotemporal domain. The term "monoscopic" or "mono" is used to refer to the fact that the left eye and the right eye see the same perspective of a given scene. One known monocular depth estimation method is performed by extracting depth information from the extent of object motion, and is therefore referred to as a depth-from-motion method.

[0003] Objects with higher extent of motion are assigned smaller depth, and objects with lower extent of motion are assigned larger depth. Another conventional monocular depth estimation method is performed by assigning larger depth to non-focused regions such as background, and is therefore referred to as a depth-from-focus-cue method.

[0004] Another conventional monocular depth estimation method is performed by detecting the intersection of vanishing lines or vanishing points. Points close to the vanishing point are assigned larger (or farther) depth, and points far from the vanishing point are assigned smaller (or closer) depth. However, a drawback of conventional methods is that the information available from the monoscopic spatiotemporal domain is very limited. Thus, unfortunately, conventional methods are unable to resolve all scene content in real-world images.

[0005] U.S. Application 20100220893 assigned to Himax Technologies Ltd. discloses a system and method relating to monocular depth estimation. The invention provides a fundamental model method and system for monocular depth estimation that can provide correct and general depth and can handle relatively large (i.e., a large amount) of various scenes as long as a depth diffusion region (DDR) exists or can be identified. The technology described in the invention decomposes an image into different entities and assigns depth to each entity. The invention provides depth estimation for an image, however, what is lacking is providing texture, color, and depth information for occluded regions that can occur during the depth estimation process.

[0006] Another US patent 8665258B2 assigned to Adobe Systems Inc. generally relates to digital image processing and more particularly to generating depth maps from a single image. The technology disclosed in this patent is advanced over prior art by providing depth maps for image data that are automatically determined according to one or more depth constraints. Furthermore, the representation of the depth map can be displayed on a computer display. However, the prior art uses a single image to predict depth and does not take into account pre-stored images and internal parameters of the camera of the electronic device for recognizing objects and labeling them. Therefore, there is a high probability of leaving unlabeled recognized objects without introducing a depth mode.

[0007] Therefore, in order to overcome the aforementioned deficiencies in the prior art, the present invention proposes a high quality depth estimation system. The high quality depth estimation system of the present invention provides a depth estimation system for providing a three-dimensional reconstruction of an image before adding augmented reality features or virtual reality features to the image.

[0008] Therefore, it is now apparent that many methods and systems suitable for various purposes have been developed in the prior art. Furthermore, even though these inventions can be accordingly suitable for the specific purposes they are intended to solve, they are not suitable for the purposes of the present invention as previously mentioned. Therefore, there is a need for the high quality depth estimation system of the present invention, which provides a depth estimation system for providing a three-dimensional reconstruction of an image before adding AR features or VR features to the image. SUMMARY

[0009] Three-dimensional photography, which enables recording of a scene in a dynamic manner, is an intriguing feature compared to conventional three-dimensional images. These photos make the image more realistic by embedding depth in the image and presenting novel views with computed order numbers. When the user moves his portable device or simply clicks with a mouse, the three-dimensional effect appears before the user. On the basis of the three-dimensional image, augmented reality (AR) or virtual reality (VR) applications with lively multi-view presentation can be added.

[0010] The main purpose of the present invention is to introduce a system that requires a single image capturing device to produce three-dimensional reconstruction and other AR or VR visual features for any display terminal including PCs, mobile phones, and wearable devices. The present invention introduces a system that employs multiple perspective features to facilitate the generation of dense and pixel-level high quality depth estimation.

[0011] Another object of the present invention is to introduce a depth estimation module that uses a single image to predict depth. Different internal camera parameters result in different fields of view, distortions and image layouts. These variations pose additional challenges to a single network. During the training process, a batch of images along with the GT depth is introduced to the network to extend the data source range. Furthermore, an additional module is designed to handle the internal parameters of different camera types. Both the GT depth and the internal parameters undergo some normalization operations before being fed into the network. An edge alignment technique is introduced to remove any edge discontinuity and displacement that can occur during depth estimation.

[0012] The depth estimated by the monocular depth estimation module will inevitably contain unreasonable depth regions, such as non-uniform depth distribution within a uniform surface, loss of depth gradient within a depth gradual region. Another object of the present invention is to propose a panoramic segmentation unit for obtaining the semantic label of each entity in a given image. The dictionary outputs the depth pattern corresponding to each instance.

[0013] In the case of low confidence regions in the image, i.e. in the case of regions in the image for which it is not possible to generate a semantic label, the entire depth map is smoothed using the vanishing points, in particular by making these low confidence points maintain the correct ordinal order consistent with the vanishing points or depth layout.

[0014] Yet another object of the present invention is to perform a inpainting operation on the image after smoothing the labeled and unlabeled entities in the image. A semantic edge detector is described for detecting actual semantic edges including occlusions and requesting inpainting. The inpainting operation detects the occluded regions and adds color, texture and depth information to the occluded regions.

[0015] Other objects and aspects of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the features of embodiments according to the present invention.

[0016] To the accomplishment of the foregoing and related objects, the present invention can take the form of implementations illustrated in the appended drawings, it being understood that the drawings provide merely examples of the invention and they are not limiting of some of the inventive aspects to the specific arrangements and instrumentality shown.

[0017] While the application has been described above by reference to various example embodiments and implementations, it will be understood that the various features, aspects and functionality described in one or more embodiments are not limited in their applicability only to the particular embodiment described and / or illustrated. Rather, the above description is intended to cover all alternatives, modifications, and equivalents of the methods and compositions described herein, whether explicitly described or not. Further, the scope of the application is intended to cover all alternatives, modifications, and equivalents of the methods and compositions described herein, whether explicitly described or not. Thus, the breadth and scope of the present application should not be limited by any of the above-described example embodiments.

[0018] The appearance in the description and in the claims of words such as "one or more", "at least", "but not limited to", or other similar phrases does not foreclose the interchangeability of that term with respect to other terms in the description and claims. For example, "one or more" can be interpreted to be equivalent to at least one. BRIEF DESCRIPTION OF DRAWINGS

[0019] The objects and features of the present application will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only typical embodiments of the application and are not therefore to be considered to be limiting of its scope, the application will be described and explained with additional specificity and detail by the use of the accompanying drawings in which:

[0020] Figure 1A A system for providing monocular CNN based depth estimation according to the present application is shown;

[0021] Figure 1B A workflow representation of a depth estimation system according to the present application is shown;

[0022] Figure 2A An initial monocular depth estimation module within a depth estimation system according to the present application is shown;

[0023] Figure 2B A workflow representation of an initial monocular depth estimation module according to the present application is shown;

[0024] Figure 3A A depth map optimization module within a depth estimation system according to the present application is shown;

[0025] Figure 3B A workflow representation of optimization with a depth map optimization module according to the present application is shown;

[0026] Figure 4 A spatial layout pattern according to the present application is shown;

[0027] Figure 5 A semantic edge guided inpainting workflow according to the present application is shown; and

[0028] Figure 6 A method for providing monocular CNN based depth estimation according to the present application is shown. DETAILED DESCRIPTION

[0029] Three-dimensional image display has gradually become popular over time, and the requirement for three-dimensional image data has also gradually increased. However, most image data is still two-dimensional (2D) data. When converting original 2D images into three-dimensional images, image depth is an important parameter in the conversion. Image depth refers to the distance from the viewpoint to the object in the image, and can be used to generate two images that are slightly different from each other for the left eye and the right eye to watch, respectively, so that the user has the feeling of watching a three-dimensional image by taking advantage of the parallax between the eyes. However, real-time two-dimensional to three-dimensional image conversion is very difficult due to the very complex algorithm. Therefore, how to quickly obtain the parameters required by two-dimensional to three-dimensional image conversion, such as the depth value of each pixel in the image, is an important topic.

[0030] In conventional techniques, in some methods, depth value estimation is accelerated by using inter-frame information. For example, the depth value is estimated by using the object moving distance in consecutive frames of a video movie. However, when the scene or the main object in the image changes or moves quickly, it is difficult to accurately estimate the depth value of each pixel. In addition, when the front-back relationship of objects in the image is not clear, for example, when the front object does not exist to shield the rear object, it is also difficult to perform image depth estimation.

[0031] According to an embodiment of the present application, an input device provides or receives one or more two-dimensional (2D) input images to be processed. The input device can generally be an electro-optical device that maps a three-dimensional object onto a 2D image plane by perspective projection. In one embodiment, the input device can be a still camera that takes 2D images or a video camera that captures multiple image frames.

[0032] In another embodiment, the input device can be a pre-processing device that performs one or more digital image processing tasks such as image enhancement, image restoration, image analysis, image compression, or image synthesis. In addition, the input device can also include a storage device, such as a semiconductor memory or a hard disk drive, which stores the processed images from the pre-processing device.

[0033] Figure 1AA depth estimation system for captured images according to the present invention is shown. The depth estimation system 100 of the present invention comprises an initial monocular depth estimation module 200 which receives an image and improves the depth quality within the image by means of a predictor unit configured within the monocular depth estimation module. The predictor unit predicts the depth of the image based on a plurality of pre-stored images and a plurality of internal parameters of the camera of the electronic device importing the prediction pipeline. Different internal camera parameters result in different FOV, distortion and image layout. These variations pose additional challenges for a single network to handle.

[0034] An additional module is designed to handle the internal parameters of different camera types. Both GT depth and internal parameters are subjected to some normalization operations before being fed into the network. In this way, the inference system not only considers the image layout with RGB (Red Green Blue) input, but also the relative scale brought by the camera parameters involved.

[0035] The initial monocular depth estimation module 200 further comprises an edge alignment quality unit. The edge alignment quality unit removes multiple edge discontinuities in the depth by introducing a semantic head. This semantic head does not impose computational burden on the computational unit as it is discarded during inference.

[0036] Limited by the model regression capability and the availability of training data, the depth estimated by the monocular depth estimation module inevitably contains unreasonable depth regions, such as uneven depth distribution within a uniform surface, loss of depth gradient within a depth gradual region. In this regard, a depth map optimization module is proposed by the present invention for performing refinement on the aligned image.

[0037] The depth map optimization module 300 comprises a panoramic segmentation unit for obtaining semantic labels of each entity in the scene. With these labeled objects, the system is able to fit different depth pattern models for each class under the guidance of prior knowledge. For example, for the sky, the depth in the scene is usually the farthest, for the ground, gradually increasing depth is an appropriate pattern, and for humans, uniform depth is suitable in most cases.

[0038] The panoramic segmentation unit generates semantic labels for those objects with depth pattern models. A dictionary unit configured within the depth map optimization module outputs the corresponding depth pattern in each instance. For example, for the ground area, the depth should gradually increase from near to far, for example, the depth distribution should increase uniformly rather than suddenly.

[0039] Vanishing points in a scene often help to indicate trends in depth variation. In the depth map, there are still some points with low confidence about the depth value. The points often do not belong to any semantic label or are not recognized as any class. The depth layout module 400 includes one or more vanishing points to indicate the depth variation pattern to facilitate the depth map to provide one or more depth patterns for one or more un-recognized semantic labels in the segmented image.

[0040] The vanishing points or even the depth layout prediction (e.g., indoor scene depth layout following the Manhattan hypothesis) can be used to smooth the entire depth map by keeping these un-labeled points in the correct ordinal order of the VP or depth layout. This operation can be seen as a global smoothing of the depth map, while the semantic label optimization in the last section serves as a local smoothing operation.

[0041] Before rendering the three-dimensional image and AR features or VR features, a repair operation needs to be performed on the single image input, which is performed by the depth repair module 500. For occluded regions within the image, a repair operation is needed that can compensate for color, texture, and depth information. A semantic edge detector for performing the repair operation is proposed in the present invention.

[0042] The semantic edge pair detects the actual semantic edge including occlusion and requests repair. The output module 600 produces a three-dimensional scene reconstruction of the repaired image and adds a plurality of augmented reality features and virtual reality features to the repaired image to generate a three-dimensional image.

[0043] Figure 1B A depth estimation system according to the present invention is shown. The image at hand can be considered as a whole image 102 or can be divided into multiple frames 102. In the main implementation, the image is considered as a whole. As shown, the depth estimation module uses a single image to predict the depth. The result is used as the initial input for the subsequent optimization and repair pipeline. Therefore, the quality of the depth is crucial for the final rendering performance.

[0044] Generally speaking, in addition to reasonable depth distribution, there are two factors that are also important for quality and practical use. The edge alignment quality unit removes the edge discontinuity within the image. A semantic head is introduced that can predict tasks such as semantic masks, semantic edges, etc.

[0045] The depth estimated by the monocular depth estimation module 104 contains unreasonable depth regions, such as non-uniform depth distribution within a uniform surface and loss of depth gradient within a depth gradual region. In this regard, a panoramic segmentation unit configured within the depth map optimization module is proposed to obtain semantic labels for each entity in the image. The segmentation of the image into different entities by the segmentation module 118 is called optimization 110 based on semantic labels. The dictionary unit is responsible for defining depth patterns for different entities of the image to form a combined depth map 112.

[0046] In the case where there are entities in the image that cannot be labeled, also known as low-confidence regions, a vanishing point correction map consisting of multiple vanishing points 120 acts on the image and performs a global smoothing operation on the image, which also helps to provide depth estimation for the unlabeled entities. This includes combining the semantic correction map with the VP correction map as shown to generate a combined depth map.

[0047] On the other hand, if no low-confidence region 108 is identified in the image, the semantic correction map or the VP correction map 122 is sufficient to immediately provide depth estimation for the entire image. The semantic edge detector 126 detects actual semantic edges including occlusions and requests a repair operation to be performed on the image 128. Finally, the repaired image 124 is combined with the LDI 114 and the AR feature or the VR feature 116 is added to the image.

[0048] Figure 2A An initial monocular depth estimation module within the depth estimation system is shown. The initial monocular depth estimation module 200 includes a predictor unit 202 that uses a single image to predict depth. The predictor unit of the depth estimation module uses a single image to predict depth. The result will be used as the initial input for the subsequent optimization and repair pipeline.

[0049] Therefore, the quality of the depth is crucial to the final rendering performance. In general, in addition to a reasonable depth distribution, two other factors are also important to the quality and practical use. The two factors are (1) the generalization ability of various input images with different camera parameters, and (2) the edge alignment quality at the semantic level about the objects.

[0050] As for the generalization ability, it is known that different internal camera parameters will result in different FOV, distortion and image layout. These changes bring additional challenges for a single network to handle. Traditionally, fine-tuning is required for the target camera input image to address this issue. In the present invention, camera internal parameters such as focal length, principal centre are employed within the prediction pipeline. These parameters are imported as input together with a batch of other images and GT depth.

[0051] By mentioning edge-aligned quality, we indicate whether the depth strictly coincides with the actual object edges without any transition area. On the contrary, it is not reasonable if the depth has discontinuity at each edge area, because these edges can be context edges without depth jump, or actual object edges containing necessary depth discontinuity.

[0052] To handle this problem, a semantic head is proposed during training to take semantic objects into account. The semantic head is able to predict tasks such as semantic masks, semantic edges, etc. This additional head does not impose computational burden on the computing unit, because it will be discarded during inference. But the semantic information that the head has learned will be implicitly retained in the remaining pipeline.

[0053] Figure 2B An initial monocular depth estimation module according to the present application is shown. In the present embodiment of the present application, the input image 206 is divided into multiple frames 208. Different intrinsic camera parameters 210 will lead to different FOV, distortion and image layout. These variations bring additional challenges for a single network to handle. Traditionally, fine-tuning is needed for the target camera input image to solve this problem. In the present application, intrinsic parameters such as focal length, principal center are introduced within the prediction pipeline. These parameters are introduced as input.

[0054] During training, a batch of images and GT depth are introduced into the network to extend the data source range. In addition, additional modules are designed to handle the intrinsic parameters of different camera types. Both GT depth and intrinsic parameters undergo some normalization operations before being fed into the network. In this way, the inference system not only takes into account the image layout with RGB input, but also takes into account the relative proportions brought by the involved camera parameters.

[0055] The depth prediction by the depth estimation module can result in images with unreasonable depth areas, forming depth branches 212. A semantic head is introduced during training to take into account semantic objects forming semantic branches 214. The semantic head is able to predict tasks such as semantic masks, semantic edges, etc. This additional head does not impose computational burden on the computing unit, because it will be discarded during inference.

[0056] Figure 3AA depth map optimization module within the depth estimation system is shown. The depth map optimization module 300 includes a panoramic segmentation unit 302 and a dictionary unit 304. The panoramic segmentation unit 302 obtains semantic labels for each entity in the scene. With these labeled objects, the depth map optimization module is able to fit different depth pattern models for each class under the guidance of prior knowledge. For example, for sky, the depth in the scene is usually the farthest, for ground, gradually increasing depth is an appropriate pattern, and for human, uniform depth is suitable in most cases.

[0057] The panoramic segmentation unit 302 produces semantic labels for those objects with depth pattern models. Then, the dictionary unit 304 outputs the corresponding depth pattern within each instance. For those objects with depth variation patterns, parameters such as minimum-maximum depth, variance of the distribution can be determined under the guidance of the estimated depth map of the last module. For example, in the ground region, the depth should gradually increase from near to far, for example, the depth distribution should increase uniformly rather than suddenly. In this way, by considering the depth nature of each semantic class, a reasonable depth will be derived.

[0058] Figure 3B A depth map optimization module according to the present application is shown. Figure 3B The image 306 in FIG. 1 shows a street scene with different objects or entities such as road, crowd, sky, vehicles, etc. As shown, the sky location 308, ground location 318, and vehicle location 314 have been labeled specifically. The label-depth dictionary 310 provides depth patterns to the mentioned entities. As shown, the sky depth pattern 312, vehicle depth pattern 316, and ground depth pattern 320 have been assigned corresponding depth patterns by the label-depth dictionary. Figure 3B

[0059] Figure 4 A spatial layout pattern according to the present application is shown. The entire depth map can be smoothed using vanishing points or even depth layout prediction (e.g., indoor scene depth layout following Manhattan assumption) by keeping these unlabeled points in the correct ordinal order with the VP or depth layout. Some spatial typical layout patterns are shown in FIGS. 402a, 402b, 402c, and 402d.

[0060] Figure 5 A semantic edge guided inpainting workflow according to the present application is shown. The depth map 502 is generated by the depth estimation module and the depth map optimization module. The generated depth map will go through an edge preserving filter 504 before the depth discontinuity edges 506 can be determined. A semantic edge detector fusion scheme 508 addresses this issue. Semantic edges will facilitate detection of actual semantic edges 514 including occlusions and request inpainting. ​

[0061] Furthermore, false depth edges can be removed to improve the inpainting speed. Based on fused depth edge detection, an initial Layered Depth Image (LDI) 510 can be created, and inpainting can be performed based on local background 512 similarity or other learning-based methods.

[0062] Figure 6 A method for providing depth estimation based on a monocular CNN is illustrated. The method includes synchronously receiving images and improving the depth quality within the images via a depth estimation system 702. Then, the system's depth estimation module synchronously predicts the depth of the images using training data 704, which includes pre-stored images and multiple internal parameters of the camera device such as FOV, primality, etc. Next, a depth map optimization module of the depth estimation system, including a panoptic segmentation unit, introduces a semantic head to consider semantic objects 706. The semantic head is capable of predicting tasks such as semantic masks and semantic edges.

[0063] Furthermore, in the next step, edge discontinuities in the image are removed from the estimated depth to create an aligned image 708. Next, in this step, the panorama segmentation unit optimizes the aligned image by classifying the image into different entities such as sky locations, vehicle locations, etc., to form a segmented image 710. Then, in step 712, the dictionary unit of the depth map optimization module assigns a depth style to each entity in the image. Next, in step 714, a depth map is generated to provide the depth styles, thereby forming a labeled image. This depth map includes a VP correction map to prevent low-confidence regions from existing within the image even after different entities have been labeled.

[0064] In this step, after generating the depth map, occluded areas in the image lacking appropriate color, depth information, and texture are repaired to form a repaired image 716. Next, in step 718, a 3D scene is reconstructed from the repaired image. Finally, multiple augmented reality and virtual reality features are added to the repaired image to generate a 3D image 720.

[0065] While various embodiments of the present invention have been described above, it should be understood that these embodiments are presented by way of example only and are not intended to limit. Similarly, the accompanying drawings may depict examples of architectures or other configurations used in the present invention, and are depicted to aid in understanding the features and functions that may be included in the present invention. The present invention is not limited to the example architectures or configurations shown, but various alternative architectures and configurations can be used to implement the desired features.

[0066] While the application has been described above in terms of various exemplary embodiments and implementations, it is to be understood that the various features, aspects and functionality described in one or more of the various embodiments are not limited in their applicability only to the particular embodiment with which they are described, but are applicable to other embodiments and implementations of the application, all of which fall within the scope of the application. Accordingly, the breadth and scope of the application should not be limited by any of the above-described exemplary embodiments.

[0067] The appearance in the description and in the claims of words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases does not exclude the meaning of "only," "exactly," "consisting only of," or "consisting exactly of."

Claims

1. A multi-channel convolutional neural network (CNN) based depth estimation system for monocular camera, characterized in that, The depth estimation system comprises: a monocular depth estimation module, wherein the monocular depth estimation module comprises: a predictor unit for predicting depth in an image based on a stored image and one or more parameters of the monocular camera; and an edge alignment quality unit for removing edge discontinuities in the depth of the image to generate an aligned image; a depth map optimization module, wherein the depth map optimization module comprises: a panoramic segmentation unit for applying one or more semantic labels to one or more portions of the aligned image to generate a segmented image; and a dictionary unit for applying a depth pattern to each of one or more portions of the segmented image based on the one or more semantic labels to generate a processed image; a depth layout module, wherein the depth layout module facilitates the depth map optimization module to form a labeled image by providing a depth pattern to one or more unidentified semantic labels in the processed image; a depth inpainting module, wherein the depth inpainting module inpaints one or more occluded regions in the labeled image to generate an inpainted image; and an output module, wherein the output module adds a plurality of augmented reality features and virtual reality features to the inpainted image to produce a three-dimensional image.

2. The depth estimation system of claim 1, wherein, The one or more parameters comprise focal length, principal center within a prediction pipeline.

3. The depth estimation system of claim 2, wherein, The stored image and the one or more parameters are imported into the prediction pipeline.

4. The depth estimation system of claim 3, wherein, The stored image and the one or more parameters constitute training data.

5. The depth estimation system of claim 1, wherein, The edge alignment quality unit introduces semantic heads for removing a plurality of the edge discontinuities to create the aligned image.

6. The depth estimation system of claim 1, wherein, The panoramic segmentation unit generates the semantic labels for one or more portions having a depth pattern model.

7. The depth estimation system of claim 1, wherein, The depth map comprises one or more vanishing points for indicating a depth variation pattern.

8. The depth estimation system of claim 1, wherein, The one or more unidentified labels maintain ordinal order.

9. The depth estimation system of claim 1, wherein, The depth inpainting module comprises a semantic edge detector.

10. The depth estimation system of claim 9, wherein, The semantic edge detector detects a plurality of semantic edges.

11. The depth estimation system of claim 9, wherein, The semantic edge detector allows for increased inpainting speed.

12. The depth estimation system of claim 1, wherein, The depth map optimization module performs a local smoothing operation.

13. The depth estimation system of claim 1, wherein, The depth layout module performs a global smoothing operation.

14. The depth estimation system of claim 9, wherein, The semantic edge detector comprises a fusion mechanism for detecting the semantic edges based on occlusions.

15. A multi-pass convolutional neural network (CNN) based depth estimation method for monocular camera, characterized in that, The method comprises: receiving an image and improving depth quality within the image; predicting depth of the image based on a stored image and one or more parameters of the monocular camera; removing a plurality of edge discontinuities in the depth to create an aligned image by introducing semantic heads during training to account for one or more semantic objects; performing optimization of the aligned image to form a segmented image by obtaining one or more semantic labels for one or more portions of the aligned image; applying a depth pattern to each entity of the one or more portions of the aligned image based on the one or more semantic labels to generate a processed image; providing one or more depth patterns to one or more unidentified semantic labels in the processed image to form a labeled image; repairing one or more occluded regions in the labeled image to create a repaired image; reconstructing a three-dimensional scene from the repaired image; and adding a plurality of augmented reality features and virtual reality features to the repaired image to generate a three-dimensional image.

16. A depth estimation system for a monocular camera, the system comprising: The system comprises: a monocular depth estimation module, wherein the monocular depth estimation module comprises: a predictor unit for predicting depth of an image based on a stored image and one or more intrinsic parameters of the monocular camera, wherein the stored image and the one or more intrinsic parameters are imported into a prediction pipeline; and an edge alignment quality unit for removing edge discontinuities in the depth by introducing a semantic head to take into account semantic objects during training to generate an aligned image; a depth map optimization module, wherein the depth map optimization module comprises: a panoramic segmentation unit for obtaining one or more semantic labels for one or more portions of the aligned image to form a segmented image; and a dictionary unit for applying a depth pattern to each of the one or more portions in the segmented image based on the one or more semantic labels to generate a processed image; a depth layout module, wherein the depth layout module comprises one or more vanishing points to indicate depth variation patterns to facilitate the depth map optimization module to provide depth patterns to one or more unidentified semantic labels in the processed image, further wherein one or more unidentified labels maintain correct ordinal order to form a labeled image; a depth repair module, wherein the depth repair module comprises a semantic edge detector for repairing one or more occluded regions in the labeled image, creating a repaired image, and removing one or more false depth edges to improve repair speed; and an output module, wherein the output module produces a three-dimensional scene reconstruction of the repaired image, and adds a plurality of augmented reality features and virtual reality features to the repaired image to generate a three-dimensional image.

Images