System and method for depth perception by monocular camera using auxiliary information

Abstract

A depth map generation unit (DMGU) receives a focus indicator, an aperture indicator, and a zoom indicator from the image signal processor; specifying a depth hint attribute for each region according to at least one of the focus indicator, the aperture indicator, and the zoom indicator; receiving, from a depth hint extraction unit, extracted depth hint information including an edge and texture map, a blurred kernel map for each color channel, and a color difference map for each color channel other than a green channel; generating a sparse depth map according to the depth hint attribute, the edge and texture map, the blurred kernel map and the color difference map; and sending the sparse depth map to a monocular depth estimator.

Description

Technical Field

[0001] The present invention relates generally to image processing, and in a specific embodiment, to techniques and mechanisms for using auxiliary information for depth perception by a monocular camera. Background Art

[0002] Most mass-produced consumer electronic (CE) products with cameras, such as smartphones, tablets, and personal computers, use monocular two-dimensional (2D) video capture technology and do not have depth perception capabilities.

[0003] Therefore, in most video signals captured, transmitted or stored for purposes such as authentication, broadcasting, streaming media, video surveillance, video conferencing, etc., only 2D video can be obtained, but depth and three-dimensional (3D) information of the scene cannot be obtained.

[0004] Estimating depth from 2D videos or images plays a role in various applications, such as 3D object detection and tracking, depth perception processing, scene reconstruction, augmented reality, robotics, and autonomous driving.

[0005] As the computing power of processors in terminal devices and cloud computing continues to increase, there is a growing desire to use advanced algorithms to improve the depth estimation accuracy of monocular 2D cameras. Summary of the invention

[0006] The embodiments of the present invention describe techniques and algorithms for improving the depth estimation accuracy of a monocular 2D camera, and generally achieve technical advantages.

[0007] According to one embodiment, a depth map generation method is provided. The method includes: a depth map generation unit (DMGU) receives a focus indicator, an aperture indicator, and a zoom indicator from an image signal processor; the DMGU specifies a depth hint attribute of each region according to at least one of the focus indicator, the aperture indicator, or the zoom indicator; the DMGU receives extracted depth hint information from a depth hint extraction unit, wherein the extracted depth hint information includes an edge and texture map, a blur kernel map of each color channel, and a color difference map of each color channel except a green channel; the DMGU generates a sparse depth map according to the depth hint attribute, the edge and texture map, the blur kernel map, and the color difference map; and the DMGU sends the sparse depth map to a monocular depth estimator.

[0008] Optionally, in any of the above aspects, the sparse depth map is generated in an area having image features by analyzing the extracted depth hint information using the depth hint attributes, wherein the depth hint attributes correspond to each input video frame, and the image features include at least one of texture, edge or line.

[0009] Optionally, in any of the above aspects, the sparse depth map is sent to a two-dimensional / three-dimensional (2D / 3D) object detector.

[0010] Optionally, in any of the above aspects, the method also includes: the DMGU divides the extracted depth hint information into blocks, wherein the blocks are smaller parts of the edge and texture map, the blur kernel map and the color difference map suitable for detecting image features; the DMGU identifies each block including the image feature based on the edge and texture map; the DMGU specifies the depth hint attribute of each block including the image feature based on at least one of the focus indicator, the aperture indicator and the zoom indicator; the DMGU generates a sparse depth map value for each block including the image feature based on the depth hint attribute and the values ​​of the edge and texture map, the blur kernel map and the color difference map.

[0011] Optionally, in any of the above aspects, the sparse depth map is generated using a deep neural network (DNN) according to N-dim shortest distance matching or N-dim regression.

[0012] Optionally, in any of the above aspects, the method further includes: the monocular depth estimator generating an estimated depth map based on the sparse depth map, wherein the estimated depth map includes much less ambiguity and error than a depth map estimated without using the sparse depth map.

[0013] Optionally, in any of the above aspects, the method also includes: a two dimensional / three dimensional (2D / 3D) object detector generates a detected object label based on at least the sparse depth map, wherein the detected object label includes much less ambiguity and error than the object label detected without using the sparse depth map.

[0014] Optionally, in any of the above aspects, the method further includes: a depth completion processor generating a densified depth map based at least on the sparse depth map, wherein the densified depth map largely avoids depth mixing between objects and reduces depth smearing at object boundaries.

[0015] Optionally, in any of the above aspects, the method also includes: an image restoration processor generates a restored image based on at least one of the sparse depth map, the blur kernel map or the color difference map, wherein the restored image largely corrects the defocus properties, L-CA properties and A-CA properties of the processed image.

[0016] According to another embodiment, another depth perception method is provided. The method includes: a depth hint extraction unit receives a processed image signal and a depth hint indicator from an image signal processor, wherein the processed image signal is generated by the image signal processor according to the original image signal, the depth hint indicator includes a focus indicator, an aperture indicator and a zoom indicator, and the depth hint indicator is determined by the image signal processor according to the original image signal; the depth hint extraction unit detects edge and texture properties according to the processed image signal; the depth hint extraction unit generates an edge and texture map according to the edge and texture properties; the depth hint extraction unit estimates a lateral chromatic aberration (L-CA) property according to the processed image signal; the depth hint extraction unit generates a color difference map for each color channel except a green channel according to the L-CA property; the depth hint extraction unit estimates a defocus blur property and an axial chromatic aberration (A-C) property according to the processed image signal. aberration (A-CA) attribute; the depth hint extraction unit generates a blur kernel map for each color channel according to the defocus blur attribute and the A-CA attribute; the depth hint extraction unit sends the extracted depth hint information to the depth map generation unit, wherein the extracted depth hint information includes the edge and texture map, the color difference map of each color channel except the green channel, and the blur kernel map of each color channel.

[0017] Optionally, in any of the above aspects, the edge and texture attributes are detected by a canny edge detector or a structured edge predictor.

[0018] Optionally, in any of the above aspects, the defocus blur attribute and the A-CA attribute are estimated based on a blur step edge model or a Gaussian prior.

[0019] Optionally, in any of the above aspects, the L-CA property is estimated based on a chromaticity alignment metric or a color difference estimator.

[0020] According to yet another embodiment, another camera calibration method is provided. The method includes: a calibration controller generates a first indicator including a focus indicator, an aperture indicator, and a zoom indicator; the calibration controller sends the first indicator to a depth hint calibration unit; the calibration controller generates a depth signal, wherein the depth signal indicates a distance between a user's head or a calibration pattern and a monocular two-dimensional (2D) camera; and the calibration controller sends the depth signal to the depth hint calibration unit.

[0021] Optionally, in any of the above aspects, the depth signal is generated according to a predetermined process to traverse a target range of depth values, and also includes: the calibration controller sends the depth signal to the mobile controller, wherein the depth signal causes the mobile controller to reduce or increase the distance between the calibration pattern and the monocular 2D camera.

[0022] Optionally, in any of the above aspects, the depth signal is generated based on a first processed image signal received by the calibration controller from an image signal processor.

[0023] Optionally, in any of the above aspects, the depth signal is generated based on a pre-registered three-dimensional (3D) head model and the first processed image signal, wherein the pre-registered 3D head model includes the dimensions of the head and a set of 3D coordinates of selected key points of prominent facial features, and the prominent facial features have associated feature descriptors.

[0024] Optionally, in any of the above aspects, the depth signal is generated based on a pre-stored two-dimensional / three-dimensional (2D / 3D) model and the first processed image signal, wherein the pre-stored 2D / 3D model includes the dimensions of a calibration pattern and a set of 2D / 3D coordinates of selected key points of salient image features, and the salient image features have associated feature descriptors.

[0025] Optionally, in any of the above aspects, the depth signal is generated based on a predetermined generic head model and the first processed image signal, the predetermined generic head model including generic dimensions of a human head.

[0026] According to another embodiment, another camera calibration method is provided. The method includes: a depth hint calibration unit receives a first indicator including a focus indicator, an aperture indicator and a zoom indicator, a depth signal and a calibration process indicator from a calibration controller; the depth hint calibration unit receives extracted depth hint information from a depth hint extraction unit, wherein the depth hint information includes an edge and texture map, a blur kernel map of each color channel and a color difference map of each color channel except a green channel; the depth hint calibration unit generates a depth hint attribute of each region according to the extracted depth hint information; the depth hint calibration unit registers the generated depth hint attribute of each region under a corresponding depth value indicated by the depth signal according to at least one of the focus indicator, the aperture indicator and the zoom indicator; the depth hint calibration unit stores the registered depth hint attribute of each region under the depth value according to at least one of the focus indicator, the aperture indicator and the zoom indicator.

[0027] Optionally, in any of the above aspects, generating the depth cue attribute comprises: estimating multi-channel defocus blur and chromatic aberration for each image block having detected edges and textures.

[0028] Optionally, in any of the above aspects, registering the depth cue attributes comprises: registering multi-channel defocus blur and chromatic aberration for each image block having detected edges and textures according to the depth signal.

[0029] Optionally, in any of the above aspects, the multi-channel defocus blur and color difference are stored as r of a red, green, blue (RGB) camera. R 、r G 、r B 、x RG ,y RG 、x BG ,y BG , where r R is the estimated blur kernel radius value of the red channel, r G is the estimated blur kernel radius value of the green channel, r B is the estimated blur kernel radius value of the blue channel, x RG is the x component of the estimated color difference value of the red channel relative to the green channel, and y RG is the y component of the estimated color difference value of the red channel relative to the green channel, and x BG is the x component of the estimated color difference value of the blue channel relative to the green channel, and y BG is the y-component of the estimated color difference value of the blue channel relative to the green channel.

[0030] Optionally, in any of the above aspects, the multi-channel defocus blur and chromatic aberration are stored as r of a red, green, blue, infrared (RGB-IR) camera. R 、r G 、r B 、r IR 、x RG ,y RG 、x BG ,y BG 、x IRG ,y IRG , where r R is the estimated blur kernel radius value of the red channel, r G is the estimated blur kernel radius value of the green channel, r B is the estimated blur kernel radius value of the blue channel, r IR is the estimated blur kernel radius value of the infrared channel, x RG is the x component of the estimated color difference value of the red channel relative to the green channel, and y RG is the y component of the estimated color difference value of the red channel relative to the green channel, and x BG is the x component of the estimated color difference value of the blue channel relative to the green channel, and y BG is the y component of the estimated color difference value of the blue channel relative to the green channel, and x IRG is the x component of the estimated color difference value of the infrared channel relative to the green channel, and y IRG is the y-component of the estimated color difference value of the infrared channel relative to the green channel.

[0031] Optionally, in any of the above aspects, the registering of the depth-implied calibration unit comprises: in response to the calibration process indicator indicating that the calibration process is manual or temporary, registering the depth-implied attribute under the measured depth value.

[0032] Optionally, in any of the above aspects, the measured depth value is determined by the calibration controller by matching features in the user's moving head image with a pre-registered three dimensional (3D) head model of the user.

[0033] Optionally, in any of the above aspects, the measured depth value is determined by the calibration controller by matching features in an image of a predetermined calibration pattern with a pre-registered two dimensional / three dimensional (2D / 3D) model of the predetermined calibration pattern.

[0034] Optionally, in any of the above aspects, the measured depth value is determined by the calibration controller by measuring features in the user's moving head image according to predetermined common dimensions of a human head.

[0035] Optionally, in any of the above aspects, the registering of the depth hint calibration unit comprises: in response to the calibration process indicator indicating that the calibration process is automatic, registering the depth hint attribute at a specified depth value.

[0036] Optionally, in any of the above aspects, the specified depth value is determined by the calibration controller according to a predetermined process to traverse a target range of depth values.

[0037] Optionally, in any of the above aspects, the depth cue properties are registered and stored by calibrating depth cue samples, the calibration being performed by an automatic process, a manual process or an ad hoc process.

[0038] Optionally, in any of the above aspects, the depth cue properties are registered and stored by learning depth cue parameters, and the learning is performed by an automatic process, a manual process or an ad hoc process.

[0039] According to another embodiment, an image restoration method is provided. The method includes: an image restoration module (IRM) receives a sparse depth map, an input image, and extracted depth hint information, wherein the extracted depth hint information includes edge and texture maps, blur kernel maps for each color channel, and color difference maps for each color channel except a green channel; the IRM performs defocus deblurring and chromatic aberration correction on the input image according to the sparse depth map and the extracted depth hint information; the IRM generates a restored output image, wherein the restored output image is one of the images whose image quality is significantly improved compared to the input image; and the IRM sends the restored output image to a loss function calculation module.

[0040] According to another embodiment, a device for executing the above method or any one of the various aspects of the method is provided. The device includes: one or more processors; a non-transitory memory including instructions, wherein when the instructions are executed by the one or more processors, the device executes the above method or any one of the various aspects of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] For a more complete understanding of the present invention and its advantages, reference is made to the following description taken in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A schematic diagram of an exemplary data communications network is shown;

[0043] 2A and 2B illustrate the principles and limitations of depth from defocus blur;

[0044] 3A and 3B illustrate characteristics of exemplary color-coded apertures;

[0045] 4A and 4B illustrate characteristics of an exemplary color lens;

[0046] 5A and 5B illustrate the principles of axial chromatic aberration and lateral chromatic aberration;

[0047] Figure 6 A schematic diagram showing an exemplary depth perception system in a machine vision application disclosed herein;

[0048] Figure 7 A schematic diagram showing an exemplary depth perception system in a photography application disclosed herein;

[0049] Figure 8 A flow chart showing an exemplary system process for depth perception using auxiliary information disclosed herein;

[0050] Fig.9A A schematic diagram showing an exemplary method of an automatic calibration process for depth perception using auxiliary information disclosed herein;

[0051] Fig. 9B A schematic diagram showing an exemplary method of a manual calibration process for depth perception using auxiliary information disclosed herein;

[0052] Fig. 9C A schematic diagram showing an exemplary method of a temporary calibration process for depth perception using auxiliary information disclosed herein;

[0053] Fig. 10A A flowchart illustrating an exemplary method of a calibration process for depth perception using auxiliary information disclosed herein;

[0054] Fig. 10B A flowchart showing an exemplary method of a single image calibration process for depth perception using auxiliary information disclosed herein;

[0055] Fig.11 A schematic diagram of an exemplary processing system is shown;

[0056] Fig.12 A schematic diagram of an exemplary edge device is shown;

[0057] Fig.13 A block diagram of a computing system provided by some embodiments that can be used to implement the devices and methods disclosed herein is shown.

[0058] Unless otherwise indicated, corresponding numerals and symbols in the different figures generally refer to corresponding parts. These figures are drawn to clearly illustrate the relevant aspects of the embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0059] The making and use of embodiments of the present invention will be discussed in detail below. However, it should be understood that the concepts disclosed herein can be embodied in a variety of specific contexts, and the specific embodiments discussed herein are merely illustrative and are not intended to limit the scope of the claims. In addition, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims.

[0060] Figure 1A network environment 100 for transmitting data is shown. The network environment 100 includes a gateway 110, a plurality of edge devices 120, and a network 130. As shown, the gateway 110 establishes an uplink (dash-dotted line) connection and / or a downlink (dash-dotted line) connection with the edge device 120, which are used to transmit data from the edge device 120 to the gateway 110 and to transmit data from the gateway 110 to the edge device 120. The data transmitted through the uplink / downlink connection may include data transmitted between the edge devices 120 and data transmitted to / from a remote terminal (not shown) through the network 130. The gateway 110 and the network 130 may be used to provide the edge devices 120 with the ability to communicate directly with each other. The term "gateway" used in this article refers to any component (or set of components) for providing wireless or wired data communication network access, for example, enhanced base station (eNB), next generation NodeB (gNB), macro cell, femtocell, Wi-Fi access point (AP), network router / switch / bridge, broadband network gateway, residential gateway or other networking equipment. The gateway can provide wireless access according to one or more wireless communication protocols such as long term evolution (LTE), advanced LTE (LTE advanced, LTE-A), high speed packet access (High Speed ​​Packet Access, HSPA), Wi-Fi 802.11a / b / g / n / ac. The gateway can also provide wired access according to one or more wired communication protocols such as Ethernet, xDSL (digital subscriber line), fiber to the home (Fiber to the Home, FTTH), fiber to the curb (Fiber to the Curb, FTTC) and fiber to the node (Fiber to the Node, FTTN). The term "edge device" as used herein refers to any component (or collection of components) capable of establishing a wireless or wired data communication connection with the gateway 110, such as user equipment (UE), mobile station (STA), smart phone, tablet computer, personal computer, HDTV, connected camera and other networked devices. In some embodiments, the network environment 100 may include various other wireless or wired devices (not shown), such as repeaters, low-power nodes, relay stations, etc.The network 130 may include one or more of the Internet, a wide area network (WAN), a local area network (LAN), analog or digital wired and wireless telephone networks (e.g., cellular networks, 3G / 4G / 5G wireless networks, and digital subscriber lines (DSL)), radio, television, cable, satellite, or any other transmission or tunneling mechanism for transmitting data, etc. The network 130 may include multiple networks or subnets, each of which may include wireless or wired data channels, etc. The network 130 may include a circuit switching network, a packet switching data network, or any other network capable of transmitting electronic communications (e.g., data or video communications). For example, the network may include a network based on the Internet protocol (IP), asynchronous transfer mode (ATM), an IP-based packet switching network, or a frame relay or other similar technology, and may support video using motion JPEG, MPEG, H.264, H.265, or other similar protocols for video communications. The network may include one or more subnets, which include wireless data channels and wireless video channels. The network 130 may be a wireless network, a broadband network, or a combination of subnetworks including wireless networks and broadband networks. In some implementations, the network 130 may be accessed via a wireless communication link and / or a wired communication link. For example, a mobile computing edge device 120 such as a smartphone may access the network 130 using a cellular network. The edge device 120 may communicate with a local system and server 140 via a gateway 110 and / or communicate with a cloud-based system and server 150 via the network 130. The local system and server 140 may be public or private, supported by local services, applications, databases, storage, and backup. The cloud-based system and server 150 may also be public or private, supported by cloud-based services, applications, databases, storage, and backup.

[0061] Traditional monocular camera systems only support two-dimensional (2D) image and video capture. These systems are unable to capture the critical depth information required for accurate perception and realistic reconstruction of three-dimensional (3D) scenes. Monocular cameras lack the baseline capabilities required for the triangulation method used for depth perception in stereo systems. In addition, most monocular cameras are not equipped with special components, such as coded apertures or color lenses, which are typically required for accurate depth estimation. In addition, traditional monocular depth estimation systems only provide relative estimates and have a limited range. Therefore, these traditional systems can only provide hierarchical depth estimation at best, but not metric depth estimation, which is very useful in applications that require high accuracy and high reliability.

[0062] Some traditional monocular camera systems use methods such as defocus deblurring and chromatic aberration correction to process images. Defocus deblurring can remove blur artifacts, but at the same time loses auxiliary information about depth cues. Chromatic aberration correction can remove edge artifacts, but at the same time loses auxiliary information about depth cues.

[0063] In monocular 2D camera systems, existing video capture and processing cannot extract existing depth cue auxiliary information from defocus blur and color difference for depth perception. These systems also cannot seamlessly fuse existing depth cue auxiliary information to improve the accuracy of depth estimation or 2D / 3D object detection. However, by applying blur kernel estimation or regression, color difference estimation or regression, camera depth cue calibration, and depth cue analysis, monocular camera systems can achieve depth perception based on image features. By fusing auxiliary depth cue information, depth estimation and 2D / 3D object detection in monocular camera systems can achieve higher accuracy.

[0064] Several technical solutions can implement the embodiments disclosed herein. By extracting auxiliary information generated by defocus blur and chromatic aberration, the exemplary system and method can provide relevant depth hint information for depth perception and / or depth estimation of a monocular 2D camera. By calibrating or learning the depth hint samples or parameters of a monocular camera using an automatic, manual or temporary camera depth hint calibration process, a set of depth hint samples or parameters can be generated and stored. By analyzing the extracted depth hint information using this set of stored depth hint samples or parameters, a sparse depth map corresponding to each input video frame can be generated in an area with image features such as texture, edges and lines. By integrating the generated sparse depth map with each corresponding input video frame as a serial input, the accuracy of monocular depth estimation and 2D / 3D object detection can be significantly improved. By appropriately partitioning the video processing and machine learning algorithms in the video pipeline, the exemplary system and method disclosed herein can be executed on any of the hardware, firmware or software on the edge, core or cloud side of the communication network.

[0065] A system and method are described herein to support depth perception and improve depth estimation by using existing auxiliary information about depth cues from a monocular camera capturing a 2D video or image of a 3D scene.

[0066] One embodiment is based on a video capture and processing device or system with a monocular camera, which may or may not have an optical component for enhanced depth discrimination, using depth cue extraction, image processing and machine learning algorithms in hardware / firmware / software based on edge devices or cloud or both in a communication network.

[0067] The exemplary system has the following features. A monocular 2D camera and an image signal processor are included in a typical video capture and processing device. An edge detection module extracts edge information and generates an edge map corresponding to each input video frame. A blur kernel estimation module extracts depth hint information from defocus blur and axial chromatic aberration and generates a blur kernel map corresponding to each input video frame. A color difference estimation module extracts depth hint information from lateral chromatic aberration and generates a color difference map corresponding to each input video frame. A depth hint description module stores calibrated or learned depth hint samples or parameters of a monocular camera. A depth hint analysis module compares the extracted depth hint information with stored calibrated or learned depth hint attributes by matching or regression, and generates a sparse depth map corresponding to each input video frame. A monocular depth estimation module and / or a 2D / 3D object detection module generate a high-precision estimated depth map and / or object detection results for machine vision applications. An image restoration module and a depth map completion module generate a restored video frame with improved image quality and a corresponding high-precision densified depth map for photography applications.

[0068] Depending on the computing power of the device, data transmission bandwidth and network delay constraints, the above-mentioned system and method can be executed on the edge side or cloud side of the communication network, or shared by the edge side and the cloud side.

[0069] Direct advantages of the above system include the following. The above system and method improve the accuracy of monocular depth perception that has performance issues due to depth ambiguity, dead zones, and reduced light intensity by using existing depth cue auxiliary information. The above system and method improves the accuracy of monocular depth estimation and 2D / 3D object detection in machine vision applications that have performance issues due to depth metamerism and domain shift errors. The above system and method supports automatic, manual, or temporary camera depth cue calibration using predetermined test patterns or pre-stored 3D object models in the factory or in the field.

[0070] Depth perception is helpful for 3D vision. Measuring distance relative to the camera is still technically challenging, but it helps enable applications such as autonomous driving, 3D scene reconstruction, augmented reality, refocusing, depth-based rendering, segmentation, and compositing. In robotics, depth perception is a prerequisite for multiple tasks such as perception, navigation, and planning. There are many consumer, enterprise, and industrial applications for depth perception, with varying requirements for range and accuracy. To meet some of these requirements, many stereo depth perception techniques have been implemented and improved.

[0071] Existing binocular depth sensing methods have various inherent advantages and disadvantages, which are described below. Stereoscopic vision methods extract depth information by matching the same points in two images. This method can use a pattern projector to add texture, but there is an occlusion problem, that is, one camera can see an object or point, but the other camera cannot see it. Correspondence matching in this method is difficult due to texture, occlusion, transparent objects and reflective surfaces. Structured light methods project a light pattern onto an object and use the deformation of the pattern on the object to reconstruct the object shape. The light source can be laser, infrared or visible light. Structured light methods can achieve higher accuracy, but require calibration and the resolution is also limited by the light source. Structured light methods perform better in low light conditions than in sunlight, and depth accuracy decreases with distance. Time of flight methods measure the time it takes for transmitted light to reflect back from an object or point. The time can be measured directly or the phase difference between the transmitted and reflected light signals can be measured. The light source can be laser, infrared or visible light. Multipath reflections must be considered, that is, the transmitted light reflects back through multiple paths. Time of flight methods are generally less expensive and do not require calibration. The method performs better in low-light conditions than in sunlight, but the range is limited by the modulation frequency.

[0072] Complementing these depth perception technologies and methods are various sensor technologies that enable measurement and data extraction. Sensor technologies such as radar, lidar, and sonar with dedicated hardware have been used in various depth perception systems, but with limited spatial resolution. Sensor fusion is the current trend in autonomous vehicle perception technology. This technology attempts to leverage the strengths of various sensor types while overcoming the weaknesses of each sensor in specific applications.

[0073] Various Deep Neural Network (DNN) training methods are commonly used to estimate the depth of objects in images. Monocular self-supervision is a method that is favored due to its simple setup and abundant data. Supervised deep network training is a method that utilizes ground truth dense depth maps, which require data collected from real sample images or synthetically generated data. However, deep networks trained on semantic and contextual information can be easily fooled or misled, producing large absolute errors. Since the statistics of the captured scene and camera parameters are very different from those of the training set, such learning algorithms are prone to poor results, known as domain shift error. In this case, the estimated depth often becomes a rank rather than a metric.

[0074] Unsupervised deep network training is another DNN training approach. This approach can take advantage of stereo cameras without ground truth depth maps to avoid domain shift errors. Even better, unsupervised deep network training with a monocular camera without ground truth depth maps is challenging, as structure from motion (SfM), ego-motion, optical flow, pose estimation, and occlusion prediction are challenging. Depth metamerism is another difficult problem for depth estimation. Many 3D scenes have the same 2D projection on the image plane, so the estimated depth is not unique.

[0075] Monocular depth estimation systems can learn capabilities based on structured information such as linear perspective, occlusion, texture, object position, and statistics.

[0076] Depth perception can use other cues, such as defocus blur and color difference. For depth perception of a 3D scene, the subjective importance of each depth cue in the input video frame is a function of the distance between the viewer and the scene. Therefore, the main depth cue for depth perception depends on the scene itself and the viewing distance between the viewer and objects in the scene.

[0077] Figure 2A illustrates various aspects of determining the depth of defocus blur. Defocus blur is caused by objects at different depths being focused on different focal planes due to a limited depth-of-field (DoF).

[0078] The single image depth from defocus (SIDFD) algorithm has two limitations: depth ambiguity and dead zone as shown in Figure 2B. Depth ambiguity occurs when similar defocus blur is measured in front of or behind the focal plane. Dead zone occurs when there is no variation in the defocus blur measured within the DoF. However, there are two common approaches to mitigate these two limitations: color-coded apertures as shown in Figures 3A and 3B and color lenses as shown in Figures 4A and 4B. In an exemplary color-coded aperture, an annular color filter is placed in the lens aperture to induce defocus blur variations on different color channels, as shown in Figure 3A. Ideally, the color filter only attenuates the green channel and is transparent to other channels. The inner radius of the color filter acts as the effective aperture radius for the green channel, while the other channels are unaffected and imaged at the full aperture radius. In this way, the defocus blur kernel in the green channel is smaller for objects at any depth captured by the camera. Therefore, the image captured using this color-coded aperture has a spectrally varying depth-of-field (DoF), with the DoF in the green channel being larger than in the red and blue channels, as shown in Figure 3B, which enhances depth discrimination and helps to recover the scene depth computationally. In the color lens, the chromatic aberration of the chromatic aberration lens is enhanced by inserting a chromatic aberration doublet. The chromatic aberration doublet acts as a plane-parallel plate with zero optical power for the nominal wavelength in the visible spectrum and non-zero optical power for other wavelengths in the visible spectrum, as shown in Figure 4A, which enhances the chromatic aberration of the lens. The main advantage of the enhanced chromatic aberration caused by the color lens is that depth ambiguity and dead zones are avoided due to the separation of R, G, and B in the focal plane, as shown in Figure 4B. Both methods require special optical components to be installed in the lens system of the monocular camera, which are not available in most deployed monocular cameras.

[0079] Model-based techniques require calibration using a point source or pattern at each depth step for each focus setting. For overall depth discrimination, larger apertures perform better, and red, green, blue, infrared (RGB-IR) cameras generally perform better than red, green, blue (RGB) cameras. This is mainly because larger lens apertures or wider sensitive spectrums of monocular cameras generally result in more severe defocus blur and chromatic aberration, which enhances depth discrimination and helps to recover scene depth computationally at the expense of reduced image quality.

[0080] Figures 5A and 5B illustrate various aspects of determining depth based on chromatic aberration. Chromatic aberration is caused by the refractive index of a lens varying with wavelength, which prevents all colors from being focused at the same focal point.

[0081] Auxiliary depth cues include axial chromatic aberration (A-CA) and lateral chromatic aberration (L-CA). A-CA causes inconsistent defocus blur between component channels, which is more obvious in the area near the optical axis, as shown in Figure 5A. L-CA causes misalignment between component channels, which is more obvious in the area far from the optical axis, as shown in Figure 5B.

[0082] For overall depth discrimination, single lenses generally perform best. Red, green, blue, infrared (RGB-IR) cameras generally perform better than red, green, blue (RGB) cameras. For lens depth sensitivity, single lenses perform better than achromatic lenses, which perform better than apochromatic lenses, which perform better than superachromatic lenses. This is mainly due to the fact that single lenses generally have more severe chromatic aberration, which enhances depth discrimination and helps to recover scene depth computationally at the expense of image quality.

[0083] Figure 6 An exemplary embodiment of a monocular camera using auxiliary depth cue information for depth perception for machine vision applications is shown. The depth cue information includes a color difference map 616, a blur kernel map 618, and an edge and texture map 620. The image signal processor 604 processes the raw image captured by the image sensor in the monocular 2D camera 602 and generates a processed image 606. The depth cue extraction unit (DCEU) 608 extracts the depth cue information from the processed image 606 and generates an edge and texture map 620, a blur kernel map 618 (one for each color channel), and a color difference map 616 (one for each color channel except the green channel). The depth map generation unit (DMGU) 622 analyzes the extracted depth cue information (616, 618, and 620) using calibrated or learned depth cue attributes through matching or regression, and generates a sparse depth map 628. The monocular depth estimator 636 and the 2D / 3D object detector 634 receive as input the processed image 606 , the sparse depth map 628 , and the edge and texture map 620 , and generate inference results, an estimated depth map 640 , and detected object labels 638 .

[0084] Figure 7Another exemplary embodiment of a monocular camera using auxiliary information for depth perception for photography applications is shown. The image signal processor 704 generates a focus setting signal, an aperture setting signal, and a zoom setting signal for auto focus (AF) and auto exposure (AE). The depth hint description module 724 stores the depth hint samples or parameters of the monocular camera for each region, at each focus setting, aperture setting, and zoom setting, which are calibrated or learned through the camera depth hint calibration process. The depth completion module 734 generates a densified depth map 738 from the sparse depth map 728 according to the edge and texture map 720 and the processed image 706, while avoiding depth mixing between objects to reduce the depth smearing problem of the object boundary. The image restoration module 736 performs defocus deblurring and chromatic aberration correction according to the sparse depth map 728, the processed image 706, and the extracted depth hint information (716, 718, 720), and generates a restored output image 740 with improved image quality.

[0085] Figure 8 A flow chart of an exemplary system process 800 for depth perception using auxiliary information as described herein is shown. In step 802, (the image signal processor) processes the raw image captured by the monocular 2D camera; in step 804, (the image signal processor) generates focus control settings, aperture control settings, and zoom control settings for the monocular 2D camera. In step 806, edges and textures are detected; then, in step 808, edge and texture maps are generated. In step 810, L-CA is estimated; in step 812, a color difference map is generated. Then, in step 814, defocus blur and A-CA are estimated; in step 816, a blur kernel map is generated. All of these generated maps are extracted from the processed input image to characterize the auxiliary information therein that can be used for depth perception.

[0086] In one embodiment, in step 818, depth hint attributes corresponding to all supported depth values ​​are loaded for processing according to the generated focus settings, aperture settings, and zoom settings, wherein the depth hint attributes include pre-calibrated and pre-stored color difference and blur kernel map values. In another embodiment, as shown in step 818, depth hint attributes of all supported depth values ​​are loaded for processing according to the generated focus settings, aperture settings, and zoom settings, wherein the depth hint attributes include pre-learned and pre-stored depth regression network parameters. The processed input image is divided into blocks. Each block is a smaller portion of the processed input image that includes objects with approximately uniform depth (i.e., approximately uniform distance from the monocular 2D camera). Each block is then analyzed to estimate representative depth values ​​of all objects within the block. Each block obtained by segmentation may overlap or not overlap in shape with each adjacent block, and the size of all blocks obtained by segmentation on the processed image may be uniform or non-uniform. The blocks obtained by segmentation may use various shapes and sizes without departing from the spirit and scope of the present invention as defined by the appended claims. In step 820, all blocks in the processed input image are traversed to determine whether the block in the processed input image includes an edge or texture based on the generated edge and texture map in step 822. For each block in the processed input image that includes an edge and texture, in step 826, the extracted depth hint information (including the generated color difference and blur kernel map values) of each block is analyzed based on the extracted map to generate a depth map value for each block. In one embodiment, the analysis is performed by finding the best match among all loaded depth hint attributes corresponding to all supported depth values, or in another embodiment, the analysis is performed by a deep regression network parameterized by the loaded depth hint attributes. The generated depth map value is recorded in a sparse depth map for each block in the processed input image that includes an edge and texture. For blocks in the processed input image that do not include an edge or texture, no depth map value is generated, as shown in step 824. In step 828, a sparse depth map value is generated based on the edge and texture map and recorded in the sparse depth map. In step 830, monocular depth estimation is performed using the generated sparse depth map values; in step 834, 2D / 3D object detection is performed; in step 838, depth completion is performed; and in step 842, image restoration is performed. Through the sparse depth map values, in step 832, an estimated depth map with much less ambiguity and error is generated; in step 836, 2D / 3D object labels with much less ambiguity and error are detected; in step 840, a densified depth map with much less depth blending and smearing is generated; in step 844, a degraded image with largely corrected defocus blur and chromatic aberration is restored.

[0087] According to an embodiment, in Fig.9AIn the illustrated automatic calibration mode, the camera calibration controller 912 automatically assigns (focus, iris, zoom) indicator signals 936 to cycle through all supported settings and sends the indicator signals 936 to the image signal processor 908 and the depth hint calibration unit 930, which in turn controls the focus setting, iris setting, and zoom setting of the monocular camera 906. The camera calibration controller 912 also automatically assigns a depth signal 938 (scalar signal) to cycle through all supported depth values ​​and sends the assigned depth signal 938 to the depth hint calibration unit 930 and the motion controller 904, which moves the predetermined calibration pattern 902 to a specified depth value away from the monocular 2D camera 906. The image signal processor 908 then processes the raw image captured by the monocular 2D camera 906 and sends the processed image 914 to the depth cue extraction unit 916, which generates an edge and texture map 928, a color difference map 924 (one for each color channel except the green channel), and a blur kernel map 926 (one for each color channel). The depth cue generation module 934 then generates depth cue attributes on the blocks or regions in the processed image 914, wherein the edges and textures are detected based on the edge and texture map 928, and the depth cue description module 932 registers the depth cue attributes at the specified depth values ​​for all blocks or regions where the edges and textures are detected under the assigned focus setting, aperture setting, and zoom setting. In one embodiment, the depth cue attributes are registered as calibrated color difference and blur kernel map values ​​at the specified depth values ​​for all blocks or regions under the assigned focus setting, aperture setting, and zoom setting. In another embodiment, the depth hint attributes are registered as learning parameter values, e.g., weights and bias values, of a deep regression neural network at the assigned focus setting, aperture setting, and zoom setting, which is trained by inputting the generated color difference and blur kernel map values ​​and outputting the specified depth values ​​of all blocks or regions. The registered depth hint attributes are stored in the depth hint description module 932 of the depth hint calibration unit 930.

[0088] According to an embodiment, in Fig. 9BIn the manual calibration mode shown, the camera calibration controller 958 automatically assigns (focus, iris, zoom) indicator signals 936 to iterate through all supported settings and sends the indicator signals 936 to the image signal processor 954 and the depth cue calibration unit 980, which in turn controls the focus setting, iris setting, and zoom setting of the monocular camera 952. The image signal processor 954 then processes the raw image captured by the monocular 2D camera 952 and sends the processed image 960 to the depth cue extraction unit 962, which generates edge and texture maps 974, color difference maps 970 (one for each color channel except the green channel), and blur kernel maps 972 (one for each color channel). The processed image 960 is also sent to the camera calibration controller 958, which generates a measured depth signal 938 (a 2D array of measured depth values ​​corresponding to each block or region in the processed image, where each block or region is indexed by two independent indices i and j along the vertical and horizontal axes, respectively) corresponding to the processed image 960 by matching features in the captured head image of the user with the registered 3D head model 956 of the user. The measured depth signal 938 is then sent to the depth cue calibration unit 980. The depth cue generation module 978 then generates depth cue attributes on the blocks or regions in the processed image 960, where edges and textures are detected based on the edge and texture map 974, and the depth cue description module 976 is to register the depth cue attributes at the measured depth values ​​corresponding to each block or region where edges and textures are detected at the assigned focus setting, aperture setting, and zoom setting. In one embodiment, the depth cue attributes are registered as calibrated color difference and blur kernel map values ​​at measured depth values ​​corresponding to each block or region at assigned focus settings, aperture settings, and zoom settings. In another embodiment, the depth cue attributes are registered as learning parameter values, such as weights and bias values, of a deep regression neural network at assigned focus settings, aperture settings, and zoom settings, which is trained by inputting generated color difference and blur kernel map values ​​and outputting measured depth values ​​corresponding to each block or region. The registered depth cue attributes are stored in a depth cue description module 976 of a depth cue calibration unit 980. Alternatively, in some embodiments, manual calibration may be performed using a 2D pattern model instead of a 3D head model. In this case, the user manually moves a predetermined calibration pattern, such as Fig.9A 902 in FIG. 902, instead of moving the user's head in front of the monocular 2D camera 952. Then, the camera calibration controller 958 compares the features in the captured image with the registered 2D pattern model (rather than the 2D pattern model) of the predetermined calibration pattern. Fig. 9B The 3D head model 956 shown is matched to generate a depth signal 938.

[0089] exist Fig. 9C In the temporary calibration mode shown, everything is the same as Fig. 9B The camera calibration controller 959 performs the same calibration as in the manual calibration mode in FIG. 9 , except that the camera calibration controller 959 performs the calibration by using a common head size 957 (e.g., an average inter-pupillary distance (IPD) of approximately 63 mm) instead of using Fig. 9B The registered 3D head model 956 of the user shown is used to measure the captured head image of the user to generate a measured depth signal 938 corresponding to the processed image 961 (a 2D array of measured depth values ​​corresponding to each block or region on the processed image, where each block or region is indexed by two independent indices i and j along the vertical axis and the horizontal axis, respectively).

[0090] According to an embodiment, Fig. 10A The flowchart of the calibration process 1000 shown covers all three calibration modes. The only difference between the three modes is the two steps after the decision step 1014 "Automatic calibration, manual calibration or temporary calibration". In the automatic calibration mode, the following step 1016 "Register depth hint attributes at specified depth values" uses the depth values ​​automatically specified by the camera calibration controller (a scalar value for all blocks or regions in the processed image). In the manual calibration mode and the temporary calibration mode, the following step 1018 "Register depth hint attributes at measured depth values" uses the measured depth values ​​generated by the camera calibration controller (in a 2D array including measured depth values ​​corresponding to each block or region in the processed image, where each block and region is indexed by two independent indices i and j along the vertical axis and the horizontal axis, respectively).

[0091] Fig. 10A A flowchart 1000 of an exemplary method of a calibration process for depth perception using auxiliary information as described herein is shown. During the calibration process, all zoom settings, aperture settings, and focus settings are traversed, as shown in steps 1004, 1006, and 1008, respectively. At least one captured image is processed at each of the traversed zoom settings, aperture settings, and focus settings. While traversing each of the at least one captured image in step 1010, in step 1012, a single image calibration is performed by traversing all blocks in the captured image. For each of the at least one captured image, a sparse depth cue attribute is generated from an extracted color difference map and an extracted blur kernel map corresponding to each of the captured images by the single image calibration performed in step 1012. Fig. 10BA flowchart of detailed steps of the single image calibration performed in step 1012 is shown. The single image calibration performed in step 1012 generates sparse depth hint attributes of the captured images, and the following steps 1014, 1016 and 1018 are performed according to the calibration mode, which can be any of automatic calibration, manual calibration or temporary calibration. Under manual calibration and / or temporary calibration, in step 1018, the generated sparse depth hint attributes under the measured depth value of each traversed block in the captured image including detected edge or texture information are registered, as shown in the extracted edge and texture map corresponding to each image in at least one captured image.

[0092] In one embodiment, the generated sparse depth hint attributes are stored as calibrated color difference and blur kernel map values ​​at the measured depth values ​​corresponding to each block or region in the current captured image at the currently traversed focus setting, aperture setting, and zoom setting. In another embodiment, the generated sparse depth hint attributes are stored as learning parameter values, such as weights and bias values, of a deep regression neural network at the currently traversed focus setting, aperture setting, and zoom setting, which is trained by inputting the extracted color difference and blur kernel map values ​​and outputting the measured depth values ​​corresponding to each block or region in the current captured image. Then, after traversing each image in the at least one captured image, in step 1020, the calibrated or learned depth hint attributes stored at each measured depth value are updated and stored at the currently traversed focus setting, aperture setting, and zoom setting.

[0093] Under automatic calibration, in step 1016, the generated sparse depth hint attributes at the specified depth value of each traversed block in the captured image including the detected edge or texture information are registered. In one embodiment, the generated sparse depth hint attributes are registered as calibrated color difference and blur kernel map values ​​at the specified depth value of all blocks or regions in the current captured image at the currently traversed focus setting, aperture setting, and zoom setting. In another embodiment, the generated sparse depth hint attributes are registered as learning parameter values, such as weights and bias values, of a deep regression neural network at the currently traversed focus setting, aperture setting, and zoom setting, which is trained by inputting the extracted color difference and blur kernel map values ​​and outputting the specified depth value of each block or region in the current captured image. Then, after traversing each image in the at least one captured image, in step 1020, the calibrated or learned depth hint attributes registered at each specified depth value are updated and stored at the currently traversed focus setting, aperture setting, and zoom setting. In contrast, in manual calibration and temporary calibration, the measured depth value is registered, while in automatic calibration, the specified depth value is registered instead of the measured depth value.

[0094] According to an embodiment, in Fig. 10B In the detailed steps of single image calibration (e.g., Fig. 10A 1012 in ) as shown in flowchart 1050. In steps 1054 and 1056, (image signal processor) processes the raw image captured by the monocular 2D camera, and (camera calibration controller) generates focus control settings, aperture control settings, and zoom control settings for the monocular 2D camera. In step 1058, edges and textures are detected; then, in step 1060, edge and texture maps are generated. In step 1062, L-CA is estimated; in step 1064, a color difference map is generated. Then, in step 1066, defocus blur and A-CA are estimated; in step 1068, a blur kernel map is generated. All of these generated maps are extracted from the processed input image to characterize the auxiliary information therein that can be used for depth perception.

[0095] In step 1072, all blocks in the processed input image are traversed to determine whether the block in the processed input image includes an edge or texture based on the generated edge and texture map in step 1074. For blocks in the processed input image that do not include edges or textures, no depth hint attributes are generated, as shown in step 1076. For each block in the processed input image that includes edges and textures, in step 1078, depth hint attributes for each block are generated based on the extracted map, wherein the depth hint attributes include generated color difference and blur kernel map values ​​and correspond to the specified or measured depth value of the block. In step 1080, for each block in the processed input image that includes edges and textures, the generated sparse depth hint attributes are recorded in the sparse depth hint attribute storage based on the edge and texture map. Single image calibration is completed in step 1082. After the single image calibration performed in step 1012, Fig. 10A Steps 1014, 1016, 1018 and 1020 in can thereby retrieve the sparse depth hint attributes of the captured image from the sparse depth hint attribute storage.

[0096] Many algorithms for edge detection, defocus blur estimation, and chromaticity alignment estimation are implemented using traditional methods. Some of these algorithms include canny edge detector, structured edge prediction, defocus map estimation, defocus chromaticity depth, chromaticity alignment metric, and color difference estimation. The canny edge detector uses a multi-stage algorithm that uses Gaussian filtering, intensity gradients, and double thresholds with hysteresis to detect various edges. The implementation of the canny edge detector is very simple and the performance is quite good. The structured edge prediction algorithm is an efficient edge detector that uses structured labels to select split points in the tree to learn a structured random decision forest. The implementation of the structured edge prediction algorithm is very simple and produces high performance results. The defocus map estimation is effective for defocus blur with spatially varying edge locations in the image. The algorithm assumes a step edge model and uses a Gaussian kernel to estimate the defocus kernel radius by the gradient ratio of the input image and the re-blurred image. The implementation of the defocus map algorithm is very simple and the performance is quite good. The defocused chroma depth algorithm estimates depth locally using a point spread function (PSF) triplet (for RGB images) or quadruple (for RGB-IR images) selection criterion derived from a maximum likelihood calculation based on a Gaussian prior. The defocused chroma depth algorithm is more complex to implement but produces high performance results. For a given pixel neighborhood, the chroma alignment metric algorithm computes the product of the eigenvalues ​​of the local covariance matrix divided by the product of the matrix diagonal elements to determine the local chroma alignment. The implementation of the chroma alignment metric algorithm is very simple and produces average performance results. The color difference estimation algorithm iterates over all acceptable color differences and relative scales to find the color difference that minimizes the collinearity metric in clusters around all selected keypoints in the image. The color difference estimation algorithm is more complex to implement but produces high performance results.

[0097] In one embodiment, an optimization-based N-dimensional shortest distance search may be used. Fig. 10B During the calibration process shown in steps 1016 and 1018 of , the depth hint attributes can be stored as calibrated sample values ​​(i.e., blur kernel map values ​​and color difference map values) at each corresponding specified or measured depth value within the supported depth range of the current image region or block (indexed along the vertical axis and the horizontal axis respectively by two independent indices i and j). Fig. 10A In step 1020, the registered depth cue attributes are stored in a depth cue description module of the depth cue calibration unit, which are respectively Fig.9A , Fig. 9B and Fig. 9C Described in.

[0098] exist Figure 6 and Figure 7In the depth hint description module of the depth map generation unit shown in FIG. Figure 8 In the normal operation of steps 826 and 828, optimal output sparse depth map values ​​are determined. These optimal output sparse depth map values ​​are determined by searching all sample depth values ​​within the supported depth range of the current image region or block (indexed by i and j) at the current focus setting, aperture setting, and zoom setting to find the optimal output sparse depth map values ​​for which the calibrated sample blur kernel map values ​​and color difference map values ​​have the shortest multidimensional distance to the extracted blur kernel map values ​​and color difference map values.

[0099] For an RGB camera in an exemplary method, an N-dimensional vector w including the depth hint attribute value of the current image region or block indexed by (i, j) can be defined as w=(r R ,r G ,r B ,x RG ,y RG ,x BG ,y BG ), i.e., a vector including multi-channel defocus blur and chromatic aberration as its components, where N=7. For an RGB-IR camera in another exemplary method, the N-dimensional vector w can be defined in a similar manner as w=(r R ,r G ,r B ,r IR ,x RG ,y RG ,x BG ,y BG ,x IRG ,y IRG ), where N = 10. In each example, a continuous function d = g(w) with an N-dimensional input vector w and a scalar output value d can be defined to model the target depth map value d according to the depth hint attribute vector w corresponding to the block (i, j).

[0100] In the function g(w)=d with a large number of input-output pairs (w s ,d s ), the sample depth map value d within the supported depth range of all image regions or blocks indexed by (i, j) can be calculated by accumulating, counting and averaging during the calibration process of depth perception using auxiliary information. k The corresponding calibrated sample depth hint attribute value For each image region or patch indexed by (i,j), Where K is a known number predetermined for the calibration process, and E is a proximity function whose non-negative value is equal to 1 if its two scalar inputs have the same value and whose non-negative value decreases monotonically from 1 as the difference between the two scalar inputs increases from 0. After performing the calibration process, for each region or patch in the current image region or patch indexed by (i, j), there is a finite set of known calibrated sample depth hint properties (i.e., multi-channel defocus blur and chromatic aberration) Each of these attributes corresponds to a sample depth map value d1, d2, ..., d in the supported depth range. K Then, we can define the calibrated parameter tensor ψ * To include all calibrated sample depth-implied attributes and all corresponding sample depth map values ​​d1, d2, ..., d within the supported depth range of all image regions or blocks indexed by (i, j) K .

[0101] For the input-output pair (w s ,d s ) in each pair, Fig. 10B In step 1080, the N-dimensional input vector w including sparse depth hint attributes (i.e., multi-channel defocus blur and chromatic aberration) is obtained based on the edge and texture map of the block (i, j). s In the automatic calibration mode, Fig. 10A In the registering process in step 1016 (i.e., the calibration process of the exemplary system), the output value d s is assigned to the specified depth value. In manual calibration mode or temporary calibration mode, Fig. 10A In the registering process in step 1018 (i.e., the calibration process of the exemplary system), the output value d s is assigned as the measured depth value. The calibrated parameter tensor ψ is obtained by the calibration process under the current focus setting, aperture setting and zoom setting. * Afterwards, if Fig. 10A As shown in step 1020, the calibrated parameter tensor ψ is updated * And stored in the depth hint description module as a depth hint attribute.

[0102] In normal operation of an exemplary system using auxiliary information for depth perception, such as Figure 8 As shown in step 818, the calibrated parameter tensor ψ for each region or block indexed by (i, j) is loaded from the depth hint module. *As the depth hint attribute under the current focus setting, aperture setting and zoom setting. For each block (i, j) in the processed input image including edges and textures, the extracted depth hint information including the generated color difference and blur kernel map value is assigned to the N-dimensional input vector And by searching all sample depth values ​​d1, d2, ..., d within the supported depth range of the current image area or block under the current focus setting, aperture setting, and zoom setting K , analyze the extracted depth hint information according to the extracted image (such as Figure 8 826) to find the calibrated sample depth hint attribute value and The optimal output sparse depth map value with the shortest N-dimensional distance For each image region or patch indexed by (i,j), Wherein, D is an N-dimensional distance function, which is defined as the L1 norm D(x,y)=|xy| or the L2 norm D(x,y)=‖xy‖2 of its two N-dimensional input vectors. Without departing from the spirit and scope of the present invention, considering the prior distribution model of the input color image, D can also be defined as a generalized likelihood function of the first input vector under the observation model represented by the second input vector. In order to improve the estimation accuracy, interpolation or curve fitting techniques can be used to calculate the optimal output sparse depth map value from P adjacent sample depth values, whose calibrated sample blur kernel map values ​​and color difference map values ​​have P shortest N-dimensional distances with the extracted blur kernel map values ​​and color difference map values, where P is a small integer, for example, 2, 3 or 4. According to the edge and texture map, the estimated output sparse depth map value Recorded in the sparse depth map, such as Figure 8 As shown in step 828.

[0103] In another embodiment using learning-based M-dimensional regression and artificial neural network (ANN), Fig. 10B The depth hint attributes are stored as learning parameters (i.e., weights and bias values) during the calibration process shown in steps 1016 and 1018 of FIG. 1014 . The same scheme can also be implemented for a deep regression neural network trained by assigning the extracted blur kernel map values ​​and color difference map values ​​as inputs and the corresponding specified or measured depth values ​​as outputs for a current image region or block indexed by two independent indices i and j along the vertical and horizontal axes, respectively, at the current focus setting, aperture setting, and zoom setting. Fig. 10A In step 1020, the registered depth cue attributes are stored in a depth cue description module of the depth cue calibration unit, which are respectively Fig.9A , Fig. 9B and Fig. 9C Described in.

[0104] exist Figure 6 and Figure 7 In the depth hint description module of the depth map generation unit shown in FIG, the output sparse depth map value is generated by a trained deep regression neural network based on the edge and texture maps under the current focus setting, aperture setting and zoom setting with the extracted blur kernel map value and color difference map value as input (in Figure 8 during normal operation as shown in steps 826 and 828 of the present invention).

[0105] The basic principle of an exemplary method is based on the universal approximation theorem, which states that if sufficient resources such as weights, neurons, and layers are provided, ANNs are capable of representing various multi-dimensional functions. According to the universal approximation theorem, if sufficient resources are provided, ANNs can be assigned to closely approximate a continuous function d=f(v) with an M-dimensional input vector v and a scalar output value d to achieve a desired level of accuracy.

[0106] For an RGB camera in an exemplary method, the M-dimensional input vector v of the function f(v) can be defined as v=(r R ,r G ,r B ,x RG ,y RG ,x BG ,y BG ,i,j), i.e., a vector including multi-channel defocus blur and chromatic aberration as its components, where M=9 and (i,j) is the index of the corresponding block in the image. For an RGB-IR camera in another exemplary method, the M-dimensional input vector v of the function f(v) can be defined in a similar way as v=(r R ,r G ,r B ,r IR ,x RG ,y RG ,x BG ,y BG ,x IRG ,y IRG ,i,j), where M=12 and (i,j) is the index of the corresponding block in the image. In each example, the scalar output value d of the function f(v) is defined as the target depth map value corresponding to block (i,j).

[0107] The learned parameter vector θ consisting of the learned weights and biases of the ANN used to approximate the function f(v) *This can be obtained by training the ANN as a deep regression network during the calibration process of an exemplary system for depth perception using auxiliary information. s ,d s ), the calibration process trains the deep regression network by minimizing the total loss function: Among them, f θ is a deep regression network with parameter vector θ, and L is the loss function, which is usually defined as the L1 norm L(x,y)=|xy| or the L2 norm L(x,y)=(xy) of its two input scalar values. 2 Without departing from the spirit and scope of the present invention, other methods such as regularized lattice regression may also be used to approximate the continuous function d=f(v). In this case, the parameter vector θ * This now includes the optimal lookup table (LUT) values ​​for the approximation function f(v) and can be obtained by optimization in a manner similar to that used to train the deep regression network described here.

[0108] For the input-output pair (v s ,d s ), such that Fig. 10B As shown in step 1080 in FIG. 1 , the M-dimensional input vector v including sparse depth hint attributes (i.e., multi-channel defocus blur and chromatic aberration) is obtained based on the edge and texture maps of the block (i, j). s In the automatic calibration mode, Fig. 10A In the registering process shown in step 1016 (i.e., the training process for the exemplary system), the output value d s is assigned to the specified depth value. In manual calibration mode or temporary calibration mode, Fig. 10A In the registering process shown in step 1018 (i.e., the training process for the exemplary system), the output value d s is assigned as the measured depth value. The learned parameter vector θ is obtained through the training process under the current focus, aperture and zoom settings. * Afterwards, if Fig. 10A As shown in step 1020 in , the learned parameter vector is updated and stored in the depth hint description module as a depth hint attribute.

[0109] In normal operation of an exemplary system for depth perception using auxiliary information, the learned parameter vector θ is loaded from the depth cue specification module. * As a depth cue property at the current focus setting, aperture setting, and zoom setting, such as Figure 8As shown in step 818 in the above. For each block (i, j) including edges and textures in the processed input image, the extracted depth hint information including the generated color difference and blur kernel map value is assigned to the M-dimensional input vector And if Figure 8 As shown in step 826 in , according to the extracted graph, by applying the loaded depth hint attribute θ * Parameterized Deep Regression Network Analysis is performed to infer the depth map value for each block: like Figure 8 As shown in step 828 in FIG. 8 , the sparse depth map value is output according to the edge and texture map. Recorded in a sparse depth map.

[0110] exist Figure 8 The tiles traversed in the normal operation of step 820 may need to be dense enough to satisfy the assumption that the depth values ​​within each tile in the processed image are approximately uniform. This is because, in most common indoor or outdoor scenes captured by a typical 2D monocular camera, the depth values ​​of the foreground object boundaries in front of the background objects far away from the camera may change suddenly. Therefore, it is desirable to use dense tiles to enable the exemplary method to generate accurate sparse depth values ​​of the sparse depth map of the foreground object boundaries, such as Figure 8 As shown in step 828.

[0111] exist Fig. 10B The blocks traversed during the calibration process of step 1072 can be coarse or downsampled to reduce the total number of blocks traversed during the calibration process and reduce the execution time and memory size required for the calibration process. This is because, for most typical 2D monocular cameras using the exemplary method, the relationship between the depth hint information (i.e., texture and edge maps, blur kernel maps, and color difference maps) and the estimated depth value is usually a smooth gradient with the change of two independent indexes i and j along the vertical axis and the horizontal axis. Therefore, it is desirable to traverse all coarser blocks, or only traverse dense blocks that have been subsampled, so the exemplary method can generate accurate sparse depth hint attributes to store sparse depth hint attributes, such as Fig. 10B As shown in step 1080.

[0112] For the optimization-based method, the use of coarser or downsampled blocks indexed by (i,j) during the calibration process does not affect the exemplary method for calibrating sample depth hint attribute values, while for dense blocks indexed by (i',j') used in normal operation, the sample depth hint attribute values ​​used to perform the N-dimensional shortest distance search should first be interpolated based on spatially neighboring calibrated sample depth hint attribute values. For example, assume that the (i',j') coordinates are D times denser than the (i,j) coordinates along the vertical and horizontal axes. For dense blocks indexed by (i',j'), each value in the sample depth hint attribute value to be searched should first be interpolated based on spatially neighboring calibrated sample depth hint attribute values. and The calibrated sample depth hint attribute values ​​of four spatially adjacent coarser or downsampled blocks indexed are bilinearly interpolated, where is the floor function representing the largest integer not greater than x, is the ceiling function representing the smallest integer not less than x.

[0113] For learning-based methods, the use of coarser or downsampled patches indexed by (i, j) during the calibration process does not affect the exemplary method for training the deep regression network or the grid regression LUT, while for dense patches indexed by (i', j') used in normal operation, their indexes should first be converted to (i, j) coordinates used in the calibration process and then sent to the trained deep regression network or the trained grid regression LUT to generate inferred depth map values. For example, assume that the (i', j') coordinates are D times denser than the (i, j) coordinates along the vertical and horizontal axes. For dense patches indexed by (i', j'), the index values ​​should first be converted to (i' / D, j' / D) including decimals before generating inferred depth map values.

[0114] Fig.11 A block diagram of an exemplary processing system 1100 for executing the methods described herein is shown, and the processing system 1100 may be installed in a host device. As shown in the figure, the processing system 1100 includes a processor 1104, a memory 1106, and interfaces 1110 to 1114, which may (or may not) be as described in the embodiment of the present invention. Fig.11Arranged as shown. Processor 1104 can be any component or collection of components for performing calculations and / or other processing-related tasks, and memory 1106 can be any component or collection of components for storing programs and / or instructions for execution by processor 1104. In one embodiment, memory 1106 includes a non-transitory computer-readable medium. Interfaces 1110, 1112, and 1114 can be any component or collection of components that supports processing system 1100 to communicate with other devices / components and / or users. For example, one or more of interfaces 1110, 1112, and 1114 can be used to send data, control, or management messages from processor 1104 to applications installed on a host device and / or a remote device. For another example, one or more of interfaces 1110, 1112, and 1114 can be used to support users or user devices (e.g., personal computers (PCs), etc.) to interact / communicate with processing system 1100. Processing system 1100 may include Fig.11 Other components not shown, such as long term storage (eg, non-volatile memory, etc.).

[0115] In some embodiments, the processing system 1100 is included in a network device that accesses a telecommunications network or otherwise becomes part of a telecommunications network. In one example, the processing system 1100 is in a network-side device in a wireless or wired telecommunications network, such as a base station, a relay station, a scheduler, a controller, a gateway, a router, an application server, an edge device, or any other device in the telecommunications network. In other embodiments, the processing system 1100 is in a user-side device that accesses a wireless or wired telecommunications network, such as a mobile station, user equipment (UE), a personal computer (PC), a tablet computer, a smartphone, a wearable communication device (e.g., a smart watch, etc.), or any other device for accessing a telecommunications network. In some embodiments, part or all of the processing system 1100 can be hosted on a local device, such as an edge device 120, one or more local servers 140, one or more cloud-based servers 150, or a combination thereof, through the gateway 110 and the network 130. Figure 1 In some embodiments, the processing system 1100 may be hosted on a cloud-based server 150, wherein the edge device 120 may communicate with the processing system 1100 via the gateway 110 and the network 130, as shown. Figure 1 shown.

[0116] In some embodiments, one or more of interfaces 1110, 1112, 1114 connect processing system 1100 to a transceiver for sending and receiving signaling over a telecommunications network. Fig.12A block diagram of a transceiver 1200 for sending and receiving signaling through a telecommunications network is shown. The transceiver 1200 can be installed in a host device. As shown, the transceiver 1200 includes a network side interface 1202, a coupler 1204, a transmitter 1206, a receiver 1208, a signal processor 1210, and a device side interface 1212. The network side interface 1202 may include any component or component set for sending or receiving signaling through a wireless or wired telecommunications network. The coupler 1204 may include any component or component set for facilitating bidirectional communication through the network side interface 1202. The transmitter 1206 may include any component or component set (e.g., upconverter, power amplifier, etc.) for converting a baseband signal into a modulated carrier signal for sending through the network side interface 1202. The receiver 1208 may include any component or component set (e.g., downconverter, low noise amplifier, etc.) for converting a carrier signal received through the network side interface 1202 into a baseband signal. The signal processor 1210 may include any component or set of components for converting baseband signals into data signals for communication through the device-side interface 1212 or converting data signals into baseband signals. The one or more device-side interfaces 1212 may include any component or set of components for transmitting data signals between the signal processor 1210 and components within a host device (e.g., the processing system 1100, a local area network (LAN) port, etc.).

[0117] The transceiver 1200 can send and receive signaling through any type of communication medium. In some embodiments, the transceiver 1200 sends and receives signaling through a wireless medium. For example, the transceiver 1200 can be a wireless transceiver for communicating according to a wireless telecommunication protocol such as a cellular protocol (e.g., long-term evolution (LTE), etc.), a wireless local area network (WLAN) protocol (e.g., Wi-Fi, etc.) or any other type of wireless protocol (e.g., Bluetooth, near field communication (NFC), etc.). In these embodiments, the network side interface 1202 includes one or more antenna / radiation units. For example, the network side interface 1202 may include a single antenna, multiple separate antennas, or a multi-antenna array for multi-layer communications such as single input multiple output (SIMO), multiple input single output (MISO), multiple input multiple output (MIMO), etc. In other embodiments, the transceiver 1200 sends and receives signaling through a wired medium such as a twisted pair cable, a coaxial cable, or an optical fiber. A particular processing system and / or transceiver may use all or only a subset of the components shown, and the level of integration may vary from device to device.

[0118] Fig.13 1300 is a block diagram of a computing system 1300 that can be used to implement the devices and methods disclosed herein, provided by some embodiments. A particular device may use all of the components shown or only a subset of the components, and the level of integration may vary from device to device. In addition, a device may include multiple instances of a component, for example, multiple processing units, multiple processors, multiple memories, multiple transmitters, multiple receivers, multiple interfaces, multiple adapters, etc. The computing system 1300 includes a processing unit 1302. The processing unit 1302 is equipped with one or more input / output (I / O) devices 1316 (e.g., speakers, microphones, touch screens, keypads, keyboards, and / or cameras) and a display 1318. The processing unit 1302 includes a central processing unit (CPU) 1314, a memory 1308, and may also include a mass storage device 1304, a video adapter 1310, and one or more I / O interfaces 1312, all of which are connected to a bus 1320.

[0119] Bus 1320 may be one or more of any type of several bus architectures, including a storage bus or storage controller, a peripheral bus, or a video bus. CPU 1314 may include any type of electronic data processor. Memory 1308 may include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In one embodiment, memory 1308 may include a ROM for use at startup and a DRAM for storing programs and data used when executing a program.

[0120] Mass storage 1304 may include any type of non-transitory storage device for storing data, programs, and other information and making the data, programs, and other information accessible via bus 1320. For example, mass storage 1304 may include one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, or an optical disk drive.

[0121] The video adapter 1310 and one or more I / O interfaces 1312 provide interfaces for coupling external input and output devices to the processing unit 1302. As shown, examples of input and output devices include a display 1318 coupled to the video adapter 1310 and an I / O device 1316 (e.g., a speaker, microphone, mouse, touch screen, keypad, keyboard, and / or camera) coupled to one or more I / O interfaces 1312. Other devices may be coupled to the processing unit 1302, and additional or fewer interface cards may be used. For example, a serial interface such as a Universal Serial Bus (USB) (not shown) may be used to provide an interface for a camera.

[0122] The processing unit 1302 also includes one or more network interfaces 1306, which may include wired links (e.g., Ethernet cables) or wireless links to access nodes or different networks 1322. The network interface 1306 enables the processing unit 1302 to communicate with remote units through the network 1322 (e.g., a video conferencing network or a remote video monitoring network). For example, the network interface 1306 can provide wireless communication through one or more transmitters / transmit antennas and one or more receivers / receive antennas. In one embodiment, the processing unit 1302 is coupled to a local area network 1322 or a wide area network 1322 for data processing and communication with remote devices such as other processing units, the Internet, or a remote storage facility.

[0123] It should be understood that one or more steps of the embodiment method provided herein can be performed by corresponding units or modules. For example, a signal can be generated by a generating unit or a generating module. The signal can be received by a receiving unit or a receiving module. The signal can be processed by a processing unit or a processing module. Other steps can be performed by an image signal processor unit / module, a depth map generation unit (depth map generation unit, DMGU) / module, a depth completion module (depth completion module, DCM) unit / module, a depth estimation unit / module, an object detection unit / module... and / or an image restoration unit / module. The corresponding unit / module can be hardware, software or a combination thereof. For example, one or more units or modules can be an integrated circuit, such as a field programmable gate array (field programmable gate array, FPGA) or an application-specific integrated circuit (application-specific integrated circuit, ASIC).

[0124] Although the present specification has been described in detail, it should be understood that various changes, substitutions and modifications may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims. In addition, the scope of the present invention is not limited to the specific embodiments described herein, as one of ordinary skill in the art will readily appreciate from this disclosure that processes, machines, manufactures, compositions of matter, components, methods or steps currently existing or later developed may perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein. Therefore, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, components, methods or steps.

[0125] Although the present invention has been described with reference to illustrative embodiments, this description is not to be interpreted in a limiting sense. With reference to this document, those skilled in the art will clearly understand various modifications and combinations of the illustrative embodiments and other embodiments of the present invention. Therefore, the appended claims cover any such modifications or embodiments.

Claims

1. A depth map generation method, characterized in that: include: A depth map generation unit (DMGU) receives a focus indicator, an aperture indicator, and a zoom indicator from an image signal processor; The DMGU specifies a depth-implied attribute for each region based on at least one of the focus indicator, the aperture indicator, or the zoom indicator; The DMGU receives extracted depth cue information from a depth cue extraction unit, wherein the extracted depth cue information includes an edge and texture map, a blur kernel map for each color channel, and a color difference map for each color channel except a green channel; The DMGU generates a sparse depth map based on the depth hint attribute, the edge and texture map, the blur kernel map and the color difference map; The DMGU sends the sparse depth map to a monocular depth estimator.

2. The method according to claim 1, characterized in that The sparse depth map is generated by analyzing the extracted depth cue information in an area having image features using the depth cue attributes corresponding to each input video frame, the image features including at least one of texture, edge or line.

3. The method according to claim 1 or 2, characterized in that: The sparse depth map is sent to a two dimensional / three dimensional (2D / 3D) object detector.

4. The method according to any one of claims 1 to 3, characterized in that Also includes: The DMGU divides the extracted depth hint information into blocks, wherein the blocks are smaller portions of the edge and texture map, the blur kernel map, and the color difference map suitable for detecting image features; The DMGU identifies each block including the image feature according to the edge and texture map; The DMGU specifies a depth cue attribute for each block including the image feature based on at least one of the focus indicator, the aperture indicator, and the zoom indicator; The DMGU generates a sparse depth map value for each block including the image feature according to the depth hint attribute and the values ​​of the edge and texture map, the blur kernel map and the color difference map.

5. The method according to any one of claims 1 to 4, characterized in that The sparse depth map is generated using a deep neural network (DNN) according to N-dim shortest distance matching or N-dim regression.

6. The method according to any one of claims 1 to 5, characterized in that Also includes: The monocular depth estimator generates an estimated depth map based on the sparse depth map, wherein the estimated depth map includes much less ambiguity and error than a depth map estimated without using the sparse depth map.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: A two dimensional / three dimensional (2D / 3D) object detector generates detected object labels based on at least the sparse depth map, wherein the detected object labels include significantly less ambiguity and error than object labels detected without using the sparse depth map.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: The depth completion processor generates a densified depth map based on at least the sparse depth map, wherein the densified depth map largely avoids depth mixing between objects and reduces depth smearing at object boundaries.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: An image restoration processor generates a restored image based on at least one of the sparse depth map, the blur kernel map, or the color difference map, wherein the restored image largely corrects the defocus property, the L-CA property, and the A-CA property of the processed image.

10. A depth perception method, characterized in that: include: The depth cue extraction unit receives a processed image signal and a depth cue indicator from an image signal processor, wherein the processed image signal is generated by the image signal processor according to the original image signal, the depth cue indicator comprises a focus indicator, an aperture indicator and a zoom indicator, and the depth cue indicator is determined by the image signal processor according to the original image signal; The depth cue extraction unit detects edge and texture attributes based on the processed image signal; The depth cue extraction unit generates an edge and texture map based on the edge and texture attributes; The depth cue extraction unit estimates a lateral chromatic aberration (L-CA) property based on the processed image signal; The depth cue extraction unit generates a color difference map for each color channel except the green channel according to the L-CA attribute; The depth cue extraction unit estimates a defocus blur property and an axial chromatic aberration (A-CA) property according to the processed image signal; The depth cue extraction unit generates a blur kernel map for each color channel according to the defocus blur attribute and the A-CA attribute; The depth hint extraction unit sends the extracted depth hint information to the depth map generation unit, wherein the extracted depth hint information includes the edge and texture map, the color difference map of each color channel except the green channel, and the blur kernel map of each color channel.

11. The method according to claim 10, characterized in that The edge and texture properties are detected by a canny edge detector or a structural edge predictor.

12. The method according to claim 10 or 11, characterized in that: The defocus blur property and the A-CA property are estimated based on a blur step edge model or a Gaussian prior.

13. The method according to any one of claims 10 to 12, characterized in that The L-CA property is estimated based on a chromaticity alignment metric or a color difference estimator.

14. A camera calibration method, characterized in that: include: The calibration controller generates a first indicator including a focus indicator, an aperture indicator, and a zoom indicator; The calibration controller sends the first indicator to a depth hint calibration unit; The calibration controller generates a depth signal, wherein the depth signal indicates a distance between a user's head or a calibration pattern and a monocular two-dimensional (2D) camera; The calibration controller sends the depth signal to the depth hint calibration unit.

15. The method according to claim 14, characterized in that The depth signal is generated according to a predetermined process to traverse a target range of depth values, and the method further comprises: The calibration controller sends the depth signal to the mobile controller, wherein the depth signal causes the mobile controller to reduce or increase the distance between the calibration pattern and the monocular 2D camera.

16. The method according to claim 14 or 15, characterized in that The depth signal is generated based on a first processed image signal received by the calibration controller from an image signal processor.

17. The method according to claim 16, characterized in that The depth signal is generated based on a pre-registered three dimensional (3D) head model and the first processed image signal, the pre-registered 3D head model including head dimensions and a set of 3D coordinates of selected key points of salient facial features having associated feature descriptors.

18. The method according to claim 16, characterized in that The depth signal is generated based on a pre-registered two dimensional / three dimensional (2D / 3D) model and the first processed image signal, wherein the pre-registered 2D / 3D model includes the dimensions of a calibration pattern and a set of 2D / 3D coordinates of selected key points of salient image features, wherein the salient image features have associated feature descriptors.

19. The method according to claim 16, characterized in that The depth signal is generated based on a predetermined generic head model and the first processed image signal, the predetermined generic head model comprising generic dimensions of a human head.

20. A camera calibration method, characterized in that: include: The depth cue calibration unit receives a first indicator including a focus indicator, an aperture indicator, and a zoom indicator, a depth signal, and a calibration process indicator from a calibration controller; The depth cue calibration unit receives the extracted depth cue information from the depth cue extraction unit, wherein the extracted depth cue information includes an edge and texture map, a blur kernel map for each color channel, and a color difference map for each color channel except a green channel; The depth cue calibration unit generates a depth cue attribute for each region according to the extracted depth cue information; The depth cue calibration unit registers the generated depth cue attribute at a corresponding depth value indicated by the depth signal for each region according to at least one of the focus indicator, the aperture indicator, and the zoom indicator; The depth cue calibration unit stores the registered depth cue properties for each region at the depth value according to at least one of the focus indicator, the aperture indicator, and the zoom indicator.

21. The method according to claim 20, characterized in that Generating the depth-hinted attributes includes estimating multi-channel defocus blur and chromatic aberration for each image block having detected edges and textures.

22. The method according to claim 20 or 21, characterized in that The registering the depth-hinted attributes includes registering multi-channel defocus blur and chromatic aberration for each image block having detected edges and textures according to the depth signal.

23. The method according to claim 22, characterized in that The multi-channel defocus blur and color difference are stored as r of the red, green, blue (RGB) camera R 、r G 、r B 、x RG ,y RG 、x BG ,y BG , where r R is the estimated blur kernel radius value of the red channel, r G is the estimated blur kernel radius value of the green channel, r B is the estimated blur kernel radius value of the blue channel, x RG is the x component of the estimated color difference value of the red channel relative to the green channel, and y RG is the y component of the estimated color difference value of the red channel relative to the green channel, and x BG is the x component of the estimated color difference value of the blue channel relative to the green channel, and y BG is the y-component of the estimated color difference value of the blue channel relative to the green channel.

24. The method according to claim 22, characterized in that The multi-channel defocus blur and color difference are stored as r of a red, green, blue, infrared (RGB-IR) camera. R 、r G 、r B 、r IR 、x RG ,y RG 、x BG ,y BG 、x IRG ,y IRG , where r R is the estimated blur kernel radius value of the red channel, r G is the estimated blur kernel radius value of the green channel, r B is the estimated blur kernel radius value of the blue channel, r IR is the estimated blur kernel radius value of the infrared channel, x RG is the x component of the estimated color difference value of the red channel relative to the green channel, and y RG is the y component of the estimated color difference value of the red channel relative to the green channel, and x BG is the x component of the estimated color difference value of the blue channel relative to the green channel, and y BG is the y component of the estimated color difference value of the blue channel relative to the green channel, and x IRG is the x component of the estimated color difference value of the infrared channel relative to the green channel, and y IRG is the y-component of the estimated color difference value of the infrared channel relative to the green channel.

25. The method according to any one of claims 20 to 24, characterized in that The registering of the depth-implied calibration unit includes registering the depth-implied attribute at a measured depth value in response to the calibration process indicator indicating that the calibration process is manual or temporary.

26. The method according to claim 25, characterized in that The measured depth value is determined by the calibration controller by matching features in the image of the user's moving head with a pre-registered three dimensional (3D) head model of the user.

27. The method according to claim 25, characterized in that The measured depth value is determined by the calibration controller by matching features in an image of a predetermined calibration pattern with a pre-registered two-dimensional / three-dimensional (2D / 3D) model of the predetermined calibration pattern.

28. The method according to claim 25, characterized in that The measured depth value is determined by the calibration controller by measuring features in an image of a user's moving head with reference to predetermined universal dimensions of a human head.

29. The method according to any one of claims 20 to 28, characterized in that The registering of the depth hint calibration unit includes registering the depth hint attribute at a specified depth value in response to the calibration process indicator indicating that the calibration process is automatic.

30. The method according to claim 29, characterized in that The specified depth value is determined by the calibration controller according to a predetermined process to traverse a target range of depth values.

31. The method according to any one of claims 20 to 30, characterized in that The depth cue properties are registered and stored by calibrating depth cue samples, the calibration being performed by an automatic process, a manual process or an ad hoc process.

32. The method according to any one of claims 20 to 31, characterized in that The depth cue properties are registered and stored by learning depth cue parameters, the learning being performed by an automatic process, a manual process or an ad hoc process.

33. An image restoration method, characterized in that: include: An image restoration module (IRM) receives the sparse depth map, the input image, and the extracted depth hint information, wherein the depth hint information includes edge and texture maps, blur kernel maps for each color channel, and color difference maps for each color channel except the green channel; The IRM performs defocus deblurring and chromatic aberration correction on the input image based on the sparse depth map and the extracted depth cue information; The IRM generates a restored output image, wherein the restored output image is one of images having significantly improved image quality compared to the input image; The IRM sends the restored output image to a loss function calculation module.

34. A device, characterized in that include: one or more processors; a non-transitory memory comprising instructions, wherein the instructions, when executed by the one or more processors, cause the apparatus to: A method according to any one of claims 1 to 9 is performed.

35. A device, characterized in that include: one or more processors; a non-transitory memory comprising instructions, wherein the instructions, when executed by the one or more processors, cause the apparatus to: A method according to any one of claims 10 to 13 is performed.

36. A device, characterized in that include: one or more processors; a non-transitory memory comprising instructions, wherein the instructions, when executed by the one or more processors, cause the apparatus to: A method according to any one of claims 14 to 19 is performed.

37. A device, characterized in that include: one or more processors; a non-transitory memory comprising instructions, wherein the instructions, when executed by the one or more processors, cause the apparatus to: Perform a method according to any one of claims 20 to 32.

38. A device, characterized in that include: one or more processors; a non-transitory memory comprising instructions, wherein the instructions, when executed by the one or more processors, cause the apparatus to: Execute the method according to claim 33.

Images