System and method for providing monocular image depth estimation based on deep neural network

Abstract

A method and system for determining depth information of an image is provided herein. According to one embodiment, the method includes: receiving an input image, classifying the input image into a depth range of a plurality of depth ranges, and determining a depth map of the image by applying depth estimation based on the depth range into which the input image is classified.

Description

[0001] priority

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 62 / 831,598 filed with the U.S. Patent and Trademark Office on April 9, 2019 and U.S. Patent Application No. 16 / 574,770 filed on September 18, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to computer vision. Specifically, the present disclosure relates to systems and methods for providing depth estimation of monocular images based on deep neural networks. Background Art

[0004] Single image depth estimation (SIDE) is an important part for understanding the geometric structure of scenes in images. Specifically, depth maps can be used to infer three-dimensional (3D) structures, which are essential elements for many topics in 3D vision, such as image reconstruction, image rendering, and shallow depth of field. However, SIDE is a problem because monocular two-dimensional (2D) images can be generated from an infinite number of different 3D scenes. By using deep convolutional neural networks (CNNs), deep learning-based methods have achieved performance improvements in solving SIDE problems such as CNNs that learn to map images to continuous depth maps.

[0005] The estimation of the real-world depth of elements in the captured scene has applications, such as the ability to separate foreground (near) objects from background (far) objects. Accurate depth estimation allows foreground objects of interest in the scene to be separated from the background. Accurate foreground-background separation allows people to process captured images to simulate effects such as Bokeh effects. Bokeh is a soft out-of-focus blur of the background, which is mastered by using the correct settings in a camera with a fast lens and a wide aperture and bringing the camera closer to the subject and moving the subject further away from the background to simulate a shallow depth of field. Therefore, accurate depth estimation allows processing images from non-professional photographers or cameras with smaller lenses (such as mobile phone cameras) to obtain more aesthetically pleasing images with a Bokeh effect focused on the subject. Other applications of accurate depth estimation include 3D object reconstruction and virtual reality applications, in which it is desired to change the background or subject and render them according to the desired virtual reality. Other applications of accurate depth estimation from captured scenes include automotive automation, surveillance cameras, autonomous driving applications, and enhancing safety by improving object detection accuracy and its estimation of distance from the camera.

[0006] There are two problems in designing deep CNNs for SIDE. The SIDE method can train different networks for individual datasets, which makes the models specific to certain domains. As a result, the huge differences between different datasets such as indoor and outdoor patterns limit the generalization ability of the SIDE network. A model that achieves impressive performance on one dataset will perform poorly on another dataset.

[0007] Furthermore, existing SIDE networks utilize very deep architectures as backbones. The scale of these networks is very large. Moreover, some SIDE networks contain fully connected layers. Therefore, a sliding window strategy is required during prediction to make the test image size the same as training. As a result, the efficiency and flexibility of such networks are not desirable. Summary of the invention

[0008] According to one embodiment, a method includes receiving an input image, classifying the input image into a depth range of a plurality of depth ranges, and determining a depth map of the image by applying depth estimation based on the depth range into which the input image is classified.

[0009] According to one embodiment, a system includes a memory and a processor configured to receive an input image, classify the input image into a depth range of a plurality of depth ranges, and determine a depth map of the image by applying depth estimation based on the depth range into which the input image is classified. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings, in which:

[0011] Figure 1 A diagram showing an image demonstrating a Bokeh effect according to an embodiment;

[0012] Figure 2 A diagram illustrating depth estimation using indoor and outdoor networks according to an embodiment;

[0013] Figure 3 A diagram showing providing depth estimation according to an embodiment;

[0014] Figure 4 A flowchart showing a method for performing depth estimation of an image according to an embodiment is shown;

[0015] Figure 5 A diagram illustrating a SIDE network according to an embodiment;

[0016] Figure 6 A diagram showing an upproj layer according to an embodiment;

[0017] Figure 7 A diagram showing an upproj layer with skip connections according to an embodiment;

[0018] Figure 8 A diagram showing depth estimation using a depth refinement SIDE network according to an embodiment;

[0019] Fig. 9 A diagram showing the use of RGB-depth (RGBD) to enhance depth estimation of a SIDE network according to an embodiment; and

[0020] Fig.10 A block diagram is shown of an electronic device in a network environment according to one embodiment. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that the same elements will be represented by the same reference numerals, although they are shown in different drawings. In the following description, specific details such as detailed configuration and components are provided only to help fully understand the embodiments of the present disclosure. Therefore, it is obvious to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. In addition, for clarity and conciseness, descriptions of well-known functions and configurations are omitted. The terms described below are defined terms in consideration of the functions in the present disclosure, and may be different according to the user, the user's intention or custom. Therefore, the definition of the terms should be determined according to the content of the entire description.

[0022] The present disclosure may have various modifications and various embodiments, wherein the embodiments will be described in detail below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the embodiments, but includes all modifications, equivalents and substitutions within the scope of the present disclosure.

[0023] Although terms including ordinal numbers such as first and second can be used to describe various elements, structural elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, the first structural element can be referred to as the second structural element without departing from the scope of this disclosure. Similarly, the second structural element can also be referred to as the first structural element. As used herein, the term "and / or" includes any one and all combinations of one or more related items.

[0024] The terms used herein are only used to describe various embodiments of the present disclosure and are not intended to limit the present disclosure. Singular forms are intended to include plural forms unless the context clearly indicates otherwise. In the present disclosure, it should be understood that the term "including" or "having" indicates the presence of a feature, number, step, operation, structural element, part or a combination thereof, and does not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, structural elements, parts or a combination thereof.

[0025] Unless defined differently, all terms used herein have the same meaning as understood by a person skilled in the art to which the present disclosure belongs. Terms such as those defined in general dictionaries will be interpreted as having the same meaning as the contextual meaning in the relevant technical field, and will not be interpreted as having an ideal or overly formal meaning unless explicitly defined in the present disclosure.

[0026] The electronic device according to one embodiment may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to one embodiment of the present disclosure, the electronic device is not limited to those described above.

[0027] The terms used in this disclosure are not intended to limit the disclosure, but are intended to include various changes, equivalents or substitutions of corresponding embodiments. With regard to the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. The singular form of the noun corresponding to an item may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may be included in all possible combinations of items enumerated together in the corresponding phrases in these phrases. As used herein, terms such as "first", "second" may be used to distinguish a corresponding component from another component, but are not intended to limit components in other respects (e.g., importance or order). It is intended that if an element (e.g., a first element) is referred to as being “coupled,” “connected,” or “coupled to,” “connected to” another element (e.g., a second element), whether or not there is the term “operably” or “communicatively,” it indicates that the element can be coupled to the other element directly (e.g., wired), wirelessly, or via a third element.

[0028] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms, such as "logic," "logic block," "portion," and "circuit." A module may be a single integral component adapted to perform one or more functions, or its smallest unit or portion. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0029] Figure 1 A diagram of an image demonstrating the Bokeh effect according to an embodiment is shown. Accurate depth estimation has many applications. Image 100 shows the Bokeh effect. Bokeh is the process of blurring background subjects in an image by defocusing distant subjects while making closer subjects clearer or in focus. The key feature of image Bokeh is the accurate foreground-background segmentation extracted from the depth map. The performance of the Bokeh effect is determined by the accuracy of the estimated depth.

[0030] Figure 2A diagram showing depth estimation using an indoor network and an outdoor network according to an embodiment. Because a monocular 2D image can be generated from an infinite number of different 3D scenes, SIDE has problems. Since it is very difficult to obtain an accurate depth map, most existing Bokeh systems use multiple cameras instead of a single camera. This limits the application scenarios. Although some existing SIDE systems can generate impressive results with a specific data set, these systems work poorly when given images from different data sets. For example, when an input image 200 belonging to an outdoor scene is provided, a neural network trained with indoor images does not perform well, as shown by the estimated depth at image 202, compared to the estimated depth at image 204 performed by an outdoor network. Therefore, it is necessary to design a robust SIDE system that works well in a variety of environments at the same time.

[0031] Figure 3 A diagram is shown providing depth estimation according to an embodiment. The depth estimation system 300 includes a two-stage robust SIDE framework. In the first stage 304, the depth estimation system 300 classifies the input image 302 into different depth ranges 306 (e.g., indoor images can be classified as "low depth range"). Although the lowest depth range, medium depth range, and highest depth range are depicted, any number and type of ranges can be utilized. In the second stage 308, for each depth range, the system applies a SIDE network optimized for the depth range into which the image is classified to obtain an accurate depth map, and outputs a depth 310.

[0032] Figure 4 A flowchart 400 of a method for performing depth estimation of an image according to an embodiment is shown. At 402, the system receives an image input. Although the embodiments described below relate to receiving an image as input, it should be understood that the system can receive a portion of an image as input, and the steps described herein can be performed on portions of an image individually or independently of each other.

[0033] At 404, the system classifies the input image into one of a plurality of depth ranges. The system may classify the image into a depth range by classifying the image into a predefined scene (e.g., sky, iceberg, room). Since scenes may be associated with specific or predefined depth ranges (e.g., sky has the highest depth range, room scenes have a relatively low depth range), the exact depth range of the input image may be inferred from the scene category.

[0034] The system can classify an image into a predefined scene based on a majority voting algorithm. In one example, a WideResNet-18 is trained on the Places-365 dataset. The dataset includes a large number of predefined or labeled scenes or scene categories. The scenes or scene categories are further labeled based on their associated depth range. Given an input image, the system uses the first specific number of predicted scenes to vote on the depth range classification of the input image. For example, if the system uses the first 15 predicted scenes, then if there are more than 7 low depth range votes among the first 15 predicted scenes, then the system classifies the image as low depth range.

[0035] The system can classify images into a predefined scene based on a weighted voting algorithm. i and m i represents the statistical median and maximum (or X-percentile) depth of scene category i. i is the probability that the current scene is classified as the i-th scene by the scene classification network. Then the median depth of the current scene can be estimated as in equation (1):

[0036]

[0037] And the estimated maximum (or X-percentile) depth of the current scene can be estimated as in equation (2):

[0038]

[0039] The sum is taken for all candidate scenes. Using the estimated median and maximum (or X-percentile) depths, the system can select quantized bins for the depth range by comparing the median depth or maximum depth of the estimated image to various predefined depth range thresholds.

[0040] Alternatively, the system can use a coarse depth estimate to classify an image into a depth range. Since it is difficult to obtain an accurate estimated depth in one go, the system can first obtain a coarse estimated depth map and then use it to decide the depth range. Any SIDE network can be used as a coarse depth estimation network. When using a coarse estimated depth map to decide the depth range, the system can refer to some statistics, such as the maximum depth or minimum depth, because images with a high depth range, such as outdoor images, will have a relatively higher maximum depth.

[0041] The system can use validation data for indoor and outdoor images (e.g., from the Robust Vision Challenge) to compute a statistic for the maximum estimated depth and then select a threshold that determines low depth range / high depth range (e.g., if the threshold is a depth value X, and the maximum depth of the estimated depth map is below X, then the system classifies the image as being in the low depth range).

[0042] In a more general way of obtaining depth ranges using a coarse depth estimate, the system can construct a mapping from a coarse depth map to a discrete depth range. This can be achieved by learning a classifier from a coarse depth map to a depth range, such as D = F(d), where D is the estimated depth range, d is the coarse depth map, and F is a classifier such as a support vector machine (SVM), multi-class boosting, and random forest.

[0043] At 406 , the system determines a depth map for the image based on the depth range into which the input image is classified.

[0044] Figure 5 A diagram showing a SIDE network according to an embodiment. The system utilizes an efficient SIDE network (depth-wise (DW)-SIDENet) based on an encoder-decoder architecture. The SIDE network 500 includes an encoding branch 502, a deep classification decoding branch 504, and a deep regression decoding branch 506.

[0045] The encoding branch 502 includes an encoding block 508 and several depth-wise separable convolutional layers (e.g., encoding blocks 510, 512, 514, and 516) to extract discriminative features 518 from the input image 501. Since low depth range images have different characteristics than high depth range images (e.g., indoor images have different contexts than outdoor images), the encoding branch 502 of the low depth range network is slightly different from that of the high depth range network. For example, the system can set the output feature map size of the high depth range encoding module to x4 downsampling and set the size of the low depth range encoding module to x32 downsampling.

[0046] The network uses a depth classification branch 504 and a depth regression branch 506 for decoding. The network architectures of these two branches are mostly the same, but the output feature maps and loss functions are different. The depth classification decoding branch 504 includes upsampling decoding blocks 520, 522, 524, and 526 that are decoded using upproj-con layers with jump connections and a decoding block 528 that is decoded using upproj layers to output a depth map 529. The depth regression decoding branch 506 includes upsampling decoding blocks 530, 532, 534, and 536 that are decoded using upproj-con layers with jump connections and a decoding block 538 that is decoded using upproj layers to output a depth map 539. In the classification branch 504, the SIDE network 500 performs depth classification after quantizing continuous depth into several bins. When calculating the loss function of the classification branch 504, the network uses the weighted sum of the quantized depth probability and the quantized depth value as the predicted quantized depth for each pixel position. In the regression branch 506, the network directly regresses the output feature map into a continuous disparity map.

[0047] The loss function of DW-SIDENet 500 is a linear combination of the depth value classification loss (Huber) and the depth regression loss, and is given by Equation (3):

[0048]

[0049] in, is the ground truth depth of image i, is the quantized ground truth depth, d i,.cls and d i,reg are the outputs from both branches. Depth map 529 is determined by classification branch 504 and depth map 539 is determined by regression branch 506.

[0050] Figure 6 A diagram of an upproj layer according to an embodiment is shown. The upproj layer 600 receives an X input 602 from the previous decoding block. At 604, the input is upsampled to the output size and then processed by a 3×3 convolution layer 606. The output of the 3×3 convolution layer 606 is processed by a first 5×5 convolution layer 608 and a second 5×5 convolution layer 610. The output of the first 5×5 convolution layer 608 is processed by a 3×3 convolution layer 612, whose output is combined with the output of the second 5×5 convolution layer 610. Then, an output layer 614 is produced, giving a final estimated depth 616.

[0051] Figure 7704. A diagram of an upproj layer with skip connections according to an embodiment is shown. The upproj layer 700 receives an X input 702 from the previous decoding block. The X input 702 is upsampled to the output size 703 and then processed by a 3×3 convolutional layer 704. Layer 700 receives a Y input 706 from the encoding module, which is processed by a 1×1 convolutional layer 708 and concatenated at the 3×3 convolutional layer 704. Layer 700 receives a ZRGB input 709. Input 709 is downsampled to the output size at 710, processed by a 1×1 convolutional layer 712, and concatenated at the 3×3 convolutional layer 704. The output of the 3×3 convolutional layer 704 is processed by a first 5×5 convolutional layer 714 and a second 5×5 convolutional layer 716. The output of the first 5x5 convolutional layer 714 is processed by a 3×3 convolutional layer 718, whose output is combined with the output of layer 716 to produce the output 720 of the decoding block.

[0052] In network 500, decoding block 528 and decoding block 538 utilize upproj layer 600. Decoding blocks 520, 522, 524, and 526 and decoding blocks 530, 532, 534, and 536 utilize upproj layer 700 with skip connections.

[0053] In examples where the system uses coarse depth estimation for classifying an image into a depth range, the coarse depth map generated during the coarse depth estimation can be used as input to the SIDE network, and a different SIDE network can be used as a result.

[0054] Figure 8 A diagram showing the use of a depth refinement SIDE network to provide depth estimation according to an embodiment. The depth estimation system 800 receives an input image 802 and classifies the image into a depth range using a coarse depth estimate 804. The system then uses the coarse depth map 806 generated from this step to perform a depth refinement of the depth estimate through a SIDE network 808.

[0055] Fig. 9 A diagram showing the use of RGB-depth (RGBD) to enhance the SIDE network to provide depth estimation according to an embodiment. The depth estimation system 900 receives an input image 902 and uses a coarse depth estimate 904 to classify the image into a depth range. The system then uses the coarse depth map 906 generated from this step and the input image 902 to perform RGBD enhanced depth estimation through the SIDE network 908. The D channel in the RGBD enhancement can use a dense RGB image but sparsely sampled pixels from the coarse depth map, or a dense RGB image and all pixels from the coarse depth map.

[0056] Fig.10FIG. 1 is a block diagram showing an electronic device 1001 in a network environment 1000 according to an embodiment. Fig.10 , the electronic device 1001 in the network environment 1000 may communicate with the electronic device 1002 via the first network 1098 (e.g., a short-range wireless communication network), or communicate with the electronic device 1004 or the server 1008 via the second network 1099 (e.g., a long-range wireless communication network). The electronic device 1001 may communicate with the electronic device 1004 via the server 1008. The electronic device 1001 may include a processor 1020, a memory 1030, an input device 1050, a sound output device 1055, a display device 1060, an audio module 1070, a sensor module 1076, an interface 1077, a haptic module 1079, a camera module 1080, a power management module 1088, a battery 1089, a communication module 1090, a subscriber identification module (SIM) 1096, or an antenna module 1097. In one embodiment, at least one of the components (e.g., display device 1060 or camera module 1080) may be omitted from electronic device 1001, or one or more other components may be added to electronic device 1001. In one embodiment, some of the components may be implemented as a single integrated circuit (IC). For example, sensor module 1076 (e.g., fingerprint sensor, iris sensor, or illumination sensor) may be embedded in display device 1060 (e.g., display).

[0057] The processor 1020 may execute, for example, software (e.g., program 1040) to control at least one other component (e.g., hardware or software component) of the electronic device 1001 coupled to the processor 1020, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 1020 may load a command or data received from another component (e.g., sensor module 1076 or communication module 1090) into the volatile memory 1032, process the command or data stored in the volatile memory 1032, and store the resultant data in the non-volatile memory 1034. The processor 1020 may include a main processor 1021 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 1023 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that operates independently of the main processor 1021 or in conjunction with the main processor 1021. Additionally or alternatively, the auxiliary processor 1023 may be adapted to consume less power than the main processor 1021, or to perform special functions. The auxiliary processor 1023 may be implemented as a separate part of the main processor 1021 or as a part of the main processor 1021.

[0058] The auxiliary processor 1023 may replace the main processor 1021 when the main processor 1021 is in an inactive (e.g., sleeping) state, or control at least some of the functions or states related to at least one of the components of the electronic device 1001 (e.g., the display device 1060, the sensor module 1076, or the communication module 1090) together with the main processor 1021 when the main processor 1021 is in an active state (e.g., executing an application). According to one embodiment, the auxiliary processor 1023 (e.g., an image signal processor or a communication processor) may be implemented as a part of another component (e.g., a camera module 1080 or a communication module 1090) that is functionally related to the auxiliary processor 1023.

[0059] The memory 1030 may store various data used by at least one component of the electronic device 1001 (e.g., the processor 1020 or the sensor module 1076). The various data may include, for example, software (e.g., program 1040) and input data or output data for commands related thereto. The memory 1030 may include a volatile memory 1032 or a non-volatile memory 1034.

[0060] The program 1040 may be stored in the memory 1030 as software, and may include, for example, an operating system (OS) 1042 , middleware 1044 , or an application 1046 .

[0061] The input device 1050 may receive commands or data from outside the electronic device 1001 (eg, a user) to be used by other components (eg, the processor 1020) of the electronic device 1001. The input device 1050 may include, for example, a microphone, a mouse, or a keyboard.

[0062] The sound output device 1055 can output sound signals to the outside of the electronic device 1001. The sound output device 1055 can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or recording, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented as a part of the speaker or a part of the speaker.

[0063] The display device 1060 can visually provide information to the outside of the electronic device 1001 (e.g., a user). The display device 1060 may include, for example, a display, a holographic device, or a projector, and a control circuit that controls a corresponding one of the display, the holographic device, and the projector. According to one embodiment, the display device 1060 may include a touch circuit adapted to detect a touch, or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0064] The audio module 1070 can convert sound into an electrical signal, and vice versa. According to one embodiment, the audio module 1070 can obtain sound via the input device 1050, or output sound via the sound output device 1055 or output sound via the earphone of the external electronic device 1002 directly (e.g., wired) or wirelessly coupled to the electronic device 1001.

[0065] The sensor module 1076 can detect the operating state (e.g., power or temperature) of the electronic device 1001 or the environmental state (e.g., the state of the user) outside the electronic device 1001, and then generate an electrical signal or data value corresponding to the detected state. The sensor module 1076 may include, for example, a posture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0066] The interface 1077 may support one or more specific protocols for direct (e.g., wired) or wireless coupling of the electronic device 1001 with the external electronic device 1002. According to one embodiment, the interface 1077 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0067] The connection terminal 1078 may include a connector via which the electronic device 1001 can be physically connected to the external electronic device 1002. According to one embodiment, the connection terminal 1078 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0068] The haptic module 1079 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via tactile or kinesthetic sense. According to one embodiment, the haptic module 1079 may include, for example, a motor, a piezoelectric element, or a point actuator.

[0069] The camera module 1080 may capture still images or moving images. According to one embodiment, the camera module 1080 may include one or more lenses, image sensors, image signal processors, or flashes.

[0070] The power management module 1088 may manage power supplied to the electronic device 1001. The power management module 1088 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0071] The battery 1089 may supply power to at least one component of the electronic device 1001. According to one embodiment, the battery 1089 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0072] The communication module 1090 can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 1001 and an external electronic device (e.g., electronic device 1002, electronic device 1004, or server 1008), and perform communication via the established communication channel. The communication module 1090 may include one or more communication processors that operate independently of the processor 1020 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 1090 may include a wireless communication module 1092 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1094 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 1098 (e.g., a short-range communication network such as Bluetooth TM , Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA) standards) or a second network 1099 (e.g., a remote communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)) to communicate with an external electronic device. These different types of communication modules can be implemented as a single component (e.g., a single IC), or can be implemented as multiple components separated from each other (e.g., multiple ICs). The wireless communication module 1092 can identify and authenticate the electronic device 1001 in the communication network (e.g., the first network 1098 or the second network 1099) using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module 1096.

[0073] The antenna module 1097 may transmit or receive a signal or power to or from the outside of the electronic device 1001 (e.g., an external electronic device). According to one embodiment, the antenna module 1097 may include one or more antennas, and thus, for example, at least one antenna applicable to a communication scheme used in a communication network (e.g., the first network 1098 or the second network 1099) may be selected by the communication module 1090 (e.g., the wireless communication module 1092). Then, a signal or power may be transmitted or received between the communication module 1090 and the external electronic device via the selected at least one antenna.

[0074] At least some of the above components can be coupled to each other and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

[0075] According to one embodiment, commands or data may be sent or received between the electronic device 1001 and the external electronic device 1004 via a server 1008 coupled to the second network 1099. Each of the electronic devices 1002 and 1004 may be a device of the same type or a different type as the electronic device 1001. All or some of the operations to be performed at the electronic device 1001 may be performed at one or more of the external electronic devices 1002, 1004, or 1008. For example, if the electronic device 1001 should automatically perform a function or service, or perform a function or service in response to a request from a user or another device, the electronic device 1001 may request one or more external electronic devices to perform at least a portion of the function or service instead of or in addition to performing the function or service. The one or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or an additional function or additional service related to the request, and transmit the result of the execution to the electronic device 1001. The electronic device 1001 may provide the result as at least a part of the reply to the request with or without further processing the result. To this end, for example, cloud computing, distributed computing or client-server computing techniques may be used.

[0076] One embodiment may be implemented as software (e.g., program 1040), including one or more instructions stored in a storage medium (e.g., internal memory 1036 or external memory 1038) readable by a machine (e.g., electronic device 1001). For example, a processor of electronic device 1001 may call at least one of the one or more instructions stored in the storage medium and execute it with or without one or more other components under the control of the processor. Thus, the machine may be operated to perform at least one function according to at least one instruction called. One or more instructions may include code generated by a compiler or code executable by a parser. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium.

[0077] According to one embodiment, the method of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or via an application store (e.g., Play Store TM ) or distributed online (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium such as a memory of a manufacturer's server, a server of an application store, or a relay server.

[0078] According to one embodiment, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities. One or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated components may still perform one or more functions of each component in the multiple components in the same or similar manner as they were performed by a corresponding component in the multiple components before integration. The operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed or omitted in a different order, or one or more other operations may be added.

[0079] Although certain embodiments of the present disclosure have been described in the detailed description of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined solely based on the described embodiments, but based on the attached claims and their equivalents.

Claims

1. A method for determining depth information of an image, comprising: receiving an input image; classifying the input image into a first depth range among a plurality of candidate depth ranges using a first network, the plurality of candidate depth ranges including the first depth range and a second depth range different from the first depth range; In response to classifying the input image into the first depth range, selecting a second network from a plurality of candidate networks, the plurality of candidate networks comprising (i) a network optimized to obtain a depth map of the first depth range and (ii) a network optimized to obtain a depth map of the second depth range; as well as A depth map of the input image is determined by applying the second network.

2. The method according to claim 1, wherein: Classifying the input image is performed based on a coarse depth estimation.

3. The method according to claim 2, wherein: Determining the depth map of the input image by applying the second network also includes: using the coarse depth map generated by the coarse depth estimation as input of a depth refinement monocular image depth estimation SIDE network.

4. The method according to claim 3, wherein: Determining the depth map of the input image by applying the second network also includes: using the coarse depth map and the input image as RGBD input of an RGB depth RGBD enhancement SIDE network.

5. The method according to claim 2, wherein: The coarse depth estimation is performed by computing a maximum depth of the input image and comparing the maximum depth to a depth range threshold.

6. The method according to claim 1, wherein: Classifying the input image includes classifying the input image into a predefined scene.

7. The method according to claim 6, wherein: Classifying the input image into a predefined scene is performed based on majority voting.

8. The method according to claim 6, wherein: Classifying the input image into a predefined scene is performed based on weighted voting.

9. The method according to claim 1, wherein: Determining the depth map of the input image by applying the second network also includes: Encoding the input image using a monocular image depth estimation SIDE network; and Using the SIDE network, the input image is decoded using a deep regression decoding branch.

10. The method according to claim 1, wherein: Determining the depth map of the input image by applying the second network also includes: Encoding the input image using a monocular image depth estimation SIDE network; and Using the SIDE network, the input image is decoded using a deep classification decoding branch.

11. A system for determining depth information of an image, comprising: Memory; and The processor is configured to: receiving an input image; classifying the input image into a first depth range among a plurality of candidate depth ranges using a first network, the plurality of candidate depth ranges including the first depth range and a second depth range different from the first depth range; In response to classifying the input image into the first depth range, selecting a second network from a plurality of candidate networks, the plurality of candidate networks comprising (i) a network optimized to obtain a depth map of the first depth range and (ii) a network optimized to obtain a depth map of the second depth range; as well as A depth map of the input image is determined by applying the second network.

12. The system according to claim 11, wherein: The processor is configured to classify the input image based on a coarse depth estimate.

13. The system according to claim 12, wherein: The processor is configured to determine a depth map of the input image by applying the second network by utilizing a coarse depth map generated by a coarse depth estimation as an input of a depth refinement monocular image depth estimation (SIDE) network.

14. The system according to claim 13, wherein: The processor is further configured to determine a depth map of the input image by applying the second network by utilizing the coarse depth map and the input image as RGBD inputs to an RGB-Depth-RGBD-Enhanced-SIDE network.

15. The system of claim 12, wherein: The coarse depth estimation is performed by computing a maximum depth of the input image and comparing the maximum depth to a depth range threshold.

16. The system according to claim 11, wherein: The processor is configured to categorize the input image by categorizing the input image into a predefined scene.

17. The system of claim 16, wherein: Classifying the input image into a predefined scene is performed based on majority voting.

18. The system of claim 16, wherein: Classifying the input image into a predefined scene is performed based on weighted voting.

19. The system of claim 11, wherein: The processor is configured to determine a depth map of the input image by applying the second network by the following steps: Encoding the input image using a monocular image depth estimation SIDE network; as well as Using the SIDE network, the input image is decoded using a deep regression decoding branch.

20. The system of claim 11, wherein: The processor is configured to determine a depth map of the input image by applying the second network by the following steps: Encoding the input image using a monocular image depth estimation SIDE network; as well as Using the SIDE network, the input image is decoded using a deep classification decoding branch.

Images