Electronic device and method for disparity estimation

Abstract

An electronic device and method for disparity estimation are disclosed herein. The electronic device includes: a first camera having a first field of view (FOV); a second camera having a second FOV narrower than the first FOV; and a processor configured to: capture a first image using the first camera, the first image having a joint FOV; capture a second image using the second camera; determine an overlapping FOV between the first image and the second image; generate a disparity estimate based on the overlapping FOV; generate a joint FOV disparity estimate; and merge the joint FOV disparity estimate with the overlapping FOV disparity estimate.

Description

[0001] This application is based on and claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 751,960 filed in the U.S. Patent and Trademark Office on October 29, 2018 and assigned Serial No. 16 / 365,167 filed in the U.S. Patent and Trademark Office on March 26, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure generally relates to an image processing system and more particularly to a method and system for disparity estimation using cameras with different fields of view. Background Art

[0003] There has been recent interest in the estimation of the real-world depth of elements in a captured scene. Depth estimation has many applications, such as the ability to separate foreground (near) objects from background (far) objects. Accurate depth estimation allows foreground objects of interest to be separated from the background in a scene. Accurate foreground-background separation allows captured images to be processed to simulate effects such as the Bokeh effect. Bokeh is a soft out-of-focus blur of the background, which is typically achieved by using the correct settings in an expensive camera with a fast lens and a wide aperture and bringing the camera closer to the subject and the subject further away from the background to simulate a shallow depth of field.

[0004] Accurate depth estimation allows images from non-professional photographers or cameras with smaller lenses (such as mobile phone cameras) to be processed to obtain more aesthetically pleasing images with a bokeh effect focused on the subject. Other applications of accurate depth estimation include 3D object reconstruction and virtual reality (VR) applications (in VR applications, it is desirable to change the background or subject and render them depending on the desired VR). Other applications of accurate depth estimation from captured scenes include: automotive automation, surveillance cameras, autonomous driving applications, and enhancing safety by improving object detection accuracy and distance from the camera using only the camera or from camera input and depth estimation from multiple sensors. Summary of the invention

[0005] According to one embodiment, an electronic device is provided. The electronic device includes: a first camera having a first field of view (FOV); a second camera having a second FOV narrower than the first FOV; and a processor configured to: capture a first image using the first camera, the first image having a joint FOV; capture a second image using the second camera; determine an overlapping FOV between the first image and the second image; generate a disparity estimate based on the overlapping FOV; generate a joint FOV disparity estimate; and merge the joint FOV disparity estimate with the overlapping FOV disparity estimate.

[0006] According to one embodiment, a method is provided. The method includes: capturing a first image using a first camera having a first field of view (FOV), the first image having a joint FOV; capturing a second image using a second camera having a second FOV narrower than the first FOV; determining an overlapping FOV between the first image and the second image; generating a disparity estimate based on the overlapping FOV; generating a joint FOV disparity estimate; and merging the joint FOV disparity estimate with the overlapping FOV disparity estimate. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is an illustration of a bokeh effect according to an embodiment;

[0009] Figure 2 is a diagram of a stereo matching system according to one embodiment;

[0010] Figure 3 is a flow chart of applying a bokeh effect to an image using two images with similar FOVs according to one embodiment;

[0011] Figure 4 is a diagram representing an image of a joint FOV and an overlapping FOV according to one embodiment;

[0012] Figure 5 is a flow chart for generating disparity of a joint FOV according to one embodiment;

[0013] Figure 6 is a diagram of a Red-Green-Blue (RGB)-Single Image Disparity Estimation (RGB-SIDE) network according to one embodiment;

[0014] Figure 7 is a diagram of a far-wide stereo matching network according to one embodiment;

[0015] Figure 8 is a diagram of a multi-task far-wide stereo matching network according to one embodiment;

[0016] Fig. 9 is a diagram of a parallax enhancement system according to one embodiment;

[0017] Fig.10 , Fig.11 , Fig.12 and Fig.13 is a diagram of a system for generating disparity estimates to be merged and post-processed according to one embodiment;

[0018] Fig.14is an illustration of parallax merging and selection according to one embodiment;

[0019] Fig.15 is an illustration of parallax merging and selection according to one embodiment;

[0020] Fig.16 is a diagram of joint FOV and overlapping FOV when there are four cameras according to one embodiment; and

[0021] Fig.17 is a block diagram of an electronic device in a network environment according to one embodiment. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that although the same elements are shown in different drawings, the same elements will be represented by the same reference numerals. In the following description, specific details such as detailed configuration and components are only provided to help the overall understanding of the embodiments of the present disclosure. Therefore, it should be clear to those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. In addition, for clarity and brevity, descriptions of well-known functions and configurations are omitted. The terms described below are defined in consideration of the functions in the present disclosure, and may be different according to the user, the user's intention or custom. Therefore, the definition of the term should be determined based on the content throughout this specification.

[0023] The present disclosure may have various modifications and various embodiments, and the embodiments thereof are described in detail below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the embodiments, but includes all modifications, equivalents and replacements within the scope of the present disclosure.

[0024] Although the term including ordinal numbers (such as, first, second, etc.) can be used to describe various elements, structural elements are not limited by the term. The term is only used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, the first structural element may be referred to as the second structural element. Similarly, the second structural element may also be referred to as the first structural element. As used herein, the term "and / or" includes any combination and all combinations of one or more related items.

[0025] The terms used herein are only used to describe various embodiments of the present disclosure and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In the present disclosure, it should be understood that the term "including" or "having" indicates the presence of features, quantities, steps, operations, structural elements, parts or combinations thereof, and does not exclude the presence or possible addition of one or more other features, quantities, steps, operations, structural elements, parts or combinations thereof.

[0026] Unless defined differently, all terms used herein have the same meaning as understood by a person skilled in the art to which the present disclosure belongs. Unless clearly defined in the present disclosure, terms (such as those defined in a general dictionary) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized or overly formal meaning.

[0027] The electronic device according to one embodiment may be one of various types of electronic devices. For example, the electronic device may include a portable communication device (e.g., a smart phone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to one disclosed embodiment, the electronic device is not limited to those described above.

[0028] The terms used in the present disclosure are not intended to limit the present disclosure, but are intended to include various changes, equivalents or replacements of corresponding embodiments. Regarding the description of the accompanying drawings, similar reference numerals may be used to represent similar or related elements. Unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to the project may include one or more things. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the project listed together with the corresponding one in the phrase. As used herein, terms such as "the 1st", "the 2nd", "the first" and "the second" may be used to distinguish the corresponding component from another component, without intending to limit the component in other aspects (e.g., importance or order). It is intended that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., a second element)”, “coupled to another element”, “connected to another element” or “connected to another element” with or without the term “operably” or “communicatively”, it indicates that the element can be coupled with another element directly (e.g., wired), wirelessly, or via a third element.

[0029] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," or "circuit"). A module may be a single integrated component or its smallest unit or component adapted to perform one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0030] The present system and method provide an estimate of the real-world depth of elements in a scene captured by two cameras with different fields of view (FOV). By calculating the disparity (e.g., horizontal displacement) between pixels in the two images, an accurate estimate of the depth from the two stereo-rectified images can be obtained.

[0031] The present system and method provide an estimate of the depth of all elements in the union of the FOVs of two images. Two cameras may be provided, one having a wide FOV typically associated with low or no optical zoom, and the other having a narrower FOV typically associated with a larger optical zoom. Recent mobile devices are equipped with two or more cameras. In order to take advantage of the additional cameras, their lenses are typically selected to have different optical zooms so that the device has good resolution for both near and far objects. In particular, the zoom level of the lens may not be changed by the user due to physical constraints of the lens and the mobile device, or for calibration and safety purposes such as in self-driving applications.

[0032] The present systems and methods are extendable to multiple (eg, more than two) cameras to determine disparity from multiple stereo cameras.

[0033] For example, one lens in a device is set to 1x zoom (e.g., no magnification) and has a wide angle FOV (e.g., wide FOV), while another lens in the device has 2x zoom (e.g., has 2x magnification) and has a narrower telephoto FOV (e.g., far FOV). The union of these two FOVs is the wide FOV. As disclosed herein, although corresponding pixels will only exist for the intersection of the FOVs as the far FOV, an estimation of the depth for the union of the FOVs, or in this case for the entire wide FOV, is performed.

[0034] Examples of the present disclosure provide a unified architecture for a deep neural network that can perform depth estimation from the union of FOVs of 2 or more cameras (rather than just the overlapping intersection of the FOVs), methods for training the unified architecture on multiple tasks simultaneously, and methods for fusing results from single image disparity estimation and stereo depth estimation algorithms / processing. Advantages include: depth estimation for the entire FOV across all cameras rather than just from the overlapping intersection of the FOVs, and generating aesthetically better images across the entire wide FOV by applying bokeh over the entire wide FOV rather than over the intersection FOV, which is the narrower telephoto FOV in the case of dual cameras with fixed preset zooms (such as wide 1x zoom and telephoto 2x fixed zoom).

[0035] Figure 1is an illustration of a bokeh effect according to one embodiment. Bokeh is the process of blurring background subjects in an image by causing distant subjects to be out of focus while making closer subjects clearer or in focus. Image 100 depicts the bokeh effect over a far FOV, while image 102 depicts the bokeh effect over a wide FOV. Overlapping FOV 104 corresponds to the entire FOV of image 100. This effect can be obtained by using more expensive cameras with large apertures (low f-stop numbers) and expensive lenses with long focal lengths to achieve a shallow depth of field. Due to the size and cost limitations of image sensors and lenses, these lenses are difficult to find in cheaper compact cameras or cameras on mobile devices.

[0036] Figure 2 is a diagram of a stereo matching system 200 according to one embodiment. Systems without accurate depth estimation may rely on stereo matching between two rectified images captured from two cameras 201 and 202 with the same FOV to determine the disparity d between two corresponding pixels (a horizontal offset as the difference between the horizontal distance x1 of point P relative to camera 201 and the horizontal distance x2 of point P relative to camera 202). For each pixel, the disparity can then be converted into a measurement of the depth z of the subject P through knowledge of the camera baseline b and focal length f.

[0037] Recent electronic devices are equipped with two or more cameras. The specifications of the two cameras may be (f / 1.7, 26 mm, 1x optical zoom) and (f / 2.4, 52 mm, 2x optical zoom), respectively. However, the first camera has a wider aperture and the second camera has a longer focal length and two times optical zoom. However, the FOV of the second camera is only the center part of the image at 2x zoom (e.g., far FOV). The FOV of the first camera is the entire wide FOV at 1x zoom. One reason for having multiple cameras with different FOVs is for diversity, where larger aperture cameras are used to obtain better images in low light settings and faster shutters, where the 2x zoom camera provides two times optical zoom and higher resolution images for the 2x FOV.

[0038] One application of stereo disparity matching is to create a bokeh effect in an image by blurring the background while keeping the object of interest in focus. However, in this case, stereo matching can only be done for the center FOV that overlaps between the two cameras. Therefore, if the user chooses to apply a bokeh effect to the captured image, it can only be done for the center far FOV, resulting in Figure 1 100 of images.

[0039] Figure 3 FIG. 3 is a flow chart 300 for applying a bokeh effect on an image according to one embodiment. At 302, the overlapping FOV between two images is determined. For example, referring to Figure 1, overlapping FOV 104 corresponds to the entire FOV of image 100 .

[0040] At 304, two images are generated with the same FOV and resolution. The images may be generated by cropping the overlapping FOV 104 from the image 102 having the larger FOV and reducing the overlapping FOV in the higher resolution image 100.

[0041] At 306, disparity information between the two images is determined. The disparity information may be determined by applying a stereo matching algorithm / process to the two images generated at 304. For overlapping FOV 104, the disparity information may correspond to a horizontal offset between image 100 and its corresponding pixel in image 102.

[0042] At 308, depth information for either image is generated. The depth information may be generated by transforming the disparity information from 306 by appropriate inversion and scaling.

[0043] At 310 , a bokeh algorithm / process is applied. The bokeh algorithm / process may blur distant objects and focus on near objects, producing a result similar to image 100 , where the bokeh effect may only be applied to the far FOV of image 100 .

[0044] As described herein, overlapping FOV may refer to the FOV present in all cameras used, even if the FOVs are at different scales. Joint FOV may refer to the FOV produced by adding the FOV of one reference camera to the FOV of another camera after adjusting the scale to that of the reference image.

[0045] Figure 4 4 is a diagram of images representing a joint FOV and an overlapping FOV, according to one embodiment. Image 400 represents the joint FOV, while image 402 represents the overlapping FOV. Image 400 includes an overlapping portion 404 with image 402. Image 402 is downsampled by one-half to match the resolution of image 400 of the overlapping FOV. Objects of interest are typically not centered on the far FOV of image 402 and are not completely captured by the far FOV of image 402. Therefore, as disclosed herein, the overlapping FOV of image 402 can be used to generate a disparity estimate from the joint FOV of image 400.

[0046] Figure 5 5 is a flow chart for generating disparity of a joint FOV according to one embodiment. At 502, a depth estimate of the joint FOV is generated, and a disparity estimate of the joint FOV is generated. The depth estimate may be generated using scene understanding using single image disparity estimation (SIDE).

[0047] Figure 6is a diagram of an RGB-SIDE network according to one embodiment. The RGB-SIDE network is implemented using a fully convolutional neural network. Disparity estimation is performed for a full wide FOV (e.g., joint FOV 400), where information from only one camera is available in certain parts of the FOV. A joint FOV image is input at 600, and a disparity estimate (i.e., single image disparity) is output at 602. The RGB-SIDE network estimates inverse depth by using the current camera settings (e.g., camera baseline, camera focal length, etc.), and implicitly scales the inverse depth to disparity within the network. In this process, a regression-based loss function can be used as a measure of disparity error, where, for each pixel, the distance between the estimated disparity and the true disparity gives an error cost function for stochastic gradient descent optimization.

[0048] Since the RGB-SIDE network needs to understand the scene to learn the relative positions of different objects with respect to the camera lens, the full width image 400 is processed at once. The advantage of this scheme is that the output provides an estimate of the disparity for the full width FOV using only one image and one block. The disadvantage of this scheme is the lack of accuracy, where the estimated disparity or estimated depth is very rough, has fuzzy edges, and is prone to large errors. The reason for this is the reliance on scene understanding and the relative positions between objects and the size of the objects to estimate the depth of objects in the FOV.

[0049] At 504, disparity estimates for overlapping FOVs are generated. This may be accomplished by using stereo matching disparity estimation (SMDE) in two images (e.g., Figure 3 A disparity estimate is generated between two images generated by 304 of FIG. 304 . The SMDE network estimates the disparity from the two input images. The SMDE network can estimate the disparity using a neural network. Various types of SMDE networks can be used, such as a far-wide SMDE network (TW-SMNet) applicable to overlapping FOVs (TW-SMNet(T)) or a far-wide SMDE network (TW-SMNet) applicable to joint FOVs (TW-SMNet(W)), and a multi-task TW-SMNet (MT-TW-SMNet) that adds an additional SIDE network to TW-SMNet(W).

[0050] Figure 7is a diagram of a TW-SMNet according to one embodiment. In the network, both image 400 and image 402 are input into the network at 700 and 702, respectively. TW-SMNet uses stereo matching to estimate the disparity from the two input images. Stereo matching by classical techniques includes matching features in the corrected images along the same horizontal line. The deep neural network can also perform stereo matching by generating offset versions of the feature maps of the images at different disparity offsets to construct a volume called a cost volume. Stereo matching deep neural networks rely on calculating cost volumes, such as the distance between depth features, or the correlation between offset depth features.

[0051] For TW-SMNet(T), stereo matching is used to provide disparity estimates only for overlapping regions. The overlapping portion 404 is the overlapping FOV in image 400 and image 402. This is done by considering only the overlapping portion 404 in the wide FOV image 400 and stereo matching it with the far FOV image 402. The algorithm or neural network is designed to regress to the true disparity using only the overlapping regions. This approach will give the best accuracy for disparity estimates in far regions.

[0052] For TW-SMNet(W), the algorithm / processing uses the full width FOV of image 400 to estimate the disparity for the joint FOV. Image 402 is scaled to match the resolution of overlap 404 in image 400. However, in this case the missing areas representing the surrounding areas outside overlap 404 are zero-filled to reach the size of the full width FOV. The output from this block is an estimate of the disparity map for the full width FOV. That is, a joint FOV disparity estimate can be generated between the zero-filled overlap FOV and the joint FOV from the stereo matching disparity estimation (SMDE) network. This scheme will give a good accuracy for disparity estimates in far areas, and a fairly rough estimate in surrounding areas.

[0053] Figure 8 is a diagram of MT-TW-SMNet according to one embodiment. Figure 8 , a wide FOV image 400 is input at 800 , and a far FOV image 402 is input at 800 . An additional SIDE network is applied to the FOV image 400 input at 800 .

[0054] The network is trained for disparity estimation by stereo matching. Before the cost volume 804, an additional head based on SIDE estimation only for image 400 is added to the network. The two heads are trained together, so the SIDE branch helps the shared layer have better scene understanding capabilities. The network stereo matching loss is regularized relative to the SIDE loss. Therefore, the loss function is a combined loss function 806 that takes into account the accuracy of the disparity estimation from the SIDE branch and the accuracy of the disparity estimation from the SMDE branch, and the system outputs the disparity at 808. That is, a joint FOV disparity estimate can be generated between the zero-filled overlapping FOV and the joint FOV from the multi-task stereo matching disparity (MT-SMDE) and single image disparity estimation network. This improves performance in the surrounding area (non-overlapping FOV), where stereo matching cannot be performed due to the lack of corresponding objects in the stereo images in this area. However, only one disparity map is output from this network, which is a function of both the disparity map from the SMDE branch and the disparity map of the SMDE branch. The network can select only the stereo matching disparity map as the final output since it usually has better accuracy.

[0055] At 506, the joint FOV disparity estimation is enhanced. Fig. 9 is a diagram of a parallax enhancement system according to one embodiment. Fig. 9 , a wide FOV image 400 is input to 900 of TW-SMNet 904, and a far FOV image 402 is input to 902 of TW-SMNet 904 which generates and outputs disparity. The wide FOV image 400 is also input to a SIDE RGB-disparity (SIDE-RGBD) estimation network 910 at 908, and the disparity output by TW-SMNet 904 is input to 906 of the SIDE-RGBD network 910.

[0056] At 508 , the estimated disparities are merged. For example, the joint FOV disparity estimate obtained at 502 or 506 and the overlapped FOV disparity estimate obtained at 504 are merged. Figures 10 to 13 is a diagram of a system for generating disparity estimates to be merged and post-processed, according to one embodiment.

[0057] Fig.10 Depicted is an RGB SIDE + TW-SMNet(T) system 1000. A wide FOV image 1002 is input to an RGB SIDE network 1006 and a TW-SMNet(T) 1008, while a far FOV image 1004 is input to a TW-SMNet(T) 1008. The resulting disparity estimates are merged at 1010, and the disparity map is processed at 1012.

[0058] Fig.11Depicted is an RGBD SIDE+TW-SMNet system 1100. A wide FOV image 1102 is input to an RGBD SIDE network 1106 and a TW-SMNet 1108, while a far FOV image 1104 is input to a TW-SMNet 1108. The resulting disparity estimates are merged at 1110, and the disparity map is processed at 1112.

[0059] Fig.12 Depicted is an RGB SIDE+RGBD SIDE+TW-SMNet system 1200. A wide FOV image 1202 is input to an RGBD SIDE network 1206, an RGB SIDE network 1208, and a TW-SMNet 1210, while a far FOV image 1204 is input to a TW-SMNet 1210. The disparity estimate from the RGB SIDE network 1208 is input to the RGBD SIDE network 1206. The resulting disparity estimates from the RGBD SIDE network 1206 and the TW-SMNet 1210 are merged at 1212, and the disparity map is processed at 1214.

[0060] Fig.13 Depicted is an RGB SIDE+RGBD SIDE+MT-TW-SMNet system 1300. A wide FOV image 1302 is input to an RGBD SIDE network 1306, an RGB SIDE network 1308, and an MT-TW-SMNet 1310, while a far FOV image 1304 is input to an MT-TW-SMNet 1310. The disparity estimate from the RGB SIDE network 1308 is input to the RGBD SIDE network 1306. The resulting disparity estimates from the RGBD SIDE network 1306 and the TW-SMNet 1310 are merged at 1312, and the disparity map is processed at 1314.

[0061] Disparity merging can be achieved by estimating two disparity maps d 1 , d 2 The bias b between the overlapping FOV (OL_FOV) of FIG. 1 and FIG. 2 is performed based on the bias adjustment, as shown in equation (1):

[0062]

[0063] Where n is the number of pixels in the overlapping FOV, and d 1 (i, j) represents the disparity map d 1 The disparity at coordinate (i, j) in 2 (i, j) represents the disparity map d 2 The disparity at the coordinate (i, j) in d 2The surrounding disparity of can be adjusted based on b, and then d can be applied 1 Weighted sum or parallax selection.

[0064] Disparity merging can be performed based on scaling, where 1 , d 2 The estimated scaling difference s between the overlapping FOVs is , where n is the number of pixels in the overlapping FOVs, as shown in Equation (2):

[0065]

[0066] d 2 The surrounding parallax of can be adjusted based on s, and then d can be applied 1 Weighted sum or parallax selection.

[0067] At 510, a disparity map is selected. Disparity merging can also be achieved through a learned non-linear function from a neural network, which can be implemented by stacking hourglass networks. Fig.14 is a diagram of parallax merging and selection according to one embodiment. Fig.14 , SIDE disparity 1402 and SMDE disparity 1404 are merged by a disparity merging network. The network is trained by regressing to ground truth disparity. One way to train the network is to quantize the disparity into multiple levels and learn a classifier that outputs a soft probability for each level. The regression output 1406 is the expected value of the disparity, where the expectation is calculated using the estimated probability. In addition, the output merged disparity 1406 is a nonlinear function of the input disparity obtained by SIDE 1402 and SMDE 1404.

[0068] By using SMDE, the parallax of the overlapping FOVs is likely to be the most accurate. Therefore, the selector 1408 can choose to select the merged parallax 1406 for the surrounding area, or select the parallax from the SMDE 1404 for the overlapping far FOV.

[0069] Fig.15 is a diagram of parallax merging and selection according to one embodiment. Fig.15In , stereo matching (SM) can be used (e.g., by using TW-SMNet(T)) to perform more accurate disparity estimates only for overlapping regions. However, SM performed on a zero-padded wide FOV can produce disparity estimates for both overlapping and surrounding regions using TW-SMNet(W) for the full-width FOV, which can be merged using a fusion network to produce more accurate disparity estimates for surrounding regions. However, if complexity permits, the best algorithm / processing for overlapping far FOVs can be utilized by utilizing a third input of a disparity block that represents a disparity estimate from an accurate SM performed on overlapping far regions. For example, SIDE disparity 1502 and TW-SMNet(W) disparity 1504 are merged into merged disparity 1508, and a selection 1510 is performed between TW-SMNet(T) disparity 1506 and merged disparity 1508.

[0070] Wide FOV and far FOV RGB image features can also be input to the disparity merging block. Other features extracted from the RGB image (such as edge maps or semantic segmentation maps) can be further concatenated with different disparity maps as input features.

[0071] At 512, the disparity map is post-processed to enhance quality (eg, via Figures 10 to 13 The post-processing block aims to provide visually pleasing disparity maps. Even after merging using non-linear operations like in neural networks, discontinuities around the borders between overlapping and non-overlapping FOVs are observable. Post-processing aims to smooth out such effects so the border effects are not noticeable. It is also used to adjust the output disparity for the desired application.

[0072] An example of post-processing to reduce the effect of sudden changes in disparity around the boundaries of overlapping regions is to smooth the disparity. However, an edge usually represents an object, and the goal is to fill the object with a smooth depth estimate. One solution is to use edge-preserving smoothing. Edge-preserving smoothing can be achieved by calculating the filtered output as a weighted average that can be implemented iteratively. Local filters such as bilateral filters can be used. A limitation of bilateral filters and other local filters is that they may not be able to resolve the ambiguity of whether to retain or smooth a specific edge in the disparity map. An RGB image can be used as a guide for smoothing, which is called edge-guided filtering, to maintain the edges in the RGB image. A global object function defined by data constraints and smoothing priors can also be optimized using a method called a fast global smoother (FGS). Therefore, the FGS filter value around the boundary depends on the entire disparity map. In one example, by deploying FGS, a global filter is used to calculate the filter value around the boundary. Then, only the strips around the boundary in the merged disparity are replaced by the filtered strips, and the remaining strips deploy the original merged values.

[0073] As described above, the present system and method utilizes only two cameras. However, this can be easily extended to more than two cameras. Fig.16 is a diagram of joint FOV and overlapping FOV when there are four cameras, according to one embodiment. Fig.16 A verged camera configuration 1602 is shown, where the dark area 1604 shows the overlapping FOVs as a common 3D space viewable from all cameras, and the dashed area 1606 shows the joint FOV.

[0074] A simple approach is to rectify each reference image and stereo match it with (number of cameras - 1) rectified images respectively. By using deep learning methods, it can be used to obtain very accurate depth estimates for FOV 1604 that overlaps across cameras. Because the positions of the cameras relative to each other are fixed, the disparity between any pair of rectified images should be converted to a specific value between the remaining pairs, which can be used to obtain more accurate results for overlapping FOV 1604. Portions of the joint FOV 1606 will overlap between the two cameras, but not between all cameras. The SM between these camera pairs can be used to obtain a good estimate of the disparity in this area. Areas in the joint FOV 1606 that are seen by only one camera will utilize single image disparity estimation. Optionally, the joint FOV 1606 can utilize disparity estimates for all input images, as well as at least the portions of the joint FOV 1606 that overlap between cameras. The other methods described above for fusion between disparity maps and post-processing smoothing are also applicable to this example.

[0075] about Fig.16 An alternative example is a system in which three cameras are utilized. The first camera may have a far FOV, the second camera may have a wide FOV, and the third camera may have an ultra-wide FOV. Figure 5 As described above, for images from the far FOV and images from the wide FOV, the joint FOV disparity estimate can be merged with the overlapped FOV disparity estimate. This method can be recursively repeated to generate a depth estimate for the ultra-wide FOV by using the previously estimated depth of the wide FOV through the SM between the ultra-wide FOV and the wide FOV. In these examples, the ultra-wide FOV can be used as the joint FOV, and the wide FOV can be used as the overlapped FOV.

[0076] Fig.17 is a block diagram of an electronic device 1701 in a network environment according to one embodiment. Fig.17, the electronic device 1701 in the network environment 1700 may communicate with the electronic device 1702 via a first network 1798 (e.g., a short-range wireless communication network), or communicate with the electronic device 1704 or the server 1708 via a second network 1799 (e.g., a long-range wireless communication network). The electronic device 1701 may communicate with the electronic device 1704 via the server 1708. The electronic device 1701 may include: a processor 1720, a memory 1730, an input device 1750, a sound output device 1755, a display device 1760, an audio module 1770, a sensor module 1776, an interface 1777, a haptic module 1779, a camera module 1780, a power management module 1788, a battery 1789, a communication module 1790, a user identification module (SIM) 1796, or an antenna module 1797. In one embodiment, at least one of the plurality of components (e.g., display device 1760 or camera module 1780) may be omitted from electronic device 1701, or one or more other components may be added to electronic device 1701. In one embodiment, some of the plurality of components may be implemented as a single integrated circuit (IC). For example, sensor module 1776 (e.g., fingerprint sensor, iris sensor, or illumination sensor) may be embedded in display device 1760 (e.g., display).

[0077] The processor 1720 may execute, for example, software (e.g., program 1740) to control at least one other component (e.g., hardware or software component) of the electronic device 1701 in combination with the processor 1720, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 1720 may load a command or data received from another component (e.g., sensor module 1776 or communication module 1790) into the volatile memory 1732, process the command or data stored in the volatile memory 1732, and store the resultant data in the non-volatile memory 1734. The processor 1720 may include a main processor 1721 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 1723 (e.g., a graphics processor (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)), and the auxiliary processor 1723 may operate independently of the main processor 1721 or operate in coordination with the main processor 1721. Additionally or alternatively, the auxiliary processor 1723 may be adapted to consume less power than the main processor 1721, or to perform specific functions. The auxiliary processor 1723 may be implemented separately from the main processor 1721, or as part of the main processor 1721.

[0078] The auxiliary processor 1723 may replace the main processor 1721 when the main processor 1721 is in an inactive (e.g., sleeping) state, or control at least some of the functions or states related to at least one component (e.g., the display device 1760, the sensor module 1776, or the communication module 1790) of the plurality of components of the electronic device 1701 together with the main processor 1721 when the main processor 1721 is in an active state (e.g., executing an application). According to one embodiment, the auxiliary processor 1723 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 1780 or a communication module 1790) that is functionally related to the auxiliary processor 1723.

[0079] The memory 1730 may store various data used by at least one component of the electronic device 1701 (e.g., the processor 1720 or the sensor module 1776). For example, the various data may include input data or output data of software (e.g., the program 1740) and commands related thereto. The memory 1730 may include a volatile memory 1732 or a nonvolatile memory 1734.

[0080] The program 1740 may be stored as software in the memory 1730 , and may include, for example, an operating system (OS) 1742 , middleware 1744 , or an application 1746 .

[0081] The input device 1750 may receive commands or data to be used by other components (eg, the processor 1720) of the electronic device 1701 from outside the electronic device 1701 (eg, a user). For example, the input device 1750 may include a microphone, a mouse, or a keyboard.

[0082] The sound output device 1755 can output sound signals to the outside of the electronic device 1701. For example, the sound output device 1755 may include a speaker or a receiver. The speaker can be used for general purposes (such as playing multimedia or recording), and the receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented as a part of the speaker or as a separate speaker.

[0083] The display device 1760 may visually provide information to the outside of the electronic device 1701 (e.g., a user). For example, the display device 1760 may include a display, a hologram device, or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to one embodiment, the display device 1760 may include a touch circuit suitable for detecting a touch, or a sensor circuit (e.g., a pressure sensor) suitable for measuring the strength of a force caused by a touch.

[0084] The audio module 1770 may convert sound into an electrical signal, and vice versa. According to one embodiment, the audio module 1770 may obtain sound via the input device 1750, or output sound via the sound output device 1755 or an earphone of an external electronic device 1702 directly (e.g., wired) or wirelessly combined with the electronic device 1701.

[0085] The sensor module 1776 may detect an operating state (e.g., power or temperature) of the electronic device 1701 or an environmental state (e.g., a state of a user) outside the electronic device 1701, and then generate an electrical signal or data value corresponding to the detected state. For example, the sensor module 1776 may include a posture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0086] The interface 1777 may support one or more specific protocols for the electronic device 1701 to be directly (e.g., wired) or wirelessly coupled with the external electronic device 1702. According to one embodiment, for example, the interface 1777 may include a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0087] The connection terminal 1778 may include a connector, and the electronic device 1701 may be physically connected to the external electronic device 1702 via the connector. According to one embodiment, for example, the connection terminal 1778 may include an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

[0088] The haptic module 1779 may convert an electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation, which may be recognized by a user via tactile or kinesthetic sense. According to one embodiment, for example, the haptic module 1779 may include a motor, a piezoelectric element, or an electrical stimulator.

[0089] The camera module 1780 may capture still images or moving images. According to one embodiment, the camera module 1780 may include one or more lenses, image sensors, image signal processors, or flashes.

[0090] The power management module 1788 may manage power supplied to the electronic device 1701. For example, the power management module 1788 may be implemented as at least part of a power management integrated circuit (PMIC).

[0091] The battery 1789 may supply power to at least one component of the electronic device 1701. According to one embodiment, for example, the battery 1789 may include a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0092] The communication module 1790 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 1701 and an external electronic device (e.g., electronic device 1702, electronic device 1704, or server 1708), and perform communication via the established communication channel. The communication module 1790 may include one or more communication processors that can operate independently of the processor 1720 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 1790 may include a wireless communication module 1792 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1794 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a first network 1798 (e.g., a short-range communication network (such as, Bluetooth TM , Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA) standards) or a second network 1799 (e.g., a long-distance communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))))) to communicate with an external electronic device. These various types of communication modules may be implemented as a single component (e.g., a single IC), or may be implemented as multiple components separated from each other (e.g., multiple ICs). The wireless communication module 1792 may use user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 1796 to identify and authenticate the electronic device 1701 in a communication network (such as, the first network 1798 or the second network 1799).

[0093] The antenna module 1797 may transmit a signal or power to the outside of the electronic device 1701 (e.g., an external electronic device), or receive a signal or power from the outside of the electronic device 1701 (e.g., an external electronic device). According to one embodiment, the antenna module 1797 may include one or more antennas, and for example, the communication module 1790 (e.g., the wireless communication module 1792) may select at least one antenna suitable for a communication scheme used in a communication network (e.g., the first network 1798 or the second network 1799) from the antennas. Then, a signal or power may be transmitted or received between the communication module 1790 and the external electronic device via the selected at least one antenna.

[0094] At least some of the above components may be coupled to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and transmit signals (e.g., commands or data) therebetween.

[0095] According to one embodiment, a command or data may be sent or received between the electronic device 1701 and the external electronic device 1704 via the server 1708 combined with the second network 1799. Each of the electronic devices 1702 and 1704 may be a device of the same type or a different type as the electronic device 1701. All or some of the operations to be performed at the electronic device 1701 may be performed at one or more of the external electronic device 1702, the external electronic device 1704, or the external electronic device 1708. For example, if the electronic device 1701 should perform a function or service automatically or in response to a request from a user or another device, the electronic device 1701 may request one or more external electronic devices to perform at least part of the function or service without performing the function or service, or the electronic device 1701 may request one or more external electronic devices to perform at least part of the function or service in addition to performing the function or service. The one or more external electronic devices receiving the request may perform at least part of the requested function or service or an additional function or additional service related to the request, and transmit the result of the execution to the electronic device 1701. The electronic device 1701 may provide the result as at least part of a reply to the request, with or without further processing the result. To this end, for example, cloud computing, distributed computing, or client-server computing techniques may be used.

[0096] One embodiment may be implemented as software (e.g., program 440), which includes one or more instructions stored in a storage medium (e.g., internal memory 1736 or external memory 1738) that can be read by a machine (e.g., electronic device 1701). For example, the processor of the electronic device 1701 may call at least one of the one or more instructions stored in the storage medium, and execute it with or without one or more other components under the control of the processor. Thus, the machine may be operated to perform at least one function according to at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between a case where data is semi-permanently stored in the storage medium and a case where data is temporarily stored in the storage medium.

[0097] According to one embodiment, the method according to the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., PlayStore). TM ) online distribution (e.g., downloading or uploading), or directly between two user devices (e.g., smartphones). If distributed online, at least part of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0098] According to one embodiment, each component (e.g., module or program) of the above-mentioned components may include a single entity or multiple entities. One or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as the corresponding components in the multiple components performed the functions before the integration. The operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed or omitted in a different order, or one or more other operations may be added.

[0099] Although certain embodiments of the present disclosure have been described in the specific embodiments of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined based solely on the described embodiments, but should be determined based on the attached claims and their equivalents.

Claims

1. An electronic device, comprising: a first camera having a first field of view; a second camera having a second field of view narrower than the first field of view; as well as The processor is configured as: capturing a first image using a first camera, the first image having a joint field of view; capturing a second image using a second camera; determining an overlapping field of view between the first image and the second image; generating an overlapping field of view disparity estimate based on the overlapping fields of view; generating a joint field of view disparity estimate based on the second image scaled to a resolution of the overlapping portion of the first image; as well as The joint view disparity estimate is merged with the overlapping view disparity estimate.

2. The electronic device according to claim 1, wherein: Disparity estimates for overlapping fields of view are generated using a stereo matching network disparity estimate using overlapping fields of view.

3. The electronic device according to claim 1, wherein: The processor is further configured to crop the determined overlapping fields of view from the first image and scale the overlapping fields of view from the second image to a resolution corresponding to the first image.

4. The electronic device according to claim 3, wherein: The processor is further configured to enhance the joint view field disparity estimate using the overlapping view field disparity estimate.

5. The electronic device according to claim 1, wherein: A joint field of view disparity estimate is generated based on the depth estimate from the first field of view of only the first image and information about a camera baseline and focal length of the first camera.

6. The electronic device according to claim 1, wherein: The processor is further configured to merge the joint view disparity estimate with the overlap view disparity estimate based on a bias in the overlap view and a scaling calibration in the joint view disparity estimate and the overlap view disparity estimate.

7. The electronic device according to claim 1, wherein: A joint field of view disparity estimate is generated based on the disparity estimate from a red-green-blue single image disparity estimation network.

8. The electronic device according to claim 1, wherein: A joint field of view disparity estimate is also generated between the zero-filled overlapping field of view and the joint field of view from the stereo matching disparity estimation network.

9. The electronic device according to claim 1, wherein: A joint field of view disparity estimate is also generated between the zero-filled overlapping field of view and the joint field of view from the multi-task stereo matching disparity and single image disparity estimation network.

10. The electronic device according to claim 1, further comprising: a third camera having a third field of view wider than the first field of view; The processor is further configured to: capturing a third image using a third camera, the third image having a third field of view; and A depth estimate for the third image is generated based on the depth estimate for the joint field of view of the first image.

11. A method for disparity estimation, comprising: capturing a first image using a first camera having a first field of view, the first image having a joint field of view; capturing a second image using a second camera having a second field of view narrower than the first field of view; determining an overlapping field of view between the first image and the second image; generating an overlapping field of view disparity estimate based on the overlapping fields of view; generating a joint field of view disparity estimate based on the second image scaled to a resolution of the overlapping portion of the first image; as well as The joint view disparity estimate is merged with the overlapping view disparity estimate.

12. The method according to claim 11, wherein: Overlapping field of view disparity estimates are generated using a stereo matching network disparity estimate utilizing overlapping fields of view.

13. The method according to claim 11, further comprising: The determined overlapping fields of view are cropped from the first image, and the overlapping fields of view from the second image are scaled to a resolution corresponding to the first image.

14. The method according to claim 13, further comprising: Enhance the joint view disparity estimation using overlapping view disparity estimates.

15. The method according to claim 11, wherein: A joint field of view disparity estimate is generated based on the depth estimate from the first field of view of only the first image and information about a camera baseline and focal length of the first camera.

16. The method according to claim 11, wherein: The step of merging the joint view disparity estimate with the overlap view disparity estimate is performed based on a bias in the overlap view in the joint view disparity estimate and the overlap view disparity estimate and a scaling calibration.

17. The method according to claim 11, wherein: A joint field of view disparity estimate is generated based on the disparity estimate from a red-green-blue single image disparity estimation network.

18. The method according to claim 11, wherein: A joint field of view disparity estimate is also generated between the zero-filled overlapping field of view and the joint field of view from the stereo matching disparity estimation network.

19. The method according to claim 11, wherein: A joint field of view disparity estimate is also generated between the zero-filled overlapping field of view and the joint field of view from the multi-task stereo matching disparity and single image disparity estimation network.

20. The method according to claim 11, further comprising: capturing a third image using a third camera having a third field of view wider than the first field of view; as well as A depth estimate for the third image is generated based on the depth estimate for the joint field of view of the first image.

Images