Depth estimation method and apparatus

Abstract

A depth estimation method and apparatus are provided. The depth estimation method includes obtaining an image, wherein pixels of the image have a plurality of sub-pixels; obtaining a plurality of sub-images based on the image, wherein pixels of the sub-images are the sub-pixels; obtaining a disparity between the sub-images; and obtaining a depth of the image based on the disparity between the sub-images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and more particularly, to a depth estimation method and device. BACKGROUND

[0002] With the development of portable electronic devices (such as smart phones, tablets, etc.), whether the camera of the device can meet the use requirements has become one of the most important indicators for users to choose a product. In addition, due to the limitations of professional cameras in terms of use difficulty, market price, and portability, how to use the camera of a portable electronic device to simulate the shooting effect of a professional camera has also become an important issue in the development of the camera function of the portable electronic device.

[0003] Among the many shooting effects of professional cameras, the diversified depth-of-field effect is one of the most distinctive features that distinguish the camera of a portable electronic device. Depth of field refers to the distance range in front of and behind the subject that the camera can capture clearly. For example, when shooting a portrait, a small depth of field can make the in-focus portrait clear, while the background outside the depth of field is blurred, making the photo present a three-dimensional sense of highlighting the portrait. For another example, when shooting a landscape photo with a person, a large depth of field can make the portrait and the distant background clear at the same time, highlighting the portrait and preserving the landscape. Based on the advantages of its hardware, a professional camera can often provide a range of depth-of-field choices, allowing users to select different aperture sizes and focal lengths according to their needs in different shooting scenarios to achieve different depth-of-field shooting effects. Due to the hardware limitations of the camera of a portable electronic device, it cannot achieve the same level of aperture and focal length as a professional camera, and thus cannot present diversified depth-of-field effects. Therefore, it is necessary to simulate such effects on a portable electronic device through algorithms.

[0004] The professional camera depth-of-field effect simulated on a portable electronic device is commonly referred to as a defocus mode or a Bokeh effect. In the prior art, the Bokeh effect can be achieved through image segmentation, monocular or binocular depth estimation, etc. The subject is separated from the image, and the content outside the subject (such as the background) is blurred. However, these existing methods have the disadvantages of being unable to obtain reasonable effects corresponding to real scenes, being unable to distinguish between real 3D scenes and false 3D scenes (such as high-quality printed photos, played videos, etc.), being unable to calculate effective depth information, and requiring high hardware and high power consumption. SUMMARY

[0005] According to an example embodiment of the present application, a depth estimation method is provided, comprising: obtaining an image, wherein pixels of the image have a plurality of sub-pixels; obtaining a plurality of sub-images based on the image, wherein pixels of the sub-images are the sub-pixels; obtaining disparity between the sub-images; and obtaining depth of the image based on the disparity between the sub-images.

[0006] The step of obtaining the disparity between the sub-images can comprise pairing the plurality of sub-images two by two, and obtaining the disparity between the two-by-two paired sub-images.

[0007] The step of obtaining the disparity between the sub-images can further comprise at least one of: obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired input sub-images, and obtaining the disparity between the sub-images based on the difference value; performing convolution on the two-by-two paired input sub-images, obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired sub-images after convolution, and obtaining the disparity between the sub-images based on the difference value; performing displacement on the two-by-two paired input sub-images, obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired sub-images after displacement, and obtaining the disparity between the sub-images based on the difference value; performing displacement on the two-by-two paired input sub-images, performing convolution on the two-by-two paired sub-images after displacement, obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired sub-images after convolution, and obtaining the disparity between the sub-images based on the difference value.

[0008] The two-by-two paired input sub-images can be displaced in opposite directions and / or with the same displacement value.

[0009] The step of obtaining the disparity between the sub-images based on the difference value can comprise comparing the difference value with a predetermined constant, and determining the disparity between the sub-images according to a comparison result.

[0010] The depth estimation method can further comprise obtaining an edge detection map and / or an image segmentation map of the image based on the plurality of sub-images.

[0011] The plurality of sub-images can be four sub-images.

[0012] The four sub-images can comprise a top-left sub-image composed of sub-pixels at a top-left position in each pixel, a bottom-left sub-image composed of sub-pixels at a bottom-left position in each pixel, a top-right sub-image composed of sub-pixels at a top-right position in each pixel, and a bottom-right sub-image composed of sub-pixels at a bottom-right position in each pixel.

[0013] According to an example embodiment of the present application, a depth estimation device is provided, comprising: an image obtaining unit, obtaining an image, wherein pixels of the image have a plurality of sub-pixels; a sub-image obtaining unit, obtaining a plurality of sub-images based on the image, wherein pixels of the sub-images are the sub-pixels; a disparity obtaining unit, obtaining disparities between the sub-images; and a depth estimation unit, obtaining depths of the image based on the disparities between the sub-images.

[0014] The disparity obtaining unit can pair the plurality of sub-images two by two, and obtain the disparities between the two-by-two paired sub-images.

[0015] The disparity obtaining unit can further perform at least one of the following operations: obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired input sub-images, and obtaining the disparity between the sub-images based on the difference value; performing convolution on the two-by-two paired input sub-images, obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired sub-images after convolution, and obtaining the disparity between the sub-images based on the difference value; performing displacement on the two-by-two paired input sub-images, obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired sub-images after displacement, and obtaining the disparity between the sub-images based on the difference value; performing displacement on the two-by-two paired input sub-images, performing convolution on the two-by-two paired sub-images after displacement, obtaining a difference value of the sub-pixels at corresponding positions of the two-by-two paired sub-images after convolution, and obtaining the disparity between the sub-images based on the difference value.

[0016] The disparity obtaining unit can displace the two-by-two paired input sub-images in opposite directions and / or with the same displacement value.

[0017] The disparity obtaining unit can compare the difference value with a predetermined constant, and determine the disparity between the sub-images according to a comparison result.

[0018] The depth estimation device can further comprise: an edge detection map and / or image segmentation map obtaining unit, obtaining an edge detection map and / or an image segmentation map of the image based on the plurality of sub-images.

[0019] The plurality of sub-images can be four sub-images.

[0020] The four sub-images can comprise a top-left sub-image composed of sub-pixels at a top-left position in each pixel, a bottom-left sub-image composed of sub-pixels at a bottom-left position in each pixel, a top-right sub-image composed of sub-pixels at a top-right position in each pixel, and a bottom-right sub-image composed of sub-pixels at a bottom-right position in each pixel.

[0021] According to an exemplary embodiment of the present application, there is provided an electronic device comprising a memory and a processor, the memory having stored thereon computer-executable instructions which, when executed by the processor, perform the method described above.

[0022] According to an exemplary embodiment of the present application, there is provided a computer-readable medium having stored thereon computer-executable instructions which, when executed, perform the method described above.

[0023] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the description which follows, and in part will be obvious from the description, or can be learned by practice of the present general inventive concept. BRIEF DESCRIPTION OF DRAWINGS

[0024] The foregoing and other objects and features of the present application will become more fully apparent from the following detailed description, taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 is a flowchart illustrating a depth estimation method according to an exemplary embodiment of the present application;

[0026] Figure 2 is a schematic diagram of an image sensor of a full-pixel quad-core camera according to an exemplary embodiment of the present application;

[0027] Figure 3 is a schematic diagram of sub-image synthesis according to an exemplary embodiment of the present application;

[0028] Figure 4 is a schematic diagram of disparity calculation according to an exemplary embodiment of the present application;

[0029] Figure 5 is a schematic diagram of four disparity calculation methods according to an exemplary embodiment of the present application;

[0030] Figure 6 is a schematic diagram of objects at different distances from a camera according to an exemplary embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a synthesized depth map according to an exemplary embodiment of the present application;

[0032] Figure 8 is a schematic diagram of an overall depth estimation method according to an exemplary embodiment of the present application;

[0033] Figure 9 is a schematic diagram of edge extraction by a third neural network according to an exemplary embodiment of the present application;

[0034] Figure 10is a schematic diagram illustrating image segmentation by a fourth neural network according to an exemplary embodiment of the present invention;

[0035] Figure 11 is a schematic diagram illustrating a multi-task learning network according to an exemplary embodiment of the present invention;

[0036] Figure 12 is a schematic diagram illustrating a training phase of an overall depth estimation method according to an exemplary embodiment of the present invention;

[0037] Figure 13 is a schematic diagram illustrating a testing phase of an overall depth estimation method according to an exemplary embodiment of the present invention;

[0038] Figure 14 is a block diagram illustrating a depth estimation apparatus according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0039] Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present invention.

[0040] Figure 1 is a flowchart illustrating a depth estimation method according to an exemplary embodiment of the present invention.

[0041] Reference Figure 1 , in step S110, an image may be acquired, wherein a pixel of the image has a plurality of sub-pixels, and here, each pixel of the image may have a plurality of sub-pixels. Here, the image may be acquired based on a camera having a phase detection (i.e., phase autofocus) function. It should be noted that the camera may be a camera installed on any portable terminal (such as a smart phone, a tablet computer, etc.). More specifically, full-pixel multi-core (such as full-pixel dual-core, full-pixel quad-core, etc.) technology as a way to implement the phase detection function can be used to acquire images by a full-pixel quad-core camera, which is only an example and not a limitation. The image sensor of a full-pixel quad-core camera can simultaneously generate four sub-pixels for each pixel.

[0042] Figure 2 is a diagram illustrating an image sensor of a full-pixel quad-core camera according to an exemplary embodiment of the present invention, Figure 2 As shown in (a), the slash and blank boxes indicate that the image sensor can use a Bayer pattern (such as an RGGB array) to provide three channels (such as R, G, and B) for each pixel. The pixels of each channel include four 2*2 sub-pixels, which can be called a quadruple Bayer array and can be Figure 2 The right side of (a) shows that the quadruple Bayer array will be transformed into an array;Figure 2 (b) also shows a four-bayer array in which one pixel is composed of four sub-pixels. Accordingly, in the case of acquiring an image by using a full-pixel four-core camera, the plurality of sub-pixels can be four sub-pixels.

[0043] At step S120, a plurality of sub-images (the sub-images can also be referred to as phase images) can be acquired based on the image, wherein the pixels of the sub-images are the sub-pixels. More specifically, the acquired image can be processed according to the rule of a Bayer array to obtain a plurality of sub-images, that is, the sub-pixels at the corresponding positions in each pixel can be combined into a sub-image. The corresponding positions can be the same positions, for example, but are not limited thereto. A camera with a phase auto-focusing function can provide a plurality of original data output modes after taking a picture of a scene, such as a color image (such as an RGB image) and a phase image. The RGB image is a common output image, and each pixel value on the final image is obtained from RGGB three-primary-color information in a Bayer arrangement, and each information of the three channels of RGB is combined from the difference values of a plurality of sub-pixels. The phase image is another data format, which can be regarded as a single-channel "gray image" and directly outputs the original pixel value. Through processing, a sub-image with the same number of sub-pixels in the Bayer array can be separated. For a full-pixel four-core technology, that is, a four-bayer array, four sub-images can be separated. Figure 3 is a schematic diagram showing sub-image combination according to an exemplary embodiment of the present application, as Figure 3 In the case of acquiring an image by using a full-pixel four-core camera as described above, four sub-images can be obtained, that is, the sub-pixels at the upper-left positions in each pixel can be combined into an upper-left sub-image, the sub-pixels at the lower-left positions in each pixel can be combined into a lower-left sub-image, the sub-pixels at the upper-right positions in each pixel can be combined into an upper-right sub-image, and the sub-pixels at the lower-right positions in each pixel can be combined into a lower-right sub-image. Therefore, the four sub-images can include an upper-left sub-image composed of the sub-pixels at the upper-left positions in each pixel, a lower-left sub-image composed of the sub-pixels at the lower-left positions in each pixel, an upper-right sub-image composed of the sub-pixels at the upper-right positions in each pixel, and a lower-right sub-image composed of the sub-pixels at the lower-right positions in each pixel. In addition, it should be understood that, in the case of acquiring an image by using a full-pixel four-core camera as described above, some or all of the upper-left sub-image, the lower-left sub-image, the upper-right sub-image, and the lower-right sub-image can be obtained for the following steps.

[0044] At step S130, the disparities between the sub-images can be acquired. More specifically, the plurality of sub-images can be paired two by two, and the disparities between the paired two-by-two sub-images can be acquired. Figure 4 is a schematic diagram showing disparity calculation according to an exemplary embodiment of the present application, as Figure 4As shown, the multiple sub-images can be inputted into the first neural network in pairs. Here, by way of example only and without limitation, the first neural network can be a convolutional neural network. Further, by way of example only and without limitation, the upper left sub-image and the upper right sub-image can be paired to calculate the vertical direction disparity d tx The lower left sub-image and the lower right sub-image can be paired to calculate the vertical direction disparity d bx The upper left sub-image and the lower left sub-image can be paired to calculate the horizontal direction disparity d ly The upper right sub-image and the lower right sub-image can be paired to calculate the horizontal direction disparity d ry The upper left sub-image and the lower right sub-image can be paired to calculate the diagonal direction disparity d lr The upper right sub-image and the lower left sub-image can be paired to calculate the diagonal direction disparity d rl After obtaining the disparity (also referred to as disparity feature), a disparity map can be further obtained. It should be appreciated that all or part of the above pairing can be used in the following steps.

[0045] More specifically, the step S130 can include inputting the multiple sub-images into multiple first neural networks in pairs to calculate the disparity by the first neural network using the following equations. Figure 5 is a schematic diagram showing four kinds of disparity calculation manners according to an exemplary embodiment of the present application, as Figure 5 shown, Figure 5 (a) in (a) corresponds to equation (1), representing obtaining the difference value of the sub-pixels at the corresponding positions of the inputted two-by-two paired sub-images, and obtaining the disparity between the sub-images based on the difference value; Figure 5 (b) in (b) corresponds to equation (2), representing performing convolution on the inputted two-by-two paired sub-images, obtaining the difference value of the sub-pixels at the corresponding positions of the convolved two-by-two paired sub-images, and obtaining the disparity between the sub-images based on the difference value; Figure 5 (c) in (c) corresponds to equation (3), representing performing displacement on the inputted two-by-two paired sub-images, obtaining the difference value of the sub-pixels at the corresponding positions of the displaced two-by-two paired sub-images, and obtaining the disparity between the sub-images based on the difference value, here, by way of example only and without limitation, the inputted two-by-two paired sub-images can be displaced in opposite directions and / or with the same displacement value; Figure 5corresponding position of the two two-pair sub-images after the convolution, and obtaining the disparity between the sub-images based on the difference. In addition, the step of obtaining the disparity between the sub-images based on the difference in the equations (1), (2), (3), (4) can include: comparing the difference with a predetermined constant a, and determining the disparity between the sub-images according to the comparison result; the step of determining the disparity between the sub-images according to the comparison result can include: taking the larger value of the difference and the predetermined constant a as the disparity. It should be understood that the corresponding position herein can be the same position only as an example but not limitation.

[0046] I diff = max (a, I1-I2) (1)

[0047] I diff = max (a, conv (I1)-conv (I2)) (2)

[0048] I diff = max (a, shift (I1, 1 / 2d)-shift (I2, -1 / 2d)) (3)

[0049] I diff = max (a, conv (shift (I1, 1 / 2d))-conv (shift (I2, -1 / 2d))) (4)

[0050] wherein I1 and I2 represent the two two-pair sub-images respectively; max represents the maximum value operation; a is a predetermined constant, which can be taken as 0 only as an example but not limitation (if a is taken as 0, then formula 0 is equivalent to taking the positive value in the difference of the sub-pixels at the corresponding positions of the sub-image I1 and the sub-image I2); conv represents the convolution operation; shift represents the shift operation; d represents the shift value of the shift operation, which is a constant (only as an example but not limitation, if d is taken as 2, then formula 0 represents moving the sub-image I1 up (or down) by 1 sub-pixel position, and moving the sub-image I2 down (or up) by 1 sub-pixel position).

[0051] If there is a zero disparity point (related to the distance of the object to the camera), the imaging position of the object at this position in the two two-pair sub-images is the same, and there is no disparity. At this time, after the above calculation, the image features of the object at this position are greatly reduced, such as Figure 5 As shown in (a) of the figure, the object in the middle is at the zero disparity point position, and the image features of the object in the disparity map are greatly reduced.

[0052] Return Figure 1At step S140, the depth of the image can be obtained based on the disparity between the sub-images. More specifically, the disparity between the sub-images can be input to a second neural network to estimate the depth of the image, thereby obtaining a depth map of the image. Here, by way of example but not limitation, the second neural network can be a convolutional neural network. More specifically, assuming that the camera parameters are fixed, for objects at different distances from the camera, the disparity and the depth are one-to-one corresponding, i.e., the different depths of the objects can be estimated according to the disparity, and thereby the depth map is obtained. Figure 6 is a schematic diagram illustrating objects at different distances from a camera according to an example embodiment of the present application. As shown in Figure 6 D(z) represents the disparity of the object S at a distance z in the two sub-images, f represents the focusing distance of the camera, it can be known that D(f) = 0, and at any distance z where z ≠ f, D(z) ≠ 0, and in the case of z1 < f < z2, D(z1) < D(f) = 0 < D(z2), thereby the relative depth of the object in the image can be obtained by the relationship of the disparity, and further the relative depth map is obtained. Figure 7 is a schematic diagram illustrating a synthesized depth map according to an example embodiment of the present application. As shown in Figure 7 By the above method, the relative depth map of the image can be obtained according to the disparity relationship of the object. In addition, the absolute depth map of the image can also be obtained according to the one-to-one correspondence between the disparity and the depth.

[0053] Figure 8 is a schematic diagram illustrating a general depth estimation method according to an example embodiment of the present application. As shown in Figure 8 The first neural network and the second neural network can be used to perform the disparity calculation and the depth estimation steps, respectively, thereby obtaining the depth map. In addition, the first neural network and the second neural network can also be trained end-to-end using the obtained depth map. During the training, since there is no calibration data of the disparity feature, the two neural networks are simultaneously trained and learned, but do not have to have the same network structure and do not share parameters.

[0054] In addition, as Figure 1The depth estimation method according to the exemplary embodiment of the present invention can also obtain an edge detection map for the image based on the multiple sub-images. Furthermore, the edge detection map for the image can also be obtained based on the disparity between the sub-images. More specifically, the multiple sub-images can be input into a third neural network to obtain the edge detection map for the image. By way of example only and not limitation, the third neural network can be a convolutional neural network. That is, when training the third neural network, a multi-task training approach can be employed to enhance image edge detection. Image edge detection is a crucial feature in computer vision tasks. In the disparity map depth estimation task, the disparity features of edges are often more pronounced. Multi-task training can enhance the extraction of effective features by the second and third neural networks. For example, when detecting disparity features in sub-images, the displacement difference between the foreground object and the background can be clearly observed. Directly using a deep neural network for end-to-end learning makes it difficult to specifically extract this feature. Incorporating supplementary supervised learning using edge maps of the foreground and background during training can help the network better extract this feature. Here, by way of example only and not limitation, the third neural network can be trained off-shelf, meaning that training of this network model need not be performed simultaneously with the training of the second neural network, and the dataset used for training the third neural network need not be the same as that used for the second neural network. The input of the third neural network can be a single color RGB image, and the output is an edge detection map of the image. However, existing edge detection algorithms extract edge information within the scene. Therefore, an artificially synthesized image can be used to synthesize an edge detection map that only detects the boundary between the foreground and background, and then the third neural network can be trained to generate the foreground and background images and the foreground and background edge detection maps as required. Figure 9 : is a schematic diagram illustrating edge extraction of a third neural network according to an exemplary embodiment of the present invention. Figure 9 As shown in (a), from left to right, they respectively represent an image containing a face, an image containing a face appearing in a real scene, the result of edge detection of the entire scene extracted using a traditional edge detection algorithm, and the result of edge detection extracted using the method provided by an exemplary embodiment of the present invention; Figure 9 As shown in (b), when the third neural network is ( Figure 9 The Encoder-Decoder in (b) is the structure of the third neural network). When training, an artificially synthesized color image is input and a matching synthetic true value edge detection map is output; when testing the third neural network, a color image of a real scene is input and a predicted edge detection map is output. The predicted edge detection map only contains the expected edges; Figure 9 As shown in (c), the trained third neural network can extract the expected effective edge features in the real data.

[0055] In addition, ifFigure 1 The depth estimation method according to the exemplary embodiment of the present invention shown can also obtain an image segmentation map of the image based on the multiple sub-images. In addition, the image segmentation map of the image can also be obtained based on the disparity between the sub-images. More specifically, the multiple sub-images can be input into a fourth neural network to obtain the image segmentation map of the image. Here, only as an example and not a limitation, the fourth neural network can be a convolutional neural network. Similar to the above-mentioned image edge detection, in the task of obtaining depth based on disparity, the segmentation of the foreground can also assist the network to improve the efficiency of extracting effective image features. Similar to the edge detection task, the fourth neural network can also be trained in an off-shelf manner. Figure 10 : is a schematic diagram showing an image segmentation graph of a fourth neural network according to an exemplary embodiment of the present invention. Figure 10 As shown, the fourth neural network can take a color image as input and output a segmentation map of the image. Here, taking the liveness detection task based on full-pixel quad-core technology as an example, the liveness detection task aims to distinguish between 3D real-person facial data and fake facial data such as 2D photos and screens. The essential difference between the two is that real-person facial data includes depth information, while fake facial data does not. Figure 11 is a schematic diagram illustrating a multi-task learning network according to an exemplary embodiment of the present invention, Figure 11 The conv in the above code represents the convolution operation. In addition, it is only used as an example and not as a limitation. Figure 11 The processing in may include displacement processing. Figure 11 The multi-task learning network shown in the figure corresponds to the feature decoder in the fourth neural network (which will be referred to later). Figure 12 and Figure 13 The feature decoder may not be located at Figure 11 The backbone network shown (i.e., the first neural network and the second neural network, corresponding to Figure 11 The data is not presented at the end of the dataset (outside the box), but rather interspersed after the intermediate results. The training data consists of 1,831 real faces and 1,840 fake faces, while the test data consists of 569 real faces and 679 fake faces. For binary classification alone, the best accuracy during the test phase reached 94.6314%. Adding auxiliary supervision for image segmentation increased the accuracy to 96.3141.

[0056] Figure 12 is a schematic diagram illustrating a training phase of an overall depth estimation method according to an exemplary embodiment of the present invention, Figure 13 is a schematic diagram illustrating a testing phase of an overall depth estimation method according to an exemplary embodiment of the present invention. Figure 12 and Figure 13The feature decoder is essentially a multi-layer neural network, serving as the decoder within the neural network encoder-decoder architecture. It decodes the required information from the features extracted by the network and produces the final output. The feature decoder is independent of the first and second neural networks used to calculate depth, but it can leverage image features calculated between them. This design allows for multi-task learning using multiple supervisory signals during training. During testing, it can simultaneously output an edge detection map and an image segmentation map, the primary output being the depth map.

[0057] Figure 14 is a block diagram illustrating a depth estimation apparatus according to an exemplary embodiment of the present invention.

[0058] like Figure 14 As shown, a depth estimation apparatus 1400 according to an exemplary embodiment of the present invention may include an image acquisition unit 1410 , a sub-image acquisition unit 1420 , a disparity acquisition unit 1430 , and a depth estimation unit 1440 .

[0059] The image acquisition unit 1410 may acquire an image, wherein a pixel of the image has multiple sub-pixels. Here, each pixel of the image may have multiple sub-pixels.

[0060] The sub-image acquisition unit 1420 may acquire a plurality of sub-images based on the image, wherein the pixels of the sub-images are the sub-pixels. Here, by way of example only and not limitation, the plurality of sub-images may be four sub-images, and the four sub-images may include an upper-left sub-image composed of sub-pixels in the upper-left position of each pixel, a lower-left sub-image composed of sub-pixels in the lower-left position of each pixel, an upper-right sub-image composed of sub-pixels in the upper-right position of each pixel, and a lower-right sub-image composed of sub-pixels in the lower-right position of each pixel.

[0061] The disparity obtaining unit 1430 can obtain the disparity between the sub-images. Further, the disparity obtaining unit 1430 can pair the plurality of sub-images two by two, and obtain the disparity between the two-by-two paired sub-images. More specifically, the disparity obtaining unit 1430 can further obtain the difference value of the sub-pixels at the corresponding positions of the input two-by-two paired sub-images, and obtain the disparity between the sub-images based on the difference value; the disparity obtaining unit 1430 can further perform convolution on the input two-by-two paired sub-images, obtain the difference value of the sub-pixels at the corresponding positions of the convolved two-by-two paired sub-images, and obtain the disparity between the sub-images based on the difference value; the disparity obtaining unit 1430 can further perform displacement on the input two-by-two paired sub-images, obtain the difference value of the sub-pixels at the corresponding positions of the displaced two-by-two paired sub-images, and obtain the disparity between the sub-images based on the difference value; the disparity obtaining unit 1430 can further perform displacement on the input two-by-two paired sub-images, perform convolution on the displaced two-by-two paired sub-images, obtain the difference value of the sub-pixels at the corresponding positions of the convolved two-by-two paired sub-images, and obtain the disparity between the sub-images based on the difference value. Further, the disparity obtaining unit 1430 can compare the difference value with a predetermined constant, and determine the disparity between the sub-images according to the comparison result. Here, merely by way of example but not by way of limitation, the disparity obtaining unit 1430 can displace the input two-by-two paired sub-images in opposite directions and / or with the same displacement value, and the predetermined constant can be 0.

[0062] The depth estimation unit 1440 can obtain the depth of the image based on the disparity between the sub-images. More specifically, the depth estimation unit 1440 can input the disparity between the sub-images to a second neural network.

[0063] Further, the depth estimation apparatus 1400 according to the exemplary embodiments of the present disclosure can further include an edge detection map and / or image segmentation map obtaining unit, which can obtain an edge detection map and / or an image segmentation map of the image based on the plurality of sub-images. Further, the edge detection map and / or image segmentation map obtaining unit can obtain the edge detection map and / or the image segmentation map of the image based on the disparity between the sub-images. More specifically, the edge detection map and / or image segmentation map obtaining unit can input the plurality of sub-images to a third neural network to obtain the edge detection map of the image, wherein the third neural network shares parameters with the first neural network and the second neural network; the edge detection map and / or image segmentation map obtaining unit can input the plurality of sub-images to a fourth neural network to obtain the image segmentation map of the image, wherein the fourth neural network shares parameters with the first neural network and the second neural network.

[0064] In addition, according to an exemplary embodiment of the present application, an electronic device can also be provided, including a memory and a processor, the memory having computer executable instructions stored thereon, the instructions, when executed by the processor, performing the method described above. According to an exemplary embodiment of the present application, a computer readable medium can also be provided, having computer executable instructions stored thereon, the instructions, when executed, performing the method described above.

[0065] According to an embodiment of the present application, depth estimation can be performed based on a single all-pixel multi-core camera (i.e., a phase detection camera), which is simpler in terms of hardware requirements, lower in power consumption, and lower in cost than the binocular camera scheme, depth sensor scheme, etc. in the prior art. In addition, through pre-processing of raw images captured by the device, equivalent multiple sub-images can be obtained, and through pairwise pairing, effective disparity in multiple directions can be calculated. Through the construction of artificial synthetic data, the problem of insufficient training samples can be solved. By adding constraints such as image segmentation and edge detection for multi-task learning, a deep neural network can be trained, and ultimately a one-to-one pixel-level correspondence from a disparity map to a depth map can be achieved.

[0066] While the present application has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A depth estimation method, comprising: Acquire an image, wherein each pixel of the image has a plurality of sub-pixels; Acquire a plurality of sub-images based on the image, wherein the plurality of sub-images respectively correspond to a plurality of sub-pixels of the image; Obtaining the disparity between the plurality of sub-images by a first neural network; and Obtaining the depth of the image based on the disparity between the multiple sub-images through a second neural network, The step of obtaining the disparity between the sub-images includes: The plurality of sub-images are paired in pairs, a shift operation is performed on each of the paired sub-images, and a disparity between the sub-images is obtained based on the sub-images after the shift operation.

2. The depth estimation method according to claim 1, wherein: The steps of obtaining the disparity between sub-images include: Obtaining the difference between the sub-pixels at corresponding positions of the paired sub-images after the shift operation; and The disparity between the paired sub-images is obtained based on the difference value.

3. The depth estimation method according to claim 1, wherein: The steps of obtaining the disparity between sub-images include: Convolution is performed on each sub-image in the paired sub-images after the shift operation; Obtaining the difference between sub-pixels at corresponding positions of the paired sub-images after convolution; and The disparity between the paired sub-images is obtained based on the difference value.

4. The depth estimation method according to claim 1, further comprising: An edge detection map of the image is obtained based on the multiple sub-images through a third neural network.

5. The depth estimation method according to claim 1, further comprising: An image segmentation map of the image is obtained based on the multiple sub-images through a fourth neural network.

6. The depth estimation method according to claim 1, wherein: The shift operation is performed on the input paired sub-images according to shift values ​​in different directions and / or the same direction.

7. The depth estimation method according to claim 1, wherein: The plurality of sub-images includes four sub-images, and the plurality of sub-pixels includes four sub-pixels; wherein the four sub-pixels included in each pixel are arranged at an upper left position, a lower left position, an upper right position, and a lower right position of each pixel; The four sub-images include an upper left sub-image corresponding to a sub-pixel at an upper left position, a lower left sub-image corresponding to a sub-pixel at a lower left position, an upper right sub-image corresponding to a sub-pixel at an upper right position, and a lower right sub-image corresponding to a sub-pixel at a lower right position.

8. A depth estimation device, comprising: An image sensor acquires an image, wherein each pixel of the image has a plurality of sub-pixels; The processor is configured to perform the following operations: Acquire a plurality of sub-images based on the image, wherein the plurality of sub-images respectively correspond to a plurality of sub-pixels of the image; Obtaining the disparity between the plurality of sub-images by a first neural network; and Obtaining the depth of the image based on the disparity between the multiple sub-images through a second neural network, The processor is further configured to perform the following operations: The plurality of sub-images are paired in pairs, a shift operation is performed on each of the paired sub-images, and a disparity between the sub-images is obtained based on the sub-images after the shift operation.

9. The depth estimation device according to claim 8, wherein: The processor is further configured to: Obtaining the difference between the sub-pixels at corresponding positions of the paired sub-images after the shift operation; and The disparity between the paired sub-images is obtained based on the difference value.

10. The depth estimation device according to claim 8, wherein: The processor is further configured to: Convolution is performed on each sub-image in the paired sub-images after the shift operation; Obtain the difference between the sub-pixels at corresponding positions of the paired sub-images after convolution; as well as The disparity between the paired sub-images is obtained based on the difference value.

11. The depth estimation apparatus according to claim 8, wherein: The processor is further configured to obtain an edge detection map of the image based on the multiple sub-images through a third neural network.

12. The depth estimation apparatus according to claim 8, wherein: The processor is further configured to obtain an image segmentation map of the image based on the multiple sub-images through a fourth neural network.

13. The depth estimation apparatus according to claim 8, wherein: The shift operation is performed on the input paired sub-images according to shift values ​​in different directions and / or the same direction.

14. The depth estimation apparatus according to claim 8, wherein: The plurality of sub-images includes four sub-images, and the plurality of sub-pixels includes four sub-pixels; wherein the four sub-pixels included in each pixel are arranged at an upper left position, a lower left position, an upper right position, and a lower right position of each pixel; The four sub-images include an upper left sub-image corresponding to a sub-pixel at an upper left position, a lower left sub-image corresponding to a sub-pixel at a lower left position, an upper right sub-image corresponding to a sub-pixel at an upper right position, and a lower right sub-image corresponding to a sub-pixel at a lower right position.

15. A non-transitory computer-readable medium having computer-executable instructions stored thereon, wherein when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images