Video residual decoding device and associated method

Abstract

A method and system for depth estimation from focus, comprising receiving input image data containing focus information, generating an intermediate attention map via an AI model, normalizing the intermediate attention map into a depth attention map via a normalization function, and deriving an expected depth value for the input image data, wherein the input image data contains focus information from the depth attention map. The AI ​​model for depth estimation can be trained unsupervised without a ground truth depth map. In some embodiments, the AI ​​model is a shared network that estimates the depth map and reconstructs an AiF image from a set of images with different focus positions.

Description

[0001] Related references

[0002] The present invention claims priority to U.S. Provisional Application No. 63 / 200,692, filed on March 23, 2021, entitled “Bridging Unsupervised and Supervised Depth from Focus via All-in-Focus Supervision,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to depth estimation in image or video processing systems, and more particularly, to depth estimation using focus information based on machine learning. Background Art

[0004] Depth estimation has been one of the most fundamental computer vision topics used for downstream tasks such as augmented reality (AR), virtual reality (VR), and autonomous driving. Depth estimation uses different cues such as color, semantics, stereo, or differences in image sequences to predict and fuse depth maps. Depth information enables a growing number of photography and video conferencing applications, such as depth of field adjustment, background replacement, and changing focus after taking a photo. In the latest developments in image or video capture devices, active or passive depth sensing technologies are embedded in each capture device to obtain depth information. Due to the need for specialized hardware, active depth sensing solutions (such as time-of-flight (ToF) and structured light) are generally more expensive and power-intensive. Compared to active depth sensing technologies, passive technologies (such as binocular or multi-view stereo) are more cost- and power-efficient, but are more prone to errors in textureless areas.

[0005] Deep learning-based stereo matching methods address this problem in a data-driven manner by learning depth estimation directly from input images. However, it requires a large amount of high-quality paired training data, which is time-consuming and expensive to obtain. Some other problems encountered by stereo matching methods include imperfect training data, synthetic and unrealistic input images, and inaccurate registered depth maps. Some unsupervised learning methods have been developed to address this issue, which rely on image reconstruction loss and consistency loss without the need for ground truth depth data. The domain gap is alleviated by directly using real-world stereo images for training without the need for corresponding registered depth maps.

[0006] Another relatively underexplored technique for depth estimation is defocus blur. The task of depth from focus (or defocus) aims to estimate a depth map of a scene from a focal stack. A focal stack consists of multiple images captured by the same camera at different focus positions. This allows consumer autofocus monocular cameras to estimate depth values ​​without additional hardware. Traditional methods based on depth from focus optimization estimate the sharpness level for each pixel and often struggle with objects with little texture or aperture. Deep learning techniques can help overcome these issues but require ground-truth depth data for supervised training. Acquiring a focal stack using registered depth maps is expensive and time-consuming, not to mention the imperfect depth data obtained with hardware solutions such as ToF sensors. Many works aim to synthesize real data to address the need for data in the target domain, for example, using synthesized depth maps to synthesize defocus blur on synthetic datasets. However, whether the thin lens models used for synthesis can accurately represent real-world optics remains questionable.

[0007] Most depth estimation methods assume that the input image is all-in-focus (AiF), whereas in real-world scenes, most images are assumed to be out-of-focus in the background or have a shallow depth of field (DoF). Few methods address depth estimation from defocused images in detail. Conventional optimization-based methods have been proposed to estimate depth from focus stacks. A variational approach has been proposed to generate an index map in which each pixel is assigned to the focus position with maximum contrast. Furthermore, deep learning-based methods can more accurately model blur and achieve better depth quality. Some methods exploit the relationship between relative blur and disparity to enhance the robustness of matching. Depth from focus-sweep videos aims to estimate depth from images with continuous focus positions. Deep learning-based methods further improve depth quality and attempt to remove defocus blur from a single defocused image. Collecting data in the target domain has become increasingly important, as it directly impacts the results of deep learning methods. However, collecting real-world data is often expensive and time-consuming. Many works aim to synthesize realistic data to address the need for data collection in the target domain. Summary of the Invention

[0008] An embodiment of a method for depth estimation using focus information for an image or video processing system includes receiving input image data containing focus information, generating an intermediate attention map from the input image data containing focus information by an artificial intelligence (AI) model, normalizing the intermediate attention map into a depth attention map by a first normalization function, and deriving an expected depth value of the input image data containing focus information from the depth attention map. The AI ​​model is trained in an unsupervised manner by one or more focus-based training materials. An example of input image data containing focus information is a focus stack comprising a stack of images with different focus positions, and another example of input image data containing focus information is a photograph that is refocused after acquisition. The photograph that is refocused after acquisition can be acquired from a light field camera.

[0009] One embodiment of focus-based training material is at least a portion of a photograph acquired with a smaller aperture than that used to acquire any image in the focus stack. Another embodiment of focus-based training material is at least a portion of a stopped-down photograph acquired from a light-field camera. Another embodiment of focus-based training material is a ground-truth All-in-Focus (AiF) image output from an AiF image generator, where the sharpness value of the ground-truth image is greater than the sharpness value of each image in the focus stack.

[0010] In some embodiments, the AI ​​model is shared by the depth estimation and AiF image reconstruction tasks, and the method of depth estimation using focus information also includes normalizing the intermediate attention map to an AiF depth attention map by a second normalization function and deriving an expected AiF image from the AiF depth attention map and the input data containing focus information. Some examples of the first normalization function normalize the intermediate attention map to a flatter probability distribution, while some examples of the second normalization function normalize the intermediate attention map to a sharper probability distribution. For example, the first normalization function is a softplus normalization function and the second normalization function is a softmax normalization function. According to various embodiments of the present invention, the AI ​​model is supervised by a ground truth depth map, or the AI ​​model is unsupervisedly trained by a ground truth AiF image. In one embodiment, when the depth estimation task is performed, the AI ​​model is supervised by a ground truth depth map. In one embodiment, the AI ​​model is trained by comparing the expected depth value with the ground truth depth map corresponding to the input image data containing focus information. In another embodiment, when the AiF image reconstruction task is performed, the AI ​​model is unsupervisedly trained using ground-truth AiF images. In this embodiment, the AI ​​model is trained by comparing the expected AiF image with the ground-truth AiF image corresponding to the input image containing focus information. For unsupervised depth estimation, according to one embodiment, the AI ​​model is trained using ground-truth AiF images by minimizing a total loss that combines the AiF training loss and the smoothness loss, where the AiF training loss represents the difference between the expected AiF image and the corresponding ground-truth AiF image.

[0011] The expected depth value of the input image data containing focus information is derived from the depth attention map and the focus position of the input image data containing focus information. In some embodiments of the present invention, the AI ​​model parameters are updated by obtaining real-time ground truth AiF images and by training the AI ​​model using real-time ground truth AiF images, and the AI ​​model is trained to adapt to the current situation or environment. The AI ​​model is a deep learning network or a machine learning network. For example, the basic architecture of the AI ​​model is an encoder-decoder network composed of three-dimensional (3D) convolutions. The AI ​​model can be trained by ground truth AiF images or ground truth depth values. After the AI ​​model parameters are updated by training the AI ​​model, at least one value pre-stored in a register or memory for the AI ​​model is modified.

[0012] Aspects of the present disclosure also provide an apparatus for depth estimation using focus information in an image or video processing system. The apparatus includes one or more electronic circuits configured to receive input image data containing focus information, generate an intermediate attention map from the input image data containing focus information using an AI model, normalize the intermediate attention map into a depth attention map using a first normalization function, and derive an expected depth value of the input image data containing focus information from the depth attention map. The AI ​​model in the apparatus can be trained using one or more focus-based training materials, such as ground truth AiF images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various embodiments of the present disclosure, presented as examples, will be described in detail with reference to the following drawings, in which like numerals represent like elements, and in which:

[0014] Figure 1 An exemplary system for depth estimation using focus information according to an embodiment of the present invention is shown.

[0015] Figure 2A Supervised learning of an AI model in a depth estimation system using focus information according to an embodiment of the present invention is illustrated.

[0016] Figure 2B Unsupervised learning of an AI model in a depth estimation system using focus information according to an embodiment of the present invention is illustrated.

[0017] Figure 3 A flowchart of estimating a depth value from input image data containing focus information according to an embodiment of the present invention is shown.

[0018] Figure 4 A flowchart is shown of estimating a depth value from input image data containing focus information according to another embodiment of the present invention. DETAILED DESCRIPTION

[0019] It will be readily understood that the elements of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a variety of different configurations. Accordingly, the following more detailed description of the embodiments of the systems and methods of the present invention, as represented in the figures, is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0020] References throughout this specification to "one embodiment," "some embodiments," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment" or "in some embodiments" appearing in various places throughout this specification do not necessarily all refer to the same embodiment, and these embodiments may be implemented alone or in combination with one or more other embodiments. In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the present invention may be practiced without one or more of the specific details, or using other methods, elements, etc. In other cases, well-known structures or operations are not shown or are not illustrated. Detailed descriptions are provided to avoid obscuring aspects of the present invention.

[0021] Overview of a Depth Estimation System Using Focus Information Some embodiments of the present invention perform depth estimation from input image data containing focus information using an artificial intelligence (AI) model. The AI ​​model of the present invention can be trained in a supervised manner using a ground-truth depth map, or in an unsupervised manner using only focus-based training material. Examples of focus-based training material include at least a portion of a ground-truth all-in-focus (AiF) image, at least a portion of a photograph taken with a small aperture, and at least a portion of a stopped-down photograph obtained from a light-field camera. Some embodiments of the present invention jointly perform depth estimation and AiF image reconstruction from input image data containing focus information. These two tasks, depth estimation and AiF image reconstruction, are accomplished by a shared AI model according to embodiments of the present invention by exploiting the relationship between the two tasks. The shared AI model can be trained using depth-based training material (e.g., a ground-truth depth map) or focus-based training material (e.g., a ground-truth AiF image). Compared to obtaining high-quality depth-based training material (e.g., a ground-truth depth map) in traditional methods, obtaining focus-based training material such as ground-truth AiF images is relatively easy. For example, real AiF images can be acquired using a small aperture and long exposure. The AI ​​model in some embodiments of the present invention learns depth estimation from input image data containing focus information in an unsupervised manner. That is, even when ground truth depth data is unavailable, the domain gap can be mitigated by test-time training on real-world data.

[0022] Network architecture of a depth estimation system using focal point information Figure 1An overview of a depth estimation system using focus information according to an embodiment of the present invention is shown. Depth estimation using focus information aims to recover depth values ​​from input image data containing focus information by means of defocus cues. Some examples of input image data containing focus information 102 are image stacks with different focus positions, and photographs acquired from light field cameras or other technologies that are refocused after acquisition. Image stacks with different focus positions are also referred to as focus stacks, for example, a focus stack is a set of images acquired with different focuses by a camera on an acquisition device. A light field camera acquires spatially distributed light. By re-rendering the digitized light field data, multi-view or variable aperture images can be displayed after acquisition. Since a light field camera provides multi-view scenes from different camera positions, it can help computational photography perform better than ordinary camera devices. Given a focus stack S∈R of F images H*W*3*F , and the F images have a gradually changing focus position P∈R H*W*1*F , the depth estimation method generates multiple attention representations through artificial intelligence (AI) models. Figure 1As shown, input image data containing focus information 102 is fed into an AI model 104 to generate an intermediate attention map M106. AI model 104 can be understood as a kernel or network of AI model parameters, trained using one or more ground-truth AiF images or one or more ground-truth depth maps. AI model 104, also known as a deep learning network or machine learning network, can share the tasks of depth estimation and AiF image reconstruction. According to some embodiments of the present invention, AI model 104 is a shared network for depth estimation and AiF image reconstruction, unlike prior art approaches that use separate networks for these two tasks. In prior art approaches, the network parameters of the separate networks are trained and determined separately, the computations performed in the separate networks are independent, and the registers or memory used to store the network parameters and the associated software programs for the separate networks are different. According to the present invention, AI model 104 is a shared network for both tasks. Therefore, the AI ​​model parameters are trained and determined separately, the computations performed in the shared network are shared by both tasks, and the registers or memory used to store the AI ​​model parameters and the software programming are the same. An example of the infrastructure of the AI ​​model 104 of the present invention is Inception3D, in which the AI ​​model 104 is an encoder-decoder network composed of three-dimensional (3D) convolutions. The encoder reduces the resolution of the input image data, and the decoder reconstructs the information to increase the resolution. In another example of the AI ​​model 104, the encoder-decoder network keeps the resolution unchanged by encoding and decoding. Through 3D convolution, defocus clues throughout the frame can be better acquired, thereby facilitating the tasks of depth estimation and AiF image reconstruction. Due to the nature of 3D convolution, the AI ​​model 104 can also process focus stacks of arbitrary size. In another example, the infrastructure of the AI ​​model 104 includes two-dimensional (2D) convolutions, which apply a global pooling layer as a communication tool between multiple weight-sharing 2D convolutional networks.

[0023] The output of the attention mechanism AI model 104 is the intermediate attention map M(M∈R H*W*1*F ) 106. The basic expectation of the intermediate attention map M 106 is that it should reflect the probability of each focus position leading to maximum sharpness. The intermediate attention map M 106 can be beneficial to the tasks of depth estimation and AiF image reconstruction. For depth estimation, the intermediate attention map M 106 is normalized to the depth attention map M by the first normalization function 110 depth 114. An example of the first normalization function 110 is a softplus normalization function, as shown in the following equation.

[0024]

[0025] in

[0026] The softplus function is a smoothed version of the ReLU function. depth 114 can also be interpreted as a probability distribution, since the softplus function ensures that there are no negative values, and the softplus function normalizes M 106 to a valid probability distribution for depth estimation. Figure 1 As shown in Equation (3), the expected depth value 120 of each pixel is obtained from the depth attention map M depth 114 and the focus position P 116 of the focus stack 102 are derived.

[0027]

[0028] wherein the expected depth value of a pixel in the input image data is according to the equation Export, where M depth is the depth attention map, P(i,j,1,t) corresponds to the focus position of the pixel, F is the number of focus positions, i is the height index of the pixel, j is the width index of the pixel, and t is the focus position index.

[0029] For AiF image reconstruction, the depth estimation system using focus information performs a similar process as depth estimation, except that the normalization function is different. AiF depth attention map M AiF 112 through Figure 1 The second normalization function 108 shown is derived from the intermediate attention map M 106. For example, the second normalization function 108 for AiF image reconstruction produces a sharper probability distribution than the first normalization function 110 for depth estimation. For example, a softmax normalization function is used for AiF image reconstruction. The second normalization function 108 is used to normalize the intermediate attention map M 106 to the AiF depth attention map M by the softmax function. AiF The equation for 112 is shown below.

[0030] M AiF =σ(M), Equation (4)

[0031] in

[0032] AiF Deep Attention Map M AiF 112 is then used for AiF image reconstruction based on the focus stack S102. Each pixel of the expected AiF image 118 is based on the AiF depth attention map M AiF 112 and focus stack S102 are expressed by equation (6).

[0033]

[0034] Depth and AiF images are estimated separately using two different normalization functions, softmax and softplus, to address the issue of sparse focus stacks. A sparse focus stack is one with a small stack size and sparsely populated focus locations within each sparse focus stack. A normalization function with a sharper distribution is selected for AiF reconstruction. For each pixel in the AiF image, the sharpest pixel along the stack dimension is selected during AiF image reconstruction. A flatter distribution results in multiple pixels with the highest sharpness. Mixing multiple pixels in a sparse stack typically does not produce the sharpest pixels. Therefore, a softmax normalization function is applied to the intermediate attention to exploit the peaking phenomenon and extract the sharpest pixel for each pixel in the AiF image. For depth estimation, a sharper distribution (such as the softmax normalization function) can lead to severe quantization of sparse focus stacks because it simply selects the nearest focus location with maximum sharpness. On the other hand, normalization functions that result in a flatter distribution, such as the softplus normalization function, are more accurate for depth prediction by interpolating between sparse focus locations. The softmax and softplus functions are examples of normalization functions used to derive depth values ​​and AiF images, and the system can apply another normalization function to the intermediate attention map to generate a depth value or AiF image.

[0035] Supervised or unsupervised depth estimation such as Figure 1As shown, the intermediate attention map M 106 generated by the AI ​​model 104 of the depth estimation system using focus information bridges the depth estimation task and the AiF image reconstruction task, requiring no intermediate supervisory signal. Furthermore, there are no learnable parameters after the intermediate attention map M 106 is generated. The expected depth values ​​120 and AiF image 118 output from the depth estimation system using focus information are simply generated by applying fixed normalization functions 110 and 108 to the intermediate attention map M 106. In some preferred embodiments, the fixed normalization function 120 used to derive the expected depth values ​​normalizes the intermediate attention map 106 to a flatter probability distribution, while the fixed normalization function used to derive the expected AiF image 118 normalizes the intermediate attention map M 106 to a sharper probability distribution. Even when depth-based training material, such as ground-truth depth maps, is unavailable, the AI ​​model 104 shared between the two tasks can still be trained unsupervised using one or more focus-based training material. An example of focus-based training material is at least a portion of one or more ground-truth AiF images. That is, the AI ​​model 104 of the depth estimation system using focus information is trained in a supervised or unsupervised manner with or without depth-based training material (e.g., a ground-truth depth map). In one embodiment, the AI ​​model 104 is trained with at least a portion of one or more ground-truth AiF images to generate an intermediate attention map M 106, and the depth value 120 of the focus stack S102 is estimated by normalizing the intermediate attention map M 106. Ground-truth AiF images can be easily acquired to train the AI ​​model 104 for depth estimation and AiF image reconstruction tasks. In other words, the AI ​​model 104 can typically be trained in a foundry with basic ground-truth depth data or basic ground-truth AiF images, and then more accurately trained on the device (also referred to as on-device training) by acquiring real-time ground-truth AiF images to adapt to the current environment, environment, or setting. One example of a ground-truth AiF image is a photograph taken with a small aperture, where the aperture used to acquire the ground-truth AiF image is smaller than the aperture used to capture any image in the focus stack. Another embodiment of a ground-truth AiF image is generated from a stopped-down photograph acquired from a light-field camera. In yet another embodiment, the ground-truth AiF image can be an image from any all-in-focus image generator, such that the sharpness of the ground-truth AiF image is greater than the sharpness of each image in the focus stack. Sharpness describes the clarity of detail in an image. By converting the image to the frequency domain, areas of the image with dense detail will have larger amplitudes at high frequencies, while areas of the image with less detail will have lower amplitudes at high frequencies. In other words, sharpness reflects the high-frequency power in the block spectrum.Focus-based training material can be defined as at least one block in an image that has a larger amplitude in high frequencies compared to the high frequency amplitudes of corresponding blocks of other images in the focus stack.

[0036] Example of training loss for supervised depth estimation, Figure 2A An AI model is shown trained by a ground truth depth map with a simple L1 loss, as shown in Equation (7).

[0037] L supervised =L depth =E[kD-D gt K1], equation (7)

[0038] Where Dgt represents the ground truth depth. The AI ​​model 204 generates an intermediate attention map M 206 of the focus stack S202. A first normalization function 210 is applied to the intermediate attention map M 206 to generate a deep attention map M depth 214. The second normalization function 208 is applied to the intermediate attention map M 206 to generate the AiF deep attention map M AiF 212. Expected depth value D 220 from the depth attention map M depth 214 and focus position 216 are derived. The expected AiF image 218 is derived from the AiF depth attention map M AiF 212 and focus stack S202 are derived. The AI ​​model 204 is trained in a supervised manner by minimizing the Lsupervised loss between the ground truth depth map 222 and the expected depth value D 220.

[0039] For an embodiment of unsupervised depth estimation, Figure 2B Shows an AI model trained by L1 loss with AiF supervision.

[0040] L AiF =E[kI-I gt K1], equation (8)

[0041] where Igt represents the ground truth AiF image 224. In addition, the depth map can be locally smoothed using edge-aware weighting as shown in Equation (9). The smoothness loss is defined as:

[0042]

[0043] in as well as

[0044] λ is a hyper-parameter for edge weighting based on the ground truth AiF image. The total loss for unsupervised depth estimation is a combination of the AiF training loss and the smoothness loss.

[0045] L unsupervised =L AiF +αL smooth , Equation (11)

[0046] where α represents the importance of smoothness loss. Figure 2B The AI ​​model 204 in receives the focus stack S202 and outputs the intermediate attention map M 206. The second normalization function 208 normalizes the intermediate attention map M 206 to the AiF deep attention map M AiF At 212, the first normalization function 210 normalizes the intermediate attention map M 206 to the deep attention map M depth 214. The expected depth value D 220 is obtained from the depth attention map M depth 214 and focus position P 216, the expected AiF image 218 is derived from the AiF depth attention map M AiF 212 and the focus stack S202 are derived. The AI ​​model 204 is trained in an unsupervised manner by minimizing the total loss that combines the AiF training loss and the smoothness loss, where the AiF training loss represents the difference between the expected AiF image 218 and the corresponding ground truth AiF image 224.

[0047] The AI ​​model from the depth estimation system using focus information is trained by minimizing the training loss of supervised or unsupervised depth estimation. The deep learning training of the present invention can be on-device training, as the device can easily obtain ground-truth AiF images without the need for any additional sensors. Some advantages of depth estimation using the focus system include low cost, as auxiliary sensors such as infrared sensors are not required to collect ground-truth depth, and depth maps are more accurate than stereo matching methods. The AI ​​model parameters of the AI ​​model are updated during the training process to achieve better depth estimation quality, and some values ​​pre-stored in the AI ​​model's registers or memory are modified. The updated model parameters are used to perform mathematical operations on the input data, or the values ​​obtained after a series of operations on the input data, which are also called features in machine learning. The output of the depth estimation of the focus system is a depth map, which is useful in many computer vision applications. Some product applications and fields that can implement the depth estimation system using focus information of the present invention include robotics, surveillance cameras, self-driving cars, and augmented reality (AR) and virtual reality (VR).

[0048] Representative flow chart of an embodiment of the present invention Figure 3 This is a flowchart illustrating an embodiment of a method for depth estimation using focus information in an image or video processing system. In step S302, the image or video processing system receives input image data containing focus information. For example, the input image data containing focus information is a focus stack. In step S304, an AI model generates an intermediate attention map for the input image data. In this embodiment, the AI ​​model is trained using one or more focus-based training data, for example, the focus-based training data is at least a portion of a ground-truth AiF image. In another embodiment, the AI ​​model is trained using one or more depth-based training materials, or in yet another embodiment, the AI ​​model is trained using both focus-based and depth-based training materials. An example of depth-based training material is a ground-truth depth map. The AI ​​model is trained in a factory, or the AI ​​model is trained on a device. In step S306, the intermediate attention map is normalized into a depth attention map using a normalization function. The depth attention map is a normalized depth attention map that represents the probability distribution of each focus position that results in maximum sharpness for depth estimation. In step S308 , the image or video processing system derives the expected depth value of the input image data containing focus information from the depth attention map.

[0049] Figure 4A flowchart illustrating another embodiment of a depth estimation method using focus information in an image or video processing system is provided. In step S402, the image or video processing system receives input image data containing focus information, such as a focus stack. In step S404, an AI model generates an intermediate attention map for the input image data. In this embodiment, the AI ​​model is trained using one or more focus-based training materials. In another embodiment, the AI ​​model is trained using one or more depth-based training materials, or in yet another embodiment, the AI ​​model is trained using both focus-based and depth-based training materials. The AI ​​model is a shared network for depth estimation and AiF image reconstruction tasks. In step S406, the intermediate attention map generated by the AI ​​model is normalized into a depth attention map using a first normalization function. In step S408, an expected depth value for the input image data containing focus information is derived from the depth attention map. In step S410, the intermediate attention map is further normalized into an AiF attention map using a second normalization function. In step S412, an expected AiF image for the input image data containing focus information is derived from the AiF attention map. In one embodiment, the expected AiF image corresponds to the entire input image data. However, it should be noted that the expected AiF image does not necessarily correspond to the entire input image data. For example, in one embodiment, the input image data consists of a first portion and a second portion, and the expected AiF image corresponds only to the first portion of the input image data. In this embodiment, another expected AiF image corresponding to the second portion of the input image data can be derived from the AiF attention map in a subsequent step of training the AI ​​model. Therefore, in this embodiment, a first ground truth image corresponding to the first portion of the input image data and a second ground truth image corresponding to the second portion of the input image data are used.

[0050] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are intended to be embraced within their scope.

Claims

1. A depth estimation method using focus information, for use in an image or video processing system, comprising: receiving input image data including focus information; generating an intermediate attention map from the input image data containing focus information by an artificial intelligence model, wherein the artificial intelligence model is trained by one or more focus-based training materials; Normalizing the intermediate attention map into a deep attention map by a first normalization function; as well as A plurality of expected depth values ​​of the input image data containing focus information are derived from the depth attention map, wherein the expected depth values ​​of pixels in the input image data are calculated according to the equation Export, where M depth is the depth attention map, P(i,j,1,t) corresponds to the focus position of the pixel, F is the number of focus positions, i is the height index of the pixel, j is the width index of the pixel, and t is the focus position index.

2. The depth estimation method using focus information according to claim 1, wherein: The input image data containing focus information is a focus stack comprising a stack of images with different focus positions.

3. The depth estimation method using focus information according to claim 2, wherein: The focus-based training material is at least a portion of a photograph acquired with an aperture that is smaller than the aperture used to acquire any image in the focus stack.

4. The depth estimation method using focus information according to claim 1, wherein: The input image data including focus information is a plurality of photos that are refocused after acquisition.

5. The depth estimation method using focus information according to claim 4, wherein: The focus-based training material is at least a portion of a stopped-down photograph acquired from a light-field camera.

6. The depth estimation method using focus information according to claim 1, wherein: The focus-based training material is at least a portion of a ground truth all-focus image output from an all-focus image generator, wherein a sharpness value of the ground truth image is greater than a sharpness value of each image in a focus stack.

7. The depth estimation method using focus information according to claim 1, wherein: The artificial intelligence model is shared for depth estimation and all-focus image reconstruction tasks, and the depth estimation method using focus information further includes normalizing the intermediate attention map to an all-focus attention map by a second normalization function, and deriving an expected all-focus image from the all-focus attention map and the input image data containing focus information, wherein the pixels of the expected all-focus image are given by the equation Indicates that M AiF is the full-focus depth attention map, and S is the focus stack.

8. The depth estimation method using focus information according to claim 7, wherein: The first normalization function normalizes the intermediate attention map into a flatter probability distribution, and the second normalization function normalizes the intermediate attention map into a sharper probability distribution.

9. The depth estimation method using focus information according to claim 7, wherein: The first normalization function is a softplus normalization function, and the second normalization function is an oftmax normalization function.

10. The depth estimation method using focus information according to claim 7, wherein: When the depth estimation task is performed, the artificial intelligence model is supervised by ground truth depth maps.

11. The depth estimation method using focus information according to claim 10, wherein: The artificial intelligence model is trained by comparing the plurality of expected depth values ​​to the ground truth depth map corresponding to the input image data containing focus information.

12. The depth estimation method using focus information according to claim 10, wherein: When the all-focus image reconstruction task is performed, the artificial intelligence model is unsupervisedly trained by the ground truth all-focus images.

13. The depth estimation method using focus information according to claim 12, wherein: The artificial intelligence model is trained by comparing the expected all-in-focus image with the ground-truth all-in-focus image, which corresponds to the input image data containing focus information.

14. The depth estimation method using focus information according to claim 7, wherein: The artificial intelligence model is trained by minimizing a total loss on the ground truth all-focus image, which combines an all-focus training loss and a smoothness loss, wherein the all-focus training loss represents the difference between the expected all-focus image and the corresponding ground truth all-focus image.

15. The depth estimation method using focus information according to claim 1, wherein: The plurality of expected depth values ​​are derived from the depth attention map and a plurality of focus positions of the input image data containing focus information.

16. The depth estimation method using focus information according to claim 1, wherein: The artificial intelligence model is trained on the device to adapt to the current situation by acquiring a plurality of real-time ground-truth all-focus images and updating a plurality of artificial intelligence model parameters by training the artificial intelligence model using the plurality of real-time ground-truth all-focus images.

17. The depth estimation method using focus information according to claim 1, wherein: The basic architecture of the artificial intelligence model is an encoder-decoder network composed of multiple three-dimensional convolutions.

18. The depth estimation method using focus information of claim 1, further comprising modifying at least one value pre-stored in a register or memory for the artificial intelligence model after training the artificial intelligence model.

19. A depth estimation device using focus information, for use in an image or video processing system, the device comprising one or more electronic circuits configured to: receiving input image data including focus information; generating an intermediate attention map from the input image data containing focus information by an artificial intelligence model, wherein the artificial intelligence model is trained by one or more focus-based training materials; Normalizing the intermediate attention map into a deep attention map by a first normalization function; as well as A plurality of expected depth values ​​of the input image data containing focus information are derived from the depth attention map, wherein the expected depth values ​​of pixels in the input image data are calculated according to the equation Export, where M depth is the depth attention map, P(i,j,1,t) corresponds to the focus position of the pixel, F is the number of focus positions, i is the height index of the pixel, j is the width index of the pixel, and t is the focus position index.

Images