Defocus blur removal and depth estimation using dual-pixel image data

Abstract

A method includes obtaining dual-pixel image data including a first sub-image and a second sub-image, and generating a focused image, a first kernel corresponding to the first sub-image, and a second kernel corresponding to the second sub-image. A loss value may be determined using a loss function that determines the difference between (i) the convolution of the first sub-image and the second kernel and (ii) the convolution of the second sub-image and the first kernel, and / or a loss function that determines the sum of (i) the difference between the convolution of the first sub-image and the focused image and the first kernel and (ii) the difference between the convolution of the second sub-image and the focused image and the second kernel. Based on the loss value and the loss function, the focused image, the first kernel, and / or the second kernel may be updated and displayed.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for defocus blur removal and depth estimation. Background Art

[0002] A portion of an image may be blurred due to a corresponding object being outside the depth of field of a camera device that captured the image. The degree of blur may depend on the location of the corresponding object relative to the depth of field, with the amount of blur increasing as the corresponding object moves farther from the depth of field in a direction toward or away from the camera. In some cases, image blur is undesirable and may be adjusted or corrected using various image processing techniques, models, and / or algorithms. Summary of the invention

[0003] The dual-pixel image data may include defocus cues that can be used to restore a focused image and / or a depth map corresponding to the dual-pixel image data. In particular, the dual-pixel image data may include a first sub-image and a second sub-image. The focused image may generate a first sub-image and / or an approximation thereof when convolved with a first blur kernel, and may generate a second sub-image and / or an approximation thereof when convolved with a second blur kernel. A loss function may be formulated based on these relationships between the focused image, the blur kernel, and the dual-pixel image data. The loss function may be used to train a mathematical model configured to generate a focused image, and in some cases, to train one or more mathematical models configured to generate the first and second blur kernels. The blur kernel may be applied to focused images and / or dual-pixel image data of multiple size scales, thereby generating a cost volume. The cost volume may be used to select the corresponding size scales to which the first and second blur kernels are applied for each corresponding pixel of the focused image, thereby indicating the depth associated with the corresponding pixel.

[0004] In a first example embodiment, a method may include obtaining dual-pixel image data including a first sub-image and a second sub-image. The method may also include determining (i) an in-focus image, (ii) a first blur kernel corresponding to the first sub-image, and (iii) a second blur kernel corresponding to the second sub-image. The method may additionally include determining a loss value using a loss function including one or more of an equivalent loss term or a data loss term: the equivalent loss term is configured to determine the difference between (i) the convolution of the first sub-image with the second blur kernel and (ii) the convolution of the second sub-image with the first blur kernel, and the data loss term is configured to determine the sum of the difference between (i) the first sub-image and the in-focus image and the convolution of the first blur kernel and (ii) the difference between the second sub-image and the in-focus image and the convolution of the second blur kernel. The method may also include updating one or more of the following based on the loss value and the loss function: (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel. The method may also include generating image data based on one or more of the following: (i) the updated in-focus image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

[0005] In a second example embodiment, a system may include a processor and a non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, causes the processor to perform an operation. The operation may obtain dual-pixel image data including a first sub-image and a second sub-image. The operation may also include determining (i) an in-focus image, (ii) a first blur kernel corresponding to the first sub-image, and (iii) a second blur kernel corresponding to the second sub-image. The operation may additionally include determining a loss value using a loss function including one or more of an equivalent loss term or a data loss term: the equivalent loss term is configured to determine the difference between (i) the convolution of the first sub-image with the second blur kernel and (ii) the convolution of the second sub-image with the first blur kernel, and the data loss term is configured to determine the sum of the difference between (i) the first sub-image and the in-focus image and the convolution of the first blur kernel and (ii) the difference between the second sub-image and the convolution of the in-focus image and the second blur kernel. The operation may also include updating one or more of the following based on the loss value and the loss function: (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel. The operations may further include generating image data based on one or more of: (i) the updated in-focus image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

[0006] In a third example embodiment, a non-transitory computer-readable medium may have instructions stored thereon that, when executed by a computing device, cause the computing device to perform an operation. The operation may obtain dual-pixel image data including a first sub-image and a second sub-image. The operation may also include determining (i) an in-focus image, (ii) a first blur kernel corresponding to the first sub-image, and (iii) a second blur kernel corresponding to the second sub-image. The operation may additionally include determining a loss value using a loss function including one or more of an equivalent loss term or a data loss term: the equivalent loss term is configured to determine the difference between (i) the convolution of the first sub-image with the second blur kernel and (ii) the convolution of the second sub-image with the first blur kernel, and the data loss term is configured to determine the sum of the difference between (i) the first sub-image and the in-focus image and the convolution of the first blur kernel and (ii) the difference between the second sub-image and the convolution of the in-focus image and the second blur kernel. The operation may also include updating one or more of the following based on the loss value and the loss function: (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel. The operations may further include generating image data based on one or more of: (i) the updated in-focus image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

[0007] In a fourth example embodiment, a system may include a device for obtaining dual-pixel image data including a first sub-image and a second sub-image. The system may also include a device for determining (i) an in-focus image, (ii) a first blur kernel corresponding to the first sub-image, and (iii) a second blur kernel corresponding to the second sub-image. The system may additionally include a device for determining a loss value using a loss function including one or more of an equivalent loss term or a data loss term: the equivalent loss term is configured to determine the difference between (i) the convolution of the first sub-image with the second blur kernel and (ii) the convolution of the second sub-image with the first blur kernel, and the data loss term is configured to determine the sum of the difference between (i) the first sub-image and the in-focus image and the convolution of the first blur kernel and (ii) the difference between the second sub-image and the convolution of the in-focus image and the second blur kernel. The system may also include a device for updating one or more of the following based on the loss value and the loss function: (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel. The system may further include means for generating image data based on one or more of: (i) the updated in-focus image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

[0008] These and other embodiments, aspects, advantages and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description and, where appropriate, referring to the accompanying drawings. In addition, this overview and the other descriptions and drawings provided herein are intended to illustrate the embodiments only by way of example, and therefore, many variations are possible. For example, structural elements and process steps may be rearranged, combined, distributed, eliminated or otherwise changed while still within the scope of the claimed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A computing device according to examples described herein is shown.

[0010] Figure 2 A computing system according to examples described herein is shown.

[0011] Figure 3 A dual pixel image sensor is shown according to examples described herein.

[0012] Figure 4A , 4B 4C illustrate aspects of systems according to examples described herein.

[0013] Figure 4D A cost volume according to examples described herein is shown.

[0014] Figure 5 A dual pixel image, blur kernel, and depth map are shown according to examples described herein.

[0015] Figure 6 An artificial neural network according to examples described herein is shown.

[0016] Figure 7 A flow chart according to examples described herein is shown. DETAILED DESCRIPTION

[0017] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" as used herein mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example," "exemplary," and / or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features unless so stated. Therefore, other embodiments may be utilized or other changes may be made without departing from the scope of the subject matter presented herein.

[0018] Therefore, the example embodiments described herein are not meant to be limiting.It will be readily understood that the various aspects of the present disclosure generally described herein and illustrated in the accompanying drawings may be arranged, substituted, combined, separated and designed in a variety of different configurations.

[0019] In addition, unless the context indicates otherwise, the features shown in each figure may be used in combination with each other. Therefore, the drawings should generally be viewed as forming aspects of one or more overall embodiments, but it should be understood that not all of the features shown are necessary for each embodiment.

[0020] In addition, any enumeration of elements, blocks, or steps in this specification or claims is for the purpose of clarity. Therefore, such enumeration should not be interpreted as requiring or implying that these elements, blocks, or steps follow a specific arrangement or are performed in a specific order. Unless otherwise specified, the drawings are not drawn to scale.

[0021] 1. Overview

[0022] A dual-pixel image sensor may include at least some pixels divided into two photosites (i.e., dual pixels), each photosite including a corresponding independently readable photodiode. Thus, the dual-pixel image sensor may be configured to generate dual-pixel image data including a first sub-image generated using a first photosite of each dual pixel (e.g., a left dual-pixel sub-image generated based on a left photosite of each dual pixel) and a second sub-image generated using a second photosite of each dual pixel (e.g., a right dual-pixel sub-image generated based on a right photosite of each dual pixel). Although the dual-pixel image data includes two sub-images, the dual-pixel image data is monoscopic because, unlike stereoscopic image data, it is generated using a single aperture.

[0023] When light reflected from a portion of a scene is focused on a corresponding portion of a dual-pixel image sensor, each photosite corresponding to the dual pixel may generate similar and / or substantially identical signals. When light reflected from the portion of the scene is out of focus with the corresponding portion of the dual-pixel image sensor, each photosite corresponding to the dual pixel may generate a different signal. Therefore, the dual-pixel image data generated by the dual-pixel image sensor may include information indicating the degree of defocus associated with each dual pixel, and may therefore be used to adjust the apparent blur associated with the dual-pixel image data.

[0024] In particular, the relationship between a first sub-image, a second sub-image, an in-focus image (e.g., an image that appears in focus along all portions of the image), a first blur kernel associated with the first sub-image, and a second blur kernel associated with the second sub-image can be used to formulate an optimization function (i.e., a loss function). For example, the in-focus image, the first blur kernel, and / or the second blur kernel can each be represented by a corresponding mathematical model. The optimization function can be used to train these mathematical models to reduce (e.g., below a threshold) and / or minimize a loss value generated using the optimization function. The optimization function can include an equivalent loss term, a data loss term, and / or one or more prior loss terms, each of which can represent a relationship between the first sub-image, the second sub-image, the in-focus image, the first blur kernel, and / or the second blur kernel.

[0025] The equivalent loss term and / or data loss term can be derived based on the relationship between the focused image, the dual-pixel sub-image and the corresponding blur kernel. In particular, the first sub-image can be represented as the convolution of the focused image and the first blur kernel. Similarly, the second sub-image can be represented as the convolution of the focused image and the second blur kernel.

[0026] Thus, the equivalence term can be configured to determine the difference between (i) the convolution of the first sub-image with the second blur kernel and (ii) the convolution of the second sub-image with the first blur kernel. Thus, the equivalence loss term can motivate optimization to generate a blur kernel that increases and / or maximizes the degree of symmetry between the convolution of the blur kernel and the two-pixel sub-image. The data loss term can be configured to determine the difference between (i) the convolution of the first sub-image and the in-focus image with the first blur kernel and (ii) the convolution of the second sub-image and the in-focus image with the second blur kernel. Thus, the data loss term can motivate optimization to generate a focused image and / or blur kernel that reduces and / or minimizes the amount of information lost due to the convolution relative to the corresponding two-pixel sub-image.

[0027] Each of the first blur kernel and the second blur kernel can be configured to deblur a portion of a scene associated with a particular depth. Thus, in order to deblur features at different depths, each of the first blur kernel and the second blur kernel can be scaled to multiple size scales, each size scale corresponding to a particular depth relative to the dual pixel camera device. An equivalent loss term and / or a data loss term can be calculated at each size scale to generate a corresponding equivalent loss image and / or a data loss image, which when stacked forms an equivalent loss cost volume and / or a data loss cost volume.

[0028] The cost volume can be reduced by selecting, for each corresponding pixel represented by a cross section of the cost volume, a corresponding pixel value of the pixel that has been most effectively deblurred from a plurality of spatially corresponding pixels along the depth of the cost volume. In some cases, the cross section of the cost volume can have the same resolution as the in-focus image. Thus, the reduction of the cost volume can indicate, for each corresponding pixel of the in-focus image, a corresponding size scale associated with the corresponding pixel. Since each size scale corresponds to a particular object depth, the reduction of the cost volume can also generate a depth map associated with the in-focus image. Thus, the optimization can simultaneously (i) generate an in-focus image, (ii) generate a corresponding depth map, (iii) select a kernel scaling for applying the first and / or second blur kernels, and in some cases, (iv) generate the first and second blur kernels.

[0029] The corresponding mathematical models that generate the in-focus image, the first blur kernel, and / or the second blur kernel can be trained on a per-image basis, rather than being trained to operate on multiple different dual-pixel images. Therefore, the training of the mathematical model is self-supervised, as the optimization function and the dual-pixel image data are sufficient to perform the training, and the training does not rely on ground truth data. In fact, the trained mathematical model can be used to generate ground truth data, which can be used to train other different machine learning models.

[0030] In some cases, since the blur kernel is not scene dependent (but depends on the shape of the aperture of the camera device), the first and second blur kernels can be calibrated for each of a plurality of different camera devices. When the first and second blur kernels are calibrated, the first and second blur kernels may not be generated from scratch by the corresponding model. Therefore, based on the specific camera device used to generate a given dual-pixel image (e.g., its specific model), the corresponding calibrated first and second blur kernels can be selected and used to train the mathematical model configured to generate the in-focus image, thereby reducing the complexity of the optimization. For example, the calibrated blur kernels can be considered to be fixed by the optimization function, or can be refined during training.

[0031] The resulting in-focus image, depth map, and / or first and second blur kernels may be used to perform various adjustments to the dual-pixel image data. For example, the degree of blur associated with various portions of the dual-pixel image data may be adjusted to a target level (e.g., by increasing and / or decreasing blur in various regions). In addition, because the depth map corresponds to the dual-pixel image data, the dual-pixel image data may be used for various augmented reality (AR) applications, such as inserting simulated objects into a scene represented by the dual-pixel image data in a depth-aware manner, among other possibilities.

[0032] 2. Example Computing Devices and Systems

[0033] Figure 1 An example computing device 100 is shown. Computing device 100 is shown in the form factor of a mobile phone. However, computing device 100 may alternatively be implemented as a laptop computer, a tablet computer, or a wearable computing device, among other possibilities. Computing device 100 may include various elements, such as a body 102, a display 106, and buttons 108 and 110. Computing device 100 may also include one or more cameras, such as a front-facing camera 104 and a rear-facing camera 112, one or more of which may be configured to generate dual-pixel image data.

[0034] Front-facing camera 104 may be positioned on a side of body 102 that generally faces a user when in operation (e.g., on the same side as display 106). Rear-facing camera 112 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to cameras as front-facing and rear-facing is arbitrary, and computing device 100 may include multiple cameras positioned on various sides of body 102.

[0035] Display 106 may represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a current image captured by front camera 104 and / or rear camera 112, an image that may be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and / or a digital representation of a modified version of one or more of these images. Thus, display 106 may be used as a viewfinder for the camera. Display 106 may also support touch screen functionality that enables adjustment of settings and / or configuration of any aspect of computing device 100.

[0036] The front camera 104 may include an image sensor and associated optical elements, such as a lens. The front camera 104 may provide a zoom capability or may have a fixed focal length. In other examples, an interchangeable lens may be used with the front camera 104. The front camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front camera 104 may also be configured to capture still images, video images, or both. In addition, the front camera 104 may represent, for example, a single-view camera. The rear camera 112 may be arranged similarly or differently. In addition, one or more of the front camera 104 and / or the rear camera 112 may be an array of one or more cameras.

[0037] One or more of the front camera 104 and / or the rear camera 112 may include or be associated with an illumination component that provides a light field to illuminate the target object. For example, the illumination component may provide a flash or constant illumination of the target object. The illumination component may also be configured to provide a light field that includes one or more of structured light, polarized light, and light with a specific spectral content. Other types of light fields that are known and used to recover a three-dimensional (3D) model from an object are also possible in the context of the examples herein.

[0038] Computing device 100 may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that may be captured by cameras 104 and / or 112. In some implementations, the ambient light sensor may be used to adjust the display brightness of display 106. Additionally, the ambient light sensor may be used to determine an exposure length for one or more of cameras 104 or 112, or to assist in such determination.

[0039] The computing device 100 may be configured to capture images of a target object using the display 106 and the front camera 104 and / or the rear camera 112. The captured images may be a plurality of still images or a video stream. Image capture may be triggered by activating a button 108, pressing a soft key on the display 106, or by some other mechanism. Depending on the implementation, images may be captured automatically at specific time intervals, for example, upon pressing the button 108, under appropriate lighting conditions for the target object, upon moving the digital camera device at a predetermined distance, or according to a predetermined capture schedule.

[0040] Figure 2 2 is a simplified block diagram illustrating some components of an example computing system 200. By way of example and not limitation, computing system 200 may be a cellular mobile phone (e.g., a smart phone), a computer (such as a desktop, notebook, tablet, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. For example, computing system 200 may represent aspects of computing device 100.

[0041] like Figure 2 As shown, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage device 208, and camera assembly 224, all of which may be communicatively linked via a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a specific physical hardware platform on which an image sensing or processing application operates in software, or other combinations of hardware and software configured to perform image capture and / or processing functions.

[0042] The communication interface 202 may allow the computing system 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, the communication interface 202 may facilitate circuit-switched and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packet communications. For example, the communication interface 202 may include a chipset and an antenna arranged to wirelessly communicate with a radio access network or access point. Furthermore, the communication interface 202 may take the form of or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI) port. The communication interface 202 may also take the form of or include a wireless interface, such as Wi-Fi, A global positioning system (GPS) or a wide area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used on the communication interface 202. In addition, the communication interface 202 may include multiple physical communication interfaces (e.g., Wi-Fi interfaces, interface and wide area wireless interface).

[0043] User interface 204 can be used for allowing computing system 200 to interact with the human or non-human user, such as receiving input from the user and providing output to the user.Therefore, user interface 204 can include input components, such as keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone etc.User interface 204 can also include one or more output components, such as display screen, which can for example be combined with touch-sensitive panel.Display screen can be based on CRT, LCD and / or LED technology, or other technologies known now or developed later.User interface 204 can also be configured to generate (one or more) audible outputs by loudspeaker, speaker jack, audio output port, audio output device, earphone and / or other similar devices.User interface 204 can also be configured to receive and / or capture (one or more) audible speech, (one or more) noise and / or (one or more) signal by microphone and / or other similar devices.

[0044] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by computing system 200. Additionally, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration and focusing of camera functions and image capture. Some or all of these buttons, switches, knobs, and / or dials may also be implemented via a touch-sensitive panel.

[0045] Processor 206 may include one or more general purpose processors - e.g., a microprocessor - and / or one or more special purpose processors - e.g., a digital signal processor (DSP), a graphics processing unit (GPU), a floating point unit (FPU), a network processor, or an application specific integrated circuit (ASIC). In some cases, the special purpose processor may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage device 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage devices, and may be integrated in whole or in part with processor 206. Data storage device 208 may include removable and / or non-removable components.

[0046] The processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in the data storage device 208 to perform various functions described herein. Therefore, the data storage device 208 may include a non-transitory computer-readable medium having stored thereon program instructions that, when executed by the computing system 200, cause the computing system 200 to perform any method, process, or operation disclosed in this specification and / or the accompanying drawings. The execution of the program instructions 218 by the processor 206 may cause the processor 206 to use the data 212.

[0047] As an example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, (one or more) device drivers, and / or other modules) and one or more applications 220 (e.g., a camera function, an address book, email, web browsing, social networking, an audio-to-text function, a text translation function, and / or a gaming application) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be primarily accessed by operating system 222, and application data 214 may be primarily accessed by one or more applications 220. Application data 214 may be arranged in a file system that is visible or hidden to a user of computing system 200.

[0048] Applications 220 may communicate with operating system 222 via one or more application programming interfaces (APIs). These APIs may facilitate, for example, applications 220 reading and / or writing application data 214, sending or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and the like.

[0049] In some cases, the application 220 may be referred to simply as an "app". In addition, the application 220 may be downloaded to the computing system 200 through one or more online application stores or application markets. However, the application may also be installed on the computing system 200 in other ways, such as through a web browser or through a physical interface on the computing system 200 (e.g., a USB port).

[0050] Camera assembly 224 may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, a shutter button, an infrared projector, and / or a visible light projector. Camera assembly 224 may include components configured to capture images in the visible light spectrum (e.g., electromagnetic radiation having a wavelength of 380-700 nanometers) and components configured to capture images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers-1 millimeter). Camera assembly 224 may be controlled, at least in part, by software executed by processor 206.

[0051] 3. Example Dual Pixel Image Sensor

[0052] Figure 3 A dual pixel image sensor 300 configured to generate dual pixel image data is shown. In particular, the dual pixel image sensor 300 includes a plurality of pixels arranged in a grid, the grid including columns 302, 304, 306, and 308 to 310 (i.e., columns 302-310) and rows 312, 314, 316, and 318 to 320 (i.e., rows 312-320). Each pixel is shown as being divided into a first (left) photosite (indicated by a corresponding shaded area) and a second (right) photosite (indicated by a corresponding white filled area). Thus, the right half of a pixel located at column 302, row 312 is labeled "R" to indicate a right photosite, and the left half of a pixel is labeled "L" to indicate a left photosite. Although the photosite of each pixel is shown as being divided into two equal vertical halves, the photosites may alternatively divide each pixel in other ways. For example, each pixel may be divided into a top photosite and a bottom photosite. The areas of the photosites may not be equal.

[0053] Each photosite of a given pixel may include a corresponding photodiode, whose output signal may be read independently of other photodiodes. In addition, each pixel of the dual pixel image sensor 300 may be associated with a corresponding color filter (e.g., red, green, or blue). A de-mosaicing algorithm may be applied to the output of the dual pixel image sensor 300 to generate a color image. In some cases, fewer pixels than all pixels of the dual pixel image sensor 300 may be divided into multiple photosites. For example, each pixel associated with a green filter may be divided into two independent photosites, while each pixel associated with a red or blue filter may include a single photosite. In some cases, the dual pixel image sensor 300 may be used to implement the front camera 104 and / or the rear camera 112, and may form part of the camera assembly 224.

[0054] The dual-pixel image sensor 300 can be configured to generate dual-pixel image data. The dual-pixel image data can include a first sub-image generated by a first photosite (e.g., only the left photosite) and a second sub-image generated by a second photosite (e.g., only the right photosite). When the imaged object or portion thereof is in focus, the corresponding signals generated by each photosite of a given pixel can be substantially the same (e.g., the two signals can be within a threshold of each other). When the imaged object or portion thereof is out of focus, the corresponding signal generated by the first photosite of a given pixel may be different from the corresponding signal generated by the second photosite of the given pixel, and the amount of difference is proportional to the degree of defocus. Therefore, the dual-pixel image data generated by the dual-pixel image sensor 300 can provide defocus cues that can be used to adjust the focus levels associated with the respective portions of the dual-pixel image data. The signals generated by each photosite of a given pixel can be combined into a single output signal to generate conventional (e.g., RGB) image data.

[0055] Example System for Generating Focused Images and Corresponding Depth Maps

[0056] Figure 4A A system 400 is shown that is configured to determine a degree of defocus and a depth map associated with dual-pixel image data and to determine the degree of defocus and the depth map based on the dual-pixel image data. In particular, the system 400 includes a focused image model 408, a first blur kernel model 410, a second blur kernel model 412, a blur kernel scaler 420, a loss function 426, and a model parameter adjuster 438. The loss function 426 may include a data loss term 428, an equivalent loss term 430, and a priori loss term 432. Each component of the system 400 may represent software instructions (e.g., configured to be executed by the processor 206), a hardware component (e.g., a specially constructed circuit), or a combination thereof.

[0057] The dual-pixel image data 402 may be generated by, for example, the dual-pixel image sensor 300. The system 400 may be configured to receive the dual-pixel image data 402 as input. The dual-pixel image data 402 may include a first sub-image 404 generated using a first subset of dual-pixel photosites (e.g., left photosites) of the dual-pixel image sensor 300 and a second sub-image 406 generated using a second subset of dual-pixel photosites (e.g., right photosites) of the dual-pixel image sensor 300. Depending on the context, the dual-pixel image data 402 may be used herein to refer to (i) the first sub-image 404 and the second sub-image 406 or (ii) image data generated by a combination of the first sub-image 404 and the second sub-image 406.

[0058] The in-focus image model 408 may be configured to generate an in-focus image 414, which may represent a modified version of the dual-pixel image data 402 in which at least a portion of the dual-pixel image data 402 is adjusted to improve the degree of focus. When the in-focus image model 408 is trained to at least a threshold level of accuracy (e.g., quantified by the loss function 426), the in-focus image 414 may represent a modified version of the dual-pixel image data 402 in which most of its portions are brought into focus (i.e., exhibiting less than a threshold degree of defocus blur). For example, the in-focus image 414 may represent a version of the dual-pixel image data 402 in which substantially all of its portions are in focus. The in-focus image model 408 may include a corresponding mathematical model that generates a corresponding pixel value for each pixel of the in-focus image 414. One or more parameters of the mathematical model may be adjusted during training (e.g., by a model parameter adjuster 438) to improve the accuracy of the in-focus image 414.

[0059] The first blur kernel model 410 may be configured to generate a first (convolution) blur kernel 416, and the second blur kernel model 412 may be configured to generate a second (convolution) blur kernel 418. In some cases, a single blur kernel model may be configured to generate a single blur kernel image representing the first blur kernel 416 (e.g., on a first half of the blur kernel image) and the second blur kernel 418 (e.g., on a second portion of the blur kernel image). The blur kernel model for each of the first blur kernel 416 and the second blur kernel 418 may be based on the aperture shape of the particular dual pixel camera device used to generate the dual pixel image data 402, and therefore may not be scene dependent. Thus, the first blur kernel model 410 and the second blur kernel model 412 may vary depending on whether the respective blur kernel has been pre-determined / calibrated for a particular dual pixel camera device.

[0060] In some implementations, the first blur kernel model 410 may include a corresponding mathematical model configured to generate a corresponding pixel value for each pixel of the first blur kernel 416. Similarly, the second blur kernel model 412 may include a corresponding mathematical model configured to generate a corresponding pixel value for each pixel of the second blur kernel 418. One or more parameters of each of these mathematical models may be adjusted during training to improve the accuracy of the first blur kernel 416 and / or the second blur kernel 418 (e.g., quantified by the loss function 426). For example, such implementations of the first blur kernel model 410 and / or the second blur kernel model 412 may be used when the corresponding blur kernels are not pre-determined / calibrated for a particular dual-pixel camera device and are therefore generated based on the loss function 426.

[0061] In other embodiments, the first blur kernel model 410 may include one or more representations of the first blur kernel 416 that have been pre-determined / calibrated for the corresponding one or more dual-pixel camera devices. Thus, the first blur kernel model 410 may be configured to select one of these representations of the first blur kernel 416 based on the specific dual-pixel camera device used to generate the dual-pixel image data 402. Similarly, the second blur kernel model 412 may include one or more representations of the second blur kernel 418 that have been pre-determined / calibrated for the corresponding first one or more dual-pixel camera devices. Thus, the second blur kernel model 412 may be configured to select one of these representations of the second blur kernel 418 based on the specific dual-pixel camera device used to generate the dual-pixel image data 402. In some cases, the first blur kernel model 410 and / or the second blur kernel model 412 may also be configured to refine the representations of the selected first blur kernel 416 and / or second blur kernel 418, respectively, as part of a training process, and thus may include corresponding mathematical models in addition to the representations of the pre-determined / calibrated kernels.

[0062] Each of the first blur kernel 416 and the second blur kernel 418 may represent a corresponding point spread function (PSF) associated with the optical components of the dual pixel camera device used to generate the dual pixel image data 402. In particular, the first blur kernel 416 may be associated with the first sub-image 404 because the first blur kernel 416 may represent the PSF associated with the first (e.g., left) photosite of the dual pixel image sensor, and the second blur kernel 418 may be associated with the second sub-image 406 because the second blur kernel 418 may represent the PSF associated with the second (e.g., right) photosite of the dual pixel image sensor.

[0063] The size of each of the first blur kernel 416 and the second blur kernel 418 can be based on the depth of the object or portion thereof being imaged. Thus, when the dual-pixel image data 402 contains representations of objects or portions thereof located at different depths relative to the dual-pixel camera device, each portion of the dual-pixel image data 402 can be associated with a first blur kernel 416 and a second blur kernel 418 that have been scaled to corresponding sizes associated with the corresponding depths.

[0064] In particular, the relationship between the focused image 414, the dual pixel image data 402, and the first blur kernel 416 can be expressed as , where I1(x) represents a specific pixel of the first sub-image 404, I(x) represents a corresponding pixel of the focused image 414, * represents a convolution operation, and represents a first blur kernel 416 scaled to a radius associated with a corresponding depth represented by a corresponding pixel of the in-focus image 414. Similarly, the relationship between the in-focus image 414, the dual pixel image data 402, and the second blur kernel 418 can be represented by , where I2(x) represents a specific pixel of the second sub-image 406, I(x) represents a corresponding pixel of the focused image 414, * represents a convolution operation, and represents a second blur kernel 418 scaled to a radius associated with a corresponding depth represented by a corresponding pixel of the in-focus image 414. In some cases, these equations may additionally model noise by including, for example, a Gaussian noise term.

[0065] Thus, the system 400 includes a blur kernel scaler 420 configured to (i) generate a first plurality of scaled blur kernels 422 based on the first blur kernel 416 and (ii) generate a second plurality of scaled blur kernels 424 based on the second blur kernel 418. The first plurality of scaled blur kernels 422 may include the first blur kernel 416 represented at a plurality of different size scales (e.g., 1.0x, 0.8x, 0.6x, 0.4x, 0.2x, 0.001x, -0.2x, -0.4x, -0.6x, -0.8x, and -1.0x). Similarly, the second plurality of scaled blur kernels 424 may include the second blur kernel 418 represented at a plurality of different size scales.

[0066] The system 400 may be configured to apply the first plurality of scaled blur kernels 422 and the second plurality of scaled blur kernels 424 to the dual-pixel image data 402 and / or the focused image 414 to determine a corresponding size scale for each pixel of the dual-pixel image data 402. In particular, the dual-pixel image data 402, the focused image 414, the first plurality of scaled blur kernels 422, and the second plurality of scaled blur kernels 424 may be processed by a loss function 426 to generate a depth map 434 and a loss value 436. The loss function 426 may include a data loss term 428 (which is Figure 4B ), the equivalent loss term 430 (which is shown in more detail in Figure 4C ) and a priori loss term 432. The corresponding size scale for each pixel of dual-pixel image data 402 can be selected based on the output of one or more terms of loss function 426. Since each size scale is associated with a corresponding depth value, selecting a corresponding size scale for each pixel of dual-pixel image data 402 can also generate a depth map 434 representing the depth of each pixel of dual-pixel image data 402.

[0067] The model parameter adjuster 438 may be configured to generate a parameter adjustment 440 based on the loss value 436 and the loss function 426. In particular, the parameter adjustment 440 may represent one or more changes to one or more parameters of the focused image model 408, the first blur kernel model 410, and / or the second blur kernel model 412. The parameter adjustment 440 may be selected to reduce the loss value 436 produced by subsequent processing iterations of the system 400. For example, the model parameter adjuster 438 may be configured to select the parameter adjustment 440 (i) at a point corresponding to the loss value 436 and (ii) based on the gradient of the loss function 426 with respect to the parameters of the focused image model 408, the first blur kernel model 410, and / or the second blur kernel model 412.

[0068] Thus, the model parameter adjuster 438 can generate parameter adjustments 440 that result in successively lower loss values ​​436. The system 400 can be configured to train a different instance of the in-focus image model 408, the first blur kernel model 410, and / or the second blur kernel model 412 with respect to each instance of the dual-pixel image data 402. Thus, the in-focus image model 408, the first blur kernel model 410, and / or the second blur kernel model 412 can be independently improved, refined, and / or optimized with respect to each instance of the dual-pixel image data 402, rather than being improved, refined, and / or optimized for a training data set that includes multiple different instances of the dual-pixel image data 402.

[0069] Once the training of the focused image model 408, the first blur kernel model 410, and / or the second blur kernel model 412 is completed, the focused image 414, the first blur kernel 416, and / or the second blur kernel 418 can be used to generate and / or display image data that modifies one or more aspects of the dual-pixel image data 402. In one example, the generated image data can include the dual-pixel image data 402 with the blur level of some portions thereof adjusted. In some cases, the blur in some portions of the dual-pixel image data 402 can be reduced, while the blur in other portions of the dual-pixel image data 402 can be increased. In other cases, all portions of the dual-pixel image data 402 can be brought into focus, and the generated image data can therefore be equivalent to the focused image 414. In another example, a representation of an object can be added to the dual-pixel image data 402 in a depth-aware manner.

[0070] 5. Example Data Loss Item Calculation

[0071] Figure 4B An example implementation of a data loss term 428 of a loss function 426 is shown. The first plurality of scaled blur kernels 422 may include scaled blur kernels 442 through 444 (i.e., scaled blur kernels 442-444) representing the first blur kernel 416 at a plurality of different size scales, and the second plurality of scaled blur kernels 424 may include scaled blur kernels 446 through 448 (i.e., scaled blur kernels 446-448) representing the second blur kernel 418 at a plurality of different size scales.

[0072] The data loss term 428 may be configured to compute, for each respective size scale in the plurality of size scales, a corresponding sum image based on a sum 462 of: (i) an absolute value (determined by an absolute value operator 454) of a difference 452 between the convolution 450 of the first sub-image 404 and the in-focus image 414 with the corresponding scaled blur kernels 442-444 and (ii) an absolute value (determined by an absolute value operator 460) of a difference 458 between the convolution 456 of the second sub-image 406 and the in-focus image 414 with the corresponding scaled blur kernels 446-448. Thus, the data loss term 428 may implement the function Wherein, I represents the focused image 414, I1 represents the first sub-image 404, and I2 represents the second sub-image 406. represents a scaled blur kernel among the scaled blur kernels 442-444 associated with the corresponding size scale, and Scaled blur kernels among the scaled blur kernels 446 - 448 are represented as being associated with corresponding size scales.

[0073] The data loss cost volume 464 may be generated by stacking a sum image corresponding to each corresponding size scale in a plurality of size scales. Thus, the data loss cost volume 464 may have a width W corresponding to the width of the sum image (which may have the same resolution as the in-focus image 414), a height H corresponding to the height of the sum image, and a depth D corresponding to a plurality of size scales. The loss value calculator 466 may be configured to determine a data loss value 470 and a data loss depth map 468 based on the data loss cost volume 464. In particular, the loss value calculator 466 may be configured to select a minimum pixel value from a plurality of pixel values ​​spatially corresponding to the corresponding pixel along the depth of the cost volume 464 for each pixel of the W pixels by H pixels of a cross-section of the data loss cost volume 464. Furthermore, the data loss value 470 may be determined by summing the values ​​selected for the corresponding pixels.

[0074] Each selected pixel value can be associated with a corresponding size scale of the corresponding pixel. Therefore, the loss value calculator 466 can be configured to select, for each corresponding pixel of the W pixels by H pixels of the cross-section of the data loss cost volume 464, a corresponding size scale of the blur kernel that most effectively deblurs the corresponding pixel. In addition, since each size scale can correspond to a specific depth relative to the dual-pixel camera device used to capture the dual-pixel image data 402, selecting a corresponding size scale for each corresponding pixel can indicate a depth value associated with the corresponding pixel. Therefore, the data loss value 470 and the data loss depth map 468 can be determined together by the described process.

[0075] 6. Example of equivalent loss calculation

[0076] Figure 4C An example implementation of an equivalent loss term 430 of the loss function 426 is shown. The equivalent loss term 430 can be configured to calculate, for each corresponding size scale in a plurality of size scales, a corresponding difference image based on the absolute value of the difference 476 (determined by the absolute value operator 478) of: (i) the convolution 472 of the second sub-image 406 with the corresponding scaled blur kernel in the scaled blur kernels 442-444 and (ii) the convolution 474 of the first sub-image 404 with the corresponding scaled blur kernel in the scaled blur kernels 446-448. Thus, the equivalent loss term 430 can implement the function Wherein, I1 represents the first sub-image 404, I2 represents the second sub-image 406, represents a scaled blur kernel among the scaled blur kernels 442-444 associated with the corresponding size scale, and represents the scaled blur kernels in the scaled blur kernels 446-448 associated with the corresponding size scale. In particular, because and but (Due to the commutative property of convolution), so Or equivalently,

[0077] The equivalent loss cost volume 480 can be generated by stacking difference images corresponding to each of a plurality of size scales. Thus, the equivalent loss cost volume 480 can have a width W corresponding to the width of the difference image, a height H corresponding to the height of the difference image, and a depth D corresponding to a plurality of size scales. A loss value calculator 482 (which can be the same or similar to the loss value calculator 466) can be configured to determine an equivalent loss value 486 and an equivalent loss depth map 484 based on the equivalent loss cost volume 480. In particular, the loss value calculator 482 can be configured to select a minimum pixel value from a plurality of pixel values ​​that spatially correspond to the corresponding pixel along the depth of the cost volume 480 for each of the W pixels by H pixels of the cross section of the equivalent loss cost volume 480. In addition, the equivalent loss value 486 can be determined by summing the values ​​selected for the corresponding pixels.

[0078] Since each selected pixel value can be associated with a corresponding size scale of the corresponding pixel, the loss value calculator 482 can be configured to select, for each pixel of the W pixels by H pixels of the cross-section of the equivalent loss cost volume 480, a corresponding size scale of the blur kernel that most effectively deblurs the corresponding pixel. In addition, since each size scale can correspond to a specific depth relative to the dual-pixel camera device used to capture the dual-pixel image data 402, selecting the corresponding size scale for each corresponding pixel can indicate a depth value associated with the corresponding pixel. Therefore, the equivalent loss value 486 and the equivalent loss depth map 484 can be determined jointly by the described process.

[0079] 7. Example Cost Volume Reduction

[0080] Figure 4D A visual illustration of a cost volume and its reduction (e.g., by a loss value calculator 466 and / or 482) to a loss value is provided. In particular, the cost volume 490 can represent, for example, the data loss cost volume 464 and / or the equivalent loss cost volume 480. The depth dimension, which includes D pixels, can correspond to and be based on the number of size scales to which each of the first blur kernel 416 and the second blur kernel 418 is scaled. The width and height dimensions, which include W pixels and H pixels, respectively, can correspond to and be based on the number of pixels present in the sum image (in the case of the data loss term 428) or the difference image (in the case of the equivalent loss term 430) used to generate the cost volume 490.

[0081] The cost volume 490 may be reduced along the depth dimension D to generate a cost image 492. In particular, each pixel position along a W pixel by H pixel cross section of the cost volume 490 may be associated with D spatially corresponding pixels along the depth dimension of the cost volume 490. In particular, Figure 4D A cost volume 490 is shown having a cross section including 100 pixel locations, each pixel location being associated with 9 spatially corresponding pixels along the depth dimension, for a total of 900 pixels. The cost volume 490 can be reduced to a cost image 492 by selecting the minimum pixel value along the depth dimension for each of the HxW (e.g., 100) pixel locations of the cross section of the cost volume 490. For example, the lowest value pixel can be selected from the pixels indicated by the shaded pattern that are part of the cost volume 490, thereby generating a corresponding pixel value of the cost image 492.

[0082] Thus, the cost image 492 may represent the lowest value pixel at each pixel position along the HxW cross section of the cost volume 490. Since each lowest value pixel is also associated with a corresponding size scale, which in turn is associated with a corresponding depth value, the cost image 492 may also represent and be used to generate a depth map (e.g., data loss depth map 468 and / or equivalent loss depth map 484). In some cases, a depth map may use unitless values ​​to represent the relative depth of each pixel therein. When a particular blur kernel associated with a given size scale is mapped to a corresponding physical depth (e.g., based on the optical properties of a dual-pixel camera device), the depth map may additionally or alternatively use values ​​corresponding to physical units (e.g., meters) to represent the depth of each pixel. The cost image 492 may be further reduced to a cost value (representing, for example, data loss value 470 and / or equivalent loss value 486) by replacing the sum of all pixel values ​​of the cost image 492.

[0083] Back to Figure 4A , loss value 436 may represent, for example, data loss value 470, equivalent loss value 486, and / or a sum of a priori loss values ​​generated by (one or more) prior loss terms 432. Depth map 434 may represent, for example, data loss depth map 468, equivalent loss depth map 484, or a combination thereof. For example, depth map 434 may represent a pixel-by-pixel average of data loss depth map 468 and equivalent loss depth map 484.

[0084] The a priori loss term(s) 432 may include one or more equations that further define aspects of the optimization problem solved by the system 400 to accelerate convergence to a solution and / or improve the accuracy of the solution. In one example, the a priori loss term(s) 432 may include a total variation denoising loss term that is configured to determine the total variation of (i) the in-focus image 414, (ii) the first blur kernel 416, and / or (iii) the second blur kernel 418. For example, the total variation denoising term may be configured to calculate the sum of the total variation of each of the in-focus image 414, the first blur kernel 416, and the second blur kernel 418. The total variation value of the image signal y may be calculated, for example, according to ∑ i,j |y i+1,j -y i,j |+|y i,j+1 -y i,j | represents, where the image signal y can represent the focused image 414, the first blur kernel 416 and / or the second blur kernel 418, and where i and j represent pixel coordinates.

[0085] In a second example, the prior loss term(s) 432 may include a deep convolution prior loss term configured to determine the result of generating and / or processing (i) the in-focus image 414, (ii) the first blur kernel 416, and / or (iii) the second blur kernel 418 by one or more artificial neural networks. For example, the prior loss term may be implicitly included by parameterizing (i) the in-focus image 414, (ii) the first blur kernel 416, and / or (iii) the second blur kernel 418 as outputs of one or more artificial neural networks, for example, as described in the following papers: "Deep Image Prior" by Dmitry Ulyanov et al. (the deep convolution prior may therefore be considered a deep image prior), and / or "Neural Blind Deconvolution Using Deep Priors" by Dongwei Ren et al. (the deep convolution prior may therefore be considered a blind deconvolution prior). Other priors based on neural network processing are also possible.

[0086] In a third example, the prior loss term(s) 432 may include a depth loss term configured to compare (i) one or more features of the depth map 434 with (ii) one or more corresponding features of the in-focus image 414. For example, the depth loss term may quantify the spatial alignment of representations of a given set of features (such as edges) in the depth map 434 and the in-focus image 414. Thus, the system 400 may be trained to generate in-focus images 414 and depth maps 434 that are consistent with each other.

[0087] 8. Example Images, Kernels, and Depth Maps

[0088] Figure 5 A visual illustration of some of the images, kernels, and depth maps used and / or generated by system 400 is provided. In particular, Figure 5 A first sub-image 404 and a second sub-image 406 are shown, each containing respective representations of an object 500 and an object 502. During image capture, the object 500 may have been within the depth of focus of the dual pixel camera device and thus appeared in focus in both the first sub-image 404 and the second sub-image 406. On the other hand, the object 502 may have been outside the depth of focus of the dual pixel camera device (e.g., out of focus) and thus appeared out of focus (i.e., blurred) in both the first sub-image 404 and the second sub-image 406. The difference in the appearance of the object 502 is visually exaggerated, and a dashed reference line is provided to illustrate the difference between the first sub-image 404 and the second sub-image 406 due to the object 502 being out of focus.

[0089] The first blur kernel 416 and the second blur kernel 418 are shown as having an approximately semicircular shape, which may be due to the dual pixel camera device having a circular aperture. Other aperture shapes may produce blur kernels of different shapes. The first blur kernel 416 and the second blur kernel 418 are not drawn to scale with the first sub-image 404 and the second sub-image 406. The first blur kernel 416 and the second blur kernel 418 are shown as each having a radius of five pixels, which may represent the kernel at full scale (i.e., 1.0x). When the first blur kernel 416 and / or the second blur kernel 418 are scaled to, for example, 0.6x, the scaled first blur kernel 416 and / or the second blur kernel 418 may instead have a radius of 3 pixels.

[0090] In-focus image 414 shows object 500 and object 502, both of which are shown in focus. In particular, the appearance of object 500 has not been adjusted because it was initially in focus, but the appearance of object 502 has been adjusted based on first blur kernel 416 and / or second blur kernel 418 as a result of convolving some portions of it with blur kernels of appropriate sizes. In depth map 434, features that are closer to the dual pixel camera device are indicated with lighter colors (with white indicating minimum depth), while features that are farther away are indicated with darker colors (with black indicating maximum depth). Therefore, depth map 434 indicates that object 500 is closer to the dual pixel camera device than object 502.

[0091] IX. Example Images and Kernel Model Parameterization

[0092] Figure 6 An example implementation of the focused image model 408 is shown. In particular, Figure 6A focused image model 408 is shown implemented as an artificial neural network (ANN) 600, which may alternatively be referred to as a multilayer perceptron (MLP). The ANN 600 includes an input layer 640, hidden layers 650 to 660, and an output layer 670. The input layer 640 includes input neurons 641 and 642. The hidden layer 650 includes neurons 651, 652, 653, 654, and 655 to 656. The hidden layer 660 includes neurons 661, 662, 663, 664, and 665 to 666. The output layer 670 includes neurons 671, 672, and 673. A given neuron may be configured to calculate a weighted sum of the outputs of the neurons in the previous layer and apply an activation function to the calculated sum, thereby generating an output of the given neuron.

[0093] ANN 600 may be configured to generate a corresponding red-green-blue (RGB) pixel value for each pixel of in-focus image 414. In particular, ANN 600 may be configured to receive a set of pixel coordinates at input layer 640, where the horizontal coordinate x of the nth pixel is n is provided to the input neuron 641 and the vertical coordinate y of the nth pixel n is provided to the input neuron 642. ANN 600 can be configured to calculate the n and n The corresponding red pixel value V is generated at the output neuron 671 RED , generating the corresponding blue pixel value V at the output neuron 672 BLUE And the corresponding green pixel value V is generated at the output neuron 673 GREEN Thus, when in-focus image 414 includes N pixels, ANN 600 may be used N times to generate corresponding RGB pixel values ​​for each respective pixel n of the N pixels forming in-focus image 414. Parameterizing in-focus image 414 using ANN 600 may be beneficial because ANN 600 may be inherently configured to generate realistic and spatially consistent image data due to its structure.

[0094] In some embodiments, the ANN 600 or a modified version thereof can be used to implement the first blur kernel model 410 and / or the second blur kernel model 412. In particular, in the case where each pixel of the first blur kernel 416 and / or the second blur kernel 418 is associated with a single value instead of an RGB value, the ANN 600 can be modified by including a single output neuron instead of three output neurons. Thus, the modified ANN 600 can be configured to generate a pixel value based on the horizontal and vertical coordinates x of the nth pixel of the first blur kernel 416 and / or the second blur kernel 418. n and n , generating the corresponding pixel value V at the output neuron 671BLUR Thus, when the first blur kernel 416 (or the second blur kernel 418) includes M pixels, the modified ANN 600 may be used M times to generate a corresponding pixel value for each respective pixel n of the M pixels forming the first blur kernel 416 (or the second blur kernel 418).

[0095] Alternatively, the in-focus image 414, the first blur kernel 416, and / or the second blur kernel 418 may be parameterized and generated using other mathematical models. In one example, the in-focus image 414, the first blur kernel 416, and / or the second blur kernel 418 may be represented using a grid parameterization, where each pixel value is represented directly and / or explicitly. Thus, the in-focus image model 408, the first blur kernel model 410, and / or the second blur kernel model 412 may store, for example, corresponding lookup tables that define and / or may be used to generate values ​​for the corresponding image, and these values ​​may be adjusted directly during model training.

[0096] 10. Additional sample operations

[0097] Figure 7 A flowchart of operations related to generating a focus image and / or blur kernel using dual pixel image data is shown. These operations can be performed by computing device 100, computing system 200, dual pixel image sensor 300 and / or system 400, among other possibilities, and / or based on outputs thereof. Figure 7 The embodiments of the present invention can be simplified by removing any one or more features shown therein. In addition, these embodiments can be combined with any of the features, aspects and / or implementations described in any of the foregoing figures or in other ways herein.

[0098] Block 700 may involve obtaining dual-pixel image data including a first sub-image and a second sub-image.

[0099] Block 702 may involve determining (i) an in-focus image, (ii) a first blur kernel corresponding to a first sub-image, and (iii) a second blur kernel corresponding to a second sub-image.

[0100] Box 704 may involve determining a loss value using a loss function that includes one or more of an equivalent loss term or a data loss term: the equivalent loss term is configured to determine the difference between (i) the convolution of the first sub-image with the second blur kernel and (ii) the convolution of the second sub-image with the first blur kernel, and the data loss term is configured to determine the sum of (i) the difference between the convolution of the first sub-image and the in-focus image with the first blur kernel and (ii) the difference between the convolution of the second sub-image and the in-focus image with the second blur kernel.

[0101] Block 706 may involve updating one or more of: (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel based on the loss value and the loss function.

[0102] Block 708 may involve generating image data based on one or more of: (i) the updated in-focus image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

[0103] In some embodiments, determining the loss value using the loss function may include receiving a first plurality of scaled blur kernels corresponding to a plurality of size scales. The first plurality of scaled blur kernels may be generated based on the first blur kernel. A second plurality of scaled blur kernels corresponding to a plurality of size scales may be received. The second plurality of scaled blur kernels may be generated based on the second blur kernel. A cost volume may be generated by determining corresponding results of one or more of an equivalent loss term or a data loss term for each of the plurality of size scales. The loss value may be determined based on the cost volume.

[0104] In some embodiments, generating a cost volume by determining a corresponding result of an equivalent loss term for each of a plurality of size scales may include generating an equivalent loss cost volume by determining a corresponding difference image for each of the plurality of size scales based on a difference between: (i) a convolution of the first sub-image with a corresponding blur kernel in the second plurality of scaled blur kernels and (ii) a convolution of the second sub-image with a corresponding blur kernel in the first plurality of scaled blur kernels. The loss value may include an equivalent loss value determined based on the equivalent loss cost volume.

[0105] In some embodiments, the corresponding difference image may have a height of H pixels and a width of W pixels. The multiple size scales may include D size scales. The cross-section of the equivalent loss cost volume may include H pixels by W pixels, and the depth of the equivalent loss cost volume may include D pixels. Determining the equivalent loss value based on the equivalent loss cost volume may include: for each corresponding pixel of the H pixels by W pixels of the cross-section of the equivalent loss cost volume, selecting the minimum pixel value from the D pixel values ​​that spatially correspond to the corresponding pixel along the depth of the equivalent loss cost volume. The equivalent loss value can be determined based on the sum of the minimum pixel values ​​selected for each corresponding pixel of the H pixels by W pixels of the cross-section of the equivalent loss cost volume.

[0106] In some embodiments, generating a cost volume by determining a corresponding result of a data loss term for each of a plurality of size scales may include generating a data loss cost volume by determining a corresponding sum image for each of the plurality of size scales based on: (i) a difference between a convolution of the first sub-image and the in-focus image with a corresponding blur kernel in a first plurality of scaled blur kernels and (ii) a difference between a convolution of the second sub-image and the in-focus image with a corresponding blur kernel in a second plurality of scaled blur kernels. The loss value may include a data loss value determined based on the data loss cost volume.

[0107] In some embodiments, the corresponding sum image may have a height of H pixels and a width of W pixels. The plurality of size scales may include D size scales. The cross-section of the data loss cost volume may include H pixels by W pixels, and the depth of the data loss cost volume may include D pixels. Determining the data loss value based on the data loss cost volume may include, for each corresponding pixel of the H pixels by W pixels of the cross-section of the data loss cost volume, selecting a minimum pixel value from the D pixel values ​​that spatially correspond to the corresponding pixel along the depth of the data loss cost volume. The data loss value may be determined based on the sum of the minimum pixel values ​​selected for each corresponding pixel of the H pixels by W pixels of the cross-section of the data loss cost volume.

[0108] In some embodiments, each of the plurality of size scales may be associated with a corresponding depth. A corresponding size scale in the plurality of size scales may be selected based on a cost volume for each corresponding pixel of the focused image. A depth map associated with the dual pixel image data may be generated based on the corresponding size scale selected for each corresponding pixel of the focused image.

[0109] In some embodiments, the loss function may include a depth loss term configured to compare (i) one or more features of the depth map with (ii) one or more corresponding features of the in-focus image.

[0110] In some embodiments, generating image data based on one or more of: (i) an updated in-focus image, (ii) an updated first blur kernel, or (iii) an updated second blur kernel may include generating a refocused image by modifying a depth of focus associated with at least a portion of the dual-pixel image data based on the depth map. The refocused image may be displayed.

[0111] In some embodiments, each of the in-focus image, the first blur kernel, and the second blur kernel can be parameterized by a corresponding mathematical model and generated using the corresponding mathematical model. Updating one or more of the following: (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel can include determining a gradient of a loss function corresponding to the loss value and with respect to one or more parameters of one or more of the corresponding mathematical models. One or more parameters of one or more of the corresponding mathematical models can be updated based on the gradient of the loss function. The updated in-focus image can be generated using the updated corresponding mathematical model, the updated first blur kernel can be generated using the updated corresponding mathematical model, and / or the updated second blur kernel can be generated using the updated corresponding mathematical model.

[0112] In some embodiments, the mathematical model corresponding to the focused image can be specific to the dual-pixel image data, such that the value of at least one parameter of the mathematical model corresponding to the focused image can be different from the value of at least one corresponding parameter of the mathematical model corresponding to different dual-pixel image data.

[0113] In some embodiments, the corresponding mathematical model configured to parameterize and generate the in-focus image may include a first artificial neural network. Generating the in-focus image may include passing the first artificial neural network based on the corresponding pixel coordinates of the in-focus image and generating corresponding pixel values ​​for each corresponding pixel coordinate of the in-focus image. Generating an updated in-focus image may include passing the updated first artificial neural network based on the corresponding pixel coordinates of the updated in-focus image and generating updated corresponding pixel values ​​for each corresponding pixel coordinate of the updated in-focus image.

[0114] In some embodiments, the mathematical model corresponding to the specific blur kernel configured to parameterize and generate the first blur kernel and the second blur kernel may include a second artificial neural network. Generating the specific blur kernel may include generating corresponding pixel values ​​based on the corresponding pixel coordinates of the specific blur kernel through the second artificial neural network and for each corresponding pixel coordinate of the specific blur kernel. Updating the specific blur kernel may include generating updated corresponding pixel values ​​based on the corresponding pixel coordinates of the updated specific blur kernel through the updated second artificial neural network and for each corresponding pixel coordinate of the updated specific blur kernel.

[0115] In some embodiments, determining the first blur kernel and the second blur kernel may include identifying a specific camera device that has been used to generate dual-pixel image data, and obtaining a first blur kernel calibrated for the specific camera device and a second blur kernel calibrated for the specific camera device based on the specific camera device.

[0116] In some embodiments, the loss function may include a total variation denoising loss term configured to determine a total variation of one or more of (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel.

[0117] In some embodiments, the loss function may include a deep convolutional prior loss term configured to determine a result of processing one or more of (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel by one or more artificial neural networks.

[0118] In some embodiments, a loss value may be updated using a loss function based on one or more of: (i) an updated in-focus image, (ii) an updated first blur kernel, or (iii) an updated second blur kernel. It may be determined that the updated loss value is below a threshold loss value. Based on determining that the updated loss value is below the threshold loss value, image data may be generated based on one or more of (i) the updated in-focus image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

[0119] In some embodiments, second dual-pixel image data including a third sub-image and a fourth sub-image may be obtained. A second focused image, a third blur kernel corresponding to the third sub-image, and / or a fourth blur kernel corresponding to the fourth sub-image may be determined. A second loss value may be determined using a loss function including one or more of an equivalent loss term or a data loss term: the equivalent loss term is configured to determine the difference between (i) the convolution of the third sub-image with the fourth blur kernel and (ii) the convolution of the fourth sub-image with the third blur kernel, and the data loss term is configured to determine the sum of the difference between (i) the convolution of the third sub-image and the second focused image with the third blur kernel and (ii) the convolution of the fourth sub-image and the second focused image with the fourth blur kernel. Based on the second loss value and the loss function, one or more of the following may be updated: (i) the second focused image, (ii) the third blur kernel, or (iii) the fourth blur kernel. Second image data may be generated based on one or more of the following: (i) the updated second focused image, (ii) the updated third blur kernel, or (iii) the updated fourth blur kernel. The second image data may be displayed.

[0120] 11. Conclusion

[0121] The present disclosure is not limited to the specific embodiments described in this application, which are intended to illustrate various aspects. Many modifications and variations can be made without departing from the scope of the present disclosure, which will be apparent to those skilled in the art. In addition to the methods and devices described herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0122] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context otherwise indicates. The example embodiments described herein and in the accompanying drawings are not intended to be limiting. Other embodiments may be used, and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure generally described herein and shown in the appended drawings may be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0123] With respect to any or all message flow diagrams, scenarios, and process diagrams in the accompanying drawings and discussed herein, each step, box, and / or communication may represent the processing of information and / or the transmission of information according to the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the operations described as steps, boxes, transmissions, communications, requests, responses, and / or messages may be performed in a different order than that shown or discussed. In addition, more or fewer boxes and / or operations may be used with any message flow diagram, scenario, and process diagram discussed herein. Figure 1 The message flow diagrams, scenarios and flowcharts may be used together, and these message flow diagrams, scenarios and flowcharts may be combined with each other in part or in whole.

[0124] The steps or boxes representing the processing of information may correspond to circuits that may be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the boxes representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to implement specific logical operations or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a random access memory (RAM), a disk drive, a solid-state drive, or other storage medium.

[0125] Computer readable media can also include non-transient computer readable media, such as computer readable media that stores data for a short period of time, such as register memory, processor cache and RAM. Computer readable media can also include non-transient computer readable media that stores program code / data for a longer period of time. Therefore, for example, computer readable media can include auxiliary or persistent long-term storage devices, such as read-only memory (ROM), optical disk or disk, solid-state drive, compact disk read-only memory (CD-ROM). Computer readable media can also be any other volatile or non-volatile storage system. Computer readable media can be considered to be computer readable storage media or tangible storage devices, for example.

[0126] In addition, the steps or boxes representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0127] The specific arrangements shown in the accompanying drawings should not be considered restrictive. It should be understood that other embodiments may include more or less of each element shown in a given figure. In addition, some of the elements shown may be combined or omitted. In addition, the example embodiments may include elements not shown in the accompanying drawings.

[0128] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope being indicated by the following claims.

Claims

1. A computer-implemented method comprising: Obtaining dual-pixel image data including a first sub-image and a second sub-image; determining (i) a focused image, (ii) a first blur kernel corresponding to the first sub-image, and (iii) a second blur kernel corresponding to the second sub-image; determining a loss value using a loss function including one or more of an equivalent loss term configured to determine a difference between (i) a convolution of the first sub-image with the second blur kernel and (ii) a convolution of the second sub-image with the first blur kernel, or a data loss term configured to determine a sum of a difference between (i) a convolution of the first sub-image and the in-focus image with the first blur kernel and (ii) a convolution of the second sub-image and the second blur kernel; Based on the loss value and the loss function, one or more of (i) the focused image, (ii) the first blur kernel, or (iii) the second blur kernel are updated; and image data is generated based on one or more of (i) the updated focused image, (ii) the updated first blur kernel, or (iii) the updated second blur kernel.

2. The computer-implemented method of claim 1 , wherein: Determining the loss value using the loss function includes: receiving a first plurality of scaled blur kernels corresponding to a plurality of size scales, wherein the first plurality of scaled blur kernels are generated based on the first blur kernel; determining a second plurality of scaled blur kernels corresponding to the plurality of size scales, wherein the second plurality of blur kernels are generated based on the second blur kernel; generating a cost volume by determining, for each of the plurality of size scales, a corresponding result of one or more of the equivalent loss term or the data loss term; and The penalty value is determined based on the cost volume.

3. The computer-implemented method of claim 2, wherein: Generating the cost volume by determining a corresponding result of the equivalent loss term for each of the plurality of size scales includes: An equivalent loss cost volume is generated by determining a corresponding difference image for each of the multiple size scales based on (i) a convolution of the first sub-image with a corresponding blur kernel among the second plurality of scaled blur kernels and (ii) a convolution of the second sub-image with a corresponding blur kernel among the first plurality of scaled blur kernels, wherein the loss value includes an equivalent loss value determined based on the equivalent loss cost volume.

4. The computer-implemented method of claim 3, wherein: The corresponding difference image has a height of H pixels and a width of W pixels, wherein the plurality of size scales includes D size scales, wherein a cross-section of the equivalent loss cost volume includes H pixels by W pixels, wherein a depth of the equivalent loss cost volume includes D pixels, and wherein determining the equivalent loss value based on the equivalent loss cost volume includes: For each corresponding pixel in the H pixels by W pixels of the cross-section of the equivalent loss cost volume, selecting a minimum pixel value from D pixel values ​​that spatially correspond to the corresponding pixel along the depth of the equivalent loss cost volume; and The equivalent loss value is determined based on a sum of the minimum pixel values ​​selected for each corresponding pixel of the H pixels by W pixels of a cross-section of the equivalent loss cost volume.

5. The computer-implemented method according to any one of claims 2 to 4, wherein: Generating the cost volume by determining a corresponding result of the data loss term for each of the plurality of size scales includes: A data loss cost volume is generated by determining a corresponding sum image for each of the multiple size scales based on the sum of (i) a difference between the convolution of the first sub-image and the focused image with a corresponding blur kernel among the first plurality of scaled blur kernels and (ii) a difference between the convolution of the second sub-image and the focused image with a corresponding blur kernel among the second plurality of scaled blur kernels, wherein the loss value includes a data loss value determined based on the data loss cost volume.

6. The computer-implemented method of claim 5, wherein: The corresponding sum image has a height of H pixels and a width of W pixels, wherein the plurality of size scales includes D size scales, wherein a cross-section of the data loss cost volume includes H pixels by W pixels, wherein a depth of the data loss cost volume includes D pixels, and wherein determining the data loss value based on the data loss cost volume includes: For each corresponding pixel of the H pixels by W pixels of the cross-section of the data loss cost volume, selecting a minimum pixel value from among the D pixel values ​​that spatially correspond to the corresponding pixel along the depth of the data loss cost volume; and The data loss value is determined based on a sum of minimum pixel values ​​selected for each corresponding pixel of the H pixels by W pixels of a cross-section of the data loss cost volume.

7. A computer-implemented method according to any one of claims 2-4, 6, wherein: Each of the plurality of size scales is associated with a corresponding depth, and wherein the computer-implemented method further comprises: selecting, for each corresponding pixel of the focused image based on the cost volume, a corresponding size scale from the plurality of size scales; and A depth map associated with the dual pixel image data is generated based on a corresponding size scale selected for each respective pixel of the in-focus image.

8. The computer-implemented method of claim 7, wherein: The loss function includes a depth loss term configured to compare (i) one or more features of the depth map with (ii) one or more corresponding features of the in-focus image.

9. The computer-implemented method of any one of claims 7, wherein: Generating the image data based on one or more of (i) an updated in-focus image, (ii) an updated first blur kernel, or (iii) an updated second blur kernel includes: generating a refocused image by modifying a depth of focus associated with at least a portion of the dual pixel image data based on the depth map; and The refocused image is displayed.

10. The computer-implemented method of any one of claims 1-4, 6, 8, 9, wherein: Each of the in-focus image, the first blur kernel, and the second blur kernel is parameterized by a corresponding mathematical model and generated using a corresponding mathematical model, wherein updating one or more of (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel comprises: determining a gradient of a loss function corresponding to the loss value and with respect to one or more parameters of one or more of the corresponding mathematical models; updating one or more parameters of one or more of the corresponding mathematical models based on the gradient of the loss function; and One or more of (i) an updated in-focus image using the updated corresponding mathematical model, (ii) an updated first blur kernel using the updated corresponding mathematical model, or (iii) an updated second blur kernel using the updated corresponding mathematical model is generated.

11. The computer-implemented method of claim 10, wherein: The mathematical model corresponding to the in-focus image is specific to the dual-pixel image data such that a value of at least one parameter of the mathematical model corresponding to the in-focus image is different from a value of at least one corresponding parameter of the mathematical model corresponding to different dual-pixel image data.

12. The computer-implemented method of claim 10, wherein: A corresponding mathematical model configured to parameterize and generate the in-focus image includes a first artificial neural network; Determining the focused image includes generating a corresponding pixel value for each corresponding pixel coordinate of the focused image through the first artificial neural network based on corresponding pixel coordinates of the focused image; and Generating an updated focused image includes generating an updated corresponding pixel value based on corresponding pixel coordinates of the updated focused image through an updated first artificial neural network and for each corresponding pixel coordinate of the updated focused image.

13. A computer-implemented method according to any one of claims 11-12, wherein: The corresponding mathematical model configured to parameterize and generate a particular blur kernel of the first blur kernel and the second blur kernel includes a second artificial neural network; Determining the specific blur kernel includes generating a corresponding pixel value for each corresponding pixel coordinate of the specific blur kernel through the second artificial neural network based on the corresponding pixel coordinate of the specific blur kernel; and Updating the specific blur kernel includes generating an updated corresponding pixel value for each corresponding pixel coordinate of the updated specific blur kernel through an updated second artificial neural network based on the corresponding pixel coordinate of the updated specific blur kernel.

14. A computer-implemented method according to any one of claims 1-4, 6, 8, 9, 11-12, wherein: Determining the first blur kernel and the second blur kernel includes: identifying a particular camera device that has been used to generate the dual pixel image data, A first blur kernel calibrated for the specific camera device and a second blur kernel calibrated for the specific camera device are obtained based on the specific camera device.

15. The computer-implemented method of any one of claims 1-4, 6, 8, 9, 11-12, wherein: The loss function includes a total variation denoising loss term configured to determine a total variation of one or more of (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel.

16. A computer-implemented method according to any one of claims 1-4, 6, 8, 9, 11-12, wherein: The loss function includes a deep convolutional prior loss term configured to determine a result of processing one or more of (i) the in-focus image, (ii) the first blur kernel, or (iii) the second blur kernel by one or more artificial neural networks.

17. The computer-implemented method of any one of claims 1-4, 6, 8, 9, 11-12, further comprising: updating the loss value using the loss function and based on one or more of (i) an updated in-focus image, (ii) an updated first blur kernel, or (iii) an updated second blur kernel; Determining that the updated loss value is below a threshold loss value; and Based on determining that the updated loss value is below the threshold loss value, the image data is generated based on one or more of: (i) an updated in-focus image, (ii) an updated first blur kernel, or (iii) an updated second blur kernel.

18. The computer-implemented method of any one of claims 1-4, 6, 8, 9, 11-12, further comprising: obtaining second dual-pixel image data including a third sub-image and a fourth sub-image; determining (i) a second focused image, (ii) a third blur kernel corresponding to the third sub-image, and (iii) a fourth blur kernel corresponding to the fourth sub-image; determining a second loss value using a loss function including one or more of an equivalent loss term or a data loss term, wherein the equivalent loss term is configured to determine a difference between (i) a convolution of the third sub-image with the fourth blur kernel and (ii) a convolution of the fourth sub-image with the third blur kernel, and wherein the data loss term is configured to determine a sum of a difference between (i) a convolution of the third sub-image and the second in-focus image with the third blur kernel and (ii) a convolution of the fourth sub-image and the second in-focus image with the fourth blur kernel; updating one or more of (i) the second focused image, (ii) the third blur kernel, or (iii) the fourth blur kernel based on the second loss value and the loss function; and Second image data is generated based on one or more of (i) the updated second in-focus image, (ii) the updated third blur kernel, or (iii) the updated fourth blur kernel.

19. A system for defocus blur removal, comprising: processor; and A non-transitory computer readable medium having stored thereon instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-18.

20. A non-transitory computer readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1-18.

Images