Image capture using dynamic lens position
Through depth map and PSF enhancement technology, combined with machine learning models, the blur problem of the camera when capturing subjects of different depths of field is solved, achieving clearer image effects and better depth of field control.
Patent Information
- Application Number
- CN202380028227.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-02-01
- Publication Date
- 2024-11-05
AI Technical Summary
When existing cameras capture subjects with different depths of field, it is difficult to effectively deal with the blur problem caused by optical deformation, especially the focus adjustment and point diffusion function (PSF) estimation between different regions is difficult to accurately perform, affecting the sharpness of the image.
By determining the focus of the camera lens based on the previously captured depth map and enhancing specific areas in the image using point diffusion function (PSF), combining machine learning models and computer vision techniques, iteratively adjusting the PSF to minimize the amount of blur, achieving different depths of field Optical correction of the area.
Improves the sharpness and depth of field effect of the image, enhances the processing capability of different depth of field areas, and ensures the sharpness and detail fidelity of the image.
Smart Images

Figure CN118901246A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to image processing. For example, aspects of the present disclosure relate to systems and techniques for capturing images (eg, with subjects at different depths of field) using dynamic lens positions. Background Art
[0002] A camera is a device that captures images (such as still images or video frames) by receiving light through a lens and bending and focusing the light onto an image sensor or light-sensitive material (such as photographic film) using the lens (and sometimes one or more mirrors). The resulting image is captured by the image sensor and stored on photographic film, which can be developed into a printed photograph, or stored digitally on a secure digital (SD) card or other storage device. In order to capture a sharp image, as opposed to a blurry image, the camera must be properly focused. Focusing a camera involves moving the lens forward and backward to ensure that light from an object that is the intended subject of the captured image is properly focused onto the image sensor or photographic film. In some cameras, the focus is usually adjusted manually by the photographer via a dial along the camera, which the photographer rotates clockwise or counterclockwise to move the lens forward or backward. Summary of the invention
[0003] In some examples, systems and techniques for capturing images (e.g., with subjects at different depths of field) using dynamic lens positions are described. According to at least one example, a method for capturing an image is provided. The method includes: determining a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; identifying a focus of a camera lens at least in part using the first distance and the second distance; capturing an image using the focus as a basis for capturing, the image including a first area corresponding to the first object and a second area corresponding to the second object; and generating a second image from the image at least in part by enhancing at least one of the first area or the second area using a point spread function (PSF).
[0004] In another example, a device for capturing an image is provided, the device comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: determine a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; use the first distance and the second distance at least in part to identify a focus of a camera lens; use the focus as a basis for capturing to capture an image, the image comprising a first area corresponding to the first object and a second area corresponding to the second object; and generate a second image from the image at least in part by enhancing at least one of the first area or the second area using a PSF.
[0005] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided that, when executed by one or more processors, causes the one or more processors to: determine a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; identify a focus of a camera lens using, at least in part, the first distance and the second distance; capture an image using the focus as a basis for capture, the image including a first region corresponding to the first object and a second region corresponding to the second object; and generate a second image from the image by, at least in part, enhancing at least one of the first region or the second region using a PSF.
[0006] In another example, a device for capturing an image is provided. The device includes: a means for determining a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; a means for identifying a focus of a camera lens using at least in part the first distance and the second distance; a means for capturing an image using the focus as a basis for capturing, the image including a first area corresponding to the first object and a second area corresponding to the second object; and a means for generating a second image from the image by enhancing at least one of the first area or the second area using a PSF, at least in part.
[0007] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further comprises selecting the PSF based on at least one of a distance between the first object and the focus and a distance between the second object and the focus.
[0008] In some aspects, the PSF is selected from a lookup table.
[0009] In some aspects, the lookup table is determined using a machine learning (ML) model that is trained using a defocused image and a loss function to correct the defocused image.
[0010] In some aspects, the lookup table is determined using a computer vision-based PSF estimate determined from a defocused image and an error calculation and iteratively modifying the computer vision-based PSF estimate until a minimum error is identified for each focal length and each blur amount.
[0011] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include: generating a modified first region at least in part by applying a deconvolution operation to the first region based on the PSF; and generating a modified second region at least in part by applying a deconvolution operation to the second region based on the PSF.
[0012] In some aspects, the first object is a face of a first person and the second object is a face of a second person.
[0013] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include: determining a first depth of field associated with the first object and a second depth of field associated with the second object; and identifying the focus as a point between the first depth of field and the second depth of field.
[0014] In some aspects, the first depth of field is determined based on a depth associated with the first object using a lookup table, and wherein the second depth of field is determined based on a depth associated with the second object using the lookup table.
[0015] In some examples, systems and techniques for capturing images are described. Systems, devices, methods, and computer-readable media for capturing images are disclosed. According to at least one example, a method for capturing an image is provided. The method includes: identifying a focus of an object; using the focus as a basis for capturing to capture a first image, the first image including a first region degraded due to optical distortion; estimating a PSF based on the focus and the optical distortion; and generating a second image from the first image at least in part by enhancing the first region of the first image using the PSF. In some aspects, the optical distortion may occur in different regions, and the method may include using the PSF to enhance a second region of the first image.
[0016] In another example, a device for capturing an image is provided, the device comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: identify a focus of an object; capture a first image using the focus as a basis for capturing, the first image comprising a first region degraded due to optical distortion; estimate a PSF based on the focus and the optical distortion; and generate a second image from the first image at least in part by enhancing the first region of the first image using the PSF. In some aspects, the optical distortion may occur in different regions, and the method may include using the PSF to enhance the second region of the first image.
[0017] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which when executed by one or more processors causes the one or more processors to: identify a focus of an object; capture a first image using the focus as a basis for capturing, the first image including a first region degraded due to optical distortion; estimate a PSF based on the focus and the optical distortion; and generate a second image from the first image at least in part by enhancing the first region of the first image using the PSF. In some aspects, the optical distortion may occur in different regions, and the method may include enhancing the second region of the first image using the PSF.
[0018] In another example, an apparatus for capturing an image is provided. The apparatus includes: means for identifying a focus of an object; means for capturing a first image using the focus as a basis for capturing, the first image including a first region degraded due to optical distortion; means for estimating a PSF based on the focus and the optical distortion; and means for generating a second image from the first image at least in part by enhancing the first region of the first image using the PSF. In some aspects, the optical distortion may occur in different regions, and the method may include using the PSF to enhance the second region of the first image.
[0019] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further comprises: determining a type of the optical deformation based on the PSF, wherein the type of deformation comprises at least one of an aberration associated with an optical setting, a motion of the object, or a tilt. In some aspects, the first region and the second region are based on the tilted PSF.
[0020] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further comprises enhancing the first region based on the determined motion of the object corresponding to the PSF.
[0021] In some aspects, one or more of the methods, devices, and computer-readable media described above also include: enhancing the first region of the first image based on the PSF tilt and tilt and the center point of the tilt; and enhancing the second region of the first image based on the tilt and the center point.
[0022] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include enhancing the first region of the first image based on an optical setting used to capture the first image, wherein the optical setting includes one of a type of lens or an aperture size of the lens.
[0023] In some aspects, the apparatus is, is part of, and / or includes a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted device (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).
[0024] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.
[0025] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Illustrative aspects of the present application are described in detail below with reference to the following drawings:
[0027] Figure 1 is a block diagram illustrating the architecture of an image capture and processing system according to some aspects;
[0028] Figure 2 is a conceptual diagram illustrating a camera capturing an image of a scene according to some aspects;
[0029] FIG. 3A to FIG. 3D illustrates a scene captured by a camera with a variable aperture lens using different focal ratios by controlling the size of the aperture according to some aspects;
[0030] Figure 4 is a graph illustrating depth of field (DOF) based on different focal length ratios from a focal length of an optical system according to some aspects;
[0031] Figure 5 illustrates a conceptual block diagram of an enhanced image capture system with extended DOF according to some aspects;
[0032] Figure 6 illustrates a conceptual diagram of capturing an image using an enhanced image capture system based on extended DOF according to some aspects;
[0033] 7A to 7D are images illustrating different point spread functions (PSFs) according to some aspects;
[0034] Fig. 8A illustrating an example of an image having an out-of-focus subject due to DOF associated with a foreground subject in accordance with some aspects;
[0035] FIG. 8B to FIG. 8E illustrates a region of an image with a defocused subject and correction based on extended DOF according to some aspects;
[0036] Fig. 9 is a conceptual diagram illustrating a process in some aspects for identifying a minimum loss function and building a lookup table for extending DOF for an optical imaging system;
[0037] Fig.10 is another conceptual diagram of a training system for training an ML model to identify a loss function for extending DOF for an optical imaging system according to some aspects;
[0038] Fig.11 illustrates another conceptual illustration of a system for building a lookup table to identify PSFs associated with a plurality of defocused images and an ML model in accordance with some aspects;
[0039] Fig.12 is a flow chart illustrating an example of a method for correcting an image based on subjects located in different DOF regions;
[0040] Fig.13 is a flow chart illustrating an example of a method for correcting an image for different types of distortion according to some aspects;
[0041] Fig.14 A block diagram of an example image processing device configured to capture images with subjects at different DOFs according to some aspects is shown; and
[0042] Fig.15 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION
[0044] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.
[0045] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.
[0046] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of exemplary aspects will provide an enabling description for implementing aspects of the present disclosure to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.
[0047] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or superior to other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.
[0048] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to an image sensor used to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, level, curve or color. For example, settings or parameters can be applied to a processor (e.g., an image signal processor (ISP)) used to process one or more image frames captured by an image sensor.
[0049] The camera may include a variable aperture lens that controls the aperture size and the amount of light emitted into the image sensor. The aperture controls the focal length ratio (sometimes referred to as the f-stop) corresponding to the depth of field (DOF) associated with the focus. Objects within the DOF will be in focus and appear sharp, while objects not within the DOF will appear blurry. The DOF is also associated with the distance of the focal point from the lens, where closer objects will be in focus, and close objects may appear blurry based on the DOF.
[0050] In some aspects, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for generating or capturing images using dynamic lens positions are described. In some aspects, the systems and techniques can capture images with subjects at different depths of field by, at least in part, selecting a focus point between two subjects in an image, determining an optimized lens position based on the focus point, extracting defocused regions from an image captured using the optimized lens position, identifying a point spread function (PSF) corresponding to each extracted region, and processing the region using a region processing operation (e.g., using a deconvolution operation, such as using a deconvolution filter or a frequency domain iterative algorithm with regularization) to enhance the extracted region. The systems and techniques can then generate a final image by compositing the captured image with the enhanced region.
[0051] In some aspects, processing of the PSF and the region (e.g., using a deconvolution operation) extends the DOF based on an optical defocus factor. In some cases, the optical defocus factor can be determined during calibration. The DOF of different subjects can partially overlap, and the focus of the image can be selected based on the overlapping area. Based on the distance between a particular subject and the focal plane associated with the same focus, the PSF can be identified, and regional processing (e.g., using a deconvolution operation) can be applied to each region based on the PSF. In some aspects, the distance between different subjects can be different and different regional processing operations (e.g., deconvolution operations) can be applied to each defocused region. In some cases, a subject can be determined to be within the DOF, and different subjects can be within the extended DOF based on the PSF. In these aspects, the subject can be positioned within the DOF, and the region corresponding to the second subject can be corrected based on the PSF.
[0052] In some cases, a machine learning (ML) model (e.g., a deep neural network) can be configured to perform at least some of the aspects described herein, such as determining a PSF from an image based on focal length. In other aspects, the ML model (e.g., a deep neural network) can be configured to modify the extracted region of the image and implement various processes, such as PSF estimation and deconvolution. An example of a deep neural network is a convolutional neural network (CNN) having multiple hidden layers (e.g., with tunable or tuned weights and / or other parameters) between an input layer and an output layer.
[0053] Figure 1 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process an image of a scene (e.g., an image of a scene 110). The image capture and processing system 100 can capture an independent image (or photograph) and / or can capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the image capture and processing system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.
[0054] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.
[0055] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to the image sensor 130 or farther away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, additional lenses may be included in the image capture and processing system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.
[0056] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.
[0057] The zoom control mechanism 125C of the control mechanism 120 can obtain the zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of the lens element assembly (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly can include a focusing lens (in some cases, the focusing lens can be lens 115), which first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. An afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.
[0058] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by the image sensor 130. In some cases, different photodiodes can be covered by different color filters in a color filter array, and thus can measure light that matches the color of the color filter covering the photodiode. Various color filter arrays can be used, including a Bayer color filter array, a four-color color filter array (also referred to as a four-color Bayer filter), and / or other color filter arrays. The four-color filter array includes a 2×2 (or "four") pattern of color filters, including a 2×2 pattern of red (R) filters, a pair of 2×2 patterns of green (G) filters, and a 2×2 pattern of blue (B) filters. The Bayer color filter array includes a repeating pattern of red filters, blue filters, and green filters. Using a four-color filter array or a Bayer filter array, each pixel of the image is generated based on red light data from at least one photodiode covered in the red filter of the filter array, blue light data from at least one photodiode covered in the blue filter of the filter array, and green light data from at least one photodiode covered in the green filter of the filter array. Other types of filter arrays may use yellow, magenta and / or cyan (also known as "emerald") filters to replace or supplement red, blue and / or green filters. Some image sensors may lack color filters entirely, and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack color filters, and therefore lack color depth.
[0059] In some cases, the image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for PDAF. The image sensor 130 may also include an analog gain amplifier to amplify the analog signal output by the photodiode and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or the analog signal amplified by the analog gain amplifier) into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms 120 may be included in the image sensor 130 alternatively or additionally. The image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.
[0060] The image processor 150 may include one or more processors, such as one or more ISPs (including ISP 154), one or more host processors (including host processor 152), and / or one or more processors in any other type of processor 1510 discussed for computing system 1500. Host processor 152 may be a digital signal processor (DSP) and / or other type of processor. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 / 1520, read-only memory (ROM) 145 / 1525, cache 1512, memory unit 1515, another storage device 1530, or some combination thereof.
[0061] In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip (SoC)) that includes host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM , global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.
[0062] The host processor 152 of the image processor 150 may configure the image sensor 130 with parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In one illustrative example, the host processor 152 may update the exposure settings used by the image sensor 130 based on the internal processing results of the exposure control algorithm from past image frames. The host processor 152 may also dynamically configure the parameter settings of the internal pipeline or modules of the ISP 154 to match the settings of one or more input image frames from the image sensor 130 so that the image data is properly processed by the ISP 154. The processing (or pipeline) blocks or modules of the ISP 154 may include modules for lens / sensor noise correction, demosaicing, color conversion, correction or enhancement / suppression of image attributes, denoising filters, sharpening filters, and the like. For example, the processing blocks or modules of ISP 154 may perform a number of tasks such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form HDR images, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The settings of the different modules of ISP 154 may be configured by host processor 152.
[0063] The image processing device 105B may include various input / output (I / O) devices 160 connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1535, any other input device 1545, or some combination thereof. In some cases, subtitles may be entered into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the image capture and processing system 100 and one or more peripheral devices, and the image capture and processing system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices through the wired connections. The I / O 160 may include one or more wireless transceivers that enable wireless connections between the image capture and processing system 100 and one or more peripheral devices, and the image capture and processing system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices through the wireless connections. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.
[0064] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.
[0065] like Figure 1 As shown, the vertical dotted line will Figure 1 1 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120 and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145 and I / O 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.
[0066] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some specific implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.
[0067] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1. The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.
[0068] As described above, the color filter array may cover one or more arrays of photodiodes (or other photosensitive elements) of the image sensor 130. In some cases, the color filter array may include a four-color filter array. In some cases, after the image is captured by the image sensor 130 (e.g., before the image is provided to the ISP 154 and processed by the ISP), the image sensor 130 may perform a merging process to merge the four-color filter array pattern into a merged Bayer pattern. For example, the four-color filter array pattern may be converted to a Bayer filter array pattern (with reduced resolution) by applying a merging process. The merging process may increase the signal-to-noise ratio (SNR), resulting in increased sensitivity and reduced noise in the captured image. In an illustrative example, the merging may be performed in a low light setting when lighting conditions are poor, which may produce a high quality image with higher brightness characteristics and less noise.
[0069] Figure 2 2 is a conceptual diagram illustrating a camera that captures an image of a scene according to some examples. In image capture system 200, the image capture system can identify various objects in the scene. For example, image capture system 200 can identify multiple objects, such as object 205 and another object 210. Although a person is illustrated as an example object, any type of animate or inanimate object can be identified, such as an animal, vegetation, landmark, vehicle, etc.
[0070] In many cases, object 205 and object 210 may be spaced apart by a distance. For example, object 205 is located at a distance D1 away from image capture system 200, and object 210 is located at a distance D2 away from the image capture system. The image capture system may also identify background plane 215 at a distance D3. When the image capture system is instructed to capture an image, the image capture system first determines a focus 220 and adjusts the lens of the image capture system 200 based on the focus and various optical properties.
[0071] An exemplary optical property is the focal ratio of the lens, which is often referred to as the f-number, f-ratio, or f-stop. The focal ratio is controlled by a variable aperture lens of the image capture system 200. The variable aperture lens may include an iris with movable blades that control the aperture of the lens. The aperture allows light to enter the lens, and exposure can be reduced by reducing the size of the aperture (e.g., radius), which in turn can affect the focal ratio. The focal ratio is often represented by "F / N," where N is equal to the focal length of the lens divided by the aperture size. The aperture size may also be referred to as the pupil size. For example, a focal length of 40 mm and an aperture size of 10 mm yield a focal ratio of F / 4.0. The focal ratio is a dimensionless number and is a quantitative measure of the speed of the lens.
[0072] The focal ratio is related to the depth of field of the captured image. Depth of field is the range from which an object is in focus, based on the distance to the focal point. A larger focal ratio has a larger depth of field and will have a larger range for objects to be in focus, and a smaller focal ratio has a smaller depth of field and will have a smaller range for objects to be in focus. FIG. 3A to FIG. 3D Examples of scenes with different focal ratios and different depths of field are illustrated in Figure 4 , a graph illustrating depth of field for different focal length ratios based on the distance to the focus point is illustrated in .
[0073] When the image capture system 200 is capturing an image, the image capture system 200 can determine the focal ratio and then expose the image sensor based on the focal ratio to generate the image. In some cases, the image capture system 200 can control the focal ratio and display a snapshot or preview image using a viewfinder. The user can sense the viewfinder to operate the image capture system 200 to focus on the correct subject of the image.
[0074] After determining the focal ratio and the focal point 220, the image capture system 200 may capture an image based on a depth of field 225 centered at the focal point and may include objects in focus within the depth of field 225. For example, Figure 4 As shown, the depth of field at a focal length of 1 meter varies from 1020 mm to 875 mm based on the focal length ratio. Figure 2As shown in , the captured image may correctly have the first object 205 in focus, but the distance (D2-D1) between the first user and the second user may be outside the depth of field 225 and the second object 210 will be blurred within the captured image.
[0075] FIG. 3A to FIG. 3D An image of a scene captured by a camera with a variable aperture lens using different focal ratios by controlling the size of the aperture according to some aspects. Figure 3A The image illustrates a scene captured at a focal ratio of F / 4.0. Figure 3B The image illustrates the Figure 3A The same scene shown in but captured at a focal ratio of F / 2.8. Figure 3C The image illustrates the Figure 3A The same scene shown in , captured at a focal ratio of F / 2.0. Figure 3D The image illustrates the Figure 3A The same scene shown in , captured at a focal ratio of F / 1.4.
[0076] As mentioned above, depth of field is related to focal ratio or f-stop. Figure 3A , Figure 3B , Figure 3C and Figure 3D Each includes foreground content 300 corresponding to the focal point of the lens and background content 305 which becomes more blurred as the focal ratio and depth of field also decrease. In particular, Figure 3A , Figure 3B , Figure 3C and Figure 3D The background content 305 shown in FIG. 1 is a drink with a straw, and the visual fidelity of the straw decreases as the focal length ratio decreases. However, the foreground content 300 Figure 3A , Figure 3B , Figure 3C and Figure 3D Each captured image shown in is similar and does not vary perceptibly to the naked human eye.
[0077] Figure 4 is a graph illustrating depth of field (DOF) based on different focal length ratios from the focal length of an optical system according to some aspects. In particular, Figure 4 The near limit and far limit for each focal length ratio corresponding to the depth of field are illustrated. Figure 4 As shown, for different focal length ratios, the depth of field at a focal length of 1 meter changes from 1020mm to 875mm. At a focal length of 4 meters, the depth of field changes from 8000mm to 1175mm.
[0078] Figure 5A conceptual block diagram of an enhanced image capture system 500 that can capture images using dynamic lens positions according to some aspects is illustrated. The image capture system 500 can effectively extend the DOF of the image capture system 500 (e.g., a camera with a smaller aperture). Figure 5 As shown, the image capture system 500 includes an object identification engine 510, an optical control engine 530, an object extraction engine 540, a defocus distance engine 550, a point spread function (PSF) estimation engine 560, a deconvolution engine 570, and a synthesis engine 580. In some examples, the image capture system 500 can be configured to enhance or deblur a blurred image using computer vision techniques and / or machine learning (ML).
[0079] In some aspects, the image capture system 500 includes one or more images 535. For example, the one or more images 535 may include a previously captured image (e.g., a preview or snapshot image), which is referred to as a first image. The first image may be obtained or retrieved by the object identification engine 510 (e.g., from a camera, from a storage device, from another device, etc.). In some examples, the first image may be displayed (e.g., as a preview image) in a display or viewfinder of the image capture system 500. The object identification engine 510 may detect whether there are two objects in the first image, such as by performing object detection. Object detection may be performed using techniques based on computer vision and / or machine learning (e.g., using a convolutional neural network (CNN) configured to detect one or more objects in an image). The object identification engine 510 may detect two different objects in the scene at different locations between the camera and the background plane.
[0080] The image capture system may obtain (e.g., from another device), retrieve (e.g., from a storage device), or generate a depth map 515 for the first image. The depth map 515 may be a two-dimensional array of pixels, where the value of each corresponding pixel in the depth map 515 identifies the distance of the corresponding pixel from the image capture system 500 (e.g., a camera of the image capture system 500) to an object in the scene to which the pixel belongs. In some aspects, when the object identification engine 510 identifies objects within a single range (e.g., a DOF of a specific focal length ratio), the image capture system 500 may capture images using conventional methods.
[0081] In some cases, depth map 515 can be provided to optical control engine 530 to determine various optical characteristics for capturing subsequent images. For example, optical control engine 530 can determine focal length ratio, exposure time, etc. In some aspects, when depth map 515 includes objects at different focal distances, optical control engine 530 can determine the distance of each object from depth map 515. Optical control engine 530 can obtain a lookup table 516 that maps distances with different DOFs. Using lookup table 516 and each corresponding distance determined by depth map 515 for each object, optical control engine 530 can determine the corresponding DOF for each object. Optical control engine 530 can then determine the focus between objects based on the determined DOFs. Reference Figure 6 An example of determining a focus between objects is further illustrated. The optical control engine 530 can use the determined focus between the objects to determine an optimal lens position for capturing an image. For example, in response to determining the optical characteristics, the image capture system 500 can control various devices (e.g., lens, image sensor, etc.) to capture a second image (captured from one or more images 535) at the optimal lens position and based on the determined characteristics (e.g., focal ratio, focal length, exposure time, etc.). Reference Fig. 8A An example of the second image is illustrated.
[0082] The second image may be provided to the object extraction engine 540 to extract at least one region from the second image. Figure 8B and Fig.8D The example from Fig. 8A 5. In some aspects, the second image can be provided to the object identification engine 510 to generate a depth map corresponding to the second image, or the depth map 515 from the first image can be used to identify the region corresponding to the object in the second image. For purposes of illustration, the image capture system 500 is illustrated as receiving the depth map 515 for the object extraction engine 540 to extract at least one object region 545 from the second image. In some aspects, the image capture system 500 generates regions for each subject in different DOF regions, such as Figure 6 shown.
[0083] The object regions 545 are provided to the defocus distance engine 550 to determine a corresponding defocus distance between each region (associated with each object) and the focus determined by the optical control engine 530. As described above, the focus is associated with the optimized lens position. For example, the defocus distance engine 550 may determine a first defocus distance between a first region (corresponding to a first object in the second image) and the focus determined by the optical control engine 530, and may determine a second defocus distance between a second region (corresponding to a second object in the second image) and the focus.
[0084] The defocus distance engine 550 can output the determined defocus distance to the PSF estimation engine 560. In some aspects, the PSF estimation engine 560 can also receive optical information related to the second image, such as focal length, focal ratio, etc. Based on the defocus distance of each object area 545 and in some cases based on the optical information of the second image, the PSF estimation engine 560 is configured to determine a PSF 565 for each object area 545. In some cases, the defocus distance engine 550 can be omitted, and the PSF estimation engine 560 can select a PSF based on a quantification of blur in the second image. The quantification of blur can be based on a sharpness value, such as a distance between intensity differences (e.g., between 10% of the minimum intensity and 90% of the maximum intensity), or can be based on a modulation transfer function (MTF).
[0085] In some aspects, the PSF specifies the response of an imaging system to a point source or point object. The PSF can be considered an optical transfer function associated with an optical imaging system, or an impulse function such as a single frequency in the frequency domain. The PSF identifies the spatial domain transformation of a single pixel. For example, a single pixel input to an optical system will be a blurred pixel based on improper focus. Here in Fig. 7A , Figure 7B and Figure 7C An example of a PSF transform function is illustrated in .
[0086] The PSF estimation engine 560 can be implemented by various mechanisms. In some aspects, the PSF estimation engine 560 can be performed using a lookup table 562. In some cases, one axis or dimension of the lookup table 562 can include defocus distances associated with regions, and another axis or dimension can be corresponding PSF values. The PSF estimation engine 560 can receive the defocus distance values from the defocus distance engine 550 and look up the corresponding PSF 565 for each region 545 from the lookup table 562 based on each corresponding defocus distance. The lookup table 562 can be generated by different mechanisms, such as an iterative process or by training a machine learning model. Fig.10 and Fig.11 An illustrative example of a mechanism for establishing lookup table 562 for PSF estimation engine 560 is illustrated in In some cases, PSF estimation engine 560 may be able to execute an algorithm to determine the PSF, such as by a distributed computer system or a single computer system that does not require low latency.
[0087] In some aspects, each object region 545 extracted by the object extraction engine 540 and each PSF 565 of each object region 545 are provided to a deconvolution engine 570. The deconvolution engine 570 uses the PSF 565 of each corresponding object region 545 and performs the inverse of the PSF 565 to generate an enhanced object region 575 from each object region 545. In an illustrative example, the deconvolution engine 570 applies a deconvolution operation to each object region 545 (e.g., using a deconvolution filter or a frequency domain iterative algorithm with regularization that iteratively updates parameters until optimal parameters are identified). Using the deconvolution operation, the deconvolution engine 570 corrects for optical properties caused by changing the focus to points between objects. Reference is made to Figure 8C and Fig. 8E An example of an enhanced object region 575 is illustrated and described.
[0088] The synthesis engine 580 receives each of the enhanced object area 575 and the second image, and synthesizes and outputs an enhanced image 590 that corrects issues such as blur, noise, etc. based on focus. In some cases, each enhanced image 590 may contain metadata identifying the exact pixel locations of the area within the second image corresponding to the object area 545, and different methods may be performed to synthesize the resulting enhanced image 590. For example, the synthesis engine 580 may combine pixels of the second image with pixels of the enhanced object area based on a saturation value to remove pixels that would reduce the sharpness value. Other types of synthesis may be implemented, such as linear interpolation, averaging, etc.
[0089] Figure 6 A conceptual diagram illustrating capturing an image using an enhanced image capture system 600 based on extended DOF according to some examples. Figure 6 In the illustrative example of , the scene includes a first object 605 located at a distance D1 from the image capture system 600 and a second object 610 located at a distance D2 from the image capture system 600. A background plane 615 is located at a distance D2 from the image capture system.
[0090] The image capture system 600 may be configured to identify a first object 605 and a second object 610 that are physically located in different DOF regions. For example, the first object 605 is located in a DOF 620 associated with a particular focal length ratio, and the second object 610 is located in a DOF 625. In this illustrative example, DOF 620 and DOF 625 are different ranges and do not overlap. When the image capture system 600 identifies the scene, the image capture system 600 may be configured to determine an extended DOF 630 that extends the distance of DOF 620 and an extended DOF 635 that extends the range of DOF 625. For example, based on the improvement provided by the deconvolution, the extended DOF 630 has a near limit 640 and a far limit 642. The extended DOF 635 also has a near limit 644 and a far limit 646, the near limit being before the far limit of the extended DOF 630. The extended DOF 630 and the extended DOF 635 may have different applicable ranges because, as shown in FIG. Figure 4 As shown, DOF is a function of focal ratio and focal length.
[0091] In some aspects, the image capture system 600 can correct blur associated with a defocused image outside of the DOF region by identifying the PSF of the defocused image and the focal length of the image. As described above, the PSF is a transfer function, and a deconvolution operation performed based on the PSF can improve the quality of the defocused region of the image. Figure 6 In the illustrative example of , the image capture system 600 can identify an overlap region 650, which is a location between the far limit 642 of the extended DOF 635 and the near limit of the extended DOF 630. Based on the overlap region, the image capture system 600 can select a focus point 660 at a location D4. In this illustrative example, both the first object 605 and the second object 610 are within the depth of field that can be corrected.
[0092] After selecting a focus 660 where both the first object 605 and the second object 610 are not within the optical DOF, the first object 605 and the second object 610 are both out of focus in the image provided from the image sensor. The image capture system 600 can determine a defocus distance between the first object 605 and the focus 660 and a defocus distance between the second object 610 and the focus 660. Using the defocus distances, the image capture system 600 can determine the PSF of each of the objects 605 and 610, as described above in Figure 5 , and applying deconvolution to enhance the image. Different variations are possible, for example, keeping one object in focus while the DOF of the object can be extended to include a second object 610. In other aspects, PSF estimation and deconvolution can be used to correct other types of distortion, such as aberrations associated with large lenses with low focal ratios, tilt associated with the lens, or motion within the image.
[0093] 7A to 7D are images illustrating different PSFs according to some examples. Fig. 7A A bitmap is illustrated that is input into an optical system to measure the PSF of the optical system. Figure 7B Illustrate when the optical system is not responding and not used Fig. 7A The output of this optical system is shown with the input bitmap distorted. Figure 7C Illustrate when the optical system is slightly defocused and slightly blurred Fig. 7A The input bitmap shown in is the output of this optical system. Fig.7D Illustrate when the optical system is severely defocused and significantly blurred Fig. 7A The input bitmap shown in is the output of this optical system.
[0094] like Figure 7C and Fig.7D As shown in , the PSF is a loss function that visually depicts how pixels spread out through a transfer function. For a known PSF, the loss can be inverted, for example, by building a matrix. In some aspects, the PSF can be determined based on an iterative algorithm or by training an ML model using the ML model with defocused and focused images.
[0095] Fig. 8A An example of an image with a defocused subject due to DOF associated with a foreground subject according to some examples is illustrated. In some aspects, Fig. 8A The image shown in FIG. 8 may correspond to a second image captured based on the focus between the first object 810 and the second object 815 (from a source relative to FIG. 814 ). Figure 5 Image 535 described).
[0096] Figure 8B 8 is an image illustrating a region extracted from an image corresponding to the first object 810. Although difficult to perceive, the shadows and highlights of the first object 810 may be slightly blurred and have lower sharpness. Figure 8C This example shows the PSF-based Figure 8B The deconvolution result of the image. Figure 8B Compared with the images in Figure 8C The image in has slightly increased sharpness of various features of the first object. In particular, Fig. 8A Compared to the image in , shadows and highlights are improved, and the edges of the glasses are more pronounced.
[0097] Fig.8D 815 is an image illustrating another area extracted from the image corresponding to the second object 815. In particular, the image illustrates that the second object 815 is more defocused based on the focus selected by the image capture system, and features of the face (such as the ears) are more blurred. Fig. 8E This example shows the PSF-based Fig.8D The deconvolution result of the image. Fig.8D Compared with the images in Fig. 8E The image in has significantly increased sharpness of various features of the second object 815. In particular, Fig.8D Compared to the image in , the facial features are significantly clearer and the face is sharper.
[0098] Fig. 9 is a conceptual diagram of an example system 900 that is configured in some aspects to identify a minimum loss function and to build a lookup table (e.g., Figure 5 A lookup table 562 for identifying an appropriate PSF for the optical imaging system to extend the DOF. In particular, the system 900 uses a computer vision (CV) based PSF estimation algorithm to perform an iterative process until a minimum loss function is identified. The iterative process performed by the system 900 is performed based on the focal length and blur applied to the training image.
[0099] In particular, the system 900 includes a validation image 905 and a training image 910. The training image 910 is provided to a CV-based estimator 915 to generate a candidate PSF 920 associated with the training image 910. In some aspects, the CV-based estimator 915 uses an initial set of parameters to generate the candidate PSF 920. The candidate PSF 920 is provided to an error calculator 925 to compare the candidate PSF 920 with the validation image. In some aspects, the error calculator 925 may perform a deconvolution of the training image 910 based on the candidate PSF 920 and perform an image comparison with the validation image 905 to determine accuracy.
[0100] The error calculator 925 quantifies the error using various metrics and provides the error to a parameter estimator 930, which uses various types of algorithms (such as gradient descent) to determine the parameters of the CV-based estimator 915. After calculating the revised set of parameters, the revised set of parameters is provided to the CV-based estimator 915 to repeat the determination of the candidate PSF 920 and the error calculation. This process continues until the error calculator 925 determines that the candidate PSF 920 corresponds to the maximum error correction of the training image 910 and outputs the PSF 940.
[0101] In some aspects, the validation image is associated with a focal distance, and the training image is defocused or distorted based on a transfer function, and the PSF 940 corresponds to a single data point of the multi-axis table for correcting the image. This process will occur for each data point of the lookup table. For example, this process will occur for each focus of the imaging system with training images 910 at various defocus distances.
[0102] Fig.10is another conceptual diagram of a training system for training an ML model in some aspects to identify a loss function for an optical imaging system to extend DOF. In some aspects, various object regions (e.g., object region 545) can be applied to the ML model, and the ML model can identify a PSF and perform deconvolution based on the PSF. Parameters such as depth of focus, distance from the depth of focus to the corresponding object region, and various camera settings such as focal ratio can be provided to the ML model. Because the resulting ML model is based on several layers of linear or nonlinear transformations, the ML model can be implemented in a mobile device to perform logic and mathematics associated with the ML model with minimal latency and power consumption.
[0103] The training system 1000 includes a validation image set 1005 and a training image set 1010 for creating an ML model 1040. The training images from the training image set 1010 are provided to an ML model trainer 1015, which configures, receives the ML model, and applies the training images to construct a candidate PSF 1020. The candidate PSF 1020 is returned to a loss function calculator 1025, which determines a loss based on a comparison of the training images with corresponding validation images from the validation image set 1005.
[0104] The loss calculated by the loss function calculator 1025 is provided to the back propagation engine 1030, which propagates the error back to the previous layer of the ML model to update the ML model. The ML model is then provided to the ML model trainer 1015, and the process continues until the minimum function of the ML model trainer 1015 determines that the minimum loss is based on the parameter identification of the ML model, and outputs the ML model 1040.
[0105] The training system 1000 is a conceptual illustration, and a specific implementation may include additional steps and processes. For example, the training system 1000 illustrates a comparison with a validation image set during training, and the ML model 1040 can be applied to an additional validation process without known validation inputs. In other aspects, an initial loss function can be applied to the backpropagation engine 1030 to reduce training time.
[0106] Fig.11 Illustrate a method for constructing a lookup table (e.g., Figure 5 Another conceptual illustration of a system 1100 for identifying a PSF associated with multiple defocused images and an ML model using a lookup table 562 of FIG. 1140. The system 1100 can be configured to establish a lookup table, such as PSF table 1140, during calibration of the image capture system. In some aspects, the system 1100 can be implemented at design time, and various factors can be determined during calibration.
[0107] In some illustrative aspects, an image may be input into the system 1100, and a lens sweep 1110 is performed to generate a plurality of defocused images 1120 for each iteration of the lens sweep. In some aspects, the image 1105 is received by the system 1100, and the system 1100 performs a lens sweep 1110 to generate a defocused image 1120 based on each iteration of the sweep. The defocused images are separately provided to the ML model 1130. The ML model 1130 may include any suitable ML system capable of generating a PSF for deconvolution to enhance the defocused image. In an illustrative example, the ML model 1130 may be a residual neural network (ResNet) ML model with 34 layers (ResNet-34), a convolutional neural network (CNN), a deep neural network (DNN), any other type of deep neural network (e.g., trained using supervised, unsupervised, or semi-supervised learning / training), any combination thereof, and / or an ML model or system. In some aspects, the ML model 1130 may be configured to identify a loss function, such as a PSF or other parameters that may be converted to a PSF.
[0108] For each defocused image in the defocused images 1120, the ML model 1130 can populate one or more data sources of the image capture system. For example, the ML model 1130 can construct a PSF table 1140 having an axis or dimension corresponding to the defocus distance and another axis or dimension corresponding to the corresponding PSF. As described in detail above, the PSF table 1140 can be used to identify a PSF (e.g., PSF 565), and a deconvolution function (e.g., deconvolution engine 570) can use the PSF to enhance the distorted bitmap. For example, the PSF can be used to enhance the sharpness of a defocused area in a single image because the object is not within a DOF that is restricted based on optical properties.
[0109] In some other aspects, the ML model 1130 can be configured to enhance the input image based on input parameters. For example, the ML model 1130 can receive an extracted portion of a bitmap image, an optical characteristic (e.g., focal length, focal ratio, etc.), and a focal difference between the position of the extracted portion of the bitmap image and the focal length. For example, an object can be 10 meters behind the focus, and an object can be outside the DOF of the lens. Based on the input, the ML model 1130 can be configured to enhance the extracted portion of the bitmap image based on, for example, the deconvolution that the ML model 1130 is trained to perform.
[0110] although Fig.11The use of a ResNet model is illustrated, but the output ML model 1130 can be trained using any suitable ML model, or can include an ML model to add additional information to the enhanced area. In one exemplary aspect, the ML model 1130 can be an adversarial network, such as a generative adversarial network (GAN) using an adverse ML model. For example, the GAN includes a generative model that creates data and an adversarial network that classifies the data as fake (e.g., from the generative model) or real and trains the generative model to add content to the image. For example, the generative model can be implemented to create details to improve the quality of the image to extend the DOF.
[0111] Fig.12 is a flow chart illustrating an example of a method 1200 for correcting an image based on a subject located in a different DOF region. The method 1200 may be performed by a computing device having one or more image sensors (e.g., a camera), such as a mobile wireless communication device, a camera, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a wirelessly enabled vehicle, or other computing device. In some cases, the computing device may include an image processing device (e.g., Fig.14 In some cases, the computing device may include or may be Fig.15 A computing system 1500 is provided.
[0112] According to some aspects, at block 1205, the computing device may determine a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image. In some aspects, the first object is a face of a first person, and the second object is a face of a second person. In some other aspects, the objects may be animate objects, landmarks, people, etc.
[0113] At block 1210, the computing device may identify a focal point of the camera lens using, at least in part, the first distance and the second distance at block 1220. In one illustrative aspect, the computing device may determine a first depth of field associated with the first object and a second depth of field associated with the second object, and identify the focal point as a point between the first depth of field and the second depth of field. The first depth of field is determined based on a depth associated with the first object using a lookup table, and the second depth of field is determined based on a depth associated with the second object using the lookup table.
[0114] At block 1215 (eg, after determining the focus), the computing device may capture an image using the focus as a basis for capturing. The captured image includes a first area corresponding to the first object and a second area corresponding to the second object.
[0115] In one illustrative aspect, a computing device selects a PSF based on at least one of a distance between a first object and a focus and a distance between a second object and a focus. In some aspects, the PSF is selected from a lookup table. An illustrative lookup table includes a defocus distance (e.g., a distance between an object and a focus) and an amount of blur. The lookup table can be determined using a deep learning model or an ML model that is trained using a defocused image and a loss function to correct the defocused image. In some other aspects, the lookup table is determined using a computer vision-based PSF estimate determined from the defocused image and an error calculation and iteratively modifying the computer vision-based PSF estimate until a minimum error is identified for each focal length and each amount of blur.
[0116] At block 1220, the computing device may generate a second image from the image at least in part by enhancing at least one of the first region or the second region using the PSF. To enhance at least one of the first region or the second region based on the PSF, the computing device may generate a modified first region at least in part by applying a deconvolution operation to the first region based on the PSF. In some aspects, the computing device may generate the modified second region at least in part by applying a deconvolution operation to the second region based on the PSF.
[0117] Fig.13 is a flow chart illustrating an example of a method 1200 for correcting an image for different types of distortion. The method 1300 may be performed by a computing device having one or more image sensors (e.g., a camera), such as a mobile wireless communication device, a camera, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a wirelessly enabled vehicle, or other computing device. In some cases, the computing device may include an image processing device (e.g., Fig.14 In some cases, the computing device may include or may be Fig.15 A computing system 1500 is provided.
[0118] According to some examples, the computing device may identify a focus of the object at block 1305. At block 1310 (eg, after identifying the focus), the computing device may capture a first image using the focus as a basis for capturing.
[0119] Using the captured image, the computing device may determine the type of optical distortion, such as blur, based on the PSF that degrades the image. Examples of optical distortion include blur, motion of an object (motion blur), tilt associated with the captured image, aberrations due to camera characteristics. Other types of optical distortion and other types of blur are possible.
[0120] At block 1315 (eg, after the image is captured), the computing device may estimate a PSF based on focus and optical distortion (eg, blur). In some aspects, the PSF may be based on different regions, and the PSF may be associated with an object that was moving when the image was captured.
[0121] At box 1320, the computing device may generate a second image from the first image at least in part by enhancing a first area of the first image using the PSF. If the distortion corresponds to motion, the computing device may remove the distortion using the determined motion of the object corresponding to the PSF. If the distortion corresponds to tilt, the computing device may enhance the first area of the first image based on the tilt and the center point of the tilt, and enhance the second area of the first image based on the tilt and the center point. The tilt may be different based on angular rotation and different points, and the computing device may divide the image into different areas and enhance the areas in different ways. If the distortion corresponds to tilt, the computing device may enhance the first area of the first image based on the optical settings used to capture the first image. For example, the optical settings include one of the type of lens or the lens aperture size.
[0122] In some examples, the processes described herein (e.g., methods 1200 and 1300 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, methods 1200 and 1300 can be performed by a computer having Fig.15 The computing device (e.g., Figure 1 The image capture and processing system 100 in FIG. 1 is used to perform the image capture and processing system 100.
[0123] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a virtual reality (VR) headset, an augmented reality (AR) headset, AR glasses, a connected watch or smart watch or other wearable device), a server computer, a computing device of an autonomous vehicle or an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the methods described herein, including methods 1200 and 1300. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the methods described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.
[0124] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.
[0125] Methods 1200 and 1300 are illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the method.
[0126] Methods 1200 and 1300 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
[0127] Fig.14 A block diagram of an example image processing device 1400 configured to capture images of subjects at different DOFs according to some aspects is shown. In some aspects, the image processing device 1400 is configured to perform one or more of the methods or processes described above. These methods may include methods 1200 and 1300 and any other processes described herein.
[0128] The image processing device 1400 may be the above reference Figure 1Example aspects of the image processing device 105B are described. For example, the ISP 1400 can be a chip, system on chip (SoC), chipset, package, or device that includes a hardware processor, hardware circuit, programmable circuit, and / or software engine or module that performs various functions. The image processing device 1400 may include an automatic exposure control (AEC) engine 1402, a zoom control engine 1404, a focus control engine 1406, an object detection engine 1408, an optical measurement engine 1410, a PSF detection engine 1412, an image correction engine 1414, and an image synthesis engine 1416.
[0129] The AEC engine 1402 may be configured to control exposure settings of the lens, such as focal ratio and exposure time. The AEC engine 1402 may also control gain settings applied to the image. In some aspects, the AEC engine 1402 may control the diaphragm to control the aperture size by causing the blades of the diaphragm to move. The AEC engine 1402 may include a function to determine when the diaphragm has completed moving. The AEC engine 1402 may also receive scene-related information (such as depth of field) and control the operation of the lens to capture an image based on the depth of field, thereby ensuring that the subject and background content are within focus. In some cases, the AEC engine 1402 may determine the focal ratio (and corresponding aperture size), exposure time, and gain of different images to be merged into a single HDR image.
[0130] The zoom control engine 1404 is configured to perform any optical zoom or digital zoom functions. The zoom control engine 1404 may also receive information related to the subject of the image and provide the subject related information and any zoom information to the focus control engine 1406. The zoom control engine 1404 is configured to perform electrical or mechanical operations to focus the lens and other processing to change the focal length to correspond to the distance from the image processing device 1400 to the subject identified by the image processing device 1400.
[0131] The object detection engine 1408 may be configured to identify at least one object in the snapshot image. For example, the object detection engine 1408 may generate a depth map to determine the distance to the object in the snapshot image. In some aspects, the object detection engine 1408 may identify that multiple objects are in the snapshot image and may be within different DOFs based on the lens settings. In this case, the object detection engine 1408 may provide information to the focus control engine 1406, which may determine the focus based on various parameters. For example, the focus control engine 1406 may identify the focus using an extended DOF. Based on the identified focus, the image capture system captures a single image and may extract portions of the image for enhancement.
[0132] In some aspects, at least some of the engines 1402, 1404, 1406, 1408, 1410, 1412, 1414, and 1416 are implemented at least in part as software stored in a memory. For example, portions of one or more of the engines 1402, 1404, 1406, 1408, 1410, 1412, 1414, and 1416 may be implemented as non-transitory instructions (or "code") that can be executed by at least one processor to perform the functions or operations of the corresponding engine. In some cases, at least some of the engines 1402, 1404, 1406, 1408, 1410, 1412, 1414, and 1416 may be implemented by circuits such as application specific integrated circuits (ASICs) or programmable circuits such as function programmable gate arrays (FPGAs).
[0133] Fig.15 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.15 An example of a computing system 1500 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1505. The connection 1505 may be a physical connection using a bus, or a direct connection into the processor 1510, such as in a chipset architecture. The connection 1505 may also be a virtual connection, a networked connection, or a logical connection.
[0134] In some aspects, computing system 1500 is a distributed system in which the functionality described in the present disclosure can be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some aspects, each component can be a physical or virtual device.
[0135] The example computing system 1500 includes at least one processing unit (CPU or processor) 1510 and connections 1505 that couple various system components including memory units 1515, such as read-only memory (ROM) 1520 and random access memory (RAM) 1525, to the processor 1510. The computing system 1500 may include a cache 1512 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1510.
[0136] Processor 1510 may include any general purpose processor and hardware or software services, such as services 1532, 1534, and 1536 stored in storage device 1530, that are configured to control processor 1510 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1510 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.
[0137] To enable user interaction, the computing system 1500 includes an input device 1545 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1500 may also include an output device 1535 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1540, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, The communication interface 1540 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1500 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.
[0138] The storage device 1530 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium capable of storing data that can be accessed by a computer, such as a cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, a Europay, an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a RAM, a static RAM (SRAM), a dynamic RAM (DRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.
[0139] Storage device 1530 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1510, enable the system to perform functions. In some aspects, hardware services that perform specific functions may include software components stored in a computer-readable medium connected to necessary hardware components (such as processor 1510, connector 1505, output device 1535, etc.) to perform functions. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as CDs or DVDs), flash memory, memory, or memory devices. Computer-readable media may have stored thereon code and / or machine-executable instructions, which may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, category, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.
[0140] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to communicate and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to communicate and / or receive wired and / or wireless data, including data in accordance with 3G, 4G, 5G, and / or other cellular standards, data in accordance with the Wi-Fi (802.11x) standard, data in accordance with the Bluetooth standard, and / or other components. TM Standard data, data according to IP standards and / or other types of data.
[0141] The components of the computing device may be implemented in circuits. For example, each component may include and / or may be implemented using electronic circuits or other electronic hardware (which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits)), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.
[0142] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.
[0143] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology can be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid confusing various aspects.
[0144] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a process, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.
[0145] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. These instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or function group. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.
[0146] The devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of further example, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.
[0147] Instructions, media for communicating such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.
[0148] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be appreciated that, in alternative aspects, each method can be performed in a different order than described.
[0149] One of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.
[0150] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.
[0151] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).
[0152] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.
[0153] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed to the entire system. Technicians may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.
[0154] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including program codes, which include instructions that perform one or more of the above methods when executed. Computer-readable data storage media can form a part of a computer program product, which can include packaging materials. Computer-readable media can include memory or data storage media, such as RAM (such as synchronous dynamic random access memory (SDRAM)), ROM, non-volatile random access memory (NVRAM), EEPROM, flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.
[0155] The program code may be executed by a processor, which may include one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.
[0156] Exemplary embodiments of the present disclosure include:
[0157] Aspect 1: A method for capturing an image, the method comprising: determining a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; using the first distance and the second distance at least in part to identify a focus of a camera lens; capturing an image using the focus as a basis for capturing, the image comprising a first area corresponding to the first object and a second area corresponding to the second object; and generating a second image from the image at least in part by enhancing at least one of the first area or the second area using a PSF.
[0158] Aspect 2: The method according to aspect 1, further comprising: selecting the PSF based on at least one of a distance between the first object and the focus and a distance between the second object and the focus. In some aspects, if the focus is on the first object, the second object can be blurred. The PSF is selected to deblur the second area. In some other aspects, if the focus point is between the first object and the second object, the first object and the second object can be blurred, and two PSFs can be selected to enhance the first area and the second area.
[0159] Aspect 3: The method according to any one of aspects 1 or 2, wherein the PSF is selected from a lookup table.
[0160] Aspect 4: The method according to aspect 3, wherein the lookup table is determined using an ML model, and the ML model is trained using a defocused image and a loss function to correct the defocused image.
[0161] Aspect 5: A method according to any one of Aspects 3 or 4, wherein the lookup table is determined using a computer vision-based PSF estimate determined from a defocused image and an error calculation and iteratively modifying the computer vision-based PSF estimate until a minimum error is identified for each focal length and each blur amount.
[0162] Aspect 6: A method according to any one of Aspects 1 to 5, wherein enhancing at least one of the first region or the second region based on the PSF includes: generating a modified first region at least in part by applying a deconvolution operation to the first region based on the PSF; and generating a modified second region at least in part by applying a deconvolution operation to the second region based on the PSF.
[0163] Aspect 7: The method according to any one of aspects 1 to 6, wherein the first object is a face of a first person, and the second object is a face of a second person.
[0164] Aspect 8: A method according to any one of Aspects 1 to 7, wherein at least partially using the first distance and the second distance to identify the focus includes: determining a first depth of field associated with the first object and a second depth of field associated with the second object; and identifying the focus as a point between the first depth of field and the second depth of field.
[0165] Aspect 9: The method of aspect 8, wherein the first depth of field is determined based on a depth associated with the first object using a lookup table, and wherein the second depth of field is determined based on a depth associated with the second object using the lookup table.
[0166] Aspect 10: A non-transitory computer-readable medium of a network entity having stored thereon instructions, which when executed by one or more processors cause the one or more processors to perform operations according to any one of aspects 1 to 9.
[0167] Aspect 11: An apparatus for wireless communication, the apparatus comprising one or more means for performing the operations according to any one of aspects 1 to 9.
[0168] Aspect 12: A method for capturing an image, the method comprising: identifying a focus of an object; capturing a first image using the focus as a basis for capturing, the first image comprising a first region degraded due to optical distortion; estimating a PSF based on the focus and the optical distortion; and generating a second image from the first image at least in part by enhancing the first region of the first image using the PSF. In some aspects, blur may be caused by distortion, and the PSF may be estimated to enhance the image caused by the distortion.
[0169] Aspect 13: The method according to aspect 12, further comprising: determining the type of the optical deformation based on the PSF, wherein the type of the deformation includes at least one of an aberration associated with an optical setting, a motion of the object, or a tilt. In some aspects, blur may be caused by distortion associated with various types of optical deformation, and the PSF may be estimated to enhance the image.
[0170] Aspect 14: The method according to any one of aspects 12 to 13, wherein generating the second image from the first image at least in part by enhancing the first region comprises: enhancing the first region based on the determined motion of the object corresponding to the PSF.
[0171] Aspect 15: A method according to any one of Aspects 12 to 14, wherein generating the second image from the first image at least in part by enhancing the first area comprises: enhancing the first area of the first image based on the tilt and the center point of the tilt; and enhancing the second area of the first image based on the tilt and the center point.
[0172] Aspect 16: A method according to any one of Aspects 12 to 15, wherein generating the second image from the first image at least in part by enhancing the first area includes: enhancing the first area of the first image based on an optical setting used to capture the first image, wherein the optical setting includes one of a type of lens or an aperture size of the lens.
[0173] Aspect 17: A non-transitory computer-readable medium of a network entity having stored thereon instructions, which when executed by one or more processors cause the one or more processors to perform operations according to any one of aspects 12 to 16.
[0174] Aspect 18: An apparatus for wireless communication, the apparatus comprising one or more means for performing the operations according to any one of aspects 12 to 16.
[0175] Aspect 19: A device for capturing an image. The device includes: at least one memory (e.g., implemented in a circuit) and at least one processor (e.g., a single processor or multiple processors) coupled to the at least one memory. The at least one processor is configured to: determine a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; use the first distance and the second distance at least in part to identify a focus of a camera lens; capture an image using the focus as a basis for capture, the image including a first area corresponding to the first object and a second area corresponding to the second object; and generate a second image from the image at least in part by enhancing at least one of the first area or the second area using a PSF.
[0176] Aspect 20: The apparatus according to aspect 19, wherein the at least one processor is configured to select the PSF based on at least one of a distance between the first object and the focus and a distance between the second object and the focus.
[0177] Aspect 21: The apparatus according to any one of aspects 19 or 20, wherein the PSF is selected from a lookup table.
[0178] Aspect 22: The apparatus according to Aspect 21, wherein the lookup table is determined using a machine learning model, the machine learning model being trained using a defocused image and a loss function to correct the defocused image.
[0179] Aspect 23: An apparatus according to any one of Aspects 21 or 22, wherein the lookup table is determined using a computer vision-based PSF estimate determined from a defocused image and an error calculation and iteratively modifying the computer vision-based PSF estimate until a minimum error is identified for each focal length and each blur amount.
[0180] Aspect 24: An apparatus according to any one of Aspects 19 to 23, wherein the at least one processor is configured to: generate a modified first region at least in part by applying a deconvolution operation to the first region based on the PSF; and generate a modified second region at least in part by applying a deconvolution operation to the second region based on the PSF.
[0181] Aspect 25: The apparatus according to any one of aspects 19 to 24, wherein the first object is a face of a first person, and the second object is a face of a second person.
[0182] Aspect 26: An apparatus according to any one of Aspects 19 to 25, wherein the at least one processor is configured to: determine a first depth of field associated with the first object and a second depth of field associated with the second object; and identify the focus as a point between the first depth of field and the second depth of field.
[0183] Aspect 27: The apparatus of aspect 25, wherein the first depth of field is determined based on a depth associated with the first object using a lookup table, and wherein the second depth of field is determined based on a depth associated with the second object using the lookup table.
[0184] Aspect 28: A device for capturing an image. The device includes: at least one memory (e.g., implemented in a circuit) and at least one processor (e.g., a single processor or multiple processors) coupled to the at least one memory. The at least one processor is configured to: identify the focus of an object; capture a first image using the focus as a basis for capture, the first image including a first area degraded due to optical deformation; estimate a PSF based on the focus and the optical deformation; and generate a second image from the first image at least in part by enhancing the first area of the first image using the PSF. In some aspects, blur can be caused by various types of distortion, and the PSF can be estimated to enhance the image caused by the distortion. In other aspects, the second area of the image can be enhanced using the PSF.
[0185] Aspect 29: The apparatus of aspect 28, wherein the at least one processor is configured to: determine the type of the optical deformation based on the PSF, wherein the type of the deformation comprises at least one of an aberration associated with an optical setting, a motion of the object, or a tilt. In some aspects, blur may be caused by distortion, and the PSF may be estimated to enhance an image caused by the distortion.
[0186] Aspect 30: The apparatus according to any one of aspects 28 to 29, wherein the at least one processor is configured to: enhance the first region based on the determined motion of the object corresponding to the PSF.
[0187] Aspect 31: An apparatus according to any one of Aspects 28 to 30, wherein the at least one processor is configured to: enhance the first region of the first image based on the tilt and the PSF tilt of the center point of the tilt; and enhance the second region of the first image based on the tilt and the center point.
[0188] Aspect 32: An apparatus according to any one of Aspects 28 to 31, wherein the at least one processor is configured to: enhance the first area of the first image based on an optical setting used to capture the first image, wherein the optical setting includes one of a type of lens or an aperture size of the lens.
Claims
1. A method for capturing an image, the method comprising: determining a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; identifying a focal point of a camera lens using, at least in part, the first distance and the second distance; capturing an image using the focus as a basis for capturing, the image including a first area corresponding to the first object and a second area corresponding to the second object; as well as A second image is generated from the image at least in part by enhancing at least one of the first region or the second region using a point spread function (PSF).
2. The method according to claim 1, further comprising: The PSF is selected based on at least one of a distance between the first object and the focus and a distance between the second object and the focus. The method of claim 2 , wherein the PSF is selected from a lookup table.
4. The method of claim 3, wherein the lookup table is determined using a machine learning (ML) model that is trained using a defocused image and a loss function to correct the defocused image.
5. The method of claim 4, wherein the lookup table is determined using a computer vision-based PSF estimate determined from a defocused image and an error calculation and iteratively modifying the computer vision-based PSF estimate until a minimum error is identified for each focal length and each blur amount.
6. The method of claim 1 , wherein enhancing at least one of the first region or the second region based on the PSF comprises: generating a modified first region at least in part by applying a deconvolution operation to the first region based on the PSF; as well as A modified second region is generated at least in part by applying a deconvolution operation to the second region based on the PSF. The method of claim 1 , wherein the first object is a face of a first person and the second object is a face of a second person.
8. The method of claim 1 , wherein identifying the focal point using at least in part the first distance and the second distance comprises: determining a first depth of field associated with the first object and a second depth of field associated with the second object; as well as The focal point is identified as a point between the first depth of field and the second depth of field.
9. The method of claim 8, wherein the first depth of field is determined based on a depth associated with the first object using a lookup table, and wherein the second depth of field is determined based on a depth associated with the second object using the lookup table.
10. A method for capturing an image, the method comprising: Identify the focus of the object; capturing a first image using the focus as a basis for capturing, the first image including a first region degraded due to optical distortion; estimating a point spread function (PSF) based on the focus and the optical deformation; and A second image is generated from the first image at least in part by enhancing the first region of the first image using the PSF.
11. The method according to claim 10, further comprising: A type of the optical distortion is determined based on the PSF, wherein the type of the distortion comprises at least one of an aberration associated with an optical setup, a motion of the object, or a tilt.
12. The method of claim 10, wherein generating the second image from the first image at least in part by enhancing the first region comprises: The first region is enhanced based on the determined motion of the object corresponding to the PSF.
13. The method of claim 10, wherein generating the second image from the first image at least in part by enhancing the first region comprises: enhancing the first region of the first image based on a tilt PSF associated with a tilt and a center point of the tilt; as well as A second region of the first image is enhanced based on the tilt PSF associated with the tilt and the center point.
14. The method of claim 10, wherein generating the second image from the first image at least in part by enhancing the first region comprises: The first area of the first image is enhanced based on optical settings used to capture the first image, wherein the optical settings include one of a type of lens or an aperture size of the lens.
15. An apparatus for capturing an image, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: determining a first distance to a first object and a second distance to a second object based on a depth map of a previously captured image; identifying a focal point of a camera lens using, at least in part, the first distance and the second distance; capturing an image using the focus as a basis for capturing, the image including a first area corresponding to the first object and a second area corresponding to the second object; as well as A second image is generated from the image at least in part by enhancing at least one of the first region or the second region using a point spread function (PSF).
16. The apparatus of claim 15, wherein the at least one processor is configured to: The PSF is selected based on at least one of a distance between the first object and the focus and a distance between the second object and the focus. The apparatus of claim 16 , wherein the PSF is selected from a lookup table.
18. The apparatus of claim 17, the lookup table being determined using a machine learning (ML) model trained using a defocused image and a loss function to correct the defocused image.
19. The apparatus of claim 18, the lookup table being determined using a computer vision-based PSF estimate determined from a defocused image and an error calculation and iteratively modifying the computer vision-based PSF estimate until a minimum error is identified for each focal length and each blur amount.
20. The apparatus of claim 15, wherein the at least one processor is configured to: generating a modified first region at least in part by applying a deconvolution operation to the first region based on the PSF; and A modified second region is generated at least in part by applying a deconvolution operation to the second region based on the PSF.
21. The apparatus of claim 15, the first object being a face of a first person, and the second object being a face of a second person.
22. The apparatus of claim 15, wherein the at least one processor is configured to: determining a first depth of field associated with the first object and a second depth of field associated with the second object; and The focal point is identified as a point between the first depth of field and the second depth of field.
23. The apparatus of claim 22, the first depth of field is determined based on a depth associated with the first object using a lookup table, and wherein the second depth of field is determined based on a depth associated with the second object using the lookup table.
24. An apparatus for capturing an image, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: Identify the focus of the object; capturing a first image using the focus as a basis for capturing, the first image including a first region blurred due to the optical distortion; estimating a point spread function (PSF) based on the focus and the optical deformation; and A second image is generated from the first image at least in part by enhancing the first region of the first image using the PSF.
25. The apparatus of claim 24, wherein the at least one processor is configured to: A type of the optical distortion is determined based on the PSF, wherein the type of the distortion comprises at least one of an aberration associated with an optical setup, a motion of the object, or a tilt.
26. The apparatus of claim 24, wherein the at least one processor is configured to: The first region is enhanced based on the determined motion of the object corresponding to the PSF.
27. The apparatus of claim 24, wherein the at least one processor is configured to: enhancing the first region of the first image based on a tilt PSF associated with a tilt and a center point of the tilt; and A second region of the first image is enhanced based on the tilt PSF associated with the tilt and the center point.
28. The apparatus of claim 24, wherein the at least one processor is configured to: The first area of the first image is enhanced based on optical settings used to capture the first image, wherein the optical settings include one of a type of lens or an aperture size of the lens.
