Image fusion for scenes with objects at multiple depths

Abstract

Image processing performed on images captured by an image capture device can be used to ameliorate some of the problems with images captured from devices, including devices with larger aperture lenses, and in some specific examples, images captured from mobile devices with larger aperture lenses. Multiple images can be captured by a capture device and processing can be applied to generate a single image by fusion of the multiple images. One potential advantage gained by fusion of multiple images is a single image with a greater effective depth of focus (DOF) than is available in a single image obtained from the capture device. The DOF of the fused image can be greater than a single image and / or include multiple distinct focus focal lengths, whereas a single image from a single capture device has only a single focus focal length.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. patent application Ser. No. 17 / 161,355, filed Jan. 28, 2021, entitled “IMAGE FUSION FOR SCENES WITH OBJECTS AT MULTIPLE DEPTHS,” which is expressly incorporated herein by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE Aspects of the present disclosure relate generally to image processing. Several features of the present disclosure may enable and provide improved processing by an image signal processor of output from an image capture device. [Background technology]

[0003] An image capture device is a device that can capture one or more digital images, whether a still image for a photograph or a sequence of images for a video. Capture devices can be incorporated into a variety of devices. By way of example, an image capture device may comprise a stand-alone digital camera or digital video camcorder, a wireless communication device handset equipped with a camera, such as a mobile phone, cellular or satellite radiotelephone, a personal digital assistant (PDA), a panel or tablet, a gaming device, a computing device such as a webcam, a video surveillance camera, or other device with digital photography or video capabilities.

[0004] The quality of an image captured by an image capture device can be improved by using a large aperture lens on one or more image sensors of the image capture device. A larger aperture lens has many benefits, including better low light performance, which can be beneficial for indoor and nighttime photography, and better bokeh blur, which can be beneficial for portrait photography. However, a large aperture lens also has a short depth of focus (DOF), resulting in a limited portion of the captured image being in focus, and the remainder of the image having varying degrees of blur. When a scene includes many objects at various distances, an image capture device with a large aperture lens captures an image in which some of the objects are out of focus. This can be undesirable, especially when the out-of-focus objects include other people.

[0005] The shortcomings mentioned herein are merely representative and are included to highlight problems the inventors have identified with existing devices and for which they have sought improvement. Aspects of the devices described below may address some or all of the shortcomings, as well as others known in the art. Aspects of the improved devices described below may offer advantages other than those described above and may be used in applications other than those described above. Summary of the Invention [Means for solving the problem]

[0006] Image processing performed on images captured by an image capture device may be used to ameliorate some of the problems with images captured from devices, including devices with larger aperture lenses, and in some specific examples, images captured from mobile devices with larger aperture lenses. Multiple images may be captured by a capture device, and processing may be applied to generate a single image by fusing the multiple images. One potential advantage gained by fusing multiple images is a single image with a greater effective depth of focus (DOF) than is available in a single image obtained from a capture device. The DOF of the fused image may be greater than a single image and / or may include multiple distinct focus focal lengths, whereas a single image from a single capture device has only a single focus focal length. Fusing multiple images may include one or more of the image processing techniques described, such as merging, copying, segmenting, and blending. Fusing multiple images may be initiated based on criteria, including certain conditions met within the captured images. In one embodiment, default image capture may include processing a first input frame from a capture device. When a condition is identified in the first image frame, such as two or more peaks in the depth data histogram separated by a sufficient threshold, an additional image frame is retrieved for processing including fusing the first image frame with the additional image frame. Retrieving the additional frame may trigger additional image capture from the image capture device.

[0007] The following summarizes some aspects of the present disclosure in order to provide a basic understanding of the described technology. This summary is not an extensive overview of all contemplated features of the disclosure, nor is it intended to identify key or critical elements of all aspects of the disclosure, nor is it intended to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in summary form as a prelude to the more detailed description that is presented later.

[0008] In different embodiments, the hardware and / or software of the digital camera may capture images, process images, trigger image fusion, and / or fuse the captured images, with some hardware and / or software performing multiple steps between capturing images and generating a fused image. In some embodiments, the hardware of the digital camera may process the captured images, determine that fusion processing should be triggered, and pass those additional images to an image signal processor and / or application processor for further processing, including processing by capturing additional images and fusing the images. Triggering of the fusion process may be performed based on a predefined criteria determined by the hardware (e.g., a difference in focal lengths of the objects of interest exceeds a threshold) and may be performed in fixed function hardware and / or a general purpose processor. In other embodiments, triggering of the fusion process may alternatively be performed based on a request from another component in the device, for example by the CPU, in response to an image processing function being performed on the CPU, in response to a user input activating a fusion processing capture mode, in response to a decision by a fixed function fusion node, or in response to a decision by an image signal processor. In some embodiments, the image signal processor may be in a device separate from the actual image capture components (e.g., image sensor and lens) and may be integrated into a system-on-chip (SoC) with one or more of the fusion node, central processing unit, memory, and / or storage.

[0009] After an output frame representing a scene is generated by fusing multiple image frames, the view of the scene may be displayed on a device display, saved as a picture or a sequence of pictures to a storage device as a video, transmitted over a network, and / or printed on an output medium. For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from one or more image frames from one or more image sensors, and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, frames of video, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization, generating a video file via the output frames, composing frames for display, composing frames for storage, etc. That is, the image signal processor may obtain incoming frames from one or more image sensors, each coupled to one or more camera lenses, and then generate a flow of output frames to output to various destinations.

[0010] In one form of the disclosure, the method includes receiving a first image frame representing a scene captured at a first focal point at a first focal length and determining whether a difference between a first depth for a first portion of the scene and a second depth for a second portion of the scene exceeds a threshold. When the difference exceeds the threshold, the method may continue with receiving a second image frame captured at a second focal length different from the first focal length and determining an output image frame based on the first image frame and the second image frame. The method for determining the output image frame may include segmenting the output image frame into a first portion and a second portion and / or applying a blurring algorithm to pixels in the second portion but not the first portion. The method for determining the output image frame may additionally or alternatively include determining a segment of the second image frame corresponding to the second depth and / or merging the segment with the first image frame to obtain the output image frame. The method for determining the output image frame may additionally or alternatively include copying first image data corresponding to the first set of pixels from a first portion of the first image frame to the output image frame, copying second image data corresponding to the second set of pixels from a segment of the second image frame to the output image frame, and / or determining third image data for the output image frame corresponding to a third set of pixels between the first and second sets of pixels, the third image data being determined based on the first and second image frames and a weighting value. The weighting value may be determined based on at least one of a contrast value of the first image frame, a distance value corresponding to a distance from a pixel in the third set of pixels to a pixel in at least one of the first or second set of pixels, a first gradient of the first image frame, and / or a second gradient of the second image frame.In some embodiments, the weighting value corresponds to selecting a value for a pixel in the third set of pixels from a first corresponding pixel in the first set of pixels or a second corresponding pixel in the second set of pixels based on whether a gradient at the first corresponding pixel is greater than the second corresponding pixel.

[0011] The method may include performing similar operations for more than two image frames. For example, the method may include determining a third depth of a third portion of the first image frame based on the depth data, determining whether a second difference between the second depth and the third depth exceeds a second threshold, sending a command to capture a third image frame at a third focal length different from the second focal length if the second difference exceeds the second threshold, receiving the third image frame captured at the third focal length, and / or determining an output image frame based on the first image frame, the second image frame, and the third image frame.

[0012] In an additional aspect of the present disclosure, an apparatus is disclosed that includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform any of the methods or techniques described herein. For example, the at least one processor may be configured to perform steps including analyzing a first image frame from a capture device and determining whether to retrieve and / or trigger capture of additional image frames, and / or fusing two or more image frames to generate an output frame. The at least one processor may include an image signal processor or a processor that includes specific functions for camera control and / or processing. The at least one processor may also or alternatively include an application processor. The methods and techniques described herein may be performed entirely by the image signal processor or application processor, or various operations may be divided between the image signal processor and the application processor, and in some embodiments, additional processors.

[0013] In an additional aspect of the present disclosure, a device configured for image capture is disclosed. The apparatus includes means for receiving a first image frame representing a scene captured at a first focal point at a first focal length, means for determining whether a difference between a first depth for a first portion of the scene and a second depth for a second portion of the scene exceeds a threshold, means for receiving a second image frame representing the scene captured at a second focal length different from the first focal length, and / or means for determining an output image frame based on the first image frame and the second image frame. The means for determining the output image frame may include means for segmenting, means for merging, means for determining a gradient map, means for determining histogram depth data, and / or means for blending. The apparatus further includes one or more means for capturing data representing the scene, such as an image sensor (including a charge-coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal-oxide semiconductor (CMOS) sensor), a time-of-flight detector, etc. The apparatus may further include one or more means (including simple lenses, compound lenses, spherical lenses, and aspherical lenses) for accumulating and / or focusing the light beam onto one or more image sensors.

[0014] In an additional aspect of the disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including those described in the methods and techniques described herein. For example, the operations may include receiving a first image frame representing a scene captured at a first focal point at a first focal length and determining whether a difference between a first depth for a first portion of the scene and a second depth for a second portion of the scene exceeds a threshold, and when the difference exceeds the threshold, the instructions may continue to receive a second image frame captured at a second focal length different from the first focal length and determine an output image frame based on the first image frame and the second image frame.

[0015] In an additional aspect of the present disclosure, a mobile device may include an image sensor, an image signal processor, a fixed function fusion node, and a central processing unit (CPU) that functions as an application processor (AP). A scene detection scheme may be executed on the AP to detect the presence of multiple regions of interest, such as using depth information described herein, during a preview operation of a camera application executed on the mobile device. When a capture request is made, such as by a user clicking a shutter button, when multiple peaks on the depth histogram are detected, the AP sends a signal to the camera hardware to capture multiple images and multiple focal planes for further processing. The camera hardware, such as an image sensor, lens, and autofocus (AF) system, executes an AF bracketing scheme to capture two images. These images may be processed by the image signal processor to enhance the individual images, and both images are transmitted from the ISP to the fusion node. The fusion node processes these two images with corresponding depth data, for example, based on the combined contour map, weighting, and compositing described herein. The fusion node then outputs the single image file to the AP for display in a camera application, stored in the mobile device's storage for later display in a gallery application, or otherwise transmitted, stored, or further processed.

[0016] In an additional aspect of the disclosure, the device may include a digital camera comprising an image sensor and an autofocus (AF) circuit coupled to the image sensor, the AF circuit configured to focus the image sensor to a first focal plane, receive first image data from the image sensor at the first focal plane, determine a distance between objects in the first image data, determine that the distance exceeds a threshold, focus the image sensor to a second focal plane, and receive second image data from the image sensor at the second focal plane. In some aspects, the distance between the objects may be determined by determining peaks in a depth data histogram and determining the distance between the peaks as the distance between the objects. The depth data histogram may be received, for example, from a depth sensor separate from the image sensor, corresponding second image data from a second image sensor capturing the scene at a different field of view, and / or corresponding second image data from non-overlapping portions of the image sensor. The first and second image data may be passed to other processing circuitry, such as an image signal processor and / or an application processor, to perform fusion processing on the first and second image data.

[0017] Other aspects, features, and implementations will become apparent to those skilled in the art upon reviewing the following description of certain exemplary aspects together with the accompanying figures. Although features may be described in conjunction with some aspects and figures below, various aspects may include one or more of the advantageous features described herein. In other words, although one or more aspects may be described as having some advantageous features, one or more of such features may also be used in accordance with various aspects. Similarly, although exemplary aspects may be described below as device aspects, system aspects, or method aspects, such exemplary aspects may be implemented in various devices, systems, and methods.

[0018] The foregoing has outlined rather broadly some of the features and technical advantages of embodiments of the present invention in order that the following Detailed Description may be better understood. Additional features and advantages of the invention will be described hereinafter which form the subject of the claims of the invention. Those skilled in the art will appreciate that the concept and specific embodiments disclosed may be readily utilized as a basis for modifying or designing other structures for the same or similar purposes. Those skilled in the art will also realize that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. Additional features will be better understood from the following description when considered in conjunction with the accompanying drawings. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended to limit the invention.

[0019] A further understanding of the nature and advantages of the present disclosure can be realized by referring to the following drawings. In the attached drawings, similar components or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. When only a first reference label is used in this specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram of a computing device configured to execute one or more of the example techniques described in this disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating the fusion of multiple images to obtain an output image frame according to an embodiment of the present disclosure. [Diagram 3] 1 is a block diagram illustrating example decisions regarding executing fusion logic, according to some embodiments of the present disclosure. [Figure 4]1 is a flowchart of a method for processing image frames with fusion logic according to some embodiments of the present disclosure. [Diagram 5] FIG. 2 is a block diagram illustrating segmentation and merging of image data from two image frames to determine an output image frame in accordance with some embodiments of the present disclosure. [Figure 6] 1 is a flowchart illustrating segmentation of image data using a contour map, according to some embodiments of the present disclosure. [Figure 7] 10 is a flowchart illustrating merging image data to determine a fused output image frame based on weighting values ​​in accordance with some embodiments of the present disclosure. [Figure 8] 1 is a flowchart illustrating generation of weight values ​​based on a gradient map for fusing two image frames according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Like reference numbers and designations in the various drawings indicate like elements.

[0022] The detailed description set forth below with reference to the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. Rather, the detailed description includes specific details to provide a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in all cases and that in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0023] The present disclosure provides systems, apparatus, methods, and computer-readable media that support improved image quality from an image capture device by capturing multiple images of a scene. The improved methods describe processing multiple captured images of a scene by combining the multiple images using fusion logic. The images may be captured using the same capture device with the same device settings, may be captured using the same capture device with different device settings, and / or may be captured using different capture devices, such that each image provides different information about the scene that differs from other captured images of the scene. The different settings for the different image captures may be the result of reconfiguring the device during capture such that a first sensor captures a different view of the scene. For example, the first sensor may be configured to focus at two different focal lengths. In some embodiments, the image frame input to the fusion process described herein is itself a fusion of images output from multiple sensors of the image capture device.

[0024] Aspects of the present disclosure may be used to capture image frames using multiple image sensors of an image capture device. The multiple image sensors may include a combination of ultra-wide (high field of view (FOV)), wide-angle, telephoto, and super-telephoto (low FOV) sensors. That is, each image sensor may be configured through hardware configuration and / or software settings to acquire images of a scene that are focused on different portions of the image. Fusion logic in the image capture device, such as in the image signal processor 112 and / or processor 104, may combine multiple images of a scene to obtain an improved image that includes multiple focal lengths, allowing objects of different focal lengths to be in focus in the fused image. This may be beneficial to improve image quality, for example, when the image capture device has a large aperture lens that results in some objects in the scene being out of focus in a single image capture.

[0025] An exemplary device for capturing image frames, such as a smartphone, may include a configuration of one, two, three, four, or more cameras on the back (e.g., opposite the user display) or front (e.g., on the same side as the user display) of the device. The device may include one or more image signal processors, computer vision processors (CVPs), or other suitable circuitry for processing images captured by the one or more image sensors. The one or more image signal processors may provide processed image frames to a memory and / or processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuitry) for further processing, such as encoding or other manipulation.

[0026] As used herein, an image sensor may refer to the image sensor itself, as well as any other suitable components coupled to the image sensor. For example, an image sensor may also refer to other components of a camera, including a shutter, a buffer, or other readout circuitry. An image sensor may further refer to an analog front end or other circuitry for converting an analog signal into a digital representation of a frame. Thus, the term "image sensor" herein may refer to any suitable component for the capture and readout of an image frame into at least one of a memory, a processor, an image signal processor, or other logic or memory circuitry.

[0027] In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, to enable a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intervening components or circuits. Furthermore, in the following description, for purposes of explanation, specific names are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the present disclosure. Some portions of the following detailed description are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc., is conceived to be a self-consistent sequence of steps or instructions that leads to a desired result. The steps are those requiring physical manipulations of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0028] It should be kept in mind, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise apparent from the following description, it should be appreciated that throughout this application, descriptions utilizing terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "setting aside," "generating," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's registers, memory, or other such information storage, transmission, or display device.

[0029] In the figures, a single block may be described as performing a function, but in reality the function performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various example components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Also, the example device may include components other than those shown, including well-known components such as a processor, memory, etc.

[0030] Aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Additionally, aspects of the present disclosure may be implemented in devices having or coupled to image sensors of the same or different capabilities and characteristics (resolution, shutter speed, sensor type, etc.).

[0031] The terms "device" and "apparatus" are not limited to one or a particular number of physical objects (e.g., one smartphone, one camera controller, one processing system, etc.). As used herein, a device may be any electronic device having one or more parts that may implement at least some parts of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the operations described.

[0032] FIG. 1 illustrates a block diagram of an exemplary device 100 for performing image capture from multiple image sensors. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from multiple image sensors, such as a first image sensor 101 and a second image sensor 102. In some implementations, the device 100 also includes or is coupled to a processor 104, and a memory 106 that stores instructions 108. The device 100 may also include or be coupled to a display 114 and several input / output (I / O) components 116. The device 100 may further include or be coupled to a power source 118 for the device 100, such as a battery or a component for coupling the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown. In one example, a wireless interface may be included for a wireless communication device, which may include several transceivers and a baseband processor. In another example, one or more other sensors (such as a gyroscope or a global positioning system (GPS) receiver) may be included in or coupled to the device. In a further example, an analog front end for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0033] The image signal processor 112 may receive image data from a local bus connection to the image sensors 101 and / or 102 or by other connections such as a wire interface to an external image sensor or a wireless interface to a remote image sensor. In some embodiments, the device 100 may include a first camera 130 with a first image sensor 101, a corresponding first lens 131, and autofocus logic 141, and a second camera 140 with a second image sensor 102, a corresponding second lens 132, and autofocus logic 142. The autofocus logic 141 and 142 may be dedicated hardware for adjusting the lenses 131 and 132, respectively, to obtain image frames from the image sensors 101 and 102, respectively, that are focused on a particular focal plane determined by the algorithms of the autofocus logic 141 and 142. In some embodiments, the device 100 may include an interface for receiving image data from the image sensors 101 and 102 located remotely from the device 100. Device 100 may perform image processing on image data from a combination of image sensors located within device 100 or remotely from device 100 .

[0034] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. For example, the first image sensor 101 and the second image sensor 102 may be included in one or more camera configurations (such as a dual camera configuration, a triple camera configuration, etc. for a smartphone or other suitable device). The image sensors 101 and 102 may also include or be coupled to one or more lenses for collecting light, one or more apertures for receiving light, one or more shutters for blocking light when outside an exposure window, one or more color filter arrays (CFAs) for filtering light outside a particular frequency range, one or more analog front ends for converting analog measurements into digital information, or other suitable components for imaging. For example, the first image sensor 101 may be coupled to a first lens 131, and the second image sensor 102 may be coupled to a second lens 132. The first lens 131 and the second lens 132 may have different fields of view, such as when the first lens 131 is an ultra-wide angle (UW) lens and the second lens 132 is a wide angle (W) lens. The device 100 may also include or be coupled to a flash, a depth sensor, a GPS, or other suitable components for imaging. An example configuration of sensors including sensors 101 and 102 in a multi-image capture device is a UW sensor having a larger FOV than a W sensor having a larger FOV than a T sensor having a larger FOV than a UT sensor. For example, a sensor configured for a wide angle FOV may capture a field of view ranging from 64 to 84 degrees, a sensor configured for an ultra-wide angle FOV may capture a field of view ranging from 100 to 140 degrees, a sensor configured for a telephoto FOV may capture a field of view ranging from 10 to 30 degrees, and a sensor configured for a super telephoto FOV may capture a field of view ranging from 1 to 8 degrees.

[0035] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although FIG. 1 illustrates device 100 as including two image sensors 101 and 102 coupled to image signal processor 112, any number of image sensors may be coupled to image signal processor 112. Additionally, any number of additional image sensors or image signal processors may be present for device 100. In some embodiments, image signal processor 112 may execute instructions from a memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in image signal processor 112, or instructions provided by processor 104. Additionally or alternatively, image signal processor 112 may execute software and / or include specific hardware (such as one or more integrated circuits (ICs)) to perform one or more operations described in this disclosure.

[0036] In some implementations, the memory 106 may include a non-transitory computer-readable medium that stores computer-executable instructions 108 for performing all or a portion of one or more operations described in this disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 to generate images or videos. The instructions 108 may also include other applications or programs executed by the device 100, such as operating systems and specific applications other than image or video generation. Execution of the camera application, such as by the processor 104, may cause the device 100 to generate images using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames or by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 is a circuit that includes the image signal processor 112, and the memory may be external to the device 100. Device 100 may be coupled to memory and configured to access the memory to write output frames for display or long-term storage.

[0037] In some embodiments, the processor 104 may include one or more general-purpose processors, sometimes referred to as "cores," capable of executing scripts or instructions of one or more software programs, such as instructions 108 stored in the memory 106. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106. In executing the camera application, the processor 104 may be configured to direct the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application executing on the processor 104 may receive a shutter capture command by tapping a button in a graphical user interface (GUI) and direct the image signal processor 112 to capture one or more image frames. Execution of the instructions 108 by the processor 104 outside of the camera application may also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include ICs or other hardware in addition to the capability of executing software to cause the device 100 to perform certain functions or operations, such as those described herein. In some other embodiments, device 100 does not include processor 104, such as when all of the described functionality is configured within image signal processor 112. In some embodiments, some image processing and / or camera functionality may be integrated into processor 104 or may be integrated into fixed function circuitry, such as a fusion node 144, which may be included elsewhere within device 100, to fuse multiple images in accordance with the embodiments described herein.

[0038] In some embodiments, at least one of the image signal processor 112 or the processor 104 may execute instructions to perform various operations described herein. For example, execution of instructions may instruct the image signal processor 112 to capture a first image frame at a first zoom at a first focal length, and then capture a second image frame at the first zoom at a second focal length. Instructions to capture image frames at a particular focal length may be interpreted by the image signal processor 112 to seed an autofocus algorithm with the indicated focal length prior to capturing the image frames.

[0039] In some embodiments, display 114 may include one or more suitable displays or screens that allow for user interaction and / or presentation of items to a user, such as a preview of image frames being captured by image sensors 101 and 102. In some embodiments, display 114 is a touch-sensitive display. I / O components 116 may be or include any suitable mechanism, interface, or device for receiving input (e.g., commands) from a user and providing output to a user. For example, I / O components 116 may include, but are not limited to, a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), sliders, switches, and the like.

[0040] Although shown coupled together via the processor 104, the processor 104, memory 106, image signal processor 112, display 114, and I / O components 116 may be coupled together in various other configurations, such as via one or more local buses not shown for simplicity. Although the image signal processor 112 is shown as separate from the processor 104, the image signal processor 112 may be an application processor unit (APU), may be included in a system on chip (SoC), or may be a core of the processor 104 that is otherwise included in the processor 104. The device 100 is referenced in the examples herein to perform aspects of the present disclosure, although some device components may not be shown in FIG. 1 to avoid obscuring aspects of the present disclosure. Additionally, other components, numbers of components, or combinations of components may be included in a device suitable for performing aspects of the present disclosure. Thus, the present disclosure is not limited to a particular device or configuration of components, including the device 100.

[0041] Methods of processing image frames captured from an image capture device described in embodiments of the present disclosure, such as the embodiment of FIG. 1, may be used to obtain improved images of a scene and objects within the scene. For example, a method for fusing image frames may obtain an image of a scene with a greater depth of focus in which multiple objects at different distances from the image capture device are in focus. One advantage of such a fusion process is that fusion may enable the use of image capture devices with larger apertures and capture devices with shorter depth of focus. These larger aperture lenses, such as lenses with apertures greater than f / 2.2 (including apertures of f / 2.0, f / 1.8, f / 1.4, and / or f / 1.2), may provide improved image quality in certain situations, such as low light situations, with loss of depth of focus. The fusion process disclosed in embodiments herein allows for obtaining improved image quality with reduced or no loss of depth of focus, for example, by generating an output image frame that includes an object of interest in focus. This may be particularly desirable when capturing a scene that includes multiple people of interest, as the fusion logic described herein can produce an image in which many or all of the people are in focus. Additionally, the use of a large aperture lens produces images that include a blurred background, sometimes referred to as a bokeh blur or shallow depth of field effect. Fusing multiple images from a larger aperture lens can produce an image in which multiple objects are in focus, including a blurred background, which is desired by many photographers.

[0042] The fusion logic in some embodiments of the present disclosure may be triggered under some conditions, which reduces unnecessary processing of images. Reducing the amount of processing may be particularly beneficial for mobile devices that operate on limited power sources such as batteries. The fusion logic may trigger further image capture and processing and merging of multiple images based on criteria that indicate the presence of multiple objects of interest in a first captured image of a scene. For example, one criterion that indicates multiple objects of interest is determining that a histogram of depth data from a scene indicates the presence of multiple objects of interest. The histogram may indicate multiple objects when multiple histogram peaks are detected and separated by a threshold value that is greater than the expected focal depth of the current camera configuration. As another example, a criterion that indicates multiple objects of interest is the detection of multiple faces or multiple objects by an artificial intelligence (AI) machine learning algorithm. As a further example, one criterion that indicates multiple objects of interest is the receipt of a user input that specifies that the fusion logic should be applied and / or that specifies multiple objects of interest in the scene. For example, a user may tap on several locations on a first image of a scene displayed on a mobile device, and the focal depths corresponding to those tapped locations may be used as the focal depths for capturing additional image frames of the scene. In some embodiments, a combination of these criteria may be used to trigger fusion image processing. As one exemplary combination, a histogram of depth data may be used to determine multiple potential objects of interest, each of which is input to a face detection algorithm to determine whether the potential objects of interest correspond to people, and then display a box on the display of the first image frame around the detected faces to allow a user to provide user input specifying which of the detected faces correspond to actual objects of interest.As another exemplary combination, a face or object detection algorithm may be used to determine potential objects of interest and their focal depths compared to peaks on a histogram of the depth data to determine whether the potential objects are a sufficient distance away to trigger the capture of additional images. As a further exemplary combination, a portrait segmentation algorithm (including portrait recognition) based on depth information may be used to determine potential objects of interest and their focal depths compared to peaks on a histogram of the depth data to determine whether the potential objects are a sufficient distance away to trigger the capture of additional images. After multiple potential objects of interest are detected, the fusion logic may command the capture of additional images at additional focal distances determined using a combination of the depth data of the scene and / or the autofocus system. If these criteria are not met, the fusion logic will not trigger the capture of additional images and subsequent processing of those images to reduce the number of operations performed by the image capture device and associated processing and memory circuits. After the additional images are captured, similar criteria may be re-evaluated to determine which of the additional images to fuse together. For example, all images captured at different focal lengths are displayed on a display and the user can select some images for fusion. As another example, images captured at different focal lengths can be input again into a face or object detection process to confirm or reject potential objects of interest as recognized faces or objects.

[0043] FIG. 2 is a block diagram illustrating fusing multiple images to obtain an output image frame according to an embodiment of the present disclosure. A scene 202 may include multiple objects 204, 206, and 208 at different distances from an image capture device. The image capture device 130 may generate a first image frame 210 capturing the scene 202. The first image frame 210 may be focused at a focal length near the object 204. As a result, the object 204 is in focus and the object 206 appears blurry. The image capture device 130 may generate a second image frame 220 capturing the scene 202. The second image frame 220 may be focused at a focal length near the object 206. As a result, the object 206 is in focus and the object 204 appears blurry or out of focus. The different image frames 210 and 220 may be captured at different focal lengths by seeding an autofocus (AF) algorithm of the image capture device 130 with focal lengths corresponding to the objects 204 and 206. The AF algorithms of image capture device 130 may process other criteria, such as local contrast or depth data, to obtain image frames 210 and 220 at focal distances near objects 204 and 206. Although the same image capture device 130 is shown capturing image frames 210 and 220, different image capture devices may be used to capture the two image frames 210 and 220.

[0044] The fusion logic 240 may be used to combine the image frames 210 and 220 to obtain an output image frame 250 that may be further processed, displayed to a user, stored in memory, and / or transmitted to another user. The output frame 250 generated by the fusion logic 240 may have a focused object 204 and a focused object 206. The output frame 250 includes the focused object at multiple focal lengths, even though image frames 210 and 220 have both objects 204 and 206 that are not in focus. Processing the image frame 220 that includes the focused object 206 may include segmenting the image frame 220 to extract the object 206 and merging the segmented object 206 into the image frame 210. The fusion logic 240 may include functionality from one or more of hardware within the cameras 130 and / or 140 (such as AFs 141 and / or 142), the image signal processor 112, the processor 104, a fusion node 144 comprising fixed function circuitry, and / or software running on the processor 104.

[0045] Producing an image frame representing a scene based on multiple image frames and a fusion process as described with reference to Fig. 2 takes longer and consumes additional processing time and battery charge level. The fusion process of Fig. 2 may be beneficial to capture image frames of several scenes. According to some embodiments, whether multiple images of the same scene are beneficial may be determined by inspecting data corresponding to the captured image frames and determining whether additional image frames, and the number of additional image frames, may improve the output image. One way to determine whether to capture additional image frames of a scene is by determining the distance between objects in the images.

[0046] 3 is a block diagram illustrating an example decision regarding executing fusion logic, according to some embodiments of the present disclosure. Image capture device 130 captures depth data 300 corresponding to scene 202. For example, depth data 300 may be determined from multiple image frames capturing scene 202 from different viewpoints. As another example, depth data 300 may be determined from a depth sensor, such as a time-of-flight (TOF) sensor, a stereo image including two or more image frames of a scene captured from different viewpoints, such as two different image sensors at different locations, and / or a light detection and ranging (lidar) using a laser illumination source. In some embodiments, image capture device 130 that captures depth data 300 may be the same image capture device that captures image frames 210 and 220 of FIG. 2. In such an embodiment, autofocus logic 141 and 142 may determine from depth data 300 whether to capture additional images and may use depth data 300 as an input to the autofocus logic to select a new focal plane for focusing and capturing a second or other subsequent image frame. In some embodiments, the image capture device for capturing the depth data 300 may be a different capture device, or the depth data 300 may be received without the use of an image capture device, such as by retrieving stored depth data 300 from memory and / or receiving depth data 300 from another device. In such embodiments, an image signal processor (ISP) and / or application processor (AP) may use the depth data 300 to determine that additional images should be captured and control one or more cameras 130 and / or 140 to acquire a second or subsequent image frame to be used in the fusion process to obtain an output image frame.

[0047] The depth data 300 may be input to the fusion logic 240 and analyzed to determine an interest focal length for capturing an image frame. For example, a histogram of the depth data 300 may be determined by determining a number of pixels at various distances from an image capture device, or by determining a certain percentage of image pixels in an image or a downsampled image based on an image captured by an image capture device. The fusion logic 240 may determine a peak in the histogram to determine the interest focal length. For example, the depth data 300 may have histogram peaks 302 and 304. The center of the peaks 302 and 304 may indicate a focal length of an object, such as a person, when the peaks meet certain criteria, such as exceeding a threshold width and / or value. The distance between the peaks 302 and 304 may be used to determine whether multiple image frames should be captured. For example, the fusion logic 240 may determine whether a difference in distance between the detected peaks 302 and 304 is greater than a threshold. The difference in distance may indicate that objects in the scene corresponding to peaks 302 and 304 may not both be in focus within a single image frame captured from image capture device 130.

[0048] A first image frame may be captured from the image capture device 130 and analyzed in the fusion logic 240 to determine whether the fusion logic 240 should trigger the capture of multiple image frames. The determination may be based on a distance of a peak in a histogram obtained from the depth data of the scene. FIG. 4 shows a flowchart of a method of processing image frames with the fusion logic according to some embodiments of the present disclosure. The method 400 begins at block 402 with receiving first image data of a scene and depth data of the scene. The first image data may correspond to a scene captured at a first focal length. For example, an autofocus (AF) algorithm may cause the image capture device to focus on an object in the scene based on contrast detection and / or distance detection. The depth data received at block 402 may correspond to the first image data. For example, the depth data may be a two-dimensional map of the depth of each pixel in the first image data.

[0049] In block 404, peaks in the depth data histogram are determined along with the distance between the peaks. In some images, each peak may indicate an object of interest in the scene captured in the first image data. In some images, the highest peak may indicate the furthest region, such as the background, and the remaining peaks may each indicate an object of interest in the scene. Thus, the number of peaks may be used to determine whether and how many additional image frames should be captured from the scene. As a first example, two peaks in the histogram may be determined and the distance between the peaks may be determined. As another example, three peaks in the histogram may be determined and the distance between each peak and the other peaks may be determined for a total of six distances.

[0050] In block 406, the distance between the peaks is evaluated to determine whether the distance of any pair of peaks exceeds a fusion threshold. Each of the distances calculated in block 404 may be compared to a fusion threshold. The fusion threshold may be a predetermined value. The fusion threshold may be predetermined based on characteristics of the image capture device that generated the first image data retrieved in block 402. For example, the fusion threshold may be a value corresponding to a depth of focus (DOF) of the image frame acquired from the image capture device. When multiple lenses are available on the image capture device, the fusion threshold may be determined based on the lens used to acquire the first image data in block 402. Thus, the fusion threshold for an image captured from a telephoto (T) lens may be lower than the fusion threshold for an image captured from an ultra-wide (UW) lens. The fusion threshold may be stored in a table, and the table is used in a lookup operation when receiving the first image data in block 402 and evaluating the data in block 406.

[0051] The distance between the peaks in the scene determined in block 404 may be used to determine whether an additional image frame should be captured and at what focal length. When the fusion logic is activated based on the distance exceeding the fusion threshold, the method 400 continues to blocks 408 and 410. In block 408, second image data corresponding to the scene captured at a second focal length is received. The second image data may be received in response to a command sent to the image capture device to obtain a second image frame of the scene at a second focal length different from the first focal length corresponding to the first image data received in block 402. For example, the command may seed an autofocus (AF) algorithm in the image capture device with a second focal length that may correspond to one of the histogram peaks determined in block 404. In some embodiments, the AF algorithm may refine the second focal length before capturing the second image data received in block 408. In some embodiments, the command may specify the same or a different image sensor to capture the second image frame.

[0052] At block 410, an output image frame is generated by the fusion logic based on the first image data and the second image data. For example, the fusion logic may segment objects from the first image data and the second image data and combine focused objects from the first and second image data with background from one or both of the first and second image data. The fusion logic may perform other operations to output the output image frame as described herein.

[0053] Additional processing may be performed on the output image frames by the fusion logic and / or outside of the fusion logic. Exemplary additional processing may include application of blur to portions of the output image frames. A blur algorithm may be applied to some portions of the output image frames based on the segmentation data, the depth data, and / or other characteristics of one or more of the input image frames. For example, blur may be applied to background pixels identified based on the depth data. As another example, blur may be applied to a portion of pixels corresponding to a foreground segment of one of the input image frames, such as when the foreground segment corresponds to a face. The blur may be a heavy blur performed to obscure the face, or a light blur performed to improve the beauty of the face. Other examples of additional processing may include color filtering, color mapping, lens effects, transformations, and / or lighting effects. In some embodiments, blur may be applied to pixels of the second portion but not the first portion.

[0054] One method of determining an output image frame from two input image frames is illustrated in FIG. 5. FIG. 5 is a block diagram illustrating segmenting and merging image data from two image frames to determine an output image frame, according to some embodiments of the present disclosure. A first input frame 210 and a second input frame 220 may be captured and input to the fusion logic 240. In some embodiments, the fusion logic 240 may command the capture of the first input frame 210 and / or the second input frame 220. The input frames 210 and 220 may be processed within the fusion logic 240 to obtain an output image frame 250. The output image frame 250 may have multiple focus regions with different depths of focus from an image capture device. For example, the output image frame 250 may have two focused objects at different depths that exceed the depth of focus of the image capture device that captures either or both of the input image frames 210 and 220.

[0055] The fusion logic 240 may determine the output image frame 250 from the input image frames 210 and 220 using segmentation and merging operations. For example, the segmentation 510 operation may determine a portion of pixels of the image frame 220 for merging into the image frame 210. The determined portion of pixels may include pixels identified as corresponding to a particular object in the image frame 220, pixels identified as foreground portions of the image frame 220, and / or pixels identified as focus portions of the image frame 220. The determined portion of pixels may be extracted from the image frame 220, e.g., resulting in a segment 512 comprised of the determined portion of pixels. In some embodiments, the segment 512 may be limited to a single contiguous region of pixels. In other embodiments, the segment 512 may include multiple regions of pixels, each region meeting certain criteria applied during the segmentation 510 operation. The segmentation 510 operation may determine the boundary between the segment 512 and the remainder of the image frame 220 by analyzing local contrast values, analyzing depth data corresponding to the scene from which the image frame 220 is acquired, performing face, object, or portrait recognition on the image frame 220, and / or applying artificial intelligence (AI) machine learning algorithms to the image frame 220.

[0056] The segment 512 may be input to a merge 516 operation to merge the segment 512 onto the image frame 210. In some embodiments, the merge 516 operation may overwrite corresponding pixels of the image frame 210 with data from the segment 512. In some embodiments, the merge 516 operation may perform other merging algorithms, which may include combinations of copying pixels from the segment 512 into the image frame 210, blending pixels from the segment 512 into the image frame 210, blending pixels around the segment 512 in the image frame 210 based on the segment 512 to reduce seams between the segment 512 and the remainder of the image frame 210, and / or other operations. The merge 516 operation may generate an output image frame 250 that includes multiple focused objects. For example, the segmentation 510 operation may generate a segment 512 that includes the focused object 206, and a subsequent merge 516 operation may overwrite the unfocused object 206 in the image frame 210 with the focused segment 512. Thus, output image frame 250 contains two focus regions containing objects 204 and 206 .

[0057] The example of FIG. 5 may be extended to additional input frames corresponding to different focal lengths. For example, processing of image frames 210 and 220 may be triggered by detection of two peaks in the depth data histogram that exceed a threshold difference. Upon detection of a third peak in the depth data histogram, a third image frame of the scene may be captured. For example, the scene may include a third object at a different focal length than objects 204 and 206. The third image frame may be captured at a focal length that is approximately the same as or the same as the focal length of the third object such that the third object is in focus in the third image frame. The in-focus portion of the third image frame may be segmented and merged with the first input frame, similar to the segmentation 510 and merging 516 of object 206 in segment 512.

[0058] One exemplary method of performing the segmentation 510 operation in the embodiment of FIG. 5 or other embodiments of the present disclosure may include segmentation based on a contour map. FIG. 6 is a flow chart illustrating segmentation of image data using a contour map according to some embodiments of the present disclosure. The segmentation method 600 may begin with receiving first depth data corresponding to a first image frame at block 602 and receiving second depth data corresponding to a second image frame at block 604. At block 606, a first depth map contour and a second depth map contour may be extracted from the first depth data and the second depth data, respectively. The contour extraction of block 606 may include applying a thresholding operation to each of the first and second depth data to determine a binarization map.

[0059] In block 608, the first and second depth map contours are merged. The merge of block 608 may include combining the two binarized depth maps of block 606 to expand the extracted contour of the foreground portion or other portion of the pixel, which may avoid artifacts during other operations in the merging logic. In block 610, a portion of the pixel may be segmented from the first image data of the first image frame based on the merged depth map contour of block 608. The merge of the two depth maps may be based on an AND operation that receives as input the two segmented depths binarized as 255 (1) and 0, where "1" may indicate a white area (corresponding to background) and "0" may indicate a black area (corresponding to foreground or portrait). When the corresponding pixels from the two depth maps are all background and the other areas are merged as foreground black areas, the output of the AND operation of the union may be "1". The segmented portion of the pixel may correspond to segment 512 of FIG. 5. The segmentation may be based on refined depth map contours generated from blocks 606 and 608, and / or other operations on the depth data.

[0060] One exemplary method of performing the merging 516 operation in the embodiment of FIG. 5 or other embodiments of the present disclosure may include a registration operation that allows data from two or more input image frames to be aligned to identify corresponding pixels between the two image frames. FIG. 7 is a flow chart illustrating merging image data to determine a fused output image frame based on weighting values ​​according to some embodiments of the present disclosure. The merging method 700 may begin at block 702 by registering the second image data from the second image frame to the first image data from the first image frame using a homography matrix. In some embodiments, the second image data may be only a portion of the second image frame, such as segment 512 in the embodiment of FIG. 5. In some embodiments, the second image data may be the entire data set of the second image frame. The registration of the first image data to the second image data may correlate corresponding pixels of the second image data with pixels of the first image data. Registration may be useful, for example, when an object in a scene moves within a frame from the capture of the first image data to the capture of the second image data. At block 704, a segmentation map is generated to identify foreground portions of pixels or other portions of pixels based on the registration of block 702. At block 706, the first image data is merged with the second image data based on the segmentation map.

[0061] One exemplary merging operation of block 706 is illustrated in blocks 712, 714, and 716. In determining the output image frame, first image data corresponding to the first set of pixels is copied from the first portion of the first scene to the output image frame in block 712. This first image data may correspond to regions of the input image frame 210 that are, for example, outside of the segment 512, such as, for example, pixels in the background and / or pixels focused around the object 206. In block 714, second image data corresponding to the second set of pixels identified as the fragment 512 is copied to the output image frame. The copying of data in blocks 712 and 714 may generate an output image frame having a merged focus region from the first and second input image frames. However, some artifacts may remain at the boundaries of the first and second sets of pixels and the surrounding pixels.

[0062] If it is desired to remove the artifacts, additional processing may be performed to reduce or remove the artifacts. For example, the third image data corresponding to the pixels at or near the boundary may be determined based on the first image data, the second image data, and the weighting value. In some embodiments, the third image data may be calculated as a blend value for each pixel at or near the boundary by combining corresponding pixel values ​​from the first image data and the second image data according to the weighting value. For example, the third image data may be calculated as a blend value for each pixel at or near the boundary by combining corresponding pixel values ​​from the first image data and the second image data according to the weighting value. b p b = ∝ p 1 +(1-∝)p 2 p 1 is the value from the corresponding pixel in the first image data, and p 2 is a value from a corresponding pixel in the second image data, and α is a weighting value. bmay be calculated based on the corresponding pixel from the first image data and the second image data including pixels around the matched pixel, which may add blurring or seam connectivity at the boundary that makes the fusion result look more natural. For example, for a boundary pixel p at (x,y), b may be determined based on pixel (x,y) from the first image data, pixel (x,y) from the second image data, pixel (x-1,y) from the first image data, pixel (x+1,y) from the first image data, pixel (x,y-1) from the first image data, pixel (x,y+1) from the first image data, pixel (x-1,y) from the second image data, pixel (x+1,y) from the second image data, pixel (x,y-1) from the second image data, and / or pixel (x,y+1) from the second image data. In calculating the third image data, each of the pixels from the first image data may be multiplied by a weighting value α and each of the pixels from the second image data may be multiplied by (1-α).

[0063] The third image data determined at block 716 may be copied to an output image frame and used to overwrite the first or second image data that would otherwise be included in the output image frame. In some embodiments, the output image frame determined from these techniques may include a first region that focuses on the foreground by copying pixel values ​​from a foreground-focused input image frame, a second region that focuses on the background by copying pixel values ​​from a background-focused input image frame, and a third region where pixel values ​​from the foreground-focused and background-focused input image frames are blended.

[0064] The weighting value α may be determined based on gradient maps from the first and second input image frames in some embodiments. One method of determining the weighting value is illustrated in FIG. 8. FIG. 8 is a flow chart illustrating generation of weighting values ​​based on gradient maps for fusing two image frames according to some embodiments of the present disclosure. At block 802, a first gradient map is determined based on the first image data. At block 804, a second gradient map is determined based on the second image data. One method for determining the gradient map from the image data, such as at block 802 and / or block 804, includes applying a Laplacian filter and / or a box filter to determine the gradient map of the foreground image and / or the background image.

[0065] At block 806, a weighting value for a pixel of the output image frame is determined based on the first gradient map and the second gradient map. The weighting value may be determined from the gradient map, for example, by normalizing the difference between the first and second gradient map values ​​to a value between [0...1] or another normalized value such as [0...255]. Alternatively, the weighting value may be a binary value such as 0 or 1 indicating whether the weighting value is a normalized value from the first gradient map or a normalized value from the second gradient map. The binary value may indicate whether to use image data from the first image data or image data from the second image data by effectively weighting the contribution from one of the sets of image data as zero. This binarization may select a value for a pixel of the third set of pixels from the corresponding pixel in the first set of pixels or the corresponding pixel in the second set of pixels based on whether the gradient at the corresponding pixel in the first image data is greater than the gradient at the corresponding pixel in the second image data. That is, the binarization may select data for a pixel from either the first image data or the second image data based on having the highest rate of change in the pixel's neighborhood. In another example, the weighting value may be determined based on a combination of values ​​in the gradient map and the distance of those values ​​from the pixel.

[0066] In some embodiments, multiple weighting values ​​may be used in determining the output image frame pixel value. For example, pixel Y in the output image frame may be determined by the following equation: Y=(W 0 *W a *Y(FG))+(W 1 *W b *Y(BG) In the formula, W 0 denotes the normalized gradient of the foreground pixels (e.g., between "0" and "1"), and W 1 denotes the normalized gradient of the background pixels (e.g., between "0" and "1"), and W ais proportional to the distance between the fused pixel in the output image frame and the segmentation boundary, and W b is proportional to the distance between the fusion pixel and the segmentation boundary. The fusion process can be applied to the joint depth map contour between the foreground and background determined by a threshold number of pixels. 0 , W 1 , W a , and W b may be determined according to the following formula:

[0067]

number

[0068] where Dist(P) is the distance of pixel P from the boundary, Grad(BG) is the gradient map value at pixel P from the background image frame, Grad(FG) is the gradient map value at pixel P from the foreground image frame, and Threshold may be a value such as 20 pixels that corresponds to the width of the fused region along with the segmented combined foreground contour. The pixels on the other two regions, foreground and background, may be copied from the original first and second image frames without blending. The weights W in these exemplary formulas a and W b The calculation of may be based on other formulas. Other exemplary formulas are: for a nearby fused pixel in the foreground that is similar to a connected foreground pixel, W a As W approaches 1 and Wb approaches zero, and when the fused pixel is a neighbor of the background pixel on the fused region, W b approaches 1, and W a Weights for the fusion process can be determined that are normalized between 0 and 1 so that σ approaches zero.

[0069] Methods of processing image frames captured from an image capture device described in embodiments of the present disclosure may be used to obtain improved images of a scene and objects within the scene. For example, a method for fusing image frames may obtain an image of a scene with a greater depth of focus in which multiple objects at different distances from the image capture device are in focus. One advantage of such a fusion process is that fusion may enable the use of image capture devices with larger apertures and capture devices with shorter depth of focus. These larger aperture lenses, such as lenses with apertures greater than f / 2.2 (including apertures of f / 2.0, f / 1.8, f / 1.4, and / or f / 1.2), may provide improved image quality in certain situations, such as low light situations, with loss of depth of focus. The fusion process disclosed in embodiments herein may enable obtaining improved image quality with reduced or no loss of depth of focus, for example, by generating an output image frame that includes the object of interest in focus. This may be particularly desirable when capturing a scene that includes multiple people of interest, as the fusion logic described herein may generate an image in which many or all of the people are in focus. Additionally, the use of large aperture lenses produces images with blurred backgrounds, sometimes referred to as bokeh blur. Fusing multiple images from larger aperture lenses can produce images in which multiple objects are in focus, including the blurred background desired by many photographers.

[0070] The fusion logic in some embodiments of the present disclosure may be triggered under certain conditions, which reduces unnecessary processing of images. Reducing the amount of processing may be particularly beneficial for mobile devices that operate on limited power sources such as batteries. The fusion logic may trigger further image capture and processing and merging of multiple images based on criteria that indicate the presence of multiple objects of interest in a first captured image of a scene. For example, when a histogram of depth data from a scene indicates that there are multiple objects of interest, such as when there are histogram peaks separated by a threshold value that is greater than the predicted focal depth for the current camera configuration, the fusion logic may command the capture of additional images at additional focal distances determined using a combination of the depth data of the scene and / or an autofocus system. If these criteria are not met, the fusion logic does not trigger the capture of additional images and subsequent processing of those images in order to reduce the number of operations performed by the image capture device and associated processing and memory circuitry.

[0071] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0072] The components, functional blocks, and modules described herein with respect to Figure 1 include, among other examples, processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software code, firmware code, etc., or any combination thereof. Additionally, features discussed herein may be implemented via dedicated processor circuitry, via executable instructions, or by a combination thereof.

[0073] Moreover, those skilled in the art will appreciate that the various exemplary logic blocks, modules, circuits, and algorithm steps described herein in connection with the present disclosure may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may realize the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Those skilled in the art will also readily appreciate that the order or combination of components, methods, or interactions described herein are merely examples, and that the components, methods, or interactions of various aspects of the present disclosure may be combined or performed in ways other than those shown and described herein.

[0074] The various example logic, logic blocks, modules, circuits, and algorithmic processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. Interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various example components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system.

[0075] The hardware and data processing devices used to implement the various example logic, logic blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed using general purpose single-chip or general purpose multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry specific to a given function.

[0076] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, or any combination thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein may also be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus.

[0077] If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code. The processes of the methods or algorithms disclosed herein may be implemented in processor-executable software modules that may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that may enable a computer program to be transferred from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection may be properly referred to as a computer-readable medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, the operations of a method or algorithm may reside on machine-readable and computer-readable media, which may be incorporated into a computer program product as one or any combination or set of code and instructions.

[0078] Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to several other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the present disclosure, the principles and novel features disclosed herein.

[0079] In addition, those skilled in the art will readily appreciate that the terms "upper" and "lower" may be used to facilitate description of the figures, and refer to relative positions that correspond to the orientation of the figure on a suitably oriented page, and may not reflect the proper orientation of any implemented device.

[0080] Also, some features described in the context of separate implementations herein may be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any suitable subcombination. Moreover, although features may be described above as acting in some combinations and may initially be claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0081] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequentially shown, or that all of the illustrated operations be performed, to achieve desirable results. Additionally, the figures may generally depict one or more exemplary processes in the form of a flow diagram. However, other operations not shown may be incorporated into the illustrative process depicted in the schematic. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. In some circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. In addition, some other implementations fall within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0082] The term "or" as used herein, including in the claims, when used in a list of two or more items means that any one of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if an apparatus is described as including components A, B, or C, the apparatus may include only A, only B, only C, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. Also, as used herein, including in the claims, in a list of items ending with "at least one of," "or" indicates a disjunctive list, such as, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C), or any of these in any combination thereof. The term "substantially" is defined as being largely, but not necessarily entirely, as understood by one of ordinary skill in the art (and includes what is specified, e.g., substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel). In any disclosed implementation, the term "substantially" may be replaced with "within [a percentage] of" what is specified, where the percentage includes 0.1, 1, 5, or 10 percent.

[0083] The above description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [Explanation of symbols]

[0084] 100 devices 101 first image sensor 102 second image sensor 104 processors 106 Memory 108 Instructions 112 Image Signal Processor 114 Display 116 Input / Output (I / O) Components 118 Power supply 130 First Camera 131 First Lens 132 Second Lens 140 Second Camera 141 Autofocus Logic 142 Autofocus Logic 144 Fusion Nodes 202 scenes 204 Objects 206 Objects 208 Objects 210 First Image Frame 220 Second Image Frame 240 Fusion Logic 250 output image frames 300 Depth Data 302 Histogram Peak 304 Histogram Peak 510 Segmentation 512 segments 516 Merge 700 Merge Method

Claims

1. receiving a first image frame representing a scene captured at a first focal length; determining whether a difference between a first depth corresponding to the first focal length for a first portion of the scene and a second depth for a second portion of the scene exceeds a threshold; When the difference exceeds the threshold value, receiving a second image frame representing the scene captured at a second focal length different from the first focal length; determining an output image frame based on the first image frame and the second image frame; The method includes:

2. When the difference exceeds the threshold value, segmenting the output image frame into the first portion and the second portion; applying a blurring algorithm to pixels in the second portion but not to the first portion; The method of claim 1, further comprising:

3. determining an output image frame, determining a segment of the second image frame that corresponds to the second depth; merging the segment with the first image frame to obtain the output image frame; 2. The method of claim 1, comprising:

4. A device comprising: A processor; a memory coupled to the processor storing instructions that, when executed by the processor, cause the device to perform operations, the operations including: receiving a first image frame representing a scene captured at a first focal length; determining whether a difference between a first depth corresponding to the first focal length for a first portion of the scene and a second depth for a second portion of the scene exceeds a threshold; When the difference exceeds the threshold value, receiving a second image frame representing the scene captured at a second focal length different from the first focal length; determining an output image frame based on the first image frame and the second image frame; Including memory and 16. A device comprising:

5. The instructions cause the device to: segmenting the output image frame into the first portion and the second portion; applying a blurring algorithm to pixels in the second portion but not to pixels in the first portion; The device of claim 4 , further comprising:

6. determining the output image frame determining a segment of the second image frame that corresponds to the second depth; and merging the segment with the first image frame to obtain the output image frame; The device of claim 4 , comprising:

7. Merging the segment with the first image frame, copying first image data corresponding to a first set of pixels from a first portion of the first image frame to the output image frame; copying second image data corresponding to a second set of pixels from the segment to the output image frame; determining third image data for the output image frame corresponding to a third set of pixels within a boundary region between the first set of pixels and the second set of pixels, the third image data being determined based on the first image frame, the second image frame and a weighting value; The device of claim 6, comprising:

8. The instructions cause the device to: a contrast value in the first image frame corresponding to a pixel in the third set of pixels; a contrast value in the second image frame corresponding to a pixel in the third set of pixels; a distance value corresponding to a distance from a pixel in the third set of pixels to a pixel in at least one of the first set of pixels or the second set of pixels; or a first gradient of the first image frame and a second gradient of the second image frame; determining the weighting value based on at least one of The device of claim 7 , configured to perform operations including:

9. The device described in claim 7, wherein the weighting value corresponds to selecting a value for a pixel in the third set of pixels from the first corresponding pixel in the first set of pixels or the second corresponding pixel in the second set of pixels based on whether a gradient at the first corresponding pixel is greater than that of the second corresponding pixel.

10. determining the first depth includes determining a depth corresponding to a first peak in a histogram based on the depth data; determining the second depth includes determining a depth corresponding to a second peak of the histogram; Optionally, the instructions cause the device to perform an operation including sending a command to capture the second image frame at the second focal length corresponding to the second depth; The device of claim 4 , wherein the step of sending the command includes seeding an autofocus algorithm with the second focal length.

11. The instructions cause the device to: determining a third depth of a third portion of the first image frame; and determining whether a second difference between the second depth and the third depth exceeds a second threshold; receiving a third image frame at a third focal length different from the second focal length when the second difference exceeds the second threshold; determining an output image frame based on the first image frame, the second image frame, and the third image frame; The device of claim 4 , further comprising:

12. The device of claim 4 , further comprising an image sensor and a lens coupled to the image sensor, the image sensor having an aperture greater than f / 2.

2.

13. The device of claim 4 , wherein the receiving the second image frame comprises receiving the second image frame captured by a second sensor.

14. The instructions cause the device to: determining a first object of interest and a second object of interest in the first image frame based on an object detection algorithm; determining a depth of the first object of interest as the first depth; determining a depth of the second object of interest as the second depth; The device of claim 4 , further comprising:

15. A non-transitory computer readable medium storing instructions which, when executed by a processor of a device, cause the device to perform the method of any one of claims 1 to 3.

Images