Image processing for blocking exposure

Abstract

The present disclosure provides systems, methods, and devices for image signal processing that support high dynamic range (HDR) image processing of image frames having time-aligned centers to reduce artifacts that result from fusing image frames having different time centers. In some aspects, a method of image processing includes capturing three or more image frames having at least two different exposure lengths. The three or more image frames are processed to obtain two image frames having a temporal alignment center, and the two image frames are processed in HDR fusion logic to obtain an output HDR image frame. Other aspects and features are also claimed and described.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 462,199, filed September 6, 2023, entitled “IMAGE PROCESSING FOR EXPOSUREBRACKETING,” which is expressly incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates generally to image processing, and more specifically to the processing of bracketed exposure image frames. Some features can be enabled in low-light multi-frame imaging, high-motion photography, or other applications of bracketed exposure and provide improved image processing, including improved image dynamic range, while reducing motion blur caused by motion during image capture. Background Technology

[0003] An image capture device is a device capable of capturing one or more digital images (whether still images for photographs or sequences of images for video). Capture devices can be integrated into a variety of devices. For example, an image capture device may include a standalone digital camera or digital video camera, a wireless communication device with a camera (such as a mobile phone, cellular, or satellite radio phone), a personal digital assistant (PDA), a panel or tablet device, a gaming device, a computing device (such as a webcam, video surveillance camera), or other devices with digital imaging or video capabilities.

[0004] Dynamic range can be important for image quality when using image capture devices to capture representations of scenes with a wide color gamut. Conventional image sensors have a limited dynamic range, which may be smaller than that of the human eye. Dynamic range can refer to the range of light between the bright and dark areas of an image. Conventional image sensors can increase exposure time to improve detail in the dark areas of an image, at the cost of saturating the bright areas. Alternatively, conventional image sensors can decrease exposure time to improve detail in the bright areas of an image, at the cost of losing detail in the dark areas. Summary of the Invention

[0005] The following summary outlines some aspects of this disclosure to provide a basic understanding of the techniques discussed. This summary is not an exhaustive overview of all the intended features of this disclosure, nor is it intended to identify key or essential elements of all aspects of this disclosure, nor to define the scope of any or all aspects of this disclosure. The sole purpose of this summary is to present, in a general form, some concepts of one or more aspects of this disclosure as a prelude to the more detailed description that follows.

[0006] High dynamic range (HDR) photography improves upon photography using conventional image sensors by combining multiple recorded representations of a scene (e.g., image frames) from the image sensor. HDR photography combines image frames of the same scene captured at different exposure lengths. Different exposure lengths capture different details in the scene. Long exposure image frames capture details in the low-light areas of the scene. Short exposure image frames capture details in the highlight areas of the scene.

[0007] Combining long-exposure and short-exposure image frames typically involves aligning the image frames and then fusing them to obtain a single output image frame with a higher dynamic range than either the long or short frame alone. Alignment allows the fusing operation to merge corresponding portions of the image frame. Alignment reduces artifacts caused by capturing different image frames at different times, during which the image capture device may have moved or objects in the scene may have moved. For example, global motion can cause motion blur in long-exposure frames, which reduces alignment accuracy. Similarly, local motion can cause uneven image distortion. The longer the duration of the long-exposure frame capture, the more difficult the alignment becomes, because longer exposures result in a greater temporal distance between the long and short image frames.

[0008] In some aspects of this disclosure, HDR image capture is performed by processing temporally aligned image frames. Arranging the capture of image data to temporally align the image frames reduces or eliminates the alignment performed on the image frames when fusing them to determine the output HDR image frames. Temporal alignment of the image frames results in spatially aligned image frames. Temporally aligned image frames can be obtained by assuming approximately linear motion within the exposure time, such that the midpoint of the exposure time coincides with the spatial midpoint of any scene motion. Temporally aligned image frames can also be obtained by assuming that the exposure is a linear operation, such that all pixel values ​​are assumed to be in a linear domain and proportional to the exposure time. Based on these assumptions, the capture of image frames using one of many different HDR techniques can be temporally arranged such that the resulting image frames are spatially aligned, and can be processed for the purpose of fusing image frames without further alignment (although other alignment operations may be performed).

[0009] Using time-aligned image frames when acquiring output HDR image frames may include, for interlaced HDR (sHDR) photography, setting successive exposures with medium, short, and medium exposure times, and summing these three exposures to determine a long image frame. Image frames with short, medium, and long exposure times may be combined to produce an HDR output image frame. Using time-aligned image frames when acquiring output HDR image frames may include, for quad-in-one HDR (qHDR) photography, setting medium and short exposure frames and combining these images to produce an HDR output image frame. In other embodiments, the techniques disclosed herein can be applied to other HDR capture techniques.

[0010] In one aspect of this disclosure, a method for image processing includes: receiving image data by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration; determining first summed image data by the at least one processor by summing corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and determining a first output image frame by the at least one processor by combining the first summed image data with the second image data.

[0011] In an additional aspect of this disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving image data comprising first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration; determining first summed image data by summing corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and determining a first output image frame by combining the first summed image data with the second image data.

[0012] In an additional aspect of this disclosure, an apparatus includes: components for receiving image data by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration; components for determining first summed image data by the at least one processor by summing corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and components for determining a first output image frame by the at least one processor by combining the first summed image data with the second image data. The components for determining may include at least one processor, such as an image signal processor, CPU, GPU, DSP, and / or NSP. The components for receiving may include at least one processor (such as those described herein) and / or network communication components.

[0013] In an additional aspect of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. These operations include: receiving image data by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration; determining first summed image data by the at least one processor by summing corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and determining a first output image frame by the at least one processor by combining the first summed image data with the second image data.

[0014] In another aspect of this disclosure, a method for image processing includes: receiving image data by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration; determining fourth image data by the at least one processor by subtracting the corresponding pixel intensity value of the third image data from the second image data; and determining a first output image frame by the at least one processor by combining the fourth image data with the first image data.

[0015] In another aspect of this disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration; determining fourth image data by subtracting corresponding pixel intensity values ​​of the third image data from the second image data; and determining a first output image frame by combining the fourth image data with the first image data.

[0016] In another aspect of this disclosure, an apparatus includes: a component for receiving image data by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration; a component for determining fourth image data by the at least one processor by subtracting corresponding pixel intensity values ​​of the third image data from the second image data; and a component for determining a first output image frame by the at least one processor by combining the fourth image data with the first image data.

[0017] In another aspect of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. These operations include: receiving image data by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration; determining fourth image data by the at least one processor by subtracting the corresponding pixel intensity value of the third image data from the second image data; and determining a first output image frame by the at least one processor by combining the fourth image data with the first image data.

[0018] The image processing methods described herein can be performed by an image capture device and / or on image data captured by one or more image capture devices. An image capture device (a device capable of capturing one or more digital images, whether still photographs or video sequences) can be incorporated into a variety of devices. By way of example, an image capture device may include a standalone digital camera or digital video camera, a wireless communication device equipped with a camera (such as a mobile phone, cellular, or satellite radio phone), a personal digital assistant (PDA), a panel or tablet device, a gaming device, a computing device (such as a webcam, video surveillance camera), or other devices with digital imaging or video capabilities.

[0019] The image processing techniques described herein may relate to a digital camera having an image sensor and processing circuitry (e.g., an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), or a central processing unit (CPU)). An image signal processor (ISP) may include one or more of these processing circuits and is configured to perform operations to acquire image data for processing according to the image processing techniques described herein and / or those involved in the image processing techniques described herein. An ISP may be configured to control the capture of image frames from one or more image sensors and to determine one or more image frames from said one or more image sensors to generate a view of a scene in an output image frame. The output image frame may be part of a sequence of image frames forming a video sequence. The video sequence may include additional image frames received from the image sensor or other image sensors.

[0020] In an example application, an image signal processor (ISP) may receive instructions for capturing a sequence of image frames in response to the loading of software, such as a camera application, to generate a preview display from an image capture device. The ISP may be configured to generate a single output image frame stream based on image frames received from one or more image sensors. The single output image frame stream may include raw image data from the image sensors, merged image data from the image sensors, or corrected image data processed by one or more algorithms within the ISP. For example, image frames may be processed by an image post-processing engine (IPE) and / or other image processing circuitry to process the image frames obtained from the image sensors (which may have undergone some processing before being output to the ISP), thereby performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc. The output image frames from the ISP may be stored in memory and retrieved by an application processor executing the camera application, which may perform further processing on the output image frames to adjust their appearance and reproduce them on a display for user viewing.

[0021] After an image signal processor and / or application processor (such as the image processing techniques described in the various embodiments herein) determines an output image frame representing a scene, the output image frame may be displayed on a device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, transmitted over a network, and / or printed to an output medium. For example, an image signal processor (ISP) may be configured to acquire input frames of image data (e.g., pixel values) from one or more image sensors and subsequently generate corresponding output image frames (e.g., preview display frames, still image captures, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output image frames to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., autofocus (AF), auto white balance (AWB), and auto exposure control (AEC)), to generate video files via the output frames, to configure frames for display, to configure frames for storage, to transmit frames via a network connection, etc. Generally, an image signal processor (ISP) can obtain incoming frames from one or more image sensors, generate an output frame stream, and output the output frame stream to various output destinations.

[0022] In some aspects, output image frames can be generated by combining various aspects of the image correction disclosed herein with other computational photographic techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, the first and second image frames are captured using different exposure times, different apertures, different lenses, and / or other characteristics that can result in improved dynamic range of the fused image when combining the two image frames. In some aspects, the method can be performed for MFNR photography, wherein the first and second image frames are captured using the same or different exposure times, and the first and second image frames are fused to generate a corrected first image frame that has reduced noise compared to the captured first image frame.

[0023] In some aspects, the device may include an image signal processor or processor (e.g., an application processor) that includes specific functionalities for camera control and / or processing, such as enabling or disabling the merging module or otherwise controlling aspects of image correction. The methods and techniques described herein may be performed entirely by the image signal processor or processor, or the various operations may be separated between the image signal processor and the processor, and in some aspects across additional processors.

[0024] The device may include one, two, or more image sensors, such as a first image sensor. When multiple image sensors are present, their configurations may differ. For example, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a long-range image sensor. In another example, the first sensor is configured to acquire an image through a first lens having a first optical axis, and the second sensor is configured to acquire an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. Any of these or other configurations may be part of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located at offset positions on the front or rear of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image processing techniques described herein can be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0025] In an additional aspect of this disclosure, an apparatus configured for image processing and / or image capture is disclosed. The apparatus includes components for capturing image frames. The apparatus also includes one or more components for capturing data representing a scene, such as image sensors (including charge-coupled device (CCD), Bayer filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, complementary metal-oxide-semiconductor (CMOS) sensors) and time-of-flight detectors. The apparatus may further include components for focusing and / or directing light onto one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0026] Other aspects, features, and specific embodiments will become apparent to those skilled in the art when they review the following description of particular exemplary aspects in conjunction with the accompanying drawings. Although features may be discussed hereinafter with reference to certain aspects and drawings, various aspects may include one or more of the advantageous features discussed herein. In other words, while one or more aspects may be discussed having certain advantageous features, one or more such features may also be used depending on the various aspects. Similarly, although exemplary aspects may be discussed hereinafter as aspects of an apparatus, system, or method, exemplary aspects can be implemented in various apparatuses, systems, and methods.

[0027] This method can be embedded as computer program code in a computer-readable medium, the computer program code including instructions that cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device including: a first network adapter configured to transmit data, such as recorded images or videos or streaming data, via a first network connection among a plurality of network connections; and a processor coupled to the first network adapter and memory. The processor enables the output image frames described herein to be transmitted via a wireless communication network, such as a 5G NR communication network.

[0028] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily used as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and manner of operation) and the associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustrative and descriptive purposes and not as a definition of limitation of the claims.

[0029] While aspects and implementations are described herein by way of example, those skilled in the art will understand that additional implementations and use cases may arise in many different arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and package arrangements. For example, aspects and / or implementations may be via integrated chip implementations and other devices based on non-modular components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchasing devices, medical devices, AI-enabled devices, etc.). While some examples may or may not specifically point to use cases or applications, applicability to various types of the described innovations is possible. The scope of implementations can range from chip-level or modular components to non-modular, non-chip-level implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more aspects of the described innovations. In some practical settings, devices incorporating the described aspects and features may also necessarily include additional components and features for implementing and practicing the claimed and described aspects. For example, the transmission and reception of wireless signals necessarily involve multiple components (e.g., hardware components, including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.) for analog and digital purposes. The innovations described herein are intended to be implemented in a variety of devices, chip-level components, systems, distributed arrangements, end-user equipment, etc., with different sizes, shapes, and constructions. Attached Figure Description

[0030] A further understanding of the nature and advantages of this disclosure can be achieved by referring to the following figures. In the figures, similar components or features may have the same reference numerals. Furthermore, various components of the same type can be distinguished by adding a dash after the reference numerals and a second reference numeral for differentiation between similar components. If only the first reference numeral is used in the specification, the description applies to any one of the similar components having the same first reference numeral, regardless of the second reference numeral.

[0031] Figure 1 A block diagram of an example device for performing image capture from one or more image sensors is shown.

[0032] Figure 2 This is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0033] Figure 3 A flowchart of an example method for processing image data to obtain high dynamic range (HDR) image frames according to some embodiments of the present disclosure is shown.

[0034] Figure 4 This is a timing diagram of example image capture operations based on various aspects of this disclosure.

[0035] Figure 5 This is a timing diagram of an example image capture operation with alternating medium and short image frames according to various aspects of this disclosure.

[0036] Figure 6 This is a timing diagram of an example image capture operation with a group of triples of image frames according to various aspects of this disclosure.

[0037] Figure 7 This is a timing diagram of an example image capture operation with an exposure quintuple according to various aspects of this disclosure.

[0038] Figure 8 This is a timing diagram of an example image capture operation with flash operation according to various aspects of this disclosure.

[0039] Figure 9 A flowchart illustrating an example method for processing image data to obtain high dynamic range (HDR) image frames, according to some aspects of this disclosure, is shown.

[0040] Figure 10 This is a timing diagram of example image capture operations based on various aspects of this disclosure.

[0041] Figure 11 This is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0042] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0043] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of this disclosure. Rather, the detailed description includes specific details for providing a thorough understanding of the subject matter of the invention. It will be apparent to those skilled in the art that these specific details are not necessary in every situation, and in some cases, well-known structures and components are shown in block diagram form for clarity of presentation.

[0044] This disclosure provides systems, apparatus, methods, and computer-readable media that support high dynamic range (HDR) image processing on image frames with temporal alignment centers to reduce artifacts resulting from fusing image frames with different temporal centers. In some aspects, an image processing method includes capturing three or more image frames having at least two different exposure lengths. The three or more image frames are processed to obtain two image frames with temporal alignment centers, and these two image frames are processed in HDR fusion logic to obtain an output HDR image frame. Other aspects and features are also claimed and described. Image frames can be captured and / or processed in a manner that produces temporally aligned image frames, such that an HDR fusion process combining two image frames operates on image frames having corresponding pixel values. Since the image frames have the same temporal center, alignment processing may not be performed when fusing the two image frames.

[0045] Specific embodiments of the subject matter described in this disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, this disclosure provides techniques for improving HDR image processing techniques. Specifically, capturing image data with temporally aligned images reduces the amount of processing during HDR image processing because certain operations (such as alignment operations including cropping and / or warping) can be omitted from the pipeline between the reception of image data and the fusion of two image frames with different exposure durations. The reduced computational requirements for processing HDR image frames can be particularly advantageous for video enabling high frame rate HDR, where the processor has significant processing demands to provide real-time processing. Furthermore, temporal alignment reduces artifacts formed in image frames by such alignment operations.

[0046] In the description of the embodiments herein, numerous specific details (such as examples of specific components, circuits, and processes) are set forth to provide a thorough understanding of this disclosure. As used herein, the term "coupled" means a direct connection or a connection via one or more intermediate components or circuits. Furthermore, specific terminology is set forth in the following description and for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that practicing the teachings disclosed herein may not require these specific details. In other instances, known circuits and devices are illustrated in block diagram form to avoid obscuring the teachings of this disclosure.

[0047] Certain portions of the following detailed description are presented using other symbolic representations of programs, logic blocks, processes, and data bit operations within computer memory. In this disclosure, programs, logic blocks, processes, etc., are conceived as a self-consistent sequence of steps or instructions that produce a desired result. These steps are those that require physical manipulation of physical quantities. Although not strictly necessary, these physical quantities typically take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated within a computer system.

[0048] Example devices (such as smartphones) for capturing image frames using one or more image sensors may include a configuration of one, two, three, four, or more camera modules on the rear side (e.g., the side opposite the main user display) and / or the front side (e.g., the same side as the main user display). These devices may include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuitry for processing the images captured by the image sensors. The one or more image signal processors (ISPs) may store the output image frames in memory (e.g., via a bus) and / or provide the output image frames to processing circuitry (e.g., an application processor). The processing circuitry may perform further processing, such as encoding, storing, transmitting, or other manipulations of the output image frames.

[0049] As used herein, a camera module may include an image sensor and certain other components coupled to the image sensor for acquiring a representation of a scene in image data comprising image frames. For example, a camera module may include other components of the camera, including a shutter, buffer, or additional readout circuitry for accessing individual pixels of the image sensor. In some embodiments, a camera module may include one or more components including an image sensor housed in a single package having an interface configured to couple the camera module to an image signal processor or other processor via a bus.

[0050] Figure 1 A block diagram of a device 100 for performing image capture from one or more image sensors is shown. Device 100 may include or be otherwise coupled to an image signal processor (e.g., ISP 112) for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some specific embodiments, device 100 may also include or be coupled to a processor 104 and a memory 106 storing instructions 108 (e.g., memory storing processor-readable code or a non-transitory computer-readable medium storing instructions). Device 100 may also include or be coupled to a display 114 and component 116. Component 116 may be used for user interaction, such as a touchscreen interface and / or physical buttons.

[0051] Component 116 may also include network interfaces for communicating with other devices, including a wide area network (WAN) adapter (e.g., WAN adapter 152), a local area network (LAN) adapter (e.g., LAN adapter 153), and / or a personal area network (PAN) adapter (e.g., PAN adapter 154). WAN adapter 152 may be a 4G LTE or 5G NR wireless network adapter. LAN adapter 153 may be an IEEE 802.11 WiFi wireless network adapter. PAN adapter 154 may be a Bluetooth wireless network adapter. Each of WAN adapter 152, LAN adapter 153, and / or PAN adapter 154 may be coupled to an antenna comprising multiple antennas configured for main and diversity reception and / or configured to receive a specific frequency band. In some embodiments, the antennas may be shared by WAN adapter 152, LAN adapter 153, and / or PAN adapter 154 for communication on different networks. In some implementations, WAN adapter 152, LAN adapter 153 and / or PAN adapter 154 may share circuitry and / or be packaged together, such as when LAN adapter 153 and PAN adapter 154 are packaged as a single integrated circuit (IC).

[0052] Device 100 may also include or be coupled to a power source 118 for use with device 100, such as a battery or an adapter for coupling device 100 to an energy source. Device 100 may also include or be coupled to... Figure 1 Additional features or components not shown. In one example, a wireless interface that may include multiple transceivers and a baseband processor in a radio frequency front-end (RFFE) may be coupled to or included in the WAN adapter 152 for use in a wireless communication device. In another example, an analog front-end (AFE) for converting analog image data to digital image data may be coupled between the first image sensor 101 or the second image sensor 102 and the processing circuitry in the device 100. In some embodiments, the AFE may be embedded in the ISP 112.

[0053] The device may include or be coupled to a sensor hub 150, which interfaces with sensors to receive data about the movement of device 100, data about the environment surrounding device 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, a device configured to measure acceleration, which can also be used to determine the speed and distance of travel by appropriately integrating the measured acceleration. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub. In another example, the non-camera sensor may be a Global Positioning System (GPS) receiver, a device used to process satellite signals, such as through triangulation and other techniques, to determine the position of device 100. Position can be tracked over time to determine additional motion information, such as velocity and acceleration. Data from one or more sensors may be accumulated by the sensor hub 150 into motion data. One or more of acceleration, velocity, and / or distance may be included in the motion data provided by sensor hub 150 to other components of device 100 (including ISP 112 and / or processor 104).

[0054] The ISP 112 can receive captured image data. In one embodiment, a local bus connection couples the ISP 112 to the first image sensor 101 and the second image sensor 102 of the first camera 103 and the second camera 105, respectively. In another embodiment, a wired interface couples the ISP 112 to an external image sensor. In yet another embodiment, a wireless interface couples the ISP 112 to either the first image sensor 101 or the second image sensor 102.

[0055] First image sensor 101 and second image sensor 102 are configured to capture image data representing scenes within the fields of view of first camera 103 and second camera 105, respectively. In some embodiments, first camera 103 and / or second camera 105 output analog data converted by an analog front-end (AFE) and / or analog-to-digital converter (ADC) in device 100 or embedded in ISP 112. In some embodiments, first camera 103 and / or second camera 105 output digital data. The digital image data may be formatted into one or more image frames, whether received from first camera 103 and / or second camera 105 or converted from analog data received from first camera 103 and / or second camera 105.

[0056] The first camera 103 may include a first image sensor 101 and a first lens 131. The second camera may include a second image sensor 102 and a second lens 132. Each of the first lens 131 and the second lens 132 may be controlled by an associated autofocus (AF) algorithm (e.g., AF 133) executed in the ISP 112, which adjusts the first lens 131 and the second lens 132 to focus on a specific focal plane located at a specific scene depth. AF 133 may be assisted by depth data received from the depth sensor 140. The first lens 131 and the second lens 132 focus light onto the first image sensor 101 and the second image sensor 102 respectively through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, and / or one or more color filter arrays (CFAs) for filtering light outside a specific frequency range. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. Multiple image sensors may include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, long-range sensors, and ultra-long-range (low FOV) sensors.

[0057] Each of the first camera 103 and the second camera 105 can be configured through hardware configuration and / or software settings to obtain different but overlapping fields of view. In some configurations, the cameras are configured using different lenses with different magnifications, resulting in different fields of view for capturing different representations of the scene. The cameras can be configured such that the UW camera has a larger FOV than the W camera, the W camera has a larger FOV than the T camera, and the T camera has a larger FOV than the UT camera. For example, a camera configured for a wide FOV can capture a field of view in the range of 64 to 84 degrees, a camera configured for an ultra-side FOV can capture a field of view in the range of 100 to 140 degrees, a camera configured for a long-range FOV can capture a field of view in the range of 10 to 30 degrees, and a camera configured for an ultra-long-range FOV can capture a field of view in the range of 1 to 8 degrees.

[0058] In some implementations, one or more of the first camera 103 and / or the second camera 105 may be a variable aperture (VA) camera, wherein the aperture can be adjusted to set a specific aperture size. Example aperture sizes include f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. The variable aperture (VA) camera may have different characteristics that produce different representations of the scene based on the current aperture size. For example, the VA camera may capture image data with a depth of focus (DOF) corresponding to the current aperture size set for the VA camera.

[0059] The ISP 112 processes image frames captured by the first camera 103 and the second camera 105. Although Figure 1 Device 100 is illustrated as including a first camera 103 and a second camera 105, but any number of cameras (e.g., one, two, three, four, five, six, etc.) may be coupled to ISP 112. In some aspects, depth sensors (such as depth sensor 140) may be coupled to ISP 112. The output from depth sensor 140 may be processed in a manner similar to that of the first camera 103 and the second camera 105. Examples of depth sensors 140 include active sensors, including one or more of indirect time-of-flight (iToF), direct time-of-flight (dToF), light detection and ranging (LiDAR), mmWave, radio detection and ranging (radar), and / or hybrid depth sensors (such as structured light sensors). In embodiments without depth sensor 140, similar information about the depth or depth map of an object may be determined based on the parallax between the first camera 103 and the second camera 105, such as by using parallax depth measurement algorithms, stereo depth measurement algorithms, phase detection autofocus (PDAF) sensors, etc. In addition, any number of additional image sensors or image signal processors may be present in device 100.

[0060] In some embodiments, ISP 112 may execute instructions from memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in ISP 112, or instructions provided by processor 104. Additionally or alternatively, ISP 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in this disclosure. For example, ISP 112 may include an image front-end (e.g., IFE 135), an image post-processing engine (e.g., IPE 136), an automatic exposure compensation (AEC) engine (e.g., AEC 134), and / or one or more engines for video analysis (e.g., EVA 137). The image pipeline may be formed by a sequence of one or more of IFE 135, IPE 136, and / or EVA 137. In some embodiments, the image pipeline in ISP 112 may be reconfigured by changing the connections between IFE 135, IPE 136, and / or EVA 137. AF 133, AEC 134, IFE 135, IPE 136 and EVA137 may each include dedicated circuitry and may be embodied as software or firmware executed by ISP 112 and / or a combination of hardware and software or firmware executed on ISP 112.

[0061] Memory 106 may include a non-transient or non-transitory computer-readable medium storing computer-executable instructions (as instructions 108) for performing all or part of one or more of the operations described in this disclosure. Instructions 108 may include a camera application (or other suitable application, such as a messaging application) to be executed by device 100 for taking pictures or videos. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system and applications other than those for image or video generation. A camera application, such as one executed by processor 104, may enable device 100 to record images using the first camera 103 and / or the second camera 105 and the ISP 112.

[0062] In addition to instruction 108, memory 106 may also store image frames. The image frames may be output image frames stored by ISP 112. The output image frames may be accessed by processor 104 for further operation. In some embodiments, device 100 does not include memory 106. For example, device 100 may be circuitry including ISP 112, and the memory may be external to device 100. Device 100 may be coupled to external memory and configured to access that memory to write output image frames for display or long-term storage. In some embodiments, device 100 is a system-on-a-chip (SoC) that integrates ISP 112, processor 104, sensor hub 150, memory 106, and / or component 116 into a single package.

[0063] In some embodiments, at least one of the ISP 112 or processor 104 executes instructions to perform various operations described herein, including HDR image processing. For example, execution of instructions may instruct the ISP 112 to begin or end capturing image frames or sequences of image frames, wherein the capture includes corrections as described in the embodiments herein. In some embodiments, processor 104 may include one or more general-purpose processor cores 104A-N capable of executing instructions to control the operation of the ISP 112. For example, cores 104A-104N may execute a camera application (or other suitable application for generating images or videos) stored in memory 106 that activates or deactivates the ISP 112 to capture image frames and / or controls the ISP 112 when HDR processing is applied to the image frames. The operation of cores 104A-N and ISP 112 may be based on user input. For example, a camera application executing on processor 104 may receive a user command to start a video preview display. Upon receiving the user command, video, including a sequence of image frames, is captured and processed via ISP 112 from first camera 103 and / or second camera 105 for display and / or storage. Image processing, such as that described herein, for determining “output” or “corrected” image frames, may be applied to one or more image frames in the sequence.

[0064] In some implementations, processor 104 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine (such as AI engine 124) or other coprocessor) to offload certain tasks from cores 104A-N. AI engine 124 may be used to offload tasks related to face detection and / or object recognition, performed, for example, using machine learning (ML) or artificial intelligence (AI). AI engine 124 may be referred to as an artificial intelligence processing unit (AI PU). AI engine 124 may include hardware configured to perform and accelerate convolutional operations involved in performing machine learning algorithms, such as by executing predictive models such as artificial neural networks (ANNs) (including multilayer feedforward neural networks (MLFFNNs), recurrent neural networks (RNNs), and / or radial basis functions (RBFs)). The ANN executed by AI engine 124 has access to predefined training weights for performing operations on user data. The ANN may optionally be trained during operation of image capture device 100, such as through reinforcement training, supervised training, and / or unsupervised training. In some other implementations, device 100 does not include processor 104, such as when all the described functionality is configured in ISP 112.

[0065] In some embodiments, display 114 may include one or more suitable displays or screens that allow the user to interact and / or present a preview of an item (such as the output of the first camera 103 and / or the second camera 105) to the user. In some embodiments, display 114 is a touch-sensitive display. Input / output (I / O) components (such as component 116) may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and provide output to the user via display 114. For example, component 116 may include (but is not limited to) a graphical user interface (GUI), keyboard, mouse, microphone, speaker, squeezable bezel, one or more buttons (such as a power button), slider, toggle key, switch, etc.

[0066] Although shown as coupled to each other via processor 104, components (such as processor 104, memory 106, ISP 112, display 114, and component 116) may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. An example of a bus used to interconnect components is a Peripheral Component Interface (PCI) Fast (PCIe) bus.

[0067] Although ISP 112 is illustrated as separate from processor 104, ISP 112 may be the core of processor 104, which is an application processor unit (APU) included in a system-on-a-chip (SoC), or otherwise included in processor 104. While device 100 is referenced in the examples herein to perform aspects of this disclosure, some device components may not be included. Figure 1 The details are shown to prevent obscuring aspects of this disclosure. Additionally, other components, the number of components, or combinations of components may be included in suitable equipment for performing aspects of this disclosure. Therefore, this disclosure is not limited to the configuration of a particular device or component, including device 100.

[0068] Operable Figure 1 An exemplary image capture device is used to obtain improved images by processing image frames to obtain a high dynamic range (HDR) representation of the scene. Figure 2 An example method of operating one or more cameras (such as a first camera 103 and / or a second camera 105) is shown and described below.

[0069] Figure 2 This is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 of system 200 communicates with ISP 112 via a bidirectional bus and / or separate control and data lines. The processor 104 can control a first camera 103 via camera control 210. Camera control 210 may be a camera driver executed by processor 104 for configuring the first camera 103 (such as activating or deactivating image capture, configuring exposure settings, and / or configuring aperture size). Camera control 210 may be managed by a camera application 204 executing on processor 104. Camera application 204 provides user-accessible settings, allowing a user to specify individual camera settings or select a profile with corresponding camera settings. Camera control 210 communicates with the first camera 103 to configure the first camera 103 according to commands received from camera application 204. Camera application 204 may be, for example, a photography application, a document scanning application, a messaging application, or other application that processes image data acquired from the first camera 103. Camera application 204 can configure the first camera 103 for HDR image capture, such as configuring a series of bracketed exposure captures to generate a set of image frames with different exposure durations.

[0070] Camera configuration may include specifying parameters such as frame rate, image resolution, readout duration, exposure level, aspect ratio, aperture size, etc. The first camera 103 may apply the camera configuration and use it to acquire image data representing the scene. In some embodiments, the camera configuration may be adjusted to obtain different representations of the scene. For example, the processor 104 may execute camera application 204 to instruct the first camera 103 to set a first camera configuration via camera control 210, acquire first image data from the first camera 103 operating with the first camera configuration, instruct the first camera 103 to set a second camera configuration, and acquire second image data from the first camera 103 operating with the second camera configuration.

[0071] In some embodiments where the first camera 103 is a variable aperture (VA) camera system, the processor 104 can execute camera application 204 to instruct the first camera 103 to be configured to a first aperture size, acquire first image data from the first camera 103, instruct the first camera 103 to be configured to a second aperture size, and acquire second image data from the first camera 103. The aperture reconfiguration and the acquisition of the first and second image data can occur with little or no change in the scene captured at the first aperture size and the second aperture size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values ​​correspond to smaller aperture sizes, and smaller aperture values ​​correspond to larger aperture sizes. That is, f / 2.0 corresponds to an aperture size larger than f / 8.0.

[0072] Image data received from the first camera 103 can be processed in one or more blocks of the ISP 112 to determine an output image frame 230 that can be stored in memory 106 and / or otherwise provided to the processor 104. The processor 104 can further process the image data to apply effects to the output image frame 230. Effects may include background blur, lighting, color cast, and / or high dynamic range (HDR) blending. In some embodiments, effects can be applied in the ISP 112.

[0073] The output image frame 230 of ISP 112 may include an improved representation of the scene through various aspects of this disclosure, resulting in a higher dynamic range that captures additional details in both the dark and bright parts of the scene. Processor 104 may display these output image frames 230 to a user, and the improvements provided by the processing described in ISP 112 and / or processor 104 improve image quality and user experience by enhancing the representation of the scene. For example, when determining the output image frame 230, HDR processing module 212 in ISP 112 may correct image data received from first camera 103.

[0074] Figure 2 System 200 can be configured to execute the reference. Figure 3 The described operation determines the output image frame 230. Figure 3 A flowchart of an example method 300 for processing image data to obtain high dynamic range (HDR) image frames according to some embodiments of the present disclosure is shown. Reference Figure 3 The described HDR processing techniques can obtain an improved digital representation of scenes captured using interlaced HDR (sHDR) capture or other techniques. (Reference) Figure 3 The described method for processing image data produces photos or videos with higher image quality (IQ). Reference Figure 3 Each of the operations described may be performed by one or a combination of processor 104 (including cores 104A to 104N and / or AI engine 124) and / or ISP 112.

[0075] At box 302, image data is received from an image sensor. This image data may include first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration. The first exposure duration may be longer than the second exposure duration, and the third exposure duration may be longer than the second exposure duration. In some embodiments, the first exposure duration is equal to the third exposure duration. Image data acquired in this manner may have a time center of the second image data, which is equidistant in time from the time centers of the first and third image data.

[0076] Figure 4 An example illustration of the data capture arrangement for generating image data received at box 302 according to the interleaved HDR configuration is shown. Figure 4 This is a timing diagram of example image capture operations according to various aspects of this disclosure. The x-axis illustrates the advance time along both the top and bottom x-axis. A first image frame 402 is captured with an exposure duration M1. After the first image frame 402, a second image frame 404 is captured with an exposure duration S, and a third image frame 406 is captured with an exposure duration M2. Similarly, the capture timing for each of image frames 402, 404, and 406 is shown. For example, with respect to the first image frame 402, time 432 marks the start of the image capture operation on the image sensor, and time 412 indicates the end of the image capture operation on the image sensor for the first image frame 402. Times 414 and 416 correspond to the end of the image capture operations for the second image frame 404 and the third image frame 406.

[0077] Figure 4A similar example of image capture operations for image frames 404 and 406 is shown below. After capturing three image frames 402, 404, and 406, a fourth image frame can be determined by summing the exposure durations of image frames 402, 404, and 406. This fourth image frame has a corresponding exposure duration L, which is the sum of M1, S, and M2. The time center of the fourth image frame with exposure duration L is the time center of the second image frame 404, which is centered within the second image frame 404 and equidistant from the time centers of the first image frame 402 and the third image frame 406.

[0078] Image data may be received at block 302, for example, from a bus coupled to the first camera 103 or from an analog front-end (AFE) coupled to the first camera 103. Image data may also be received from a wireless camera, wherein the image data is received via one or more of WAN adapter 152, LAN adapter 153, and / or PAN adapter 154. Image data may also be received from a memory location or network storage location, such as when image data was previously captured and is now retrieved from memory 106 and / or from a remote location via one or more of WAN adapter 152, LAN adapter 153, and / or PAN adapter 154. In some embodiments, the capture of image data may be initiated by a camera application executing on processor 104, which causes camera control 210 to activate the first camera 103 to capture image data. The image data retrieved at block 302 may then be processed by ISP 112 and / or processor 104 or other components for processing the image data according to the operations described in one or more of the following boxes.

[0079] Using references Figure 2 The described camera configuration, when configured to capture an image sensor, can capture image data received at block 302. In this embodiment, the method may include, prior to block 302, configuring the image sensor by at least one processor to capture first image data for a first exposure duration, capture second image data for a second exposure duration, and capture third image data for a third exposure duration. The camera configuration may configure the image sensor to have: a first exposure duration that may be longer than the second exposure duration; and a third exposure duration that may be longer than the second exposure duration.

[0080] After acquiring various image data corresponding to different image frames with specific exposure durations, the image frames can be processed to obtain output HDR image frames, wherein these image frames are organized to obtain temporally aligned image data. HDR processing may include the operations described in reference boxes 304 and 306.

[0081] At block 304, the method can include determining first summed image data by at least one processor by summing corresponding pixel intensity values of first image data, second image data, and third image data. The summing forms a fourth image frame corresponding to a long exposure image frame.

[0082] At block 306, the method can include determining a first output image frame by at least one processor by combining the first summed image data (which is the sum of the first image frame, the second image frame, and the third image frame) with the second image data. The temporal center of the fourth image frame is the same temporal center as the temporal center of the second image data. Thus, in some embodiments, when combining two image frames, no alignment operation is applied to the fourth image data (long exposure image frame) or the second image data (short exposure image frame).

[0083] For image frames 402, 404, and 406 corresponding to inputs M1, S, and M2, example operations performed according to Figure 3 the method shown are as follows: input (M1, S, M2) for u in [0, L.height): for v in [0, L.width): L(u,v) ← M1(u,v) + S(u,v) + M2(u,v) HDR_fusion(L, S).

[0084] Although one example of image processing is shown as M1 > S and M2 > S, other embodiments of the image processing techniques described herein can include other exposure durations of bracketed exposures of input image frames M1, S, and M2. In another such embodiment, M1 < S and M2 < S, where M1 = M2, such that the S image frame is temporally centered between M1 and M2.

[0085] When obtaining an output HDR image frame, additional image data can be processed, such as by configuring an image sensor to capture additional image data corresponding to the fourth image data, the fifth image data, and / or additional image data. Figure 5 An example of capturing such additional image data is shown in Figure 5 is a timing diagram of an example image capture operation having alternating medium image frames and short image frames in accordance with aspects of the present disclosure. The image capture operation, an example of which is in Figure 5(As shown in the figure) may include capturing alternating image frames with medium and short exposure durations. As shown, image capture may involve zero shutter lag, where image frames are captured seamlessly in succession, but other embodiments may involve some shutter lag or another predetermined delay between image captures or sets of image captures.

[0086] The fourth exposure duration of the fourth image frame 508 can be equal to the second exposure duration of the second image frame 504; and the fifth exposure duration of the fifth image frame 510 can be equal to the third exposure duration of the third image frame 506, which is equal to the first exposure duration of the first image frame 502. The time center of the fourth image data can be equidistant from the time center of the third image data and the time center of the fifth image data.

[0087] As part of the preview operation, the image capture operation shown can continuously capture image frames. When the shutter is activated, it can be performed according to the reference... Figure 3 One or more techniques are described to process the most recent triplets of image frames. For example, an image capture device may acquire a first image frame 502, a second image frame 504, a third image frame 506, a fourth image frame 508, a fifth image frame 510, a sixth image frame 512, and a seventh image frame 514 before activating the shutter. Image frames 510, 512, and 514 may be input to... Figure 3 The operation determines the output HDR image frame, wherein the sum of image frames 510, 512 and 514 is a fourth image frame that can be combined with image frame 512 to obtain the output HDR image frame.

[0088] Figure 6 Another example configuration for capturing image data is shown in the figure. Figure 6 This is a timing diagram of an example image capture operation with a group of triples of image frames according to various aspects of this disclosure. Figure 6 The image capture operation groups captured image frames into triplets, where intermediate short-exposure image frames are spatially and temporally positioned by anchoring the triplets. Any one of these triplets can be processed and used as a reference. Figure 3 Input to any of the methods described. Reference Figure 6 ,like Figure 4 As shown, image frames 402, 404, and 406 are captured. Subsequently, after a delay of 610, additional triplets, such as image frames 402, 404, and 406, are captured with the same or different exposure times as the triplets in image frames 602, 604, and 606. Triplets can be captured continuously as part of a preview or live image, and when the shutter is pressed, the most recent triplet is input to the HDR processing logic to determine the output HDR image frame.

[0089] use Figure 6In the capture configuration, the data received by block 302 may further include fourth image data, fifth image data, and sixth image data received by at least one processor. Image data processing may include: at least one processor determining second summed image data by summing the corresponding pixel intensity values ​​of the fourth, fifth, and sixth image data; and at least one processor determining a second output image frame by combining the second summed image data with the fifth image data. In some embodiments, the HDR output frame may be the result of combining two or more HDR image frames generated from two or more triplets. For example, Figure 3 The method may include determining a third output image frame by at least one processor by combining a first output image frame with a second output image frame.

[0090] Figure 7 Another example configuration for capturing image data is shown in the figure. Figure 7 Image capture operations have been extended to include, for example Figure 4 and Figure 6 The triples shown are used to include additional image frames. The set of image frames can be processed and used as a reference. Figure 3 Input to any of the methods described. Figure 7 This is a timing diagram of an example image capture operation with an exposure quintuple according to various aspects of this disclosure. Reference Figure 7 Image frames 702, 704, 706, 708, and 710 are captured by an image capture device. In the example shown, image frames 702, 704, 708, and 710 have equal exposure durations corresponding to exposure duration M, and image frame 706 is captured with an exposure duration S.

[0091] The long exposure image frame 712 can be determined by summing image frames 702, 704, 706, 708, and 710. The intermediate duration image frame can be determined by summing image frames 704, 706, and 708. Although image frames 702 and 704 are shown as having the same duration as image frames 708 and 710, these image frames can also have different durations, as long as the durations are symmetrical, such that the time center of the long image frame 712 remains the time center of image frame 706. For example, image frames 702 and 710 can have a duration of M2, and image frames 704 and 708 can have a duration of M1.

[0092] Figure 7 The image data from the capture operation shown can be input into the reference. Figure 3 One or more of the methods described. For example, the long image frame 712 can be determined by an operation corresponding to box 304, and the output HDR image frame can be determined by an operation corresponding to box 306, which combines the long image frame 712 with the short image frame 706.

[0093] Figure 8 Another example configuration for capturing image data is shown in the figure. Figure 8 The image capture operation is extended to include a flash to illuminate details as part of capturing a short-exposure image frame. Figure 8 This is a timing diagram of an example image capture operation with flash operation according to various aspects of this disclosure. Triplets of image frames 802, 804, and 806 are captured with corresponding exposure durations M, S, and M. The duration of image frame 804 may be long enough to provide a time window 808 for flash activation. In such embodiments, the camera configuration provided to the first camera may also include an instruction to activate the flash as part of the image capture process. Image frames 802 and 806 may form a triplet together with image frame 804, which may be input to a reference... Figure 3 One of the methods described for determining the output HDR image frame.

[0094] refer to Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 The described HDR processing demonstrates an example involving interlaced HDR (sHDR) capture techniques. Aspects of this disclosure are applicable to other HDR capture techniques. For example, see reference... Figure 9 and Figure 10 Similar techniques applied to 4-in-1 HDR (qHDR) capture technology are illustrated.

[0095] Figure 9 A flowchart illustrating an example method for processing image data to obtain high dynamic range (HDR) image frames according to some aspects of this disclosure is shown. Method 900 begins at block 902 with receiving first image data, second image data, and third image data, similar to... Figure 3 Box 302. A first, second, and third image can be captured from the qHDR capture operation, such as... Figure 10 As described in [the text]. Figure 10 This is a timing diagram of example image capture operations according to various aspects of this disclosure. A single readout can be performed at time 1012, wherein the single readout from the image sensor provides first image data 1002 corresponding to exposure duration L, second image data 1004 corresponding to exposure duration M', and third image data 1006 corresponding to exposure duration S'.

[0096] At box 904, the first image data, the second image data, and the third image data are de-mosaiced to separate the red channel, green channel, and blue channel from the corresponding four-in-one CFA image data.

[0097] At block 906, a fourth image data is determined based on second image data and third image data, such as by subtracting the third image data from the second image data. The resulting operation obtains an image frame corresponding to a short exposure duration centered in time within the image data 1002.

[0098] At block 908, a first output HDR image frame is determined by combining the fourth image data with the first image data. The first image data corresponds to a long exposure duration, and the fourth image data corresponds to a short exposure duration centered in time within the long exposure duration. The temporal alignment between the fourth image data and the first image data allows the image data to be combined without alignment operations.

[0099] For the image data 1002, 1004, and 1006 corresponding to the inputs L, M', and S', example operations performed according to the Figure 9 method shown are as follows: input (L, M’, S’) L ← demosaic(L) M’ ← demosaic(M’) S’ ← demosaic(S’) for u in [0, S.height): for v in [0, S.width): S(u, v) ← M'(u, v) - S'(u, v) HDR_fusion(L, S).

[0100] Although one example of image processing is shown as L - M' > S and S' > M' - S', other implementations of the image processing techniques described herein may include other exposure durations of the bracketed exposures of the input image frames. In another such implementation, L - M' < S and S' < M' - S'.

[0101] Figure 11 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 (e.g., one or more processor cores, graphics processing unit, image signal processor, digital signal processor, neural signal processor, or a combination thereof) or other processing circuitry may be configured to operate on the image data to perform reference Figure 3 and / or Figure 9One or more operations of the described method. Image data can be processed to determine one or more output HDR image frames 1110. The processor 104 can be configured using frame processing 1104A processing logic, which may include summing, subtracting, de-mosaicing, or otherwise processing the image data to obtain two image frames, which will be fused to determine the output HDR image frame 1110. These two image frames are provided to HDR fusion 1104B processing logic. HDR fusion 1104B can combine the two image frames by determining a combination of pixel values ​​from a first image frame or a second image frame, or pixel values ​​of corresponding pixels in the first and second image frames, for each pixel of the output HDR image frame 1110. For example, when a pixel is determined to be in a shadow region (e.g., intensity below a threshold level), a corresponding pixel from a longer duration image frame can be selected for the output HDR image frame, and when a pixel is determined to be in a bright region (e.g., intensity above a threshold level), a corresponding pixel from a shorter duration image frame can be selected for the output HDR image frame. In some implementations, tone mapping operations can be used to combine corresponding pixel values ​​of the first and second image frames to determine the pixel values ​​of the output HDR image frame 1110.

[0102] Processor 104 receives first image data and second image data. In some embodiments, the first image data may be received directly from an image sensor or a memory coupled to the image sensor. In some embodiments, the first image data may be retrieved from a long-term storage device (such as a flash memory device or a network location) that stores previously captured or generated images.

[0103] In one or more aspects, the technology for supporting image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include a means configured to perform image processing in at least one processor (e.g., CPU, GPU, DSP, NSP, ISP). The means is also configured to perform an operation to perform a method of image processing, the method comprising: receiving image data by the at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein a second time center of the second image data is temporally equidistant between the first time center of the first image data and the third time center of the third image data; determining first summed image data by the at least one processor by summing corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and determining a first output image frame by the at least one processor by combining the first summed image data with the second image data.

[0104] Additionally, the apparatus may perform or operate according to one or more aspects described below. In some embodiments, the apparatus includes a wireless device, such as a UE. In some embodiments, the apparatus includes a remote server (such as a cloud-based computing solution) that receives image data, processes it, and determines output image frames. In some embodiments, the apparatus may include at least one processor and memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other embodiments, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, and the program code may be executable by a computer to cause the computer to perform the operations described herein with reference to the apparatus. In some embodiments, the apparatus may include one or more components configured to perform the operations described herein. In some embodiments, a method of wireless communication may include one or more operations described herein with reference to the apparatus.

[0105] In a second aspect, in conjunction with the first aspect, the first exposure duration is equal to the third exposure duration.

[0106] In a third aspect, in conjunction with one or more of the first or second aspects, the first exposure duration is longer than the second exposure duration; and the third exposure duration is longer than the second exposure duration.

[0107] In the fourth aspect, in conjunction with one or more of the first to third aspects, the steps of determining the first summed image data and determining the first output image frame do not include spatial image alignment processing.

[0108] In the fifth aspect, in combination with one or more of the first to fourth aspects, the combination includes performing an HDR fusion algorithm.

[0109] In a sixth aspect, in conjunction with one or more of the first to fifth aspects, the method further includes configuring the image sensor by the at least one processor to capture the first image data during the first exposure duration, capture the second image data during the second exposure duration, and capture the third image data during the third exposure duration, wherein the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration.

[0110] In a seventh aspect, in conjunction with one or more of the first to sixth aspects, the method further includes receiving fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration by the at least one processor, wherein: the fourth exposure duration is equal to the second exposure duration, the fifth exposure duration is equal to the third exposure duration, and the time center of the fourth image data is equidistant in time from the time center of the third image data and the time center of the fifth image data; the at least one processor determines second summed image data by summing the corresponding pixel intensity values ​​of the third image data, the fourth image data, and the fifth image data; and the at least one processor determines a second output image frame by combining the second summed image data with the fourth image data.

[0111] In an eighth aspect, in combination with one or more of the first to seventh aspects, the method further includes: receiving fourth image data, fifth image data, and sixth image data by the at least one processor; determining second summed image data by the at least one processor by summing corresponding pixel intensity values ​​of the fourth image data, the fifth image data, and the sixth image data; determining a second output image frame by the at least one processor by combining the second summed image data with the fifth image data; and determining a third output image frame by the at least one processor by combining the first output image frame with the second output image frame.

[0112] In a ninth aspect, in conjunction with one or more of the first to eighth aspects, the method further includes: receiving, by the at least one processor, fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein the fourth exposure duration is longer than the second exposure duration, and the fifth exposure duration is longer than the second exposure duration; the method further includes receiving, by the at least one processor, fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein; and the fifth exposure duration is longer than the second exposure duration, the time center of the second image data is equidistant in time from the first time center of the fourth image data and the second time center of the fifth image data; and determining, by the at least one processor, second summed image data by summing the corresponding pixel intensity values ​​of the first summed image data, the fourth image data, and the fifth image data.

[0113] In the tenth aspect, in conjunction with one or more of the first to ninth aspects, the time center of the second exposure duration is aligned with the activation of the flash.

[0114] In an eleventh aspect, in conjunction with one or more of the first to tenth aspects, the apparatus may include: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor being configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving image data by the at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein the time center of the second image data is equidistant in time between the first time center of the first image data and the second time center of the third image data; determining first summed image data by the at least one processor by summing corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and determining a first output image frame by the at least one processor by combining the first summed image data with the second image data.

[0115] In the twelfth aspect, in conjunction with one or more of the first to eleventh aspects, the first exposure duration is equal to the third exposure duration.

[0116] In the thirteenth aspect, in conjunction with one or more of the first to twelfth aspects, the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration.

[0117] In the fourteenth aspect, in conjunction with one or more of the first to thirteenth aspects, the steps of determining the first summed image data and determining the first output image frame do not include spatial image alignment processing.

[0118] In the fifteenth aspect, in combination with one or more of the first to fourteenth aspects, the combination includes performing a fusion algorithm.

[0119] In a sixteenth aspect, in conjunction with one or more of the first to fifteenth aspects, the at least one processor is further configured to perform operations including: configuring the image sensor to capture the first image data during the first exposure duration, capture the second image data during the second exposure duration, and capture the third image data during the third exposure duration, wherein: the first exposure duration is longer than the second exposure duration, and the third exposure duration is longer than the second exposure duration.

[0120] In a seventeenth aspect, in conjunction with one or more of the first to sixteenth aspects, the at least one processor is further configured to perform operations including: receiving fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein: the fourth exposure duration is equal to the second exposure duration, the fifth exposure duration is equal to the third exposure duration, and the time center of the fourth image data is equidistant in time between the first time center of the third image data and the second time center of the fifth image data; determining second summed image data by summing the corresponding pixel intensity values ​​of the third image data, the fourth image data, and the fifth image data; and determining a second output image frame by combining the second summed image data with the fourth image data.

[0121] In the eighteenth aspect, in combination with one or more of the first to seventeenth aspects, the at least one processor is further configured to perform operations including: receiving fourth image data, fifth image data, and sixth image data by the at least one processor; determining second summed image data by the at least one processor through summing corresponding pixel intensity values ​​of the fourth image data, the fifth image data, and the sixth image data; determining a second output image frame by the at least one processor through combining the second summed image data with the fifth image data; and determining a third output image frame by the at least one processor through combining the first output image frame with the second output image frame.

[0122] In the nineteenth aspect, in conjunction with one or more of the first to eighteenth aspects, the at least one processor is further configured to perform operations including: receiving fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein: the fourth exposure duration is longer than the second exposure duration, the fifth exposure duration is longer than the second exposure duration, and the time center of the second image data is equidistant in time between the first time center of the fourth image data and the second time center of the fifth image data; and determining second summed image data by the at least one processor by summing the corresponding pixel intensity values ​​of the first summed image data, the fourth image data, and the fifth image data.

[0123] In the twentieth aspect, in conjunction with one or more of the first to nineteenth aspects, the time center of the second exposure duration is aligned with the activation of the flash.

[0124] In the twenty-first aspect, in combination with one or more of the first to twentieth aspects, the image capturing device includes an image sensor, wherein the at least one processor is coupled to the image sensor.

[0125] In the accompanying drawings, a single block can be described as performing one or more functions. The one or more functions performed by this block can be performed in a single component or across multiple components, and / or can be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps are described below in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as causing a departure from the scope of this disclosure. Additionally, the example device may include components other than those shown, including well-known components such as processors, memory, etc.

[0126] The aspects of this disclosure are applicable to any electronic device that includes, is coupled to, or otherwise processes data from one, two, or more image sensors capable of capturing image frames (or “frames”). The terms “output image frame,” “modified image frame,” and “corrected image frame” can refer to an image frame that has been processed by any of the techniques disclosed to adjust the raw image data received from the image sensor. Furthermore, aspects of the disclosed techniques can be implemented for processing image data received from image sensors having the same or different capabilities and characteristics, such as resolution, shutter speed, or sensor type. Additionally, aspects of the disclosed techniques can be implemented in devices for processing image data, whether or not the device includes or is coupled to an image sensor. For example, the disclosed techniques may include operations performed by a processing device in a cloud computing system that retrieves image data previously recorded by a separate device having an image sensor for processing.

[0127] Unless explicitly stated otherwise in the following discussion, it should be understood that throughout this application, the use of terms such as “access,” “receive,” “transmit,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” “set,” “generate,” etc., refers to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the registers and memories of the computer system and transform them into other data similarly represented as physical quantities in the registers, memories, or other such information storage, transmission, or display devices of the computer system. The use of different terms to refer to actions or processes of a computer system does not necessarily indicate different operations. For example, “determining” data can refer to “generating” data. Similarly, “determining” data can refer to “retrieving” data.

[0128] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of this disclosure. Although the description and examples herein use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus can include a device or part of a device for performing the described operations.

[0129] Certain components in a device or apparatus described as “parts for access,” “parts for receiving,” “parts for transmitting,” “parts for using,” “parts for selecting,” “parts for determining,” “parts for normalizing,” “parts for multiplying,” or other similarly named terms referring to one or more operations on data (such as image data) may refer to processing circuitry (e.g., application-specific integrated circuit (ASIC), digital signal processor (DSP), graphics processing unit (GPU), central processing unit (CPU), computer vision processor (CVP), or neural signal processor (NSP)) configured to perform the described functions by means of hardware, software, or a combination of hardware configured by software.

[0130] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0131] The components, functional blocks, and modules described herein with respect to the accompanying figures cited above include processors, electronic devices, hardware devices, electronic components, logic circuits, memory, software code, firmware code, and so on, or any combination thereof. Software should be interpreted broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, programs, and / or functions, whether referred to as software, firmware, middleware, microcode, hardware description languages, or other terms. Furthermore, the features discussed herein may be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

[0132] Those skilled in the art should understand that: reference Figure 3 and Figure 4 One or more boxes (or operations) described may be combined with one or more boxes (or operations) described in another figure in the reference diagram. For example, Figure 3 One or more boxes (or operations) can be connected with Figures 1 to 2 A combination of one or more boxes (or operations). For example, with... Figure 11 One or more associated boxes can be combined with Figures 1 to 2 A combination of one or more associated boxes (or operations).

[0133] Those skilled in the art will also recognize that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this disclosure. Those skilled in the art will also readily recognize that the order or combination of components, methods, or interactions described herein is merely illustrative, and that components, methods, or interactions of various aspects of this disclosure can be combined or performed in ways other than those illustrated and described herein.

[0134] The various exemplary logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various exemplary components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0135] Hardware and data processing means for implementing the various exemplary logic units, logic blocks, modules, and circuits described herein can be implemented or executed using general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic units, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some embodiments, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some embodiments, specific processes and methods may be performed by circuitry specific to a given function.

[0136] In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuits, computer software, firmware, including the structures disclosed in this specification and their structural equivalents or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0137] If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted through a computer-readable medium. The processes of the methods or algorithms disclosed herein can be implemented in a processor-executable software module that can reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that can be implemented to transfer a computer program from one location to another. Storage media can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible to a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically reproduce data, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, the operation of a method or algorithm may reside as a set of code and instructions or any combination of code and instructions on a machine-readable medium and a computer-readable medium that may be incorporated into a computer program product.

[0138] Various modifications to the specific embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the specific embodiments shown herein, but are to be granted the broadest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0139] Additionally, those skilled in the art will readily recognize that, for the convenience of describing the accompanying drawings, contrasting terms such as “upper” and “lower” or “front” and “back” or “top” and “bottom” or “forward” and “backward” are sometimes used, indicating relative positions on a correctly oriented page corresponding to the orientation of the drawings, and may not reflect the correct orientation of any device as implemented.

[0140] Certain features described in this specification in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even originally claimed in this way, one or more features from the claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0141] Similarly, although operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the indicated specific order or sequential order, or to perform all illustrated operations to achieve the desired result. Furthermore, the figures may schematically depict one or more example processes in the form of flowcharts. However, other operations not depicted may be combined with the schematically illustrated example processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any illustrated operation. In some contexts, multitasking and parallel processing are advantageous. Moreover, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other embodiments also fall within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result.

[0142] As used herein (including the claims), the term "or" in a list of two or more items means that any one of the listed items may be used alone, or any combination of two or more listed items may be used. For example, if a composition is described as containing component A, B, or C, the composition may contain A alone; B alone; C alone; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C. Additionally, as used herein (including the claims), "or" in a list of items beginning with "at least one of" indicates a separate list, such that a list such as "at least one of A, B, or C" refers to A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these items.

[0143] The term “substantially” is defined as being largely but not necessarily entirely what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any specific implementation of the disclosure, the term “substantially” may be used in place of the “[percentage]” of the specified content, where the percentage includes 0.1%, 1%, 5%, or 10%.

[0144] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, the method comprising: Image data is received by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration; The at least one processor determines the first summed image data by summing the corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and The at least one processor determines the first output image frame by combining the first summed image data with the second image data.

2. The method according to claim 1, wherein the first exposure duration is equal to the third exposure duration.

3. The method according to claim 1, wherein the second time center of the second image data is equidistant in time from the first time center of the first image data and the third time center of the third image data.

4. The method of claim 1, wherein the steps of determining the first summed image data and determining the first output image frame do not include spatial image alignment processing.

5. The method of claim 1, wherein the combination includes performing an HDR fusion algorithm.

6. The method according to claim 1, further comprising: The at least one processor configures the image sensor to capture the first image data during the first exposure duration, capture the second image data during the second exposure duration, and capture the third image data during the third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration.

7. The method according to claim 1, further comprising: The at least one processor receives fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein: The fourth exposure duration is equal to the second exposure duration. The fifth exposure duration is equal to the third exposure duration, and The time center of the fourth image data is equidistant in time from the time center of the third image data and the time center of the fifth image data; The at least one processor determines the second summed image data by summing the corresponding pixel intensity values ​​of the third image data, the fourth image data, and the fifth image data; and The at least one processor determines the second output image frame by combining the second summed image data with the fourth image data.

8. The method according to claim 1, further comprising: The at least one processor receives the fourth image data, the fifth image data, and the sixth image data; The at least one processor determines the second summed image data by summing the corresponding pixel intensity values ​​of the fourth image data, the fifth image data, and the sixth image data; The at least one processor determines the second output image frame by combining the second summed image data with the fifth image data; as well as The at least one processor determines the third output image frame by combining the first output image frame with the second output image frame.

9. The method according to claim 1, further comprising: The at least one processor receives fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein: The fourth exposure duration is longer than the second exposure duration. The fifth exposure duration is longer than the second exposure duration, and The time center of the second image data is equidistant in time from the first time center of the fourth image data and the second time center of the fifth image data; and The at least one processor determines the second summed image data by summing the corresponding pixel intensity values ​​of the first summed image data, the fourth image data, and the fifth image data.

10. The method of claim 1, wherein the time center of the second exposure duration is aligned with the activation of the flash.

11. An apparatus comprising: Memory, the memory storing processor-readable code; and At least one processor coupled to the memory, the at least one processor being configured to execute processor-readable code to cause the at least one processor to perform operations including: Image data is received by the at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration; The at least one processor determines the first summed image data by summing the corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and The at least one processor determines the first output image frame by combining the first summed image data with the second image data.

12. The apparatus of claim 11, wherein the first exposure duration is equal to the third exposure duration.

13. The apparatus of claim 11, wherein the time center of the second image data is equidistant in time from the first time center of the first image data and the second time center of the third image data.

14. The apparatus of claim 11, wherein the steps of determining the first summed image data and determining the first output image frame do not include spatial image alignment processing.

15. The apparatus of claim 11, wherein the combination includes performing a fusion algorithm.

16. The apparatus of claim 11, wherein the at least one processor is further configured to perform operations including: The at least one processor configures the image sensor to capture the first image data during the first exposure duration, capture the second image data during the second exposure duration, and capture the third image data during the third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration.

17. The apparatus of claim 11, wherein the at least one processor is further configured to perform operations including: The at least one processor receives fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein: The fourth exposure duration is equal to the second exposure duration. The fifth exposure duration is equal to the third exposure duration, and The time center of the fourth image data is equidistant in time from the first time center of the third image data and the second time center of the fifth image data; The at least one processor determines the second summed image data by summing the corresponding pixel intensity values ​​of the third image data, the fourth image data, and the fifth image data; as well as The at least one processor determines the second output image frame by combining the second summed image data with the fourth image data.

18. The apparatus of claim 11, wherein the at least one processor is further configured to perform operations including: The at least one processor receives the fourth image data, the fifth image data, and the sixth image data; The at least one processor determines the second summed image data by summing the corresponding pixel intensity values ​​of the fourth image data, the fifth image data, and the sixth image data; The at least one processor determines the second output image frame by combining the second summed image data with the fifth image data; as well as The at least one processor determines the third output image frame by combining the first output image frame with the second output image frame.

19. The apparatus of claim 11, wherein the at least one processor is further configured to perform operations including: The at least one processor receives fourth image data for a fourth exposure duration and fifth image data for a fifth exposure duration, wherein: The fourth exposure duration is longer than the second exposure duration. The fifth exposure duration is longer than the second exposure duration, and The time center of the second image data is equidistant in time from the first time center of the fourth image data and the second time center of the fifth image data; and The at least one processor determines the second summed image data by summing the corresponding pixel intensity values ​​of the first summed image data, the fourth image data, and the fifth image data.

20. The apparatus of claim 11, wherein the time center of the second exposure duration is aligned with the activation of the flash.

21. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations, the operations including: Image data is received by at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration; The at least one processor determines the first summed image data by summing the corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and The at least one processor determines the first output image frame by combining the first summed image data with the second image data.

22. The non-transitory computer-readable medium of claim 21, wherein the first exposure duration is equal to the third exposure duration.

23. The non-transitory computer-readable medium of claim 21, wherein the time center of the second image data is equidistant in time from the first time center of the first image data and the second time center of the third image data.

24. The non-transitory computer-readable medium of claim 21, wherein the steps of determining the first summed image data and determining the first output image frame do not include spatial image alignment processing.

25. The non-transitory computer-readable medium of claim 21, wherein the combination includes performing a fusion algorithm.

26. An image capturing device, the image capturing device comprising: Image sensor; Memory, the memory storing processor-readable code; and At least one processor, coupled to the memory and the image sensor, is configured to execute processor-readable code to cause the at least one processor to perform operations, said operations including: Image data is received by the at least one processor, the image data including first image data for a first exposure duration, second image data for a second exposure duration, and third image data for a third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration; The at least one processor determines the first summed image data by summing the corresponding pixel intensity values ​​of the first image data, the second image data, and the third image data; and The at least one processor determines the first output image frame by combining the first summed image data with the second image data.

27. The image capture device of claim 26, wherein the first exposure duration is equal to the third exposure duration.

28. The image capture device of claim 26, wherein the combination includes performing a fusion algorithm.

29. The image capture device of claim 26, wherein the at least one processor is further configured to perform operations including: The image sensor is configured by the at least one processor to capture the first image data during the first exposure duration, capture the second image data during the second exposure duration, and capture the third image data during the third exposure duration, wherein: The first exposure duration is longer than the second exposure duration, and The third exposure duration is longer than the second exposure duration.

30. The image capturing device of claim 26, wherein the time center of the second exposure duration is aligned with the activation of the flash.

Images