Single-exposure multi-frame image capture method for producing HDR video

Abstract

An example method for single-exposure multi-frame image capture for producing HDR video includes: capturing a series of frames at an underexposed setting; selecting a reference image from the series of frames; aligning the series of frames with the reference image; merging the series of frames into a single frame based on the reference image; indexing the reference image to the next frame in the series of frames, thereby generating an indexed reference image; realigning the series of frames with the indexed reference image; and remerging the series of frames into a single frame based on the indexed reference image. A method for multiple-exposure multi-frame image capture for producing HDR video is also provided.

Description

Technical Field

[0001] The present disclosure relates to generating video, and in particular to generating high dynamic range video using a low dynamic range image sensor. Background Art

[0002] The dynamic range of a scene can be much greater than what current low dynamic range (LDR) image sensors can capture. Current LDR image sensors used to capture high dynamic range (HDR) scenes will saturate brightly lit areas or fail to capture a sufficient number of photons in poorly lit areas. Currently, capturing HDR scenes can be achieved using specially designed HDR sensors.

[0003] Currently, the use of high dynamic range sensors is rarely reported due to the difficulty in expanding the capacity of the sensor. Generally speaking, the use of high dynamic range sensors results in high fixed pattern noise, and the implementation is expensive and limited. Summary of the Invention

[0004] An example method for single-exposure multi-frame image capture for producing HDR video includes: capturing a series of frames at an underexposed setting; selecting a reference image from the series of frames; aligning the series of frames with the reference image; merging the series of frames into a single frame based on the reference image; indexing the reference image to the next frame in the series of frames, thereby generating an indexed reference image; realigning the series of frames with the indexed reference image; and remerging the series of frames into a single frame based on the indexed reference image.

[0005] Another example method for multi-exposure multi-frame image capture for producing HDR video includes: capturing a first series of frames at a first underexposure setting; selecting a first reference image from the first series of frames; aligning the first series of frames with the first reference image; merging the first series of frames into a first frame based on the first reference image; capturing a second series of frames at a second underexposure setting; selecting a second reference image from the second series of frames; aligning the second series of frames with the second reference image; merging the second series of frames into a second frame based on the second reference image; and fusing the one first frame and the one second frame.

[0006] Another example method for multi-exposure multi-frame image capture for producing HDR video includes: capturing a first series of frames at a first underexposure setting; selecting a first reference image from the first series of frames; aligning the first series of frames with the first reference image; merging the first series of frames into a first frame based on the first reference image; capturing a second series of frames at a second underexposure setting; selecting a second reference image from the second series of frames; aligning the second series of frames with the second reference image; merging the second series of frames into a second frame based on the second reference image; capturing a third series of frames at a third underexposure setting; selecting a third reference image from the third series of frames; aligning the third series of frames with the third reference image; and merging the third series of frames into a third frame based on the third reference image, and fusing a second frame and a third frame into a first fused frame; and fusing the one first frame and the first fused frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the attached figure:

[0008] Figure 1 depicts a first example system diagram according to one embodiment of the present disclosure;

[0009] Figure 2 depicts a second example system diagram according to one embodiment of the present disclosure;

[0010] Figure 3 Depicting an example streaming video mode with single-exposure multi-frame fusion of rolling reference images according to one embodiment of the present disclosure;

[0011] Figure 4 depicts an example combination of a single frame of high dynamic range and low dynamic range data according to one embodiment of the present disclosure;

[0012] Figure 5 Depicts an example combination of different exposure value data using a fusion of high and low exposure values ​​according to one embodiment of the present disclosure;

[0013] Figure 6 Depicts an example two-exposure flow for multi-exposure multi-frame video according to one embodiment of the present disclosure;

[0014] Figure 7 Depicts an example triple exposure process for multi-exposure multi-frame video with high dynamic range and low dynamic range fusion according to one embodiment of the present disclosure;

[0015] Figure 8 Depicts an example triple exposure process for multi-exposure multi-frame video with high and low exposure value fusion according to one embodiment of the present disclosure;

[0016] Figure 9is a first example method according to one embodiment of the present disclosure;

[0017] Figure 10 is a second example method according to one embodiment of the present disclosure;

[0018] Figure 11 is a third example method according to one embodiment of the present disclosure; and

[0019] Figure 12 is a fourth example method according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The embodiments listed below are written only to illustrate the application of the device and method, rather than to limit the scope. Modifications of equivalent forms of the device and method should be classified as within the scope of the claims.

[0021] Certain terms are used in the following description and claims to refer to specific system components. As will be appreciated by those skilled in the art, different companies may refer to components and / or methods by different names. This document is not intended to distinguish between components and / or methods that have the same function but differ in name.

[0022] In the following discussion and in the claims, the terms "including" and "comprising" are used in an open-ended fashion, and thus can be interpreted to mean "including, but not limited to... ." Likewise, the terms "couple" or "coupled" are intended to mean either an indirect or direct connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0023] Figure 1 An example hybrid computing system 100 is depicted that may be used to implement a neural network associated with the operations of one or more portions or steps of process 600. In this example, processors associated with the hybrid system include a Field Programmable Gate Array (FPGA) 122, a Graphical Processor Unit (GPU) 120, and a Central Processing Unit (CPU) 118.

[0024] CPU 118, GPU 120, and FPGA 122 have the ability to provide neural networks. A CPU is a general-purpose processor that can perform many different functions. Its versatility results in the ability to perform a variety of different tasks. However, a CPU is limited in its ability to process multiple data streams, and its capabilities with respect to neural networks are limited. A GPU is a graphics processor with many small processing cores capable of sequentially processing parallel tasks. An FPGA is a field-programmable device that has the ability to be reconfigured and hardwired to perform any function that can be programmed into a CPU or GPU. Because FPGAs are programmed in circuit form, they are many times faster than CPUs and significantly faster than GPUs.

[0025] There are other types of processors that can be included in a system, such as an Accelerated Processing Unit (APU), which includes both CPU and GPU elements on a chip, and a Digital Signal Processor (DSP), which is designed to perform high-speed digital data processing. Application-Specific Integrated Circuits (ASICs) can also perform the hard-wired functions of an FPGA; however, the lead time for designing and producing ASICs is on the order of several quarters of a year, rather than the quick turnaround available in programming FPGAs.

[0026] Graphics processor unit 120, central processing unit 118, and field programmable gate array 122 are connected and connected to memory interface controller 112. The FPGA is connected to the memory interface via programmable logic circuit to memory interconnect 130. This additional device is used due to the fact that FPGAs operate at a very large bandwidth and is used to minimize the electronic circuitry used by the FPGA to perform memory tasks. Memory and interface controller 112 is also connected to permanent storage disk 110, system memory 114, and read-only memory (ROM) 116.

[0027] Figure 1 The system can be used to program and train the FPGA. The GPU works well with unstructured data and can be used for training. Once the data has been trained, a deterministic inference model can be found, and the CPU can use the model data determined by the GPU to program the FPGA.

[0028] The memory interfaces and controllers are connected to a central interconnect 124, which is further connected to GPU 120, CPU 118, and FPGA 122. Central interconnect 124 is further connected to input and output interfaces 128 and network interface 126.

[0029] Figure 2 Depicted is a second example hybrid computing system 200 that may be used to implement a neural network associated with the operations of one or more portions or steps of process 1000. In this example, processors associated with the hybrid system include a field programmable gate array (FPGA) 210 and a central processing unit (CPU) 220.

[0030] The FPGA is electrically connected to an FPGA controller 212, which interfaces with a Direct Memory Access (DMA) 218. The DMA is connected to an input buffer 214 and an output buffer 216, which are coupled to the FPGA to buffer data entering and leaving the FPGA, respectively. The DMA 218 includes two First In First Out (FIFO) buffers, one for the host CPU and the other for the FPGA. The DMA allows data to be written to and read from the appropriate buffers.

[0031] On the CPU side of the DMA is a master switch 228 that shuttles data and commands to the DMA. The DMA is also connected to an SDRAM controller 224, which allows data to be shuttled to and from the FPGA to the CPU 220. The SDRAM controller is also connected to external SDRAM 226 and the CPU 220. The master switch 228 is connected to a peripheral interface 230. A flash memory controller 222 controls permanent storage and is connected to the CPU 220.

[0032] In some advanced sensors, different pixels within the sensor can be exposed with different integration times, gains, and / or sensitivities. The outputs of the individual pixels are then fused into a single output frame. The different pixels act as one, and this is transparent to the user, so it behaves like a single high dynamic range sensor. These sensors are also expensive due to the complex pixel design and timing control. Typically, the different exposures cover different spatial spans within the pixel array area, and the full-scale high dynamic range output sacrifices the sensor's native resolution for the increased dynamic range.

[0033] Methods have been proposed for capturing high dynamic range images using non-high dynamic range sensors, where the sensor captures the scene multiple times and then combines the information into a high dynamic range output.

[0034] One method currently used is to capture a scene multiple times with different exposure times and / or ISOs using Multiple Exposure (ME). Shorter exposure times and lower ISOs capture the scene in well-lit areas, allowing for higher sensitivity with longer exposure times and higher ISOs in poorly lit areas. This method allows the most relevant information to be selected from the different exposures and merged into a single frame.

[0035] Currently, there is growing interest in processing photographic data using raw data due to the benefits of using raw data-based linearity and noise models. These methods capture a scene in a linear raw format using multiple short, identical exposures, a method known as single-exposure multi-frame (SEMF) methods. These short-exposure images can be aligned and accumulated. As a result of this accumulation, well-illuminated areas in a single short-exposure image can be extended to higher dynamic range values, while poorly illuminated areas can be accumulated to higher signal levels, just as if they were exposed using a long exposure.

[0036] Globally and locally aligned frames aligned with the reference frame are essentially accumulated into an intermediate image along with weights for local variations. Equation (1)

[0037]

[0038] Where x represents the coordinate of the pixel on the image plane, F represents the number of frames, i is the index of the frame, and W i (x) represents the weight of the i-th image of the pixel at x, Im_i(x) represents the i-th image value of the pixel at x, and CumImage(x) is the cumulative image of the pixel at x.

[0039] Essentially, the accumulated image is normalized to the weights of the local variations. In addition, another global normalizer K is introduced to facilitate the use between dynamic range expansion and noise reduction. The merged image can be expressed as Equation (2)

[0040]

[0041] Where MergedImage(x) represents the merged image of the pixel at x.

[0042] To simplify the analysis of K, for x, W i (x) = 1, and 1≤i≤F. Therefore, the combined image becomes equation (3)

[0043]

[0044] where SimpleMergedImage(x) is the merged image of the pixel at x obtained after simplifying the analysis of K.

[0045] From equation (3), several terms about the normalizer K become known.

[0046] 1. When K is 1, the merged image has the same signal level as each input image, making the output essentially a denoised version of the input image. Therefore, the output has the same EV value as the input image. This can be called single-exposure multi-frame denoising.

[0047] 2. When K is less than 1, the maximum value of the merged image is approximately 1 / K times the maximum value of the input individual images. As a simple example, when K = 0.25, the dynamic range of the merged image is approximately 4 times that of the individual images. The product of K and F is generally greater than 1, otherwise the simple merged image becomes the sum of images with a digital gain of 1 / (KF), which increases the signal level without any practical benefit.

[0048] 3. When K equals 1 / F, the merged image is simply the sum of the input images, which is approximately F times longer exposure than the individual short exposures while preserving good lighting detail. This means the dynamic range is extended by a factor of F. Compared to the individual input images, the equivalent EV value increases by approximately log2(F). This is called single-exposure multi-frame fusion.

[0049] High Dynamic Range Plus (HDR+) is a single-exposure, multi-frame approach. Linear data generation is performed at full resolution, aligned, and merged, with noise reduction and / or dynamic range expansion. Subsequent image signal processing, such as white balancing, demosaicing, chroma denoising, tone mapping, and sharpening, is typically performed in conventional image signal processors or tone mapping of high dynamic range data. HDR+ is robust to alternating reference frame selection and can be converted to denoised video.

[0050] The reference exposure selection method is similar to single exposure multi-frame denoising, in that the sensor captures the raw data stream at an underexposed setting. In one example, the individual exposure value (EV) can be -T (T>0), i.e., the data value is approximately 1 / 2 of the common exposure. T If the single exposure multi-frame fusion process is 2 T If two frames are used, then after aligning these frames and accumulating them, the accumulated image in equation (3) is approximately 2 times that of a single image. T The output is similar to the output of the common exposure (EV=0), where the saturation level is increased to 2 times of the individual frame EV=-T. T times.

[0051] In another example, a sensor may capture 16 raw data frames with an exposure of EV=-4, 10-bit frames. After alignment and accumulation, the well-lit area data can be expanded to 14 bits, and the poorly lit areas of the same scene have values ​​close to EV=0. Therefore, the accumulated data is a dynamic range-extended version.

[0052] Extending this concept to video mode, the sensor can capture 10-bit raw frames at EV=-T (with an example value of T=4). Set input frame N-1 as the reference and accumulate input frames (N-2 T )~(N-1), single exposure multi-frame fusion can output (10+T)-bit high dynamic range data with EV=0, aligned with input frame N-1. When the next original frame (frame N) is captured, it can be set as the new reference frame and the accumulated input frame (N-2) can be used to generate the reference frame. T +1)~(N), single-exposure multi-frame fusion can output (10+T)-bit high dynamic range data aligned with frame N, and so on.

[0053] Putting these output frames together forms a (10+T)-bit high dynamic range video, where the brightness level is made similar to EV=0. Figure 3 The processing flow is shown.

[0054] Figure 3 An example streaming video mode is depicted with single-exposure multi-frame fusion of scrolling reference images. In a first iteration, 10-bit low dynamic range 328 frames 310-316 are fused in single-exposure multi-frame module 320, where the input frames are 10-bit low dynamic range and the output 322 is 14-bit high dynamic range 330. The frames are indexed one by one so that in the next iteration, low dynamic range 328 frames 312-318 are input in raw data format at 10 bits, single-exposure multi-frame fusion module 324 is executed, and output frame 326 is 14-bit high dynamic range 330.

[0055] By applying a conventional image signal processor and high dynamic range tone mapping methods, the video can be previewed on a low dynamic range display.

[0056] As mentioned above, one of the steps in the single-exposure multi-frame method is to align and merge multiple low dynamic range raw frames into one. By choosing different regularizers K≤1 in equation (2), denoising and / or dynamic range expansion can be achieved.

[0057] In the single-exposure multi-frame method, the underexposure EV may have a lower limit due to the limitation of capturing a large number of raw frames on a mobile phone. In addition, when the EV value is low, the captured raw data itself may be noisy, which limits the possible dynamic range extension of single-exposure multi-frame fusion.

[0058] Continuing with the previous example, if 16 frames of 10-bit raw data are captured at EV=-4, and single-exposure multi-frame fusion produces 14-bit, single-frame high dynamic range data with an equivalent EV=0. The dynamic range can be extended to approximately 16 times that of the direct frame with EV=0. However, in low-light conditions at night, the dynamic range between well-lit areas and poorly lit areas, such as those illuminated by neon lights, can be much higher than 16 times, so even when EV=-4, the well-lit areas lose detail / color. In this case, the exposure value for the well-lit areas needs to be further reduced. In this case, an additional short exposure can be captured at EV=-8 to cover the well-lit areas. This then creates a potential problem when combining the 10-bit raw data with EV=-8 into a 14-bit single-frame high dynamic range data with EV=0. This combination of different exposure data with different EV values ​​is called high dynamic range-low dynamic range fusion and can accommodate different bit depths.

[0059] In the case of a single frame, the high dynamic range data with EV=0 is H0, the h bits with the maximum value are MaxH≈2 h ; and low dynamic range data with EV=-E as L -E , the l-bit with the maximum value is MaxL≈2 l The definition of EV is logarithmic, H0 and L -E The exposure ratio between can be given by equation (4).

[0060]

[0061] For an unsaturated pixel X in H0, due to the linearity of the original data, H0(X) and RL -E The values ​​of (X) may be comparable, but may not be the same due to the presence of noise. Typically, the signal-to-noise ratio (SNR) of H0(X) may be higher than that of RL -E (X) signal-to-noise ratio.

[0062] For saturated pixel X in H0, the data in the combined result can be selected as RL -E (X), which implies the maximum value of H0 and L -E Another constraint on the maximum value of , that is, R·MaxL≥MaxH, in other words, l+E≥h.

[0063] so, Figure 4 The high-dynamic range - low-dynamic range fusion block shown can be described by Equation (5).

[0064]

[0065] According to Equation (5), when H0(x) < MaxH, HDR out (x) = H0(x), and conversely, HDR out (x) = RL -E (X).

[0066] The maximum value of the combined data can be extended to R·MaxL, where the bit depth is (l + E) bits and the equivalent EV = 0.

[0067] Figure 4 An example of the combination of single-frame high-dynamic range and low-dynamic range data is depicted. The linear low-dynamic range, low-bandwidth, low-exposure value input 410 is fused with the linear, high-dynamic range data input 412. The fusion occurs at the HDR-LDR fusion module 414, and the output is the high-dynamic range linear raw data output 416. The output at 416 has a higher bandwidth and has the same exposure value as the high-dynamic range data input 412.

[0068] High-dynamic range - low-dynamic range fusion extends the data of low exposure value (LEV) by the exposure ratio and places it in the well-lit area of the high exposure value (HEV) data, which is called HEV-LEV fusion.

[0069] EV = ev h The single-frame HEV data with EV = ev is used as HEV, and the h bits with the maximum value are MaxH ≈ 2 h ; and the LEV data with EV = ev l is used as LEV (ev h = ev l + E, E > 0), and the l bits with the maximum value are MaxL ≈ 2 l . The definition of EV is logarithmic, and the exposure ratio between HEV and LEV is given by Equation (6).

[0070]

[0071] For the unsaturated pixels X in HEV, due to the linearity of the raw data, the values of HEV(X) and R·LEV(X) can be comparable, but may not be the same due to the presence of noise. Generally, the signal-to-noise ratio of HEV(X) may be higher than that of R·LEV(X).

[0072] For a saturated pixel X in the HEV, the data in the combined result can be selected as R·LEV(X), which implies another constraint on the maximum value of HEV and the maximum value of LEV, that is, R·MaxL≥MaxH, or equivalently, l+E≥h, having the same constraint as the high dynamic range - low dynamic range fusion.

[0073] Thus, Figure 5 the HEV-LEV fusion block shown can be described by Equation (7).

[0074]

[0075] According to Equation (7), when HEV(x)<MaxH, HDR out (x)=HEV(x), and conversely, HDR out (x)=R·LEV(X).

[0076] The maximum value of the combined data can be extended to R·MaxL, with a bit depth of (l+E) bits and an equivalent EV = EV h .

[0077] Figure 5 FIG. depicts an example combination of data with different exposure values using high exposure value and low exposure value fusion. In this example, the linear low exposure value raw data 510 and the linear high exposure value input data 512 are fused in the HEV-LEV fusion module 514 to output high dynamic range raw high-bitwidth data 516 with the same exposure value as the high exposure value input data 512.

[0078] Using the high dynamic range - low dynamic range fusion method, the dynamic range can be further increased and the output frames can be made into a smooth video.

[0079] Figure 6 An example of a two-exposure mode is shown in. The camera captures 10-bit raw data with two different exposures. When the 2Nth frame is about to come, N frames with EV = -4 are input into single-exposure multi-frame fusion. The single-exposure multi-frame fusion outputs 14-bit high dynamic range data with EV = 0. N frames with 10-bit EV = -8 are used to implement the temporal three-dimensional (3D) denoising module and output 10-bit denoised low dynamic range raw data with EV = -8. The 14-bit high dynamic range data with EV = 0 and the 10-bit denoised low dynamic range data with EV = -8 are input into the high dynamic range - low dynamic range fusion and output 18-bit high dynamic range data with EV = 0. When the sensor captures another two frames with EV = -8 and EV = -4, the same processing flow can produce another output frame. These output frames can create a high dynamic range video with a frame rate reduced by 1 / 2.

[0080] Compared to the timing of the input frames, the interleaved input frame structure (i.e., EV = -8, -4, -8, -4, -8, -4, -8, -4...) can provide evenly distributed output frames, i.e., one frame is output for every two input frames, which allows smooth video output and low dynamic range image input to high dynamic range - low dynamic range fusion allows denoising, and therefore well-illuminated areas exhibit good signal-to-noise ratio.

[0081] Figure 6 Depicted is an example double exposure process for multiple exposure multi-frame video. Frames 610, 614, 618 of lower exposure value (EV=-8) are input with 10-bit low dynamic range 634 and are routed to a three-dimensional (3D) de-noising (DNS) module 622, which outputs a low dynamic range 10-bit output at a lower exposure value EV=-8. Frames 612, 616, 620 of higher exposure value are 10-bit low dynamic range 634 and are routed to a single exposure multi-frame (SEMF) fusion module 624, which outputs a 14-bit high dynamic range frame at the equivalent value of EV=0. The outputs from the three-dimensional (3D) de-noising (DNS) module 622, which outputs 10-bit LDR frames with EV=-8, and the SEMF fusion module 624, which outputs 14-bit HDR frames with EV=0, are fused in an HDR-LDR fusion module 626, which outputs an output frame 628 of 18-bit HDR EV=0 data 636.

[0082] In addition, single-exposure multi-frame (SEMF) is a kind of 3D DNS, where the streams can share almost the same structure as single-exposure multi-frame fusion.

[0083] The multi-exposure multi-frame method flow for video can be extended to more than two exposures to achieve even higher dynamic range. Figure 7 As shown in the example, a sequence of three exposure input frames can produce even higher dynamic range output data, where the output frame rate is 1 / 3 the input frame rate.

[0084] Figure 7An example triple exposure process for multi-exposure multi-frame video with high dynamic range and low dynamic range fusion is depicted. The first set of very low exposure value frames 710, 716, 722 has EV=-12 and is input into the 3D DNS module 728, which outputs an LDR 10-bit output with EV=-12. The second set of low exposure value frames 712, 718, 724 has EV=-8 and is input into the 3D DNS module 730, which outputs a 10-bit LDR with EV=-8. The third set of higher exposure value frames 714, 720, 726 has EV=-4 and is input into the single exposure multi-frame (SEMF) fusion module 732, which outputs a 14-bit HDR with EV=0. Frames 710-726 are raw data 744 with 10-bit values. The 3D DNS module 730 and the SEMF fusion module 732 are input into the HDR-LDR fusion module 734, which outputs an 18-bit HDR with EV=0. The output of the HDR-LDR fusion module 734 and the output of the 3D DNS module 728 are input to the HDR-LDR fusion module 736, which outputs (738) a 22-bit (746) with EV = 0. The previous output from the previous cycle is found in the HDR-LDR fusion module 740, which outputs a 22-bit (742) with EV = 0.

[0085] The multi-exposure multi-frame method flow for video can be extended to more than two exposures using the HEV-LEV structure to achieve even higher dynamic range. Figure 8 As shown in the example, a triple exposure input frame sequence can produce even higher dynamic range output data, where the output frame rate is 1 / 3 of the input frame rate. The function and output are similar to the disclosed multiple exposure multi-frame method with high dynamic range-low dynamic range structure.

[0086] Figure 8Depicted is an example triple exposure process for a multi-exposure multi-frame video with high and low exposure value fusion. The first set of very low exposure value frames 810, 816, 822 has EV = -12 and is input into the 3D DNS module 828, which outputs an LDR 10-bit output with EV = -12. The second set of low exposure value frames 812, 818, 824 has EV = -8, is input into the 3D DNS module 830, and outputs a 10-bit LDR with EV = -8. The third set of higher exposure value frames 814, 820, 826 has EV = -4 and is input into the single exposure multi-frame (SEMF) fusion module 832, and outputs a 14-bit HDR with EV = 0. Frames 810, 826 are raw data 844 with 10-bit values. In this example, the 3D DNS module 828 and the 3D DNS module 830 are input into the HEV-LEV fusion module 834, which fuses and outputs 14-bit HDR raw data with EV = -8. The outputs of the HEV-LEV fusion module 834 and the SEMF fusion module 832 are input into the HEV-LEV fusion module 836, which outputs 22 bits (838) with EV = 0. The output of the index output of the raw frame is 22 bits (846). The previous output from the previous cycle is found in the HHEV-LEV fusion module 840, which outputs 22 bits (842) with EV = 0.

[0087] A first example method for multiple-exposure multi-frame image capture includes: capturing (910) a series of frames at an underexposed setting; selecting (912) a reference image from the series of frames; and aligning (914) the series of frames with the reference image. The method also includes: merging (916) the series of frames into a single frame based on the reference image; indexing (918) the reference image to the next frame in the series of frames, thereby generating an indexed reference image; realigning (920) the series of frames with the indexed reference image; and re-merging (922) the series of frames into a single frame based on the indexed reference image.

[0088] A second method for multiple-exposure multi-frame image capture includes: capturing (1010) a first series of frames at a first underexposure setting; selecting (1012) a first reference image from the first series of frames; aligning (1014) the first series of frames with the first reference image; and merging (1016) the first series of frames into a single first frame based on the first reference image. The method includes: capturing (1018) a second series of frames at a second underexposure setting; selecting (1020) a second reference image from the second series of frames; aligning (1022) the second series of frames with the second reference image; merging (1024) the second series of frames into a single second frame based on the second reference image; and fusing (1026) the single first frame and the single second frame.

[0089] The method may additionally include: indexing the capture of another first series of frames at the first underexposure setting as N; and indexing the capture of another second series of frames at the first underexposure setting as N. The method may alternatively include: iteratively indexing the capture of another first series of frames at the first underexposure setting as N; and iteratively indexing the capture of another second series of frames at the first underexposure setting as N.

[0090] A third method for multiple-exposure, multi-frame image capture includes: capturing (1110) a first series of frames at a first underexposure setting; selecting (1112) a first reference image from the first series of frames; aligning (1114) the first series of frames with the first reference image; and merging (1116) the first series of frames into a first frame based on the first reference image. The method also includes: capturing (1118) a second series of frames at a second underexposure setting; selecting (1120) a second reference image from the second series of frames; aligning (1122) the second series of frames with the second reference image; and merging (1124) the second series of frames into a second frame based on the second reference image. The method includes: capturing (1126) a third series of frames at a third underexposure setting; selecting (1128) a third reference image from the third series of frames; aligning (1130) the third series of frames with the third reference image; and merging (1132) the third series of frames into a third frame based on the third reference image. The method further includes: fusing (1134) a second frame and a third frame into a first fused frame; and fusing (1136) a first frame and the first fused frame.

[0091] The method may further include: capturing another first series of frames at the first underexposure setting with an index of N; capturing another second series of frames at the second underexposure setting with an index of N; and capturing another third series of frames at the third underexposure setting with an index of N.

[0092] The method may alternatively include iteratively indexing as N the capture of another first series of frames at a first underexposure setting; iteratively indexing as N the capture of another second series of frames at a second underexposure setting; and iteratively indexing as N the capture of another third series of frames at a third underexposure setting.

[0093] A fourth method for multiple-exposure, multi-frame image capture includes: capturing (1210) a first series of frames at a first underexposure setting; selecting (1212) a first reference image from the first series of frames; aligning (1214) the first series of frames with the first reference image; and merging (1216) the first series of frames into a first frame based on the first reference image. The method also includes: capturing (1218) a second series of frames at a second underexposure setting; selecting (1220) a second reference image from the second series of frames; aligning (1222) the second series of frames with the second reference image; and merging (1224) the second series of frames into a second frame based on the second reference image. The method also includes: capturing (1226) a third series of frames at a third underexposure setting; selecting (1228) a third reference image from the third series of frames; aligning (1230) the third series of frames with the third reference image; and merging (1232) the third series of frames into a third frame based on the third reference image. The method further includes: fusing (1234) a first frame and a second frame into a first fused frame; and fusing (1236) a third frame with the first fused frame.

[0094] The method may further include: capturing another first series of frames at the first underexposure setting with an index of N; capturing another second series of frames at the second underexposure setting with an index of N; and capturing another third series of frames at the third underexposure setting with an index of N.

[0095] The method may alternatively include iteratively indexing as N the capture of another first series of frames at a first underexposure setting; iteratively indexing as N the capture of another second series of frames at a second underexposure setting; and iteratively indexing as N the capture of another third series of frames at a third underexposure setting.

[0096] In the disclosed multiple-exposure multi-frame method, the input image is in linear raw format, and there are several frames for any different exposures.

[0097] Those skilled in the art will appreciate that the various illustrative blocks, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, parts, methods and algorithms have been generally described above in terms of their functions. Whether this function is implemented as hardware or software depends on the specific application and the design constraints imposed on the system. The technician can implement the described function in different ways for each specific application. Various components and blocks can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of this subject technology.

[0098] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an illustration of example approaches. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the processes may be rearranged. Some steps may be performed simultaneously. The accompanying method claims present elements of the various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0099] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. The previous description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects may be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be given the full scope consistent with the language claims, wherein, unless otherwise stated, reference to an element in the singular is not intended to mean "one and only one", but rather "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. Masculine pronouns (e.g., his) include feminine and neuter (e.g., her and its), and vice versa. Titles and subtitles (if any) are used merely for convenience and do not limit the present invention. The predicate words "configured to", "operable to", and "programmed to" do not imply any specific tangible or intangible modification to the subject matter, but are intended to be used interchangeably. For example, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation, or that the processor is operable to monitor and control the operation. Likewise, a processor configured to execute code may be construed as a processor programmed to execute code or operable to execute code.

[0100] Phrases such as "aspects" do not imply that such aspects are essential to the subject technology or that such aspects apply to configurations of the subject technology. Disclosure relating to an aspect may apply to a configuration, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Phrases such as "embodiments" do not imply that such embodiments are essential to the subject technology or that such embodiments apply to configurations of the subject technology. Disclosure relating to an embodiment may apply to an embodiment, or one or more embodiments. An embodiment may provide one or more examples. Phrases such as "embodiments" may refer to one or more embodiments, and vice versa. Phrases such as "configurations" do not imply that such configurations are essential to the subject technology or that such configurations apply to configurations of the subject technology. Disclosure relating to a configuration may apply to a configuration, or one or more configurations. A configuration may provide one or more examples. Phrases such as "configurations" may refer to one or more configurations, and vice versa.

[0101] The word “exemplary” is used herein to mean “serving as an example or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0102] All structural and functional equivalents to the elements described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is expressly recited in the claims. Furthermore, to the extent the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive in a manner similar to the term "comprising," as interpreted when "including" is used as a transitional word in the claims.

[0103] References to "one embodiment," "an embodiment," "some embodiments," "various embodiments," etc. indicate that a particular element or feature is included in at least one embodiment of the present invention. Although these phrases may appear in various places, they do not necessarily refer to the same embodiment. In conjunction with this disclosure, those skilled in the art will be able to design and incorporate any of a variety of mechanisms suitable for implementing the aforementioned functions.

[0104] It should be understood that this disclosure teaches only one example of illustrative embodiments and that those skilled in the art may readily devise many variations of the invention after reading this disclosure, and that the scope of the invention is determined by the appended claims.

Claims

1. A method for capturing multiple exposure and multiple frame images for producing HDR video, comprising: capturing a first series of frames at a first underexposed setting; selecting a first reference image from the first series of frames; aligning the first series of frames with the first reference image; merging the first series of frames into a first frame based on the first reference image; capturing a second series of frames at a second underexposed setting; selecting a second reference image from the second series of frames; aligning the second series of frames with the second reference image; merging the second series of frames into a second frame based on the second reference image; as well as The one first frame and the one second frame are fused.

2. The method for capturing multiple-exposure images according to claim 1 , further comprising: indexing the capture of another first series of frames at the first underexposure setting as N; as well as The capture of another second series of frames at the first underexposure setting is indexed as N.

3. The method for capturing multiple-exposure images according to claim 1 , further comprising: iteratively indexing N the capture of another first series of frames at the first underexposure setting; as well as The capture of another second series of frames at the first underexposure setting is iteratively indexed as N.

4. The method for capturing multiple exposure and multiple frame images according to claim 1, wherein: The first series of frames and the second series of frames are raw data.

5. The method for capturing multiple exposure and multiple frame images according to claim 1, wherein: The first series of frames and the second series of frames are low dynamic range.

6. A method for capturing multiple exposures and multiple frames of images for producing HDR video, comprising: capturing a first series of frames at a first underexposed setting; selecting a first reference image from the first series of frames; aligning the first series of frames with the first reference image; merging the first series of frames into a first frame based on the first reference image; capturing a second series of frames at a second underexposed setting; selecting a second reference image from the second series of frames; aligning the second series of frames with the second reference image; merging the second series of frames into a second frame based on the second reference image; A third series of frames is captured at a third underexposed setting; selecting a third reference image from the third series of frames; aligning the third series of frames with the third reference image; merging the third series of frames into a third frame based on the third reference image; fusing the second frame and the third frame into a first fused frame; as well as The one first frame and the first fused frame are fused.

7. The method for capturing multiple-exposure multiple-frame images according to claim 6 , further comprising: indexing the capture of another first series of frames at the first underexposure setting as N; indexing the capture of another second series of frames at the second underexposure setting as N; and The capture of another third series of frames at the third underexposure setting is indexed as N.

8. The method for capturing multiple-exposure and multiple-frame images according to claim 6 , further comprising: iteratively indexing N the capture of another first series of frames at the first underexposure setting; iteratively indexing N the capture of another second series of frames at the second underexposure setting; as well as The capture of another third series of frames at the third underexposure setting is iteratively indexed as N.

9. The method for capturing multiple exposure and multiple frame images according to claim 6, wherein: The first series of frames, the second series of frames, and the third series of frames are raw data.

10. The method for capturing multiple exposure and multiple frame images according to claim 6, wherein: The first series of frames, the second series of frames, and the third series of frames are low dynamic range.

11. The method for capturing multiple-exposure and multiple-frame images according to claim 6 , further comprising: fusing the first frame and the second frame into a first fused frame; as well as The one third frame and the first fused frame are fused.

12. The method for capturing multiple-exposure multiple-frame images according to claim 11 , further comprising: indexing the capture of another first series of frames at the first underexposure setting as N; indexing the capture of another second series of frames at the second underexposure setting as N; and The capture of another third series of frames at the third underexposure setting is indexed as N.

13. The method for capturing multiple-exposure multiple-frame images according to claim 11 , further comprising: iteratively indexing N the capture of another first series of frames at the first underexposure setting; iteratively indexing N the capture of another second series of frames at the second underexposure setting; as well as The capture of another third series of frames at the third underexposure setting is iteratively indexed as N.

14. The method for capturing multiple exposure and multiple frame images according to claim 11, wherein: The first series of frames, the second series of frames, and the third series of frames are raw data.

15. The method for capturing multiple exposure and multiple frame images according to claim 11, wherein: The first series of frames, the second series of frames, and the third series of frames are low dynamic range.

Images