Image generation circuitry and image generation method

Abstract

The disclosure pertains to image generation circuitry that is configured to: obtain a set of input image frames that represent a first and at least a subsequent capture; iteratively merge two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generate an output frame based on the merged set of input image frames.

Description

[0001] IMAGE GENERATION CIRCUITRY AND IMAGE GENERATION

[0002] METHOD

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to image generation circuitry and an image generation method.

[0005] TECHNICAL BACKGROUND

[0006] It is generally known to align and merge a sequence of images such that a motion blur is reduced as compared to averaging over the sequence of images.

[0007] Although there exist techniques for aligning and merging a sequence of images, it is generally desirable to provide improved image generation circuitry and an improved image generation method.

[0008] SUMMARY

[0009] According to a first aspect, the disclosure provides image generation circuitry that is configured to: obtain a set of input image frames that represent a first and at least a subsequent capture; iteratively merge two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generate an output frame based on the merged set of input image frames.

[0010] According to a second aspect, the disclosure provides an image generation method that includes: obtaining a set of input image frames that represent a first and at least a subsequent capture; iteratively merging two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generating an output frame based on the merged set of input image frames.

[0011] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0014] Fig. 1 illustrates an electronic device according to an embodiment; Fig. 2 schematically illustrates a generation of an output frame based on binary frames according to an embodiment;

[0015] Fig. 3 illustrates a first embodiment of an image generation method;

[0016] Fig. 4 illustrates a second embodiment of an image generation method;

[0017] Fig. 5 illustrates an example of a flat accumulation of image frames;

[0018] Fig. 6 illustrates a first embodiment of a hierarchical accumulation;

[0019] Fig. 7 illustrates a second embodiment of a hierarchical accumulation;

[0020] Fig. 8 illustrates a third embodiment of an image generation method;

[0021] Fig. 9 illustrates a fourth embodiment of an image generation method; and

[0022] Fig. 10 illustrates an embodiment of a general-purpose computer.

[0023] DETAILED DESCRIPTION OF EMBODIMENTS

[0024] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0025] As mentioned in the background, it is generally known to align and merge a sequence of images such that a motion blur is reduced as compared to averaging over the sequence of images.

[0026] For example, for quanta burst imaging (QBI), a large number of image frames may be acquired at a high frame rate and low exposure, wherein each of the acquired image frames may include little light information and a low signal-to-noise ratio (SNR). The large number of acquired image frames may be merged into one output frame that may exhibit more light information and a high SNR. The merging of the image frames may include compensating for a movement of a camera that acquires the image frames and / or for a motion in an imaged scene in order to reduce a motion blur in the output frame.

[0027] In order to stabilize a camera image and achieve high low-light sensitivity, each image frame may be analyzed to detect motion, and then image frames, which are neighboring or consecutive in time (there may also be further image frames between two neighboring image frames, i.e. the present disclosure is not limited to aligning of image frames which are all directly consecutive), may be aligned on top of each other to generate a clear (e.g., sharp) image. For example, an extreme amount of image frames (e.g., around 1000 frames per second (fps), without limiting the disclosure to this number) with little light information may be received. The image frames may be subject to camera movement and / or may exhibit individual areas of motion in a scene that is represented by the image frames.

[0028] In some instances, a dense optical flow measurement is used, wherein all pixels in each image frame may be analyzed and tracked for motion. Individual pixels of the image frames may also be aligned with a pixel in a previous image frame that originates from a same point (e.g., from a same feature) in the imaged scene. This alignment of pixels of different neighboring or consecutive image frames may also be performed in a way that a pixel at a spatially same location in an image frame is not necessarily summed with the spatially same pixel in a neighboring frame, but such pixels may refer to a corresponding feature in neighboring frames. Such an alignment of pixels may, in some instances, work well under good light conditions, where features may easily be extracted from the image frames, but in almost complete darkness, there may be very few (or even no) features to detect. It has been recognized that in some embodiments, for the alignment of pixels, a hierarchical pyramid approach may be used, which may combine pattern matching and optical flow into a high performance solution.

[0029] Consequently, some embodiments of the disclosure pertain to image generation circuitry that is configured to: obtain a set of input image frames that represent a first and at least a subsequent capture; iteratively merge two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generate an output frame based on the merged set of input image frames.

[0030] Some embodiments pertain to an image generation method that includes: obtaining a set of input image frames that represent a first and at least a subsequent capture; iteratively merging two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generating an output frame based on the merged set of input image frames.

[0031] The image generation circuitry may be configured to perform the image generation method, and the image generation method may be performed by the image generation circuitry.

[0032] The image generation circuitry may include a processing unit such as a programmed microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or the like. For example, the image generation circuitry may include a central processing unit (CPU), a graphics processing unit (GPU) and / or a tensor processing unit (TPU). The image generation circuitry may be configured to execute instructions (e.g., software and / or firmware) that cause the image generation circuitry to perform the image generation method.

[0033] The image generation circuitry may include a memory unit that may store the instructions executed by the image generation circuitry, the obtained input image frames, the generated output frame, and / or temporary data. The memory unit may be based on Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Read-Only Memory (ROM), flash memory (e.g., NOR or NAND flash), Video Random Access Memory (VRAM), Synchronous Graphics Random Access Memory (SGRAM), Graphics Double Data Rate (GDDR), or the like.

[0034] The image generation circuitry may include a communication unit for communicating with other devices. The communication unit may include an interface according to a specification of the Mobile Industry Processor Interface (MIPI) Alliance (e.g., Camera Serial Interface (CSI), C- PHY, D-PHY, etc.), a Peripheral Component Interconnect (PCI) interface, a Universal Serial Bus (USB) interface, a parallel port (IEEE 1284) interface, a serial port (RS-232) interface, an Ethernet interface, a wireless interface (e.g., Wi-Fi, IEEE 802.11), a mobile telecommunications interface (e.g., High Speed Packet Access (HSPA), Long Term Evolution (LTE), New Radio (NR), etc.), or the like. The image generation circuitry may receive via the communication unit the set of input image frames from an imaging device (e.g., from a semiconductor chip that includes an image sensor). The image generation circuitry may transmit via the communication unit the generated output frame to another device and / or may store the generated output frame in the memory unit.

[0035] For example, the image generation circuitry may be implemented by a general-purpose computer, as described with respect to Fig. 10.

[0036] As mentioned, the image generation circuitry, when performing the image generation method, may obtain the set of input image frames by receiving the set of input image frames from an imaging device. The image generation circuitry and the imaging device may be included in an electronic device such as a camera, a smartphone, a tablet, a notebook, smartglasses, a headmounted display (HMD), or the like.

[0037] The imaging device may include an image sensor with an array (e.g., one-dimensional (ID) or two-dimensional (2D); e.g., arranged in one or more rows and / or one or more columns) of photosensitive elements that may be configured to generate signals which may indicate incident light (e.g., detected photons). The photosensitive elements may include single-photon avalanche diodes (SPADs) that may detect single photons based on avalanche multiplication. The photosensitive elements may include jots that may detect single photons based on a high conversion gain due to a low capacitance. Thus, the image sensor may generate image data even at low light conditions and / or at short exposures, where only few or single photons may be incident to the image sensor.

[0038] The imaging device may further include one or more optical elements (e.g., lens or mirror) that may focus incident light on the array of photosensitive elements such that each photosensitive element of the array may receive light from a predefined direction (e.g., solid angle).

[0039] The imaging device may read the signals from the photosensitive elements at a predefined rate (e.g., at 10,000 fps or 100,000 fps, without limiting the disclosure to these values or to this range) and may generate image frames based on the signals read out from the photosensitive elements. The image frames may include pixels, wherein each pixel may be associated with one photosensitive element or with several (binning; e.g., two or four, without limiting the disclosure to these values) photosensitive elements. The pixels may be arranged in the image frames like their associated photosensitive elements (e.g., for a 2D array of photosensitive elements, the pixels may also be arranged in a 2D array). The imaging device may assign to each pixel of an image frame a value that may indicate whether the associated photosensitive element(s) has / have detected a photon in a time interval represented by the image frame (binary frame) or how many photons the associated photosensitive element(s) has / have detected in the time interval.

[0040] The image sensor may include color filters for detecting light of predefined colors (e.g., wavelength intervals), such as red, green and blue. The color filters may be arranged according to a predefined pattern (e.g., Bayer pattern) such that the photosensitive elements may detect incident light of predefined colors, and the image frames may represent color (e.g., red-green- blue (RGB)) images. However, the disclosure is not limited to color images. For example, all photosensitive elements may be configured to detect light in a same wavelength interval (e.g., in the visible range, in an infrared range, etc. or a combination thereof), and the image frames may represent grayscale images.

[0041] For reducing a required transmission data rate from the imaging device to the image generation circuitry, the imaging device may merge (accumulate) a predefined number of image frames (e.g., 4, 6, 8, 9, 10, 12, 16, 20, 32, 50, 100, 128 or the like, without limiting the disclosure to these values or to this range) into one merged image frame, and may transmit to the image generation circuitry the merged image frames generated this way. For example, the imaging device may simply add up values of corresponding pixels of the predefined number of image frames (e.g., pixels at a corresponding coordinate in the image frames), based on the assumption that an impact of potential motions within an imaged scene or of potential movements of the imaging device with respect to the scene across the predefined number of image frames does not (or not significantly) extend beyond a size of one photosensitive element (and, thus, of one pixel) due to the high frame rate.

[0042] The image generation circuitry may receive the (merged) image frames from the imaging device as input image frames. The input image frames may represent a first and at least a (or multiple) subsequent captures of a scene imaged by the imaging device, wherein a capture may correspond to a period in which the imaging device acquires the predefined number of image frames of one merged image frame. For example, the image generation circuitry may receive 1000 input image frames per second from the imaging device (which are based, for example on a corresponding number of captures which are consecutive in time). However, the disclosure is not limited to 1000 input image frames. The image generation circuitry may as well receive the input image frames at a frame rate of 100 fps, 500 fps, 900 fps, 1500 fps or the like, or at any other suitable frame rate within or outside of this range.

[0043] Due to the high frame rate and, thus, the short exposure, the input image frames may have little light information and may suffer from a high shot noise and a low SNR. Therefore, for generating an output frame with high light information, low shot noise and a high SNR, the image generation circuitry may merge the obtained set of input image frames (e.g., based on adding up values of corresponding pixels or values derived from corresponding pixels).

[0044] However, a portion of the scene that is represented by the input image frames may change between the input image frames. For example, an object in the scene may move such that it may be imaged at different positions in the scene in different captures. For example, the imaging device may be moved (e.g., rotated and / or translated) with respect to the scene such that the different input image frames may show different parts of the scene, and / or the different input image frames may show a part of the scene at different coordinates in the image frame.

[0045] Therefore, simply adding up values of pixels at a same coordinate in the set of input image frames may cause motion blur. For reducing or avoiding a motion blur in the output frame, the image generation circuitry may align the pixels of the input image frames before adding them up. The aligning of the pixels may include determining which pixels in the set of input image frames correspond to a same object, feature or coordinate in the scene. The image generation circuitry may then sum up values of such pixels that correspond to the same object, feature or coordinate in the scene even if the pixels have different coordinates in the set of input image frames. The aligning of the pixels may include determining an optical flow. In some embodiments, an optical flow defines how a pixel changes from a first image (or image frame) to second image (or image frame), for example from one image frame to a subsequent image frame (wherein also here, one or more image frames may be present between a first and a subsequent image frame, although in some embodiments the first image frame and the subsequent image frame are directly consecutive in time). The change in the optical flow may be defined in the form of a (data) vector or any other kind of format which is suitable to represent the change in the optical flow. The optical flow may indicate which pixels (or portions) of two image frames correspond to each other and should be added. The image generation circuitry may determine a dense optical flow, e.g., the image generation circuitry may determine for each pixel of an image frame to which pixel of a preceding image frame it corresponds. For example, the optical flow may indicate a motion vector that may represent a motion from a position that may correspond to a first coordinate in a first image frame to a position that may correspond to a second coordinate in a second image frame. If no corresponding pixel in another image frame is found for a pixel, the image generation circuitry may ignore the pixel such that no optical flow may be determined for the pixel. For example, the image generation circuitry may determine the optical flow based on a standard (or traditional) optical flow algorithm, e.g., phase correlation, sum of absolute differences, the Lucas-Kanade method, the Hom-Schunck method, or the like. The image generation circuitry may also determine the optical flow based on template matching, which may be suitable for image frames with low light information and a low SNR. The image generation circuitry may also determine the optical flow based on a machine learning algorithm (e.g., decision trees, support vector machines (SVM), an artificial neural network such as a convolutional neural network (CNN), a forward propagation network, a multilayer perceptron (MLP) network, etc.), which may require fewer merging iterations and / or may provide a higher quality output.

[0046] The iterative merging may include merging the set of input image frames pairwise in multiple merging iterations. The two neighboring frames may be image frames that may correspond to two neighboring time points or time intervals (e.g., that may have been acquired at two subsequent time points or time intervals, or at two time points or time intervals within a predefined time range from among time intervals at which the imaging device may acquire image frames) and that should be merged in a same merging iteration. The merging may include determining a dense optical flow between the two neighboring frames, and merging values of corresponding pixels of the two neighboring frames. The optical flow may indicate which respective pixels of the two neighboring frames correspond to each other. The merging of the values may include adding the values, determining an average or median of the values or the like. The merging may include generating a merged frame whose pixels may have the merged values of corresponding pixels of the two neighboring pixels.

[0047] A merged value may be assigned to a pixel at a coordinate in the merged frame that corresponds to a coordinate of one of the corresponding pixels of the two neighboring frames. For example, the merged values may be assigned to pixels of the merged frame at coordinates that correspond to the earlier one of the two neighboring frames (e.g., to the one that corresponds to an earlier time) or to the later one of the two neighboring frames (e.g., to the one that corresponds to a later time). For example, the merged values may be assigned to pixels of the merged frame at coordinates that correspond to the one of the two neighboring frames that is closer to a time to which the output frame should correspond. In a case where no merged value is available for a pixel of the merged frame because no corresponding optical flow could be determined, the pixel of the merged frame may be assigned the value of the pixel at a corresponding coordinate in one of the two neighboring frames (e.g., the earlier one, the later one, or the one that is closer to the output frame). A normalization may be performed that may account for such pixels of a merged frame.

[0048] After each merging iteration, the image generation circuitry may determine whether the criterion is met. If the criterion is not met, the image generation circuitry may perform a subsequent merging iteration. If the criterion is met, the image generation circuitry may generate the output frame.

[0049] In the output frame, pixel values from all input image frames of the set of input image frames may be merged such that a SNR of the output frame may be increased and a shot noise in the output frame may be reduced as compared to the input image frames.

[0050] For example, the image generation circuitry may use, as the output frame, a merged frame from a last merging iteration before the criterion was determined to be met. For example, the image generation circuitry may generate the output frame by merging the set of input image frames according to the optical flow determined in the merging iterations.

[0051] The image generation circuitry may normalize the pixel values of the output frame. The normalizing may include, for example, dividing the pixel values by the number of input image frames in the set of input image frames, or dividing each pixel value by the number of pixel values on which it is based (which may be fewer than the number of input image frames if an optical flow could not be determined for at least one pair of neighboring frames in a merging iteration). Instead of dividing the pixel values by a number, the normalizing may also include multiplying the pixel values with a corresponding scaling factor, e.g., in order to scale the pixel values to a predefined value range.

[0052] The image generation circuitry (and / or the electronic device that includes the image generation circuitry) may store the output frame, display the output frame, transmit the output frame via the communication unit to another device and / or perform further processing (e.g., analyzing the output frame and / or controlling a machine, robot, autonomous vehicle or the like based on the output frame). The image generation circuitry may output the output frame as a frame of a stream (e.g., video stream, movie) and / or may, after obtaining the set of input image frames, obtain a further set of input image frames and generate, based on the further set of input image frames, a further output frame for the stream.

[0053] In some embodiments, the iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration.

[0054] For example, in a first merging iteration, the image generation circuitry may determine an optical flow for pairs of neighboring frames in the set of input image frames and generate a merged frame for each of the pairs of neighboring frames based on the determined optical flow. In a subsequent merging iteration, the image generation circuitry may determine an optical flow for pairs of neighboring frames that have been generated as merged frames in a preceding merging iteration.

[0055] The disclosure is not limited to generating the merged frames based on merging only two neighboring frames. The merged frames are generated based on merging three, four or any other suitable number of image frames in a predefined neighborhood in some embodiments.

[0056] Thus, each merging iteration may reduce a number of image frames while increasing light information of the merged frames, such that an impact of a shot noise may be reduced and a SNR may be increased in the merged frames.

[0057] The iterative merging is in some embodiments referred to as a hierarchical accumulation, wherein the term “hierarchical” may refer to the iterative merging of image frames and the term “accumulation” may refer to increasing light information in the image frames (e.g., by adding up values of corresponding pixels). In some embodiments, the generating of the output frame includes merging the set of input image frames based on the determined optical flow between the respective neighboring frames.

[0058] The image generation circuitry may perform the merging iterations for obtaining the optical flow between the respective neighboring frames. When the iterative merging of the set of input image frames is completed (e.g., when all necessary optical flows for merging the set of input image frames have been determined), the image generation circuitry may generate the output frame by merging the set of input image frames according to the determined optical flows.

[0059] For example, the iterative merging of the set of input image frames may be performed for the purpose of obtaining the optical flows. The image generation circuitry may adapt the set of input image frames such that the iterative merging can be performed faster. For example, the image generation circuitry may downscale the input image frames (e.g., reduce a pixel resolution (number of pixels) of the input image frames), convert a colorspace of the input image frames (e.g., convert color / RGB images into grayscale images), and / or apply an image processing operation (e.g., edge detection, contrast amplification, sharpening etc.) prior to the iterative merging such that less computational effort and / or less time may be needed for the iterative merging.

[0060] After the iterative merging, the image generation circuitry may discard the merged frames and may merge the set of input image frames (e.g., in their original quality) based on the determined optical flow to generate the output frame.

[0061] In some embodiments, the image generation method further includes: downscaling the set of input image frames; and performing the iterative merging on the downscaled frames.

[0062] The downscaling may include reducing a resolution (number of pixels) of the input image frames, e.g., reducing a number of rows and / or columns of pixels in the frames. For example, the image generation circuitry may halve the number of rows of pixels and halve the number of columns of pixels such that a number of pixels in the downscaled frames may be a quarter of a number of pixels in the input image frames. The iterative merging (and the determining of the optical flow) may therefore be performed on fewer pixels than the input image frames and, thus, may be faster than for the number of pixels of the input image frames.

[0063] For generating the output frame, the image generation circuitry may then merge the input image frames at their original resolution based on the optical flow, wherein the image generation circuitry may scale the optical flow according to the downscaling ratio. The image generation circuitry may also interpolate the optical flow between input image frames based on the optical flow determined for the downscaled frames.

[0064] For downscaling, the image generation circuitry may add up the values of pixels in a 2-by-2 neighborhood of an input image frame and may assign the resulting value to a pixel at a corresponding coordinate in the downscaled frame. Thus, a light information and a SNR in the downscaled frames may be increased.

[0065] However, the disclosure is not limited to adding up the pixel values of the input image frames. The image generation circuitry determines, in some embodiments, pixel values of the downscaled frames based on a mean or a median of pixel values in the pixel neighborhood of the input image frames or based on selecting a predefined or random pixel value from the pixel neighborhood of the input image frames.

[0066] Further, the disclosure is not limited to downscaling by halving a number of rows and columns of pixels. In some embodiments, the number of rows and / or columns or pixels is reduced to a third or fourth with regard to the input image frames. The skilled person may find further suitable downscaling ratios. The number of pixel rows and the number of pixel columns may be downscaled at a same ratio or at different ratios. The number of pixel rows may also be kept while the number of pixel columns may be reduced or vice versa.

[0067] In some embodiments, the iterative merging includes: using a first algorithm for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration.

[0068] The image generation circuitry may apply an algorithm for determining an optical flow according to an amount of light information in the image frames that may be expected for the respective merging iteration according to a number of previous merging iterations.

[0069] For example, if the first merging iteration is an earlier merging iteration in which the image frames exhibit less light information and a lower SNR than in the second merging iteration, the first algorithm may be an algorithm that may be more suitable (e.g., fine-tuned or optimized) for determining an optical flow in low-light frames, and the second algorithm may be an algorithm that may be more suitable (e.g., fine-tuned or optimized) for determining an optical flow in highlight frames. For example, the first algorithm may include a template matching algorithm for determining an optical flow at low-light conditions, and the second algorithm may be a standard optical flow algorithm for determining an optical flow at high-light conditions. The image generation circuitry may switch from the first algorithm to the second algorithm after a predefined number of merging iterations and / or after detecting that an amount of light information, a SNR, an amount of shot noise or the like satisfies a predefined condition (e.g., a threshold).

[0070] Thus, the optical flow may be determined by an algorithm that may be fine-tuned or optimized for a light level of the image frames in the respective merging iterations. Therefore, the optical flow may be determined more accurately, more robustly and / or faster than with a single algorithm for all merging iterations.

[0071] In some embodiments, the iterative merging includes merging a plurality of neighboring frames based on a determined optical flow between respective ones of the plurality of neighboring frames and a reference frame of the plurality of neighboring frames.

[0072] The reference frame may be selected from the plurality of neighboring frames. For example, the image generation circuitry may select, as the reference frame, an imaging frame of the plurality of neighboring frames that may correspond to an earliest time, to a latest time, to a time that may be closest to a mean or median of the plurality of neighboring frames, or to a time that may be closest to the output frame.

[0073] The image generation circuitry may then determine an optical flow between the selected reference frame and each respective remaining frame of the plurality of neighboring frames, and may generate a merged frame based on merging the plurality of neighboring frames according to the determined optical flow.

[0074] Thus, the image generation circuitry may generate a merged frame based on merging more than two neighboring frames in a same merging iteration. Therefore, a number of merging iterations may be reduced, such that the image generation method may be faster. Further, the optical flows between the reference frame and the further frames of the plurality of neighboring frames may be performed in parallel (e.g., by different processing units, such as CPU cores or GPU shaders), such that the image generation circuitry may perform the image generation method faster.

[0075] The image generation circuitry may also generate the merged frames based on merging different numbers of neighboring image frames in different merging iterations. For example, the image generation circuitry may merge more than two (e.g., three, four, eight or the like, without limiting the disclosure to these numbers) image frames into a merged frame in an earlier merging iteration, and may merge two image frames into a merged frame in a later merging iteration.

[0076] In some embodiments, the criterion indicates a predefined number of frames. The criterion may indicate a predefined number of frames that should be merged. When the image generation circuitry determines that the predefined number of frames is merged (e.g., that optical flows for merging the predefined number of frames have been determined), or that the predefined number of merged frames has been generated, or that a merged frame is based on the predefined number of frames from preceding merging iterations, the image generation circuitry may generate the output frame.

[0077] In some embodiments, the criterion indicates a predefined image quality.

[0078] The image quality may correspond to an amount of light, a SNR, a shot noise, or the like in the merged frames. The image generation circuitry may generate the output frame when detecting that the merged frames meet the image quality.

[0079] The criterion may also indicate a combination of the predefined number of frames and the predefined image quality. For example, the image generation circuitry may merge at least the predefined (minimum) number of frames before determining whether the predefined image quality is met, such that a minimum number of merged frames may be ensured. For example, the image generation circuitry may generate the output frame if a predefined (minimum) number of frames is merged even if the predefined image quality is not (yet) reached, such that a predefined frame rate of output frames may be ensured even at extremely low light.

[0080] In some embodiments, the merging includes summing associated portions of the frames.

[0081] As mentioned, the image generation circuitry may add up corresponding pixels of the frames to be merged (e.g., when merging the two neighboring frames in a merging iteration, or when merging the set of input image frames to generate the output frame), e.g., according to an optical flow between the frames to be merged.

[0082] However, the portions of the frames are not limited to single pixels. In some embodiments, the merging includes summing associated patches of pixels (e.g., N-by-M neighborhoods of pixels, where N and M may be integers and may be equal to each other or differ from each other, such as 2-by-2, 3-by-3, l-by-2, 2-by-l, l-by-3, 3-by-l, 2-by-3, 3-by-2 or the like). The summing of associated patches may include summing pixels at corresponding coordinates in the patches separately, summing sums / means / medians or the like of the patches, or the like.

[0083] The portions may be associated according to an optical flow between the frames. For example, a first portion of a first frame may be associated with a second portion of a second frame if an optical flow between the first frame and the second frame indicates that the first portion corresponds to the second portion (e.g., that the first portion and the second portion represent a same object / feature that has moved from a coordinate of the first portion in the first frame to a coordinate of the second portion in the second frame).

[0084] In some embodiments, the image generation method further includes: converting the set of input image frames to grayscale; and performing the iterative merging on the converted grayscale frames.

[0085] For example, the set of input image frames may include color (e.g., RGB) frames (e.g., may include red, green and blue channels). The image generation circuitry may convert the color frames into grayscale frames (e.g., with a single grayscale channel) according to any suitable known method. Thus, a size of the image frames in memory may be reduced, and the iterative merging may be faster.

[0086] For generating the output frame, the image generation circuitry may merge the set of input image frames (e.g., in their original quality) according to the optical flows determined based on the converted grayscale frames, such that the output frame may include color information (e.g., red, green and blue channels).

[0087] It is noted that the color frames are not limited to RGB and may as well use HSV, YUV or any other suitable color model for representing colors.

[0088] It is noted that the features described for the image generation circuitry may be features of the image generation method as well, and vice versa. Further, the features described above may be combined in any suitable way.

[0089] Some embodiments pertain to an electronic device that includes an imaging device and the image generation circuitry described above, wherein the imaging device is configured to acquire the set of input image fames.

[0090] The imaging device may be configured as described above. For example, the imaging device may include the image sensor with an array of photosensitive elements (e.g., SPADs or jots) as well as one or more optical elements that may focus incident light on the array of photosensitive elements. The imaging device may generate a plurality of image frames based on signals read out from the photosensitive elements. The imaging device may accumulate a predefined number of (e.g., subsequent) generated image frames to a merged image frame, as described above, and may provide a plurality of such merged image frames to the image generation circuitry as the set of input image frames. In some embodiments, the accumulating of image frames to a merged image frame is omitted, and the imaging device provides a plurality of the generated image frames to the image generation circuitry as the set of input image frames. The electronic device may be configured as a camera, a smartphone, a tablet, a notebook, smartglasses, a HMD, or the like, as described above.

[0091] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0092] Returning to Fig. 1, Fig. 1 illustrates an electronic device 1 according to an embodiment. The electronic device 1 includes an imaging device 2 and image generation circuitry 3.

[0093] The imaging device 2 includes an image sensor 2a with an array of SPADs. The imaging device 2 further includes a lens 2b that focuses incident light on the array of SPADs of the image sensor 2a. The SPADs convert the incident light to electric signals. The imaging device 2 generates image frames based on the electric signals from the SPADs and transmits the generated image frames to the image generation circuitry 3.

[0094] The image generation circuitry 3 includes a processing unit 3a, a memory unit 3b and a communication unit 3c. The processing unit 3a includes a CPU and a GPU, and performs image processing such as downscaling image frames, converting color image frames to grayscale image frames, determining an optical flow between image frames, merging (accumulating) image frames, generating output frames etc., as described herein. The memory unit 3b stores software instructions for causing the processing unit 3 a to perform the image processing. The memory unit 3b also stores temporary data that are necessary for the image processing, such as downscaled image frames, grayscale image frames, an optical flow between image frames, merged (accumulated) image frames etc. The communication unit 3 c receives the image frames from the imaging device 2.

[0095] The electronic device 1 (including the image generation circuitry 3) is configured to perform an image generation method, e.g., any one of the methods as described with respect to Fig. 2, 3, 4, 6, 7, 8 or 9.

[0096] Fig. 2 schematically illustrates a generation of an output frame based on binary frames according to an embodiment. A camera sensor 11 (e.g., the image sensor 2a of Fig. 1) on a camera chip 12 (e.g., the imaging device 2 of Fig. 1) generates binary frames at a high frame rate of 104fps. In some embodiments, the camera sensor 11 generates the binary frames at a frame rate of 105fps. Other frame rates (below 104fps, between 104fps and 105fps, or above 105fps) are also used in some embodiments. Each pixel of the binary frames has one of two possible values. A first value indicates that the SPAD that corresponds to the pixel did not detect a photon during the time interval that corresponds to the binary frame, and a second value indicates that the SPAD that corresponds to the pixel did detect a photon during the time interval that corresponds to the binary frame. The camera chip 12 accumulates n subsequent binary frames (i.e., adds the binary frames pixel-wise) to one input frame, where n is a suitable integer number. Fig. 2 illustrates five input frames (input frame 0 to input frame 4).

[0097] The camera chip 12 transmits the input frames to image generation circuitry 13 (e.g., the image generation circuitry 3 of Fig. 1) at a frame rate of 960 fps (without limiting the disclosure to this value). The image generation circuitry 13 performs a hierarchical accumulation 14 (e.g., the hierarchical accumulation at 22 of Fig. 3 or at 34 of Fig. 4, or the hierarchical accumulation of Fig. 6 or of Fig. 7) on the received input frames to generate an output frame. It is noted that the number of five input frames is only chosen for illustrational purposes; the image generation circuitry 13 may generate an output frame based on hierarchical accumulation of any suitable number of input frames, such as 3, 4, 8, 12, 16, 20, 32 etc. input frames, without limiting the disclosure to these values. The image generation circuitry 13 generates output frames at a frame rate of 30 fps (without limiting the disclosure to this value).

[0098] Fig. 3 illustrates a first embodiment of an image generation method 20. The image generation method 20 is an example of a method performed by the image generation circuitry 3 of Fig. 1.

[0099] At 21, the image generation circuitry 3 obtains a set of input image frames (e.g., the input frames of Fig. 2) from the imaging device 2. The input image frames represent a first and at least a subsequent capture by the imaging device 2.

[0100] At 22, the image generation circuitry 3 performs a hierarchical accumulation on the set of input image frames. The hierarchical accumulation is an example of iteratively merging two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met. The iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration. The merging includes summing associated portions of the frames. Examples of a hierarchical accumulation are described with respect to Fig. 6, 7, 8 and 9. The iterative merging includes using a first algorithm 22a for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm 22b different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration. The image generation circuitry 3 uses as the first algorithm 22a a template matching algorithm and as the second algorithm 22b a standard optical flow algorithm that is known in the art. Here, the first merging iteration, in which the first algorithm 22a is applied, is performed earlier than the second merging iteration, in which the second algorithm 22b is applied.

[0101] The criterion indicates a predefined number of frames that should be iteratively merged. When the predefined number of frames has been merged, the image generation circuitry 3 determines that a predefined image quality of an output frame is reached. Thus, the criterion also indicates a predefined image quality.

[0102] At 23, when the criterion is met, the image generation circuitry generates an output frame based on the merged set of input image frames (i . e. , based on the hierarchical accumulation at 22).

[0103] Fig. 4 illustrates a second embodiment of an image generation method 30. The image generation method 30 is an example of a method performed by the image generation circuitry 3 of Fig. 1.

[0104] At 31, the image generation circuitry 3 obtains a set of input image frames (e.g., the input frames of Fig. 2) from the imaging device 2. The input image frames represent a first and at least a subsequent capture by the imaging device 2. The set of input image frames includes color image frames (RGB frames).

[0105] At 32, the image generation circuitry 3 downscales the set of input image frames.

[0106] At 33, the image generation circuitry 3 converts the set of input image frames to grayscale.

[0107] At 34, the image generation circuitry 3 performs a hierarchical accumulation on the (downscaled and grayscale-converted) set of input image frames. The hierarchical accumulation is an example of iteratively merging two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met. The iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration. The merging includes summing associated portions of the frames. Examples of a hierarchical accumulation are described with respect to Fig. 6, 7, 8 and 9. The criterion indicates a predefined number of frames that should be iteratively merged. When the predefined number of frames is merged, the image generation circuitry 3 determines that a predefined image quality of an output frame is reached. Thus, the criterion also indicates a predefined image quality.

[0108] During the hierarchical accumulation at 34, the image generation circuitry 3 generates intermediate output frames 35, which are grayscale frames and have a pixel resolution that corresponds to the downscaled frames. The image generation circuitry 3 uses the intermediate output frames 35 for determining an optical flow 36 between frames. The optical flow 36 indicates motion vectors.

[0109] At 37, the image generation circuitry 3 generates an output frame 38 in a simple flow-based accumulation. The generating of the output frame 38 includes merging the set of input image frames based on the optical flow 36 determined at 34 between the respective neighboring frames. Thus, the generating of the output frame 38 is based on the merged set of input image frames because the generating of the output frame 38 uses the optical flow 36.

[0110] It is noted that, in some embodiments, the hierarchical accumulation at 34 includes using a first algorithm for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration, as described with respect to the hierarchical accumulation at 22 of Fig. 3.

[0111] It is noted that, in some embodiments, the downscaling at 32 is omitted. It is further noted that, in some embodiments, the converting to grayscale at 33 is omitted (e.g., if the set of input image frames includes grayscale images but no color / RGB images).

[0112] Fig. 5 illustrates an example of a flat accumulation of image frames. Input frames 41 (denoted as F_0 to F_15) are merged into one output frame 42 in a simple flow-based accumulation. To this end, an optical flow between the first input frame F_0 and each further input frame is determined. For example, the optical flow between F_0 and F l is denoted as A_0_l, the optical flow between F_0 and F_2 is denoted as A_0_2, and so on. The output frame is then generated by adding portions (e.g., pixels) of the input frames F l to F l 5 to the corresponding portions of the input frame F_0 according to the determined optical flow.

[0113] Fig. 6 illustrates a first embodiment of a hierarchical accumulation (iterative merging), which is an example of the hierarchical accumulation 14 of Fig. 2, of the hierarchical accumulation at 22 of Fig. 3 and of the hierarchical accumulation at 34 of Fig. 4. A set 51 of input image frames F_0 to F_15 corresponds to a light level of K lux. Note that in the embodiment of Fig. 4, the set 51 of input image frames corresponds to downscaled grayscale frames obtained by the downscaling at 32 and converting to grayscale at 33.

[0114] In a first merging iteration, the image generation circuitry 3 determines an optical flow between two neighboring frames (e.g., the optical flow A_0_l between F_0 and F l, the optical flow A_2_3 between F_2 and F_3 etc.) and merges the two neighboring frames based on the determined optical flow. For the merging, the image generation circuitry 3 adds up corresponding pixels (examples of portions) of the two neighboring frames, wherein the corresponding pixels are associated with each other by the optical flow.

[0115] With the merging in the first merging iteration, the image generation circuitry 3 generates first merged frames. For example, a frame F_0_l is generated by merging frames F_0 and F l according to the optical flow A_0_l, a frame F_2_3 is generated by merging frames F_2 and F_3 according to the optical flow A_2_3, etc. Due to the merging, the first merged frames correspond to a light level of 2K lux.

[0116] In a second merging iteration, the image generation circuitry 3 determines an optical flow between two neighboring frames of the first merged frames (e.g., the optical flow A_0_2 between the frames F_0_l and F_2_3, the optical flow A_4_6 between the frames F_4_5 and F_6_7 etc.) and merges the two neighboring frames based on the determined optical flow.

[0117] With the merging in the second merging iteration, the image generation circuitry 3 generates second merged frames. For example, a frame F_0_3 is generated by merging frames F_0_l and F_2_3 according to the optical flow A_0_2, a frame F_4_7 is generated by merging frames F_4_5 and F_6_7 according to the optical flow A_4_6, etc. Due to the merging, the second merged frames correspond to a light level of 4K lux.

[0118] In a third merging iteration, the image generation circuitry 3 determines an optical flow between two neighboring frames of the second merged frames (e.g., the optical flow A_0_4 between the frames F_0_3 and F_4_7, and the optical flow A_8_12 between the frames F_8_ll and F_12_15) and merges the two neighboring frames based on the determined optical flow.

[0119] With the merging in the third merging iteration, the image generation circuitry 3 generates third merged frames. For example, a frame F_0_7 is generated by merging frames F_0_3 and F_4_7 according to the optical flow A_0_4, and a frame F_8_15 is generated by merging frames F_8_l 1 and F_12_15 according to the optical flow A_8_12. Due to the merging, the third merged frames correspond to a light level of 8K lux. In a fourth merging iteration, the image generation circuitry 3 determines an optical flow between two neighboring frames of the third merged frames (e.g., the optical flow A_0_8 between the frames F_0_7 and F_8_15) and merges the two neighboring frames based on the determined optical flow.

[0120] With the merging in the fourth merging iteration, the image generation circuitry 3 generates a fourth merged frame F_0_15 by merging frames F_0_7 and F_8_15 according to the optical flow A_0_8. Due to the merging, the fourth merged frame corresponds to a light level of 16K lux.

[0121] In a first stage 52 (i.e., during the first and second merging iteration), the image generation circuitry 3 uses a first algorithm for determining the optical flow between two neighboring frames. The first algorithm is fine-tuned for low light and includes a template matching algorithm.

[0122] In a second stage 53 (i.e., during the third and fourth merging iteration), the image generation circuitry 3 uses a second algorithm for determining the optical flow between two neighboring frames. The second algorithm is fine-tuned for normal light and includes a standard optical flow algorithm.

[0123] It is noted that the first and second algorithm are not limited to a template matching algorithm and a standard optical flow algorithm, respectively. The skilled person may find another suitable algorithm for determining the optical flow in the first stage 52 or second stage 53, respectively.

[0124] It is further noted that the image generation circuitry 3 may use a same algorithm for determining the optical flow in the first stage 52 and the second stage 53 in the hierarchical accumulation at 34 of Fig. 4.

[0125] The frame F_0_15 is based on the input image frames F_0 to F l 5 and is used as output frame 54, e.g., as the output frame generated at 23 of Fig. 3.

[0126] It is noted that in the hierarchical accumulation at 34 of Fig. 4, the image generation circuitry 3 may omit generating the frame F_0_15 and may instead, after determining the optical flow A_0_8, generate the output frame 38 at 37 by merging the set of input image frames according to the optical flow 36 determined in the hierarchical accumulation at 34.

[0127] For example, when the output frame 38 should be aligned to the first input image frame F_0, an optical flow between the frame F_0 and the output frame 38 is the identity function, i.e., no motion between F_0 and the output frame 38. An optical flow between F l and the output frame 38 corresponds to A_0_l. An optical flow between F_2 and the output frame 38 corresponds to A_0_2. An optical flow between F_3 and the output frame 38 is based on A_0_2 and A_2_3. An optical flow between F_4 and the output frame 38 corresponds to A_0_4. An optical flow between F_5 and the output frame 38 is based on A_0_4 and A_4_5. An optical flow between F_6 and the output frame 38 is based on A_0_4 and A_4_6. An optical flow between F_7 and the output frame 38 is based on A_0_4, A_4_6 and A_6_7. An optical flow between F_8 and the output frame 38 corresponds to A_0_8. An optical flow between F_9 and the output frame 38 is based on A_0_8 and A_8_9. An optical flow between F_10 and the output frame 38 is based on A_0_8 and A_8_10. An optical flow between F l 1 and the output frame 38 is based on A_0_8, A_8_10 and A_10_ll. An optical flow between F_12 and the output frame 38 is based on A_0_8 and A_8_12. An optical flow between F_13 and the output frame 38 is based on A_0_8, A_8_12 and A_12_13. An optical flow between F_14 and the output frame 38 is based on A_0_8, A_8_12 and A_12_14. An optical flow between F_15 and the output frame 38 is based on A_0_8, A_8_12, A_12_14 and A_14_15.

[0128] It is noted that in some embodiments, another number of merging iterations is performed, another number of input image frames is merged into one output frame, and / or a switching between the first algorithm and the second algorithm for determining the optical flow is performed after another merging iteration (e.g., after the first or third merging iteration).

[0129] Fig. 7 illustrates a second embodiment of a hierarchical accumulation (iterative merging), which is an example of the hierarchical accumulation 14 of Fig. 2, of the hierarchical accumulation at 22 of Fig. 3 and of the hierarchical accumulation at 34 of Fig. 4.

[0130] Instead of pair-wise accumulation as shown in Fig. 6, Fig. 7 shows another combination, wherein four neighboring frames are merged (accumulated) in a first merging iteration, and a pair-wise merging (accumulation) is performed in subsequent merging iterations. By merging more than two neighboring frames in one merging iteration, a tree depth is reduced.

[0131] As described with respect to Fig. 6, a set 61 of input image frames F_0 to F l 5 corresponds to a light level of K lux. Note that in the embodiment of Fig. 4, the set 61 of input image frames corresponds to downscaled grayscale frames obtained by the downscaling at 32 and converting to grayscale at 33.

[0132] In a first merging iteration, the image generation circuitry 3 determines an optical flow between respective ones of four neighboring frames and a reference frame of the four neighboring frames, and generates first merged frames by merging the four neighboring frames based on the determined optical flow between the respective ones of the four neighboring frames. For example, the frame F_0 is used as reference frame of the neighboring frames F_0 to F_3, and the image generation circuitry 3 determines an optical flow A_0_l between F_0 and F l, an optical flow A_0_2 between F_0 and F_2, and an optical flow A_0_3 between F_0 and F_3. The image generation circuitry 3 then generates a merged frame F_0_3 by merging F_0, F l, F_2 and F_3 according to A_0_l, A_0_2, and A_0_3, respectively. Likewise, the image generation circuitry 3 uses F_4 as reference frame of the neighboring frames F_4 to F_7, determines optical flows A_4_5, A_4_6 and A_4_7 between F_4 and F_5, F_6 and F_7, respectively, and generates a merged frame F_4_7 by merging F_4 to F_7 according to A_4_5, A_4_6 and A_4_7, respectively. Due to the merging, the first merged frames correspond to a light level of 4K lux.

[0133] In a second merging iteration, the image generation circuitry 3 determines an optical flow between two neighboring frames of the first merged frames and generates second merged frames by merging the two neighboring frames according to the determined optical flow. For example, the image generation circuitry 3 determines an optical flow A_0_4 between the frames F_0_3 and F_4_7, and generates a merged frame F_0_7 by merging F_0_3 and F_4_7 according to A_0_4. Basically, the second merging iteration of Fig. 7 corresponds to the third merging iteration of Fig. 6. Due to the merging, the second merged frames correspond to a light level of 8K lux.

[0134] In a third merging iteration, the image generation circuitry 3 determines an optical flow between two neighboring frames of the second merged frames and generates a third merged frame by merging the two neighboring frames based on the determined optical flow. For example, the image generation circuitry 3 determines an optical flow A_0_8 between the frames F_0_7 and F_8_15, and generates a frame F_0_15 by merging F_0_7 and F_8_15 according to A_0_8. Basically, the third merging iteration of Fig. 7 corresponds to the fourth merging iteration of Fig. 6. Due to the merging, the third merged frame corresponds to a light level of 16K lux.

[0135] As described with respect to the first stage 52 and the second stage 53 of Fig. 6, the image generation circuitry also uses the first algorithm in a first stage 62 (i.e., during the first merging iteration in Fig. 7) and the second algorithm in a second stage 63 (i.e., during the second and third merging iteration in Fig. 7) for determining the optical flow. Also in Fig. 7, the first algorithm is fine-tuned for low light and includes a template matching algorithm, and the second algorithm is fine-tuned for normal light and includes a standard optical flow algorithm.

[0136] It is noted that the first and second algorithm are not limited to a template matching algorithm and a standard optical flow algorithm, respectively. The skilled person may find another suitable algorithm for determining the optical flow in the first stage 62 or second stage 63, respectively. It is further noted that the image generation circuitry 3 may use a same algorithm for determining the optical flow in the first stage 62 and the second stage 63 in the hierarchical accumulation at 34 of Fig. 4.

[0137] The frame F_0_15 is based on the input image frames F_0 to F l 5 and is used as output frame 64, e.g., as the output frame generated at 23 of Fig. 3.

[0138] It is noted that in the hierarchical accumulation at 34 of Fig. 4, the image generation circuitry 3 may omit generating the frame F_0_15 and may instead, after determining the optical flow A_0_8, generate the output frame 38 at 37 by merging the set of input image frames according to the optical flow 36 determined in the hierarchical accumulation at 34.

[0139] For example, when the output frame 38 should be aligned to the first input image frame F_0, an optical flow between the frame F_0 and the output frame 38 is the identity function, i.e., no motion between F_0 and the output frame 38. An optical flow between F l and the output frame 38 corresponds to A_0_l. An optical flow between F_2 and the output frame 38 corresponds to A_0_2. An optical flow between F_3 and the output frame 38 corresponds to A_0_3. An optical flow between F_4 and the output frame 38 corresponds to A_0_4. An optical flow between F_5 and the output frame 38 is based on A_0_4 and A_4_5. An optical flow between F_6 and the output frame 38 is based on A_0_4 and A_4_6. An optical flow between F_7 and the output frame 38 is based on A_0_4 and A_4_7. An optical flow between F_8 and the output frame 38 corresponds to A_0_8. An optical flow between F_9 and the output frame 38 is based on A_0_8 and A_8_9. An optical flow between F_10 and the output frame 38 is based on A_0_8 and A_8_10. An optical flow between F l 1 and the output frame 38 is based on A_0_8 and A_8_l 1. An optical flow between F_12 and the output frame 38 is based on A_0_8 and A_8_12. An optical flow between F_13 and the output frame 38 is based on A_0_8, A_8_12 and A_12_13. An optical flow between F_14 and the output frame 38 is based on A_0_8, A_8_12 and A_12_14. An optical flow between F_15 and the output frame 38 is based on A_0_8, A_8_12 and A_12_15.

[0140] It is noted that in some embodiments, another number of merging iterations is performed, another number of input image frames is merged into one output frame, and / or a switching between the first algorithm and the second algorithm for determining the optical flow is performed after another merging iteration (e.g., after the second merging iteration). Further, in some embodiments, another number of neighboring frames (e.g., three, five, six, seven, eight etc.) is merged in the first merging iteration. In some embodiments, more than two neighboring frames are merged in the second and / or third merging iteration instead of or in addition to the first merging iteration. Also, in some embodiments, another frame than an earliest frame in the plurality of neighboring frames is used as reference frame (e.g., a latest frame, a middle frame, or a frame that corresponds to a time closest to an output frame).

[0141] Fig. 8 illustrates a third embodiment of an image generation method 70, which is an example of a method performed by the image generation circuitry 3 of Fig. 3. The image generation method 70 is an example of generating a merged frame by merging two neighboring frames, e.g., in a merging iteration of the hierarchical accumulation in Fig. 6 or in the second or third merging iteration of Fig. 7.

[0142] The method 70 starts at 71.

[0143] At 72, an accumulation buffer is cleared.

[0144] At 73, it is determined whether there are more unprocessed image frames. In a first merging iteration, the unprocessed image frames correspond to the set of input image frames. In a subsequent merging iteration, the unprocessed image frames correspond to the merged frames from the respective preceding merging iteration.

[0145] At 74, if there are more unprocessed image frames, flow vectors (which represent an optical flow) are determined, and an image represented by one of the more unprocessed image frames is unwarped using the determined flow vectors.

[0146] At 75, the un warped image is added to the accumulation buffer.

[0147] After the processing at 75, the processing proceeds at 73.

[0148] At 76, if there are no more unprocessed image frames, the accumulation buffer is normalized.

[0149] The method 70 ends at 77. The accumulation buffer stores the generated merged frame.

[0150] Fig. 9 illustrates a fourth embodiment of an image generation method 80, which is an example of a method performed by the image generation circuitry 3 of Fig. 3. The image generation method 80 is an example of generating a merged frame by merging more than two neighboring frames, e.g., in the first merging iteration of Fig. 7.

[0151] The method 80 starts at 81

[0152] At 82, an accumulation buffer is cleared.

[0153] At 83, a first input image frame of a plurality of neighboring input image frames is loaded and set as reference. At 84, it is determined whether there are more unprocessed image frames in the plurality of neighboring input image frames.

[0154] At 85, if there are more unprocessed image frames, flow vectors (which represent an optical flow) are determined between the reference and one of the more unprocessed image frames (current frame).

[0155] At 86, an image represented by the current frame (current image) is unwarped using the determined flow vectors.

[0156] At 87, the unwarped image is added to the accumulation buffer.

[0157] After the processing at 87, the processing proceeds at 84.

[0158] At 88, if there are no more unprocessed image frames, the accumulation buffer is normalized.

[0159] The method 80 ends at 89. The accumulation buffer stores the generated merged frame.

[0160] As described above, the present technology performs hierarchical accumulation on a set of input image frames. In some embodiments, a first sub-frame (e.g., a first input image frame in the set of input image frames) is used as reference frame. In very low light situations, the reference frame may be very dark and noisy.

[0161] The hierarchical accumulation may first merge input image frames pairwise (e.g., run an optical flow algorithm between frames 0 and 1, between frames 2 and 3, between frames 4 and 5, etc. The pairs of image frames may be accumulated (merged) into new images (merged images) according to the determined optical flow. The merged images may have less noise and a double photon count as compared to the input image frames (in case of merging two frames). The merged frames may further be merged together pairwise etc.

[0162] At different hierarchy levels of the hierarchical accumulation (e.g., at different merging iterations), different parameters for determining an optical flow, or even different algorithms for determining an optical flow may be used. For example, at very low light situations, the hierarchical accumulation may be more robust when the optical flow is determined by template matching. When a suitable number of images is accumulated, less noise and a higher photon count may be achieved, so that at higher levels of the hierarchical accumulation, a standard optical flow algorithm may be used for determining the optical flow (as described with respect to Fig. 6 and 7).

[0163] For robustness, inputted full resolution images may be downscaled (which may allow a faster processing, less noise and a better robustness) before performing the hierarchical accumulation. The hierarchical accumulation may compute motion vectors (which indicate an optical flow) for each downscaled frame. A final accumulation for generating an output frame may be performed on full resolution RGB frames (e.g., on the input image frames without downscaling and without converting to grayscale) based on the motion vectors (optical flow) determined for the downscaled and grayscale-converted frames.

[0164] In some embodiments, the downscaling sums up pixel values instead of using a mean.

[0165] In some embodiments, two lowest levels of the hierarchical accumulation use template matching for determining the optical flow, and use a sum for the accumulation instead of a mean. In some embodiments, higher levels use a standard optical flow algorithm.

[0166] The hierarchical accumulation may be performed on a CPU and / or on a GPU, using OpenCV, Vulkan or the like.

[0167] Fig. 10 illustrates an embodiment of a general-purpose computer 150. The general -purpose computer 150 can be implemented such that it can basically function as any type of electronic device (such as the electronic device 1 of Fig. 1), for example, a smartphone, smartglasses, a head-mounted display, a smartwatch, a mobile phone, a mobile tablet, a notebook, a terminal device or the like. The general-purpose computer 150 is an example of an information processing apparatus that includes circuitry that is configured to perform the method according to the present technology (e.g., the method of Fig. 2, 3, 4, 6, 7, 8, or 9). The computer has components 151 to 161, which can form a circuitry, such as the image generation circuitry 3, as described herein.

[0168] Embodiments which use software, firmware, programs or the like for performing the methods as described herein can be installed on computer 150, which is then configured to be suitable for the concrete embodiment.

[0169] The computer 150 has a CPU 151 (Central Processing Unit), which can execute various types of procedures and methods as described herein, for example, in accordance with programs stored in a read-only memory (ROM) 152, stored in a storage 157 and loaded into a random-access memory (RAM) 153, stored on a medium 160 which can be inserted in a respective drive 159, etc.

[0170] Furthermore, the computer 150 includes an artificial intelligence (Al) processor 151a. The Al processor 151a may include a graphics processing unit (GPU) and / or atensor processing unit (TPU). The Al processor 151a may be configured to execute an Al model (e.g., an artificial neural network). The CPU 151, the ROM 152 and the RAM 153 are connected to a bus 161, which in turn is connected to an input / output interface 154. The number of CPUs, memories and storages is only exemplary, and the skilled person will appreciate that the computer 150 can be adapted and configured accordingly for meeting specific requirements which arise when it functions as an information processing apparatus (e.g., an electronic device and / or an image generation circuitry) according to the present technology.

[0171] At the input / output interface 154, several components are connected: an input 155, an output 156, the storage 157, a communication interface 158 and the drive 159, into which a medium 160 (compact disc (CD), digital video disc (DVD), universal serial bus (USB) flash drive, secure digital (SD) card, CompactFlash (CF) memory, or the like) can be inserted.

[0172] The input 155 can be a pointer device (mouse, graphic table, or the like), a keyboard, a microphone, a camera, a touchscreen, an eye-tracking unit etc.

[0173] The output 156 can have a display (liquid crystal display (LCD), cathode ray tube (CRT) display, light-emitting diode (LED) display, electronic paper, etc.; e.g., included in a touchscreen), loudspeakers, etc.

[0174] The storage 157 can have a hard disk drive (HDD), a solid-state drive (SSD), a flash drive and the like.

[0175] The communication interface 158 can be adapted to communicate, for example, via universal serial bus (USB), a serial port (RS-232), parallel port (IEEE 1284), a local area network (LAN; e.g., ethemet), wireless local area network (WLAN; e.g., Wi-Fi, IEEE 802.11), mobile telecommunications system (GSM, UMTS, LTE, NR etc.), Bluetooth, near-field communication (NFC), ZigBee, infrared, etc.

[0176] It should be noted that the description above only pertains to an example configuration of computer 150. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces or the like. For example, the communication interface 158 may support other radio access technologies than the mentioned UMTS, LTE and NR.

[0177] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. For example, the ordering of 32 and 33 in the embodiment of Fig. 4 may be exchanged. Other changes of the ordering of method steps may be apparent to the skilled person. Please note that the division of the image generation circuitry 3 into units 3a to 3c is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the image generation circuitry 3 could be implemented by a respective programmed processor, field programmable gate array (FPGA) and the like.

[0178] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0179] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0180] Note that the present technology can also be configured as described below.

[0181] (1) Image generation circuitry, configured to: obtain a set of input image frames representing a first and at least a subsequent capture; iteratively merge two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generate an output frame based on the merged set of input image frames.

[0182] (2) The image generation circuitry of (1), wherein the iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration.

[0183] (3) The image generation circuitry of (1) or (2), wherein the generating of the output frame includes merging the set of input image frames based on the determined optical flow between the respective neighboring frames.

[0184] (4) The image generation circuitry of any one of (1) to (3), wherein the image generation circuitry is further configured to: downscale the set of input image frames; and perform the iterative merging on the downscaled frames.

[0185] (5) The image generation circuitry of any one of (1) to (4), wherein the iterative merging includes: using a first algorithm for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration.

[0186] (6) The image generation circuitry of any one of (1) to (5), wherein the iterative merging includes merging a plurality of neighboring frames based on a determined optical flow between respective ones of the plurality of neighboring frames and a reference frame of the plurality of neighboring frames.

[0187] (7) The image generation circuitry of any one of (1) to (6), wherein the criterion indicates a predefined number of frames.

[0188] (8) The image generation circuitry of any one of (1) to (7), wherein the criterion indicates a predefined image quality.

[0189] (9) The image generation circuitry of any one of (1) to (8), wherein the merging includes summing associated portions of the frames.

[0190] (10) The image generation circuitry of any one of (1) to (9), wherein the image generation circuitry is further configured to: convert the set of input image frames to grayscale; and perform the iterative merging on the converted grayscale frames.

[0191] (11) Image generation method, comprising: obtaining a set of input image frames representing a first and at least a subsequent capture; iteratively merging two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generating an output frame based on the merged set of input image frames.

[0192] (12) The image generation method of (11), wherein the iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration.

[0193] (13) The image generation method of (11) or (12), wherein the generating of the output frame includes merging the set of input image frames based on the determined optical flow between the respective neighboring frames. (14) The image generation method of any one of (11) to (13), wherein the image generation method further comprises: downscaling the set of input image frames; and performing the iterative merging on the downscaled frames.

[0194] (15) The image generation method of any one of (11) to (14), wherein the iterative merging includes: using a first algorithm for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration.

[0195] (16) The image generation method of any one of (11) to (15), wherein the iterative merging includes merging a plurality of neighboring frames based on a determined optical flow between respective ones of the plurality of neighboring frames and a reference frame of the plurality of neighboring frames.

[0196] (17) The image generation method of any one of (11) to (16), wherein the criterion indicates a predefined number of frames.

[0197] (18) The image generation method of any one of (11) to (17), wherein the criterion indicates a predefined image quality.

[0198] (19) The image generation method of any one of (11) to (18), wherein the merging includes summing associated portions of the frames.

[0199] (20) The image generation method of any one of (11) to (19), wherein the image generation method further comprises: converting the set of input image frames to grayscale; and performing the iterative merging on the converted grayscale frames.

[0200] (21) An electronic device, comprising: an imaging device; and the image generation circuitry of any one of (1) to (10); wherein the imaging device is configured to acquire the set of input image frames.

[0201] (22) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (20), when being carried out on a computer. (23) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (20) to be performed.

Claims

CLAIMS1. Image generation circuitry, configured to: obtain a set of input image frames representing a first and at least a subsequent capture; iteratively merge two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generate an output frame based on the merged set of input image frames.

2. The image generation circuitry of claim 1, wherein the iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration.

3. The image generation circuitry of claim 1, wherein the generating of the output frame includes merging the set of input image frames based on the determined optical flow between the respective neighboring frames.

4. The image generation circuitry of claim 1, wherein the image generation circuitry is further configured to: downscale the set of input image frames; and perform the iterative merging on the downscaled frames.

5. The image generation circuitry of claim 1, wherein the iterative merging includes: using a first algorithm for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration.

6. The image generation circuitry of claim 1, wherein the iterative merging includes merging a plurality of neighboring frames based on a determined optical flow between respective ones of the plurality of neighboring frames and a reference frame of the plurality of neighboring frames.

7. The image generation circuitry of claim 1, wherein the criterion indicates a predefined number of frames.

8. The image generation circuitry of claim 1, wherein the criterion indicates a predefined image quality.

9. The image generation circuitry of claim 1, wherein the merging includes summing associated portions of the frames.

10. The image generation circuitry of claim 1, wherein the image generation circuitry is further configured to: convert the set of input image frames to grayscale; and perform the iterative merging on the converted grayscale frames.

11. Image generation method, comprising: obtaining a set of input image frames representing a first and at least a subsequent capture; iteratively merging two neighboring frames of the set of input image frames based on a determined optical flow between them until a criterion is met; and generating an output frame based on the merged set of input image frames.

12. The image generation method of claim 11, wherein the iterative merging includes merging two neighboring frames based on a determined optical flow between them, wherein the two neighboring frames have been obtained by merging frames in a previous merging iteration.

13. The image generation method of claim 11 , wherein the generating of the output frame includes merging the set of input image frames based on the determined optical flow between the respective neighboring frames.

14. The image generation method of claim 11, wherein the image generation method further comprises: downscaling the set of input image frames; and performing the iterative merging on the downscaled frames.

15. The image generation method of claim 11 , wherein the iterative merging includes: using a first algorithm for determining the optical flow between two neighboring frames in a first merging iteration; and using a second algorithm different from the first algorithm for determining the optical flow between two neighboring frames in a second merging iteration.

16. The image generation method of claim 11, wherein the iterative merging includes merging a plurality of neighboring frames basedon a determined optical flow between respective ones of the plurality of neighboring frames and a reference frame of the plurality of neighboring frames.

17. The image generation method of claim 11, wherein the criterion indicates a predefined number of frames.

18. The image generation method of claim 11 , wherein the criterion indicates a predefined image quality.

19. The image generation method of claim 11, wherein the merging includes summing associated portions of the frames.

20. The image generation method of claim 11 , wherein the image generation method further comprises: converting the set of input image frames to grayscale; and performing the iterative merging on the converted grayscale frames.

Images