Deep learning based fusion techniques for high resolution images

Abstract

Electronic devices, methods, and program storage devices for performing high-resolution and low-latency image fusion and / or noise reduction using machine learning are disclosed. An incoming image stream can be obtained from an image capture device, wherein the incoming image stream includes multiple captures of different resolutions and / or different exposures received according to a specific mode, such as EV0 images, EV- images, EV+ images, long-exposure images, etc. When a capture request is received, a deep neural network can be used to generate two or more intermediate clips from the images from the incoming image stream, and the intermediate clips can then be fed into a neural network that has been trained to pass additional image details from one intermediate clip to another. In some embodiments, the resulting output image generated from two or more intermediate clips may have a higher resolution than at least one of the intermediate clips.

Description

Technical Field

[0001] This disclosure relates in its entirety to the field of digital image processing. More specifically, but not as a limitation, this disclosure relates to techniques for using machine learning to perform high-resolution and low-latency image fusion and noise reduction on captured images with multiple resolutions and / or exposure times. Background Technology

[0002] Fusing multiple images of the same captured scene is an effective way to increase the signal-to-noise ratio (SNR) in the resulting fused image. This is especially important for small and / or thin devices such as mobile phones, tablets, laptops, wearables, etc., where the image sensor's pixel size is typically very small. Smaller pixel size means that each pixel captures relatively less light (i.e., compared to a full-size standalone camera with a larger pixel size), resulting in more visible noise in the captured image—especially in low-light conditions.

[0003] Multiple image captures used in a given fusion operation may include: multiple images captured at the same exposure (e.g., for the purpose of freezing motion), a process that may be referred to as static image stabilization (SIS); multiple images captured at different exposures (e.g., for the purpose of highlight recovery, such as in the case of high dynamic range (HDR) imaging); or a combination of multiple images captured at shorter and longer exposures, as can be captured when an optical image stabilization (OIS) system of the image capture device is coupled (e.g., for the purpose of estimating moving pixels from a shorter exposure and estimating static pixels from a longer exposure). Furthermore, the captured images to be fused may come from, for example, the same camera, multiple cameras with different image sensor characteristics (e.g., cameras with different lens and / or native sensor resolutions (such as relatively high-resolution image sensors and relatively low-resolution image sensors)), or different image processing workflows from the same image sensor (e.g., a full or “high-resolution” output image from a given image sensor and a merged or “low-resolution” output image from the same image sensor).

[0004] In some existing image fusion schemes, design engineers may need to calculate, tune, and / or optimize multiple image heuristics (e.g., based on a relatively small number of test images) to attempt to achieve satisfactory fusion results across a wide variety of image capture scenarios. However, such calculations and optimizations are inherently limited by the small size of the test image set on which they are derived. Furthermore, the more complex these calculations and optimizations become, the more computationally expensive it is to perform such fusion techniques on real-world image capture devices.

[0005] Therefore, there is a need for a method to improve fusion and noise reduction for bracketing captures at arbitrary exposure levels and varying resolutions using machine learning-based techniques. This improved fusion and noise reduction technique is optimized based on a much larger image training set and can be executed in a memory-efficient and low-latency manner. However, as increasingly higher resolution image sensors become available for use in consumer electronics, new technical challenges arise in terms of constraints such as power, memory, and system performance. Furthermore, the additional latency involved in capturing such high-resolution images may prevent users from capturing images of the scene at the exact intentional moment of the image capture request. This can be particularly noticeable when shooting highly dynamic scenes such as sporting events, moving children, pets, etc.

[0006] In such cases, the camera's ability to capture the exact moment of intent in such a scene may be just as important (or even more important) to the user as the noise level, color reproduction quality, or resolution level of the final image. Ideally, the user wants a high-resolution photo that also captures the exact moment of intent from the scene being captured. Therefore, this paper presents techniques for performing image capture and neural network-based image fusion that avoid (or reduce) the effects of system latency and provide the user with a high-resolution (and high-quality) output image that accurately represents the scene at the moment of intent, i.e., without exhibiting unwanted shutter lag. Summary of the Invention

[0007] This document discloses devices, methods, and nontransitory program storage devices for performing high-resolution and low-latency image fusion and / or noise reduction using machine learning (ML) and other artificial intelligence (AI) technologies (e.g., deep neural networks (DNN)) to generate low-noise and high dynamic range (HDR) images from images captured by multiple lenses and / or with multiple resolutions and exposure times.

[0008] More specifically, an incoming image stream can be obtained from one or more image capture devices, wherein the incoming image stream includes, for example, multiple image captures of different resolutions and / or surrounded in different ways, received in a specific sequence and / or according to a specific pattern. Upon receiving an image capture request, the method can then generate two or more intermediate materials in response to the capture request, wherein at least two of these intermediate materials include "image-based" intermediate materials, such as images generated using one or more determined images from the incoming image stream (e.g., images generated by one or more trained deep neural networks).

[0009] According to some implementations, one or more high-resolution image clips (e.g., images with a higher resolution than the constituent images from the incoming image stream) can also be captured in response to receiving an image capture request. As will be described herein, deep neural networks can also be used to intelligently and memory-efficiently transfer additional details from such high-resolution image clips to other lower-resolution image clips used in the neural image fusion process.

[0010] In some cases, the final output image may have the same resolution as the one or more high-resolution image materials. In other cases, the one or more high-resolution image materials may be downsized before the final neural image fusion process with other lower-resolution image materials, resulting in a final output image that still has a higher resolution than the other lower-resolution image materials, although not as high as the original resolution of the originally captured high-resolution image materials. According to other embodiments, one or more long-exposure image materials may also be captured in response to receiving an image capture request, and then used intelligently in the neural image fusion process.

[0011] As described above, embodiments of various electronic devices are disclosed herein. Such electronic devices may include one or more image capture devices, such as an optical image sensor / camera unit; a display; a user interface; one or more processors; and a memory coupled to the one or more processors. Instructions may be stored in the memory, which cause the one or more processors to execute instructions to: obtain an incoming image stream (e.g., an incoming image stream including images having two or more different resolutions and / or exposure values) from the one or more image capture devices; receive an image capture request via the user interface; and generate two or more intermediate materials in response to the image capture request, wherein: a first intermediate material among the two or more generated intermediate materials includes an image generated by a first neural network configured to perform a fusion operation on a determined first or more images from the incoming image stream, and wherein the first intermediate material has a first resolution; and a second intermediate material among the two or more generated intermediate materials includes an image generated by a second neural network configured to perform an image enhancement operation (e.g., denoising and / or de-mosaicing) on ​​at least a second image from the incoming image stream, wherein the second image has a second resolution, and wherein the second resolution is greater than the first resolution.

[0012] Next, these instructions enable the one or more processors to execute instructions to: feed the first intermediate material and the second intermediate material into a third neural network, wherein the third neural network is configured to combine the first intermediate material and the second intermediate material to generate an output image with a resolution greater than the first resolution; and finally, use the third neural network to generate the output image.

[0013] In some implementations, one or more of the first or one or more images are captured before an image capture request is received, while in other implementations, one or more of the first or one or more images (and / or the second image) may also be captured after an image capture request is received.

[0014] In some implementations, the second neural network is further configured to perform a demosaicing operation on a second image from the incoming image stream.

[0015] In some implementations, the second neural network is further configured to perform image enhancement operations on a cropped region (e.g., a center cropped region) of a second image from the incoming image stream.

[0016] In some implementations, the second resolution is n times larger than the first resolution (e.g., 2 times, 4 times, 8 times, 9 times, 16 times, etc.), where n is greater than or equal to 2.

[0017] In some implementations, the output image has an improved level of detail and / or a lower level of noise compared to the first intermediate material.

[0018] In some implementations, the third neural network is further configured to operate on tiles of the first intermediate material and tiles of the second intermediate material (where each tile includes a sub-part of pixels in the image, such as a rectangular sub-part).

[0019] In some such implementations, the one or more processors are further configured to execute instructions that cause the one or more processors to perform per-tile homography estimation between tiles of the second intermediate material and tiles of the first intermediate material.

[0020] In other such embodiments, the one or more processors are further configured to execute instructions that cause the one or more processors to perform the following operation: for each tile in the first intermediate material, identify a guide tile in the second intermediate material (e.g., the guide tile may include a tile from one image that most closely matches a tile from another image in the feature space).

[0021] In other such implementations, the third neural network is further configured to pass details from each tile in the first intermediate material to a corresponding guide tile in the second intermediate material to each tile in the first intermediate material.

[0022] In some implementations, the third neural network is further configured to generate an output image with a resolution that is configured to simulate a specific fixed-focus lens (e.g., simulating the field of view (FOV) of a 24mm equivalent fixed-focus camera, a 28mm equivalent fixed-focus camera, a 35mm equivalent fixed-focus camera, a 48mm equivalent fixed-focus camera, etc.), which is subjected to any further desired post-processing, such as hardware scaling, rotation, etc.

[0023] In some implementations, the output image has a second resolution.

[0024] Based on the various electronic device implementations listed above, this document also discloses various methods for performing high-resolution and low-latency machine learning-enhanced image fusion and / or denoising. This document also discloses a non-transitory program storage device that can store instructions for causing one or more processors to perform operations according to the various electronic device implementations listed above. Attached Figure Description

[0025] Figure 1 An exemplary incoming image stream according to one or more embodiments is illustrated, which can be used to generate one or more intermediate materials to be used in machine learning-enhanced image fusion and / or noise reduction methods.

[0026] Figure 2 An overview of a process for performing high-resolution and low-latency machine learning-enhanced image fusion and / or noise reduction, according to one or more embodiments, is illustrated.

[0027] Figure 3 Examples of neural network architectures, based on one or more implementations, that can be used to perform high-resolution and low-latency machine learning-enhanced image fusion and / or noise reduction.

[0028] Figure 4 This is a flowchart illustrating a method for performing high-resolution and low-latency machine learning-enhanced image fusion and / or noise reduction using one or more intermediate materials according to one or more embodiments.

[0029] Figure 5 This is a block diagram illustrating a programmable electronic computing device in which one or more of the techniques disclosed herein may be implemented. Detailed Implementation

[0030] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention disclosed herein. However, it will be apparent to those skilled in the art that the invention may be practiced without these specific details. In other instances, structures and devices are shown in block diagram form to avoid obscuring the invention. References to numbers without subscripts or suffixes should be construed as references to all subscripts and suffixes corresponding to the reference numerals in the drawings. Furthermore, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, and therefore may therefore require recourse to the claims to determine such inventive subject matter. References to “an embodiment” or “an embodiment” in the specification refer to a particular feature, structure, or characteristic included in at least one embodiment of an invention described in connection with that embodiment, and multiple references to “an embodiment” or “an embodiment” should not be construed as necessarily referring to the same embodiment in all cases.

[0031] The naming conventions used in this paper to refer to the various images exposed in different ways from an incoming image stream will now be discussed. Similar to conventional bounding notation, “EV” represents an exposure value and refers to a given exposure level of the image (which can be controlled by one or more settings of the device, such as the shutter speed and / or aperture setting of the image capture device). Different images can be captured at different EVs, where an EV difference (also called a “stop”) between images is equal to a predefined difference in exposure energy. Typically, stops are used to represent a power of two difference between exposures. Therefore, changing the exposure value changes the amount of light received for a given image, depending on whether the EV is increased or decreased. For example, a stop doubles (or halves) the amount of light received for a given image, depending on whether the EV is increased (or decreased).

[0032] In the general context, an "EV0" image refers to an image captured using an exposure value determined by the exposure algorithm of the image capture device (e.g., specified by an automatic exposure (AE) mechanism). Typically, it is assumed that an EV0 image has an ideal exposure value (EV) for a given lighting condition. It should be understood that, in the context of an EV0 image herein, the term "ideal" refers to the ideal exposure value calculated for a given image capture system. In other words, it is the ideal exposure for the system-specific version. Different image capture systems may have different versions of ideal exposure values ​​for a given lighting condition, and / or different constraints and analyses may be used to determine the exposure settings for capturing EV0 images.

[0033] The term "EV-" image refers to an underexposed image captured at a lower stop (e.g., 0.5, 1, 2, or 3 stops lower) than the stop used to capture an EV0 image. For example, an "EV-1" image is an underexposed image captured one stop lower than the exposure of an EV0 image, and an "EV-2" image is an underexposed image captured two stops lower than the exposure of an EV0 image. The term "EV+" image refers to an overexposed image captured at a higher stop (e.g., 0.5, 1, 2, or 3) than an EV0 image. For example, an "EV+1" image is an overexposed image captured one stop higher than the exposure of an EV0 image, and an "EV+2" image is an overexposed image captured two stops higher than the exposure of an EV0 image.

[0034] For example, according to some implementations, the incoming image stream may include a combination of the following: EV-, EV0, EV+, and / or other longer exposure images. It should also be noted that the image stream may also include any combination of exposures as desired for a given specific implementation or operating conditions, such as EV+2, EV+4, EV-3 images, etc.

[0035] As described above, in image fusion, one of the images to be fused is typically designated as the reference image for the fusion operation, and other candidate images involved in the fusion operation are registered to this reference image. The reference image is usually selected based on its temporal proximity to the moment the user intends to "freeze" in the captured image. To more effectively freeze motion in the captured scene, the reference image may have a relatively short exposure time (e.g., shorter than a long-exposure image) and therefore contain an undesirable amount of noise. Thus, the reference image can benefit from fusion with one or more additional images to improve the original noise characteristics of the reference image while still sufficiently freezing the desired moment in the scene.

[0036] Therefore, according to some embodiments, the enhanced image (also referred to herein as "intermediate material") can be synthesized, for example, from multiple captured images by one or more deep neural networks, which are fused together in a feature space by such deep neural networks. According to other embodiments, high-resolution captured images can also be enhanced as needed, for example, by one or more deep neural networks, such as for denoising, de-mosaicing, and / or further enlargement and / or reduction, and the resulting enhanced image can also be referred to herein as "intermediate material". Two or more such intermediate materials (e.g., generated by deep neural networks) can themselves be used as inputs to a neural image fusion process that is ultimately designed to intelligently and multi-scale transfer details from higher-resolution intermediate materials to corresponding portions of lower-resolution intermediate materials, thereby generating an output fused image with a resolution and level of detail greater than that of at least one of the initially generated intermediate materials.

[0037] According to some embodiments, a long exposure image may include an image frame captured as overexposed relative to an EV0 exposure setting. In some cases, this exposure setting may be a predetermined EV+ value (e.g., EV+1, EV+2, etc.). In other cases, the exposure setting for a given long exposure image may be calculated on the fly at the capture time (e.g., within a predetermined range). A long exposure image may originate from a single image captured from a single camera, or in other cases, it may be synthesized from multiple captured images fused together (the result may be referred to as a "synthesized long image," "synthesized long exposure image," or "SL" image). According to other embodiments, a synthesized long image may also simply be the result of selecting a single bracketing capture (i.e., not fused with one or more other bracketing captures). For example, in a given embodiment, a single EV+2 long exposure image may be used as a synthesized long image.

[0038] This paper uses the term "intermediate material" to refer to the fact that a particular material is not typically an image captured directly by an image sensor (e.g., except in scenarios where a specific single bracketing image capture can be selected as intermediate material). Instead, intermediate material is typically synthesized by a deep neural network and / or fused from two or more images captured directly by an image sensor. Intermediate material is called "intermediate," for example, because it can be generated and used during the intermediate time period between when the image is captured in real time by the device's image sensor and when the final fused output image is generated. Intelligent use of intermediate material allows the fusion operation to benefit (at least to some extent) from both the additional light information captured by a larger number of bracketing exposures and the additional details that can be recovered from higher-resolution image captures, while still maintaining the processing and memory efficiency benefits of performing the actual fusion operation using only a smaller number of intermediate materials (e.g., leveraging potentially processing-intensive deep learning techniques).

[0039] When the image capture device is capable of performing OIS (Optical Image Stabilization), OIS can actively stabilize the camera and / or image sensor during the capture of one or more long-exposure images and / or other captured images. (In other embodiments, OIS stabilization may not be employed during the capture of other images (i.e., non-long-exposure images), or different stabilization control techniques may be employed for such non-long-exposure images.) In some cases, the image capture device may use only one type of long-exposure image. In other cases, the image capture device may capture different types of long-exposure images, for example, depending on the capture conditions. For example, in some embodiments, composite long-exposure images may be created when the image capture device does not perform or cannot perform OIS, while a single long-exposure image may be captured when the OIS system is available and engaged at the image capture device.

[0040] According to some implementations, in order to recover a desired amount of shadow detail in the captured image, a certain degree of overexposure (e.g., EV+2) can be intentionally employed in bright scenes and scenes with moderate brightness. Therefore, under certain bright ambient light levels, long exposure images themselves may also comprise images that are overexposed by one or more stops relative to EV0 (e.g., EV+3, EV+2, EV+1, etc.). To maintain a consistent brightness level across long exposure images, the gain can be reduced proportionally with increasing capture exposure time, since, according to some implementations, brightness can be defined as the product of gain and exposure time.

[0041] In some embodiments, a long exposure image may include an image captured with an exposure time greater than a minimum threshold exposure time (e.g., 50 milliseconds (ms)) and less than a maximum threshold exposure time (e.g., 250 ms, 500 ms, or even 1 second). In other embodiments, a long exposure image may include an image captured with an exposure time relatively longer than a corresponding normal or “short” exposure image of the image capture device (e.g., an exposure time 4 to 30 times longer than the exposure time of a short exposure image). In still other embodiments, the specific exposure time (and / or system gain) of a long exposure image may be further based at least in part on the ambient light level around the image capture device, wherein brighter ambient conditions allow for relatively shorter long exposure times, and darker ambient conditions allow for relatively longer long exposure times. In still other embodiments, the specific exposure time (and / or system gain) of a long exposure image may be further based at least in part on whether the image capture device is using an OIS system during the capture operation.

[0042] It should be noted that the noise level in a given image can be estimated, at least in part, based on the system's gain level (where a larger gain results in a larger noise level). Therefore, to achieve low noise, an image capture system may desire to use a small gain. However, as discussed above, the brightness of an image can be determined by the product of the exposure time and the gain. Therefore, to maintain image brightness, a large exposure time is typically used to compensate for low gain. However, longer exposure times can lead to motion blur, for example, in the absence of an OIS system in the camera and / or in the presence of significant camera shake during long-exposure image capture. Therefore, for cameras with OIS systems, the range of exposure times can be up to the maximum threshold exposure time in low-light environments, which will allow for the use of small gains and thus allow for less noise. However, for cameras without OIS systems, using very long exposure times may produce motion-blurred images, which is generally undesirable. Therefore, as can now be understood, the exposure time for long-exposure images may not always be the maximum threshold exposure time allowed by the image capture device.

[0043] According to some embodiments, the incoming image stream may include a specific exposure sequence and / or a specific exposure mode. For example, according to some embodiments, the incoming image sequence may include: EV0, EV-, EV0, EV-, etc. In other embodiments, the incoming image sequence may only include EV0 images. In response to a received capture request, according to some embodiments, the image capture device may capture one (or more) "high-resolution" images and one (or more) long-exposure images. After the long-exposure capture, the image capture device may return to the specific sequence of incoming image exposures, for example, the aforementioned sequence: EV0, EV-, EV0, EV- sequence. The exposure sequence may continue in this manner, for example, until a subsequent capture request is received, the camera stops capturing images (e.g., when the user turns off the device or disables the camera application), and / or when one or more operating conditions may change. In yet another embodiment, the image capture device may, in response to an image capture request received during image streaming mode, capture one or more additional EV0 images (referred to herein as "pre-enclosed" or PB capture) before obtaining appropriate enclosed capture in response to the image capture request. The device can then use a deep neural network to fuse additional EV0 exposure images (and optionally, one or more additional EV0 images captured prior to the received capture request, and long exposure images, if needed) into intermediate material, as discussed above. This intermediate material can be used for additional downstream machine learning-enhanced image fusion and / or noise reduction processes, such as those described herein. According to some implementations, images in the incoming image stream may be captured as part of the device's preview operation, or otherwise captured while the device's camera is active, allowing the camera to respond more quickly to the user's image capture requests. Returning to the incoming image sequence ensures that the device's camera is ready for the next image capture request.

[0044] According to some implementations, the terms "high resolution" (or "higher resolution") and "low resolution" (or "lower resolution") may be used herein to refer to the relative difference in the number of pixel values ​​produced by an image sensor for a particular captured image. For example, a "high resolution" image may refer to an image captured with a greater number of pixels than a "low resolution" image in the same incoming image stream. In some implementations, a high resolution image may include an image captured at a resolution greater than a minimum threshold resolution (e.g., greater than 12 megapixels (MP), greater than 24 MP, etc.).

[0045] In other embodiments, as described above, high-resolution images may include images captured natively at a resolution level relatively higher than the corresponding normal or "low" resolution image of the image capturing device (e.g., a resolution level 2, 4, 8, or 9 times higher than the resolution of a so-called low-resolution image). In some cases, the higher-resolution image sensor may have a pixel color mode that mirrors an existing color filter array (CFA) mode (e.g., the Bayer color filter array mode used by a "low-resolution" image sensor), but with finer-grained detail. For example, if a typical Bayer mode follows the pixel pattern:

[0046] BGBG…

[0047] GRGR…

[0048] The 4x higher resolution image sensor can follow the same Bayer color filter pattern, but instead of the "low resolution" Bayer CFA image sensor pattern, each pixel location is further subdivided into a 2×2 grid of pixels of the same color. This means that the 8 pixels in the example pattern above are represented on the sensor as 8×4 or 32 pixels on the higher resolution image sensor, as shown in the pattern for example:

[0049] BBGGBBGG…

[0050] BBGGBBGG…

[0051] GGRRGGRR…

[0052] GGRRGGRR…

[0053] Other color filter array patterns and methods for achieving higher resolution images are also possible, with the above example being just one such option. In yet another implementation, as will be explained in more detail below, the specific resolution of the high-resolution image used in the neural image fusion process may be further based, at least in part, on the amount of merging applied to the image capture device, where a higher level of merging results in a smaller representation of the high-resolution image. In some cases, determining the amount of merging may be a trade-off between the loss of image detail level and the gain in overall processing / memory / power efficiency obtained by being able to operate on images with a smaller total memory footprint.

[0054] Example incoming image stream

[0055] Now for reference Figure 1An exemplary incoming image stream 100, according to one or more embodiments, is illustrated as a source image stream 100 that can be used to generate one or more intermediate materials for use in machine learning-enhanced image fusion and / or noise reduction methods. Images from the incoming image stream 100 may be captured along a timeline (e.g., exemplary image capture timeline 102), which... Figure 1 The timeline extends from left to right. It should be understood that this timeline is presented for illustrative purposes only, and a given incoming image stream may be captured for seconds, minutes, hours, days, etc., depending on the capabilities and / or needs of a given specific implementation.

[0056] According to some implementations, EVO image frames in the incoming image stream may be captured by default at a first frame rate (e.g., 15 frames per second (fps), 30 fps, 60 fps, etc.). In some implementations, this frame rate may remain constant and uninterrupted unless (or until) an image capture request 106 is received at the image capture device. In other implementations, the capture frame rate of the EVO image frames may vary over time based on, for example, one or more device conditions (such as device operating mode, available processing resources, ambient lighting conditions, device thermal conditions, etc.).

[0057] In other embodiments, one or more captured EV0 images may be paired with another image as part of a so-called “secondary frame pair” (SFP). According to some embodiments, the SFP may comprise images captured and read from by an image sensor sequentially (e.g., immediately following the capture of the corresponding EV0 image). In some embodiments, the SFP may comprise an EV0 image and: an EV-1 image frame, an EV-2 image frame, or an EV-3 image frame, etc. EV-images will have shorter exposure times, and are therefore slightly darker and noisier than their EV0 counterparts, but they may be better at freezing motion and / or representing detail in darker areas of the image.

[0058] exist Figure 1 In the example shown, the image capture device sequentially captures 104 SFPs (e.g., 1041, 1042, 1043, 1044, etc.), where each SFP includes two images with different exposure values, such as an EV0 image and a corresponding EV- image. Note that... Figure 1 The illustrated EV0 and EV- images use subscript symbols (e.g., EV-1, EV-2, EV-3, EV-4, etc.). These subscripts are intended only to indicate different instances of the image being captured (not different numbers of exposure stops). It should be understood that, although in Figure 1The example is illustrated as an EV0 image and an EV-2 image pair, but any desired exposure level pair can be used for images in an SFP, such as an EV0 image and an EV-2 image, or an EV0 image and an EV-3 image, etc. In other embodiments, depending on the capabilities of the image capture device, an SFP may even include more than two images (e.g., three or four images).

[0059] In some implementations, the relative exposure settings of the image capture device during image capture (including each SFP) may follow the image capture device's AE mechanism. Therefore, in some cases, the exposure settings for each SFP can be determined independently of other captured SFPs. In some cases, the AE mechanism may have a built-in delay or hysteresis in its response to changes in ambient lighting conditions, causing the camera's AE settings not to change too quickly, resulting in undesirable flickering or brightness variations. Therefore, the exposure settings for a given captured image (e.g., EV0 image, EV- image, and / or EV+ image) may be based on the camera's current AE settings. Since image readout in an SFP is continuous, each image in an SFP may follow the same AE settings (i.e., will be captured relative to the same EV0 settings calculated for the current lighting conditions). However, if the delay between captured images in the SFP is long enough and / or if the camera's AE mechanism reacts quickly enough to changes in ambient lighting, in some cases, images in a given SFP may follow different AE settings (i.e., the first image in the SFP may be captured relative to a first calculated EV0 setting, and the second image in the SFP may be captured relative to a second calculated EV0 setting). Of course, outside the context of the SFP, it may also be possible to capture consecutively captured images (e.g., from an incoming image stream) relative to different calculated EV0 settings, based on, for example, changing ambient lighting conditions and the rate at which the camera's AE mechanism updates its calculated EV0 settings.

[0060] According to some implementations, the capture frame rate of the incoming image stream can be varied based on the ambient light level (e.g., capturing at 30 frames per second (fps) in bright light conditions and at 15 fps in low light conditions). In one example, assuming the image sensor streams captured images at a rate of 30 fps, consecutive SFP image pairs (e.g., EV0, EV-) are also captured at 30 fps. The time interval between any two such SFP captures will be 1 / 30 of a second, and such an interval can be allocated between the capture of two images (e.g., EV0 image and EV- image) in the SFP. According to some implementations, the first portion of this interval can be used to capture the EV0 image in the pair, and the last portion of the interval can be used to capture the EV- image in the pair. Of course, in this 30 fps example, the sum of the exposure times of the EV0 image and the EV- image in a given pair cannot exceed 1 / 30 of a second. In yet other implementations, the capture of the EV- image from each SFP can be disabled based on the ambient light level. For example, when the scene lux level is below the threshold, it is easy to disable the capture of EV-images from each SFP, since any information captured under such exposures may be too noisy to be used for subsequent fusion operations.

[0061] Moving forward along timeline 102 to capture request 106, according to some implementations, one or more additional EV0 images and EV-image pairs (e.g., pre-banded or “PB” image pairs 104) may be captured during system time delay interval 108 before the image sensor switches modes to capture one (or more) high-resolution images 110. PB (For example, an EV0 high-resolution image has 2, 4, or even more pixels than a potentially generated SFP 104, as described above, which has a relatively low resolution due to image sensor merging.)

[0062] In some implementations, the image capture device may also capture one or more additional long exposure images, such as long exposure image 112, in response to receiving a capture request 106 (e.g., after capturing any desired high-resolution image material such as 110). As described above, according to some implementations, a system latency delay 108 may exist in the image capture stream after receiving the image capture request 106. In some cases, an additional intentional delay may also be built into the image capture process after receiving the image capture request, for example, to reduce any jitter or vibration caused by the user touching or selecting a capture button (e.g., a physical button or a software-based user interface button or other graphical element) on the image capture device before initiating any high-resolution image and / or long exposure image capture. For example, while long exposure images are more likely to produce low-noise images, they are also more prone to blurring and therefore lack sharpness due to the amount of time the shutter remains open during the capture of a long exposure image. As is now understood, due to various system latency (and the photographer's reaction time), the image enclosure that best represents the moment the user presses the shutter to indicate the desired image capture can actually be the image enclosure captured before the shutter is pressed. In other words, in Figure 1 In the example, the image enclosing can actually be an image from one of the SFP104 (e.g., 1041, 1042, 1043, 1044) that best captures or “freezes” the moment the photographer intended to capture (and / or has the best sharpness score), and is therefore used as the best “reference” image, with other image materials used in the fusion process aligned with this best “reference” image.

[0063] Based on an evaluation of one or more capture conditions, the image capture device may then select two or more images captured before image capture request 106 (e.g., represented by optional dashed line 114) and one or more images captured after image capture request 106 (e.g., represented by optional dashed line 116) to be included in the image fusion operation performed by the first neural network 1181 to generate first intermediate material, such as relatively low-resolution intermediate material, which successfully “freezes” the motion of the scene close to the time of image capture request 106 and is able to reduce noise / increase detail compared to any single image included in the fusion operation performed by the first neural network 1181.

[0064] The image capture device can also select one or more additional relatively high-resolution images (e.g., such as...). Figure 1The high-resolution EV0 image 110 is used for processing by the second neural network 1182 to generate a second intermediate material, such as a high-resolution image that has been enhanced (i.e., denoised, de-mosaiced (i.e., in the case where the original image material is input into network 1182) and / or downsized (e.g., in the case where the original or high-resolution size of the image captured by the image sensor needs to be reduced before being processed by the third neural network 1183) as will be described in more detail below. In some embodiments, the image comes from the pre-enclosed image pair 104 PB The EV-image may be included in the HDR fusion process along with the high-resolution EVO image 110 before being processed by the second neural network 1182 (e.g., indicated by optional dashed line 117). In yet another embodiment, if sufficient time / processing resources are available, a high-resolution EV-image (not illustrated) may also be captured and used to perform highlight restoration on the high-resolution EVO image 110. Finally, it should be understood that one or more images captured as part of the incoming image stream (or in response to image capture request 106) may be processed as needed, for example, by a hardware image signal processor, to perform highlight / shadow restoration on them before further processing by any of the neural networks in neural network 118.

[0065] Once the outputs (i.e., intermediate materials) of the first neural network 1181 and the second neural network 1182 have been created, these outputs can be further fused, for example, by intelligently transferring additional details from the higher-resolution second intermediate material from 1182 to the appropriate and corresponding portions of the lower-resolution intermediate material from 1181 via a third neural network 1183 (e.g., indicated by dashed line 120) to form the final neurally fused image 122. In some embodiments, both the first neural network 1181 and the second neural network 1182 can generate linear RGB output image data, each of which can be individually subjected to local tone mapping, sharpening, subject relighting, and / or other post-processing operations as needed before being fed to the third neural network 1183.

[0066] Please refer to the following: Figure 3 To explain further in detail, machine learning techniques (e.g., deep neural networks) can be used to determine the preferred or optimal way to fuse the images (e.g., intermediate materials) used to generate the final fused image 122 and / or to pass relevant details between corresponding sub-parts of these images.

[0067] The third neural network 1183 processes the selected images and / or intermediate materials from the incoming image stream 100 (e.g., the outputs of the first neural network 1181 and the second neural network 1182, such as...). Figure 1The final fusion operation (as illustrated) will produce a final neural fusion output image 122 (wherein, in this context, the modifier "neural" refers to the fact that the output image 122 is generated using one or more deep neural networks (DNNs). The decision on how to ultimately fuse the various images and / or image-based intermediate materials included in the final network fusion operation can be made by one or more trained deep neural networks (e.g., 1183). Similarly, as... Figure 1 As illustrated in the examples, in some implementations, after capturing a long exposure image 112 following capture request 106, the image capture stream can return to capturing SFP 104. N The EVO image or any other image mode required for a given specific implementation, for example, until the next capture request is received, thereby triggering the capture of another high-resolution image and / or long-exposure image (and / or the generation of one or more synthetic intermediate materials to be used in the final neural image fusion operation), or until the camera functionality of the device is deactivated.

[0068] Overview of the Derivation of Details in Deep Neural Networks

[0069] Now go to Figure 2 This illustrates an overview of a process 200 for performing high-resolution and low-latency machine learning-enhanced image fusion and / or denoising, according to one or more embodiments. (See above references...) Figure 1 As described, the image capture device can be placed in image capture mode to capture one or more "live" capture clips 201 (e.g., Figure 2 Image capture from the incoming image stream in 2021 to 2022 N At a given time, the user of the image capture device may issue an image capture request 203, for example, by pressing a physical or virtual shutter button, instructing one or more image sensors of the image capture device to capture and generate an image of the scene (i.e., how the scene appears at the moment of the image capture request (or as close as possible to that moment)).

[0070] It should be noted that in some specific implementations, one or more image captures in image capture 202 may actually be captured before image capture request 203 (e.g., image captures 2021 and 2022), and one or more image captures in image capture 202 may be captured after image capture request 203 (e.g., image captures 2023 and 2024). In some implementations, at least one image among the images captured after image capture request 203 (e.g., in...) Figure 2Example 2024) may include one of the aforementioned high-resolution image clips captured by the image sensor of the image capture device. These image captures 2021 and 2022 are referred to herein as “real-time” or “streaming” capture clips 201, indicating that they were obtained at a rate set at the frame rate (e.g., 30 fps) of the video images captured by the image sensor. Image captures following capture request 203 (e.g., image captures 2023 and 2024) are referred to herein as “encircling” captures, indicating that they were obtained immediately and continuously in response to capture request 203 until the image capture device can return to capturing “real-time” or “streaming” capture clips (e.g., 2024). N Until then. Enclosing image capture 202 may include, for example, the above reference. Figure 1 One or more of the SFP104, high-resolution image 110, and / or long-exposure image 112 discussed.

[0071] Now turning to the deferred processing and deep network detail transfer section (205) of process 200, one or more intermediate materials may be generated, for example, based on network processing 204 of a combination of two or more image captures in image capture 202, according to the implementation described herein. Figure 2 (Intermediate material 2061 to 2062). See above for reference. Figure 1 As described, according to some implementation schemes, an intermediate material (e.g., Figure 2 Intermediate material 1 (2061) may include the result of a neural fusion operation on two or more lower-resolution real-time captured materials (201) at network processing block 2041, and another intermediate material (e.g., Figure 2 Intermediate material 2 (2062) may include the result of neural processing operations performed at network processing block 2042 on one or more high-resolution real-time captured materials (201) (e.g., image capture 2024).

[0072] One or both of the intermediate materials 206 may be processed and / or scaled (if needed) and divided into tiles (i.e., sub-parts) for more efficient comparison between corresponding sub-parts of the respective intermediate materials 206. In some embodiments, to reduce the overall memory footprint of the fusion operation, one of the image-based intermediate materials (e.g., 2061) may intentionally have a lower resolution than another image-based intermediate material (e.g., 2062), which may require scaling up, i.e., so that it may be applied to or used in network processing block 2043 together with other image-based intermediate materials.

[0073] As will be explained in further detail below, in some embodiments, a guide tile generation process 208 may be performed first between the second intermediate material and (optionally, a magnified) the first intermediate material before the additional detail transfer occurs in network processing block 2043. According to some embodiments, the guide tile generation process 208 can be used to find the optimal guide tile for feature transfer from the higher-resolution second intermediate material to the first intermediate material. In some cases, the appropriate guide tile from the second intermediate material for a given tile from the first intermediate material will simply be a tile from the second intermediate material (e.g., after estimating optical flow and then registering the two intermediate materials) that most closely contains the same capture scene content as the tile from the first intermediate material. However, in other cases, such as when the optical flow estimate between the two intermediate materials is quite large, there may not be a corresponding tile in the second intermediate material containing the same capture scene content as a given tile in the first intermediate material. In such instances, the guide tile may be identified as the most similar tile in the feature space, i.e., without considering the output of any per-tile homography estimation process, etc. However, such guide blocks may still contain relevant details, and it may be helpful to try to pass that details to the corresponding part of the first intermediate material.

[0074] Once any necessary scaling operations have been performed and the guide tile has been identified at block 208, a depth network detail transfer operation can be performed on a tile-by-tile basis at network processing block 2043. In some implementations, each tile may share an overlapping boundary pixel region with each adjacent tile, wherein pixels in such overlapping boundary pixel regions may be blended (e.g., according to a predetermined weighting function) before the individual tiles are reassembled into a full-size second intermediate material.

[0075] At block 210, any desired adjustments or post-processing can be applied to the merged high-resolution image to generate the final merged high-resolution output image 212. Examples of types of adjustments and post-processing operations that can be performed on the merged image include: brightness filtering (e.g., to maintain a consistent color appearance between intermediate materials), sharpening operations, determining the percentage of high-resolution or texture details to be added back to the merged image (e.g., based on estimated blur amounts, brightness values, skin / face segmentation regions, etc.), subject relighting, rotation, and / or additional hardware zooming or scaling (e.g., based on user-requested zoom / resolution levels).

[0076] Deep network-enhanced image detail delivery

[0077] As mentioned above, according to some implementations, a deep network-enhanced image detail transfer process is performed using two or more intermediate images as input, which does not introduce unwanted ghosting / blurring into the final output image. One way to perform such a deep network-enhanced image detail transfer process involves a multi-scale feature fusion process, in which details are transferred from a higher-resolution image portion of one intermediate image to a corresponding lower-resolution image portion of another intermediate image in the feature space.

[0078] As mentioned above, one way to improve the efficiency of this deep network-enhanced image detail transfer process is to operate on image patches (i.e., sub-regions, e.g., 800-pixel × 800-pixel rectangular areas) rather than the entire image. This might involve identifying and / or generating optimal high-resolution "guide" patches from higher-resolution images (e.g., second intermediate materials, as described above) for each patch, including lower-resolution images (e.g., the first intermediate material described above). According to some implementations, the guide patch generation process may preferably be based on computing a homography estimate per patch and avoiding any dense distortion operations, which can be computationally expensive and / or introduce undesirable artifacts, such as distortion of occluded areas, ghosting, etc.

[0079] Therefore, according to some implementation schemes, the guided tile generation process can be initiated by performing a global registration operation between a first intermediate image and a second intermediate image. This process can be used to determine a single global homography matrix that optimally aligns the majority of pixels in the second intermediate image with the corresponding pixels in the first intermediate image.

[0080] Next, a dense optical flow (OF) field can be estimated between (optionally, scaled-down) versions of the first and second intermediate images. According to some implementations, a network-based OF field estimation can be used. Then, for example, RANSAC or other model-fitting techniques can be used to extract per-tile homography estimates from the OF estimates. The determined global and local tile homography can then be concatenated and applied as a final homography transformation for each tile, thus requiring only a single alignment transformation for each tile.

[0081] In some implementations, the homography of each per-tile can be checked (e.g., using principal component analysis (PCA) techniques) to ensure that the local per-tile homography is valid (e.g., not considered a statistical outlier among the determined per-tile homography). If, for a given tile, the local homography estimate fails the PCA check (e.g., is considered a statistical outlier because the calculated distance between the estimated homography and the centers of a pre-estimated set of homography in the feature space projected by PCA is greater than a determined threshold), the implementation can simply fall back to the aforementioned global homography estimation. Once the homography check has passed and the higher-resolution guide tile from the second intermediate material corresponding to each first intermediate material tile has been identified, information from the first and second intermediate materials (i.e., including guide tile information) can be fed into a third neural network (e.g., as will be referred to below). Figure 3 (described in more detail), wherein the third neural network is configured to combine the first intermediate material and the second intermediate material (e.g., passing additional details from the second intermediate material to the first intermediate material) to generate an output image with a resolution greater than that of the image from the incoming image stream used to form the first intermediate material (e.g., the output image may have the resolution of the second intermediate material itself, or the resolution of the higher-resolution image captured initially used to generate the second intermediate material).

[0082] Now for reference Figure 3 Example 300 of a neural network architecture, according to one or more embodiments, is illustrated for performing high-resolution and low-latency machine learning-enhanced image fusion and / or denoising. The exemplary neural network 300 exhibits several properties that allow it to efficiently transfer details from higher-resolution image guide tiles to corresponding lower-resolution image tiles in the feature space. For example, network 300 is configured to perform nonlocal feature matching, for example, by using attention modules (e.g., transformers). Such modules allow the network to recognize and understand how geographically distant data elements influence and depend on each other. Network 300 may also exhibit improved detail preservation, for example, by using multiple (and optionally, decoupled) encoders. Finally, the design of network 300 may also increase the degree of ghosting artifact mitigation by using its robust fusion module.

[0083] As will now be described in more detail, Network 300 achieves higher computational efficiency through operations including, but not limited to, the following: using a small number of residual blocks in deeper layers of the network; utilizing multi-head attention modules to determine nonlocal context correspondences; performing fusion operations at multiple scales (which improves detail delivery, especially for static scenes); reducing the detail delivery burden on the attention modules; and performing kernel pruning and quantization (e.g., 8-bit quantization) where appropriate. Additional details regarding the nonlocal attention modules and neural network architectures that can be used to perform high-resolution feature aggregation and detail delivery from portions of a first image to corresponding low-resolution features from other captured images can be found in the co-assigned U.S. Patent Application No. 17 / 658,706 (hereinafter referred to as the “706 Application”), filed April 11, 2022, entitled “Reference-Based Super-Resolution for Image and VideoEnhancement,” the entire contents of which are hereby incorporated by reference.

[0084] like Figure 3 As shown, one or more input intermediate image materials (e.g., 302 / 304) can be combined by network 300 to produce a high-resolution output image 368. Starting with a first intermediate material (e.g., a lower-resolution image 302), image data for the lower-resolution first intermediate material 302 (e.g., divided into tile-based sub-parts, as described above) is shown following a first processing path through network 300 from one or more convolutional layers 308 / 310, which may, for example, have 16 channels (or some other multiple of eight channels, depending on the requirements of a given specific implementation).

[0085] Similarly, image data for the higher-resolution second intermediate material 304 (e.g., divided into tile-based sub-parts, as described above) follows a second processing path through network 300, starting from one or more convolutional layers 318 / 320. Next, data from each intermediate material is processed at a first scale by path 321. Specifically, data from the first intermediate material 302 from convolutional layer 310 may be processed by one or more additional convolutional layers 328, while data from the second intermediate material 304 from convolutional layer 320 may be processed by one or more additional convolutional layers 322, and then each output is combined at a low-resolution (LR) / high-resolution (HR) feature fusion block 334.

[0086] In parallel, data from each intermediate source is processed at a second scale by path 331 (e.g., a path with a smaller resolution but greater feature depth than path 321). Specifically, data from the first intermediate source 302 from convolutional layer 328 may be processed by one or more additional convolutional layers 330, while data from the second intermediate source 304 from convolutional layer 322 may be processed by one or more additional convolutional layers 324, and then each output is combined at the LR / HR feature fusion block 336.

[0087] In the third parallel processing path, data from each intermediate material is processed at a third scale by path 341 (e.g., a path with a smaller resolution but greater feature depth than paths 321 or 331). Specifically, the first intermediate material 302 data from convolutional layer 330 can be processed by one or more additional convolutional layers 332, while the second intermediate material 304 data from convolutional layer 324 can be processed by one or more additional convolutional layers 326. Each output is then combined at an LR / HR attention and feature fusion block 338 (e.g., a nonlocal multi-head attention module). The output of block 338 (representing a fusion of HR and LR details based on nonlocal context correspondence) can then be processed by one or more convolutional layers (340), residual blocks (342), and additional convolutional layers (344), and then amplified (346) as needed, so that the output channel can be concatenated at block 348 with the output of the LR / HR feature fusion block 336 from path 331.

[0088] Then, the output of the splicing block 348 can be processed by one or more convolutional layers (350), residual blocks (352), and additional convolutional layers (354), and then amplified (356) as needed so that the output channel can be spliced ​​at block 358 with the output of the LR / HR feature fusion block 334 from path 321.

[0089] Then, the output of stitching block 358 can be processed by one or more convolutional layers (360), residual blocks (362), and additional convolutional layers (364), and then amplified (366) as needed, so that the output channel can be stitched at block 312 with the output from convolutional layer 310 (i.e., the original features from the lower-resolution first intermediate material 302). Then, the output of block 312 itself can be passed through one or more convolutional layers 314 before it is combined at block 316 with a version of the first intermediate material image data 302 that has been smoothed (e.g., via Gaussian smoothing) at block 306 (e.g., in an element-wise addition manner). Then, the output of block 316 is the aforementioned final high-resolution fused output image 368.

[0090] Compared to the LR / HR attention and feature fusion block 338, the LR / HR feature fusion blocks 334 and 336 do not utilize a multi-head attention module, but instead perform direct feature fusion, which is well-suited for static scenes (i.e., cases where the homography alignment between the first and second intermediate footage is very good). As is now understood, this design choice allows network 300 to more aggressively transfer details from the HR intermediate footage (304) to the LR intermediate footage (302) at certain scales. This design choice also allows for more aggressive quantization through network 300, which speeds up the overall network latency.

[0091] Methods for performing high-resolution and low-latency machine learning-enhanced image fusion

[0092] Now for reference Figure 4 The diagram illustrates another method 400, exemplified by using one or more intermediate materials to perform high-resolution and low-latency machine learning-enhanced image fusion and / or denoising according to one or more embodiments. Method 400 may be performed at step 402 by obtaining an incoming image stream (e.g., Figure 1 The method 400 may begin with an image stream 100. Next, at step 404, the method 400 may receive an image capture request (e.g., ...). Figure 1 Image capture request 106). In response to the image capture request, at step 406, method 400 may generate two or more intermediate materials based on the incoming image stream. In some embodiments, the first intermediate material may be generated by a first neural network configured to perform a fusion operation on a determined first one or more images from the incoming image stream, wherein the first intermediate material has a first resolution (step 408), and the second intermediate material may be generated by a second neural network configured to perform an image enhancement operation (e.g., denoising and / or demosaicing) on ​​a second image from the incoming image stream, wherein the second image has a second resolution, and wherein the second resolution is greater than the first resolution (step 410).

[0093] At step 412, method 400 may feed the first intermediate image and the second intermediate image into a third neural network, wherein the third neural network is configured to combine the first intermediate image and the second intermediate image to generate an output image with a resolution greater than the first resolution. See above for example, [reference needed]. Figure 3 Network 300 describes in more detail the neural networks mentioned in step 412, such as the third neural network.

[0094] At step 414, the third neural network can be used to pass additional details from the second intermediate material to the corresponding part (e.g., a tile) of the first intermediate material to produce an output image with a resolution greater than the first resolution (e.g., up to and including the second resolution).

[0095] If necessary, at step 416, optional post-processing and / or adjustments may be performed on the image to generate the final fused output image. For example, additional scaling, rotation, etc., may be required based on the specific “zoom level” / resolution of the image requested by the user, such as the “native” zoom level / resolution based on the output of the third neural network. For example, according to some embodiments, the third neural network may be configured to generate an output image with a resolution configured to simulate a specific fixed-focus lens (e.g., simulating the field of view (FOV) of a 24mm equivalent fixed-focus camera, a 28mm equivalent fixed-focus camera, a 35mm equivalent fixed-focus camera, a 48mm equivalent fixed-focus camera, etc.). Then, if the user has requested a specific zoom level that falls between one of the zoom levels in the existing pre-trained neural network (e.g., the zoom level equivalent to a 30mm fixed focal length camera), the method can simply select and utilize a pre-trained network configured to simulate the closest fixed focal length lens to the requested zoom level (e.g., a neural network configured to generate a 28mm fixed focal length image equivalent FOV) as a third neural network, and then use a high-performance hardware accelerator to perform any additional scaling, rotation, etc. required to upscale the output image to the specific zoom level requested by the user (e.g., in this example, upscale to a 30mm fixed focal length image equivalent).

[0096] In other embodiments, at some user-requested zoom level, the second neural network may be configured to perform image enhancement operations (e.g., denoising operations) on a cropped region of a second image from the incoming image stream. For example, at zoom levels higher than a certain requested zoom level, the second neural network may instead operate on a central cropped region of the second image (e.g., a central crop of approximately 70% of the width and approximately 70% of the height of the second image would yield approximately 50% of the original number of image pixels), that is, instead of performing a naive 2x downscaling operation on the second image (which would also reduce the original number of pixels by 50%, but would reduce the level of detail in the central region of the image in the process of downscaling). As described above, once the central cropped region of the second image has been processed by an appropriate second neural network (e.g., a pre-trained neural network configured to simulate a prime lens closest to the requested zoom level, such as a network for generating images at a 35mm equivalent FOV) to generate second intermediate material, the hardware accelerator can perform any additional scaling required to achieve the precise zoom level requested by the user, thereby producing a final output image with better detail than if the higher-resolution second image had not been used at all during the generation process, while still achieving improved latency and freezing the capture scene as close as possible to the moment the user requested the image to be captured.

[0097] At step 418, if the image capture device has been instructed by the user to continue acquiring the incoming image stream (i.e., "Yes" at step 418), then method 400 may return to step 402. Conversely, if the image capture device has been instructed by the user to stop acquiring the incoming image stream (i.e., "No" at step 418), then method 400 may terminate.

[0098] Exemplary electronic computing device

[0099] See now Figure 5 A simplified functional block diagram of an exemplary programmable electronic computing device 500 according to one embodiment is shown. The electronic device 500 may be, for example, a mobile phone, personal media device, portable camera, or a tablet computer, laptop computer, or desktop computer system. As shown, the electronic device 500 may include a processor 505, a display 510, a user interface 515, graphics hardware 520, device sensors 525 (e.g., proximity sensor / ambient light sensor, accelerometer, inertial measurement unit, and / or gyroscope), a microphone 530, an audio codec 535, a speaker 540, communication circuitry 545, an image capture device 550 (e.g., which may include multiple camera units / optical image sensors with different characteristics or capabilities (e.g., still image stabilization (SIS), high dynamic range (HDR), optical image stabilization (OIS) system, optical zoom, and digital zoom, etc.), a video codec 555, a memory 560, a storage device 565, and a communication bus 570.

[0100] Processor 505 can execute instructions necessary for implementing or controlling various functions performed by electronic device 500 (e.g., image generation and / or processing according to the various embodiments described herein). Processor 505 can, for example, drive display 510 and can receive user input from user interface 515. User interface 515 can take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, and / or touchscreen. User interface 515 can, for example, be a channel through which a user can view a captured video stream and / or indicate a specific image frame that the user wants to capture (e.g., by clicking a physical or virtual button at the moment the desired image frame is being displayed on the device's display screen). In one embodiment, display 510 can display the captured video stream while processor 505 and / or graphics hardware 520 and / or image capture circuitry simultaneously generate and store the video stream in memory 560 and / or storage device 565. Processor 505 can be a system-on-a-chip (SoC), such as those present in mobile devices, and can include one or more dedicated graphics processing units (GPUs). Processor 505 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 520 may be dedicated computing hardware for processing graphics and / or assisting processor 505 in performing computational tasks. In one embodiment, graphics hardware 520 may include one or more programmable graphics processing units (GPUs) and / or one or more dedicated system-on-a-chip (SoCs), such as SoCs specifically designed to implement neural networks and machine learning operations (e.g., convolutions) in a more energy-efficient manner than a main device central processing unit (CPU) or a typical GPU, such as Apple's Neural Engine processing cores.

[0101] For example, according to this disclosure, image capture device 550 may include one or more camera units configured to capture images, for example, images that can be processed to aid in further calibration of the image capture device in field use. Image capture device 550 may include two (or more) lens assemblies 580A and 580B, wherein each lens assembly may have a separate focal length. For example, lens assembly 580A may have a shorter focal length relative to the focal length of lens assembly 580B. Each lens assembly may have a separate associated sensor element, for example, sensor elements 590A / 590B. Alternatively, two or more lens assemblies may share a common sensor element. In some embodiments, sensor element 590 may be configured to perform a pixel-binding operation, for example, outputting an image having a native (e.g., high-resolution) image or a downsized (e.g., low-resolution) image as a result of performing the pixel-binding operation on sensor hardware. Image capture device 550 may capture still images and / or video images. The output from the image capture device 550 can be processed at least in part by: a video codec 555, and / or a processor 505, and / or graphics hardware 520, and / or a dedicated image processing unit or image signal processor integrated within the image capture device 550. The captured images can then be stored in a memory 560 and / or a storage device 565.

[0102] Memory 560 may include one or more different types of media used by processor 505, graphics hardware 520, and image capture device 550 to perform device functions. For example, memory 560 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 565 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 565 may include one or more non-transitory storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). Memory 560 and storage device 565 can be used to hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. For example, when executed by processor 505, such computer program code may implement one or more of the methods or processes described herein. The power source 575 may include electronic components and associated circuitry for managing electronic device 500 and / or a rechargeable battery (e.g., a lithium-ion battery, etc.) or other electrical connections to a power source (e.g., to a mains power source) for supplying power to the electronic components and associated circuitry of the electronic device 500.

[0103] It should be understood that the above description is intended to be illustrative and not restrictive. For example, the above embodiments may be used in combination with each other. Many other embodiments will be apparent to those skilled in the art upon review of the above description. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. An apparatus, the apparatus comprising: Memory; user interface; Image capture device; and One or more processors, operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to perform the following operations: Obtain the incoming image stream from the image capture device; Receive image capture request via the user interface; In response to the image capture request, two or more intermediate clips are generated, wherein: The first intermediate material among two or more generated intermediate materials comprises an image generated by a first neural network configured to perform a fusion operation on a determined first one or more images from the incoming image stream, wherein the first intermediate material has a first resolution; and The second intermediate material among the two or more generated intermediate materials includes an image generated by a second neural network configured to perform an image enhancement operation on at least a second image from the incoming image stream, wherein the second image has a second resolution and wherein the second resolution is greater than the first resolution; The first intermediate image and the second intermediate image are fed into a third neural network, wherein the third neural network is configured to combine the first intermediate image and the second intermediate image to generate an output image with a resolution greater than the first resolution; and The third neural network is used to generate the output image.

2. The device of claim 1, wherein one or more of the first or more images were captured prior to receiving the image capture request.

3. The device of claim 2, wherein at least one of the following is captured after the image capture request is received: a) the first one or more images; or b) the second image.

4. The device according to claim 1, wherein the image enhancement operation includes at least one of the following: denoising operation or de-mosaic operation.

5. The device of claim 1, wherein the second neural network is further configured to perform the image enhancement operation on a cropped region of the second image from the incoming image stream.

6. The device of claim 1, wherein the second resolution is n times greater than the first resolution, wherein n is greater than or equal to 2.

7. The device of claim 1, wherein the output image has an improved level of detail compared to the first intermediate material.

8. The device of claim 1, wherein the third neural network is further configured to operate on the tiles of the first intermediate material.

9. The device of claim 8, wherein the one or more processors are further configured to execute instructions that cause the one or more processors to perform the following operations: Perform per-tile homography estimation between the tiles of the second intermediate material and the tiles of the first intermediate material.

10. The apparatus of claim 9, wherein the one or more processors are further configured to execute instructions that cause the one or more processors to perform the following operations: For each tile in the first intermediate material, a guide tile is identified in the second intermediate material.

11. The apparatus of claim 10, wherein the third neural network is further configured to pass details from each tile in the first intermediate material to a corresponding guide tile in the second intermediate material to each tile in the first intermediate material.

12. The apparatus of claim 1, wherein the third neural network is further configured to generate the output image having a resolution configured to simulate a particular fixed-focus lens.

13. The device of claim 1, wherein the output image has the second resolution.

14. A non-transitory program storage device, the non-transitory program storage device comprising instructions stored thereon to cause one or more processors to perform the following operations: Obtain the incoming image stream from the image capture device; Receive image capture request; In response to the image capture request, two or more intermediate clips are generated, wherein: The first intermediate material among two or more generated intermediate materials comprises an image generated by a first neural network configured to perform a fusion operation on a determined first one or more images from the incoming image stream, wherein the first intermediate material has a first resolution; and The second intermediate material among the two or more generated intermediate materials includes an image generated by a second neural network configured to perform an image enhancement operation on at least a second image from the incoming image stream, wherein the second image has a second resolution and wherein the second resolution is greater than the first resolution; The first intermediate image and the second intermediate image are fed into a third neural network, wherein the third neural network is configured to combine the first intermediate image and the second intermediate image to generate an output image with a resolution greater than the first resolution; as well as The third neural network is used to generate the output image.

15. The non-transitory program storage device of claim 14, wherein the third neural network is further configured to operate on the tiles of the first intermediate material.

16. The non-transitory program storage device of claim 15, wherein the instructions stored in the non-transitory program storage device further cause the one or more processors to: Perform per-tile homography estimation between the tiles of the second intermediate material and the tiles of the first intermediate material.

17. The non-transitory program storage device of claim 16, wherein the instructions stored in the non-transitory program storage device further cause the one or more processors to: For each tile in the first intermediate material, a guide tile is identified in the second intermediate material.

18. The non-transitory program storage device of claim 17, wherein the third neural network is further configured to pass details from each tile in the first intermediate material to a corresponding guide tile in the second intermediate material to each tile in the first intermediate material.

19. The non-transitory program storage device of claim 14, wherein the third neural network is further configured to generate the output image having a resolution configured to simulate a specific fixed-focus lens.

20. An image processing method, the image processing method comprising: Obtain the incoming image stream from the image capture device; Receive image capture request; In response to the image capture request, two or more intermediate clips are generated, wherein: The first intermediate material among two or more generated intermediate materials comprises an image generated by a first neural network configured to perform a fusion operation on a determined first one or more images from the incoming image stream, wherein the first intermediate material has a first resolution; and The second intermediate material among the two or more generated intermediate materials includes an image generated by a second neural network configured to perform an image enhancement operation on at least a second image from the incoming image stream, wherein the second image has a second resolution and wherein the second resolution is greater than the first resolution; The first intermediate image and the second intermediate image are fed into a third neural network, wherein the third neural network is configured to combine the first intermediate image and the second intermediate image to generate an output image with a resolution greater than the first resolution; and The third neural network is used to generate the output image.

Images