Temporal filtering for motion distortion correction

Abstract

The present disclosure provides systems, methods, and devices for image signal processing that support improved temporal filtering of image frames to correct for motion distortion. In a first aspect, a method of image processing includes receiving image data including a first image frame. One or more of (i) a gain map of the first image frame and / or (ii) a motion map of the first image frame may be calculated and may be used to determine at least one parameter for the temporal filtering process. The corrected image frame may be determined by temporally filtering the first image frame based on the at least one parameter. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure generally relate to image processing, and more particularly to processing images to correct motion artifacts and other errors. Some features can implement and provide improved image processing, including processing images using temporal filtering. Background Art

[0002] An image capture device is a device that can capture one or more digital images (either still images for photos or sequences of images for video). The capture device can be incorporated into various devices. By way of example, the image capture device can include a standalone digital camera or digital video camera, a wireless communication device handset equipped with a camera (such as a mobile phone, cellular or satellite radiotelephone), a personal digital assistant (PDA), a panel or tablet device, a gaming device, a computing device (such as a webcam, a video surveillance camera), or other devices having digital imaging or video capabilities.

[0003] Movement of an object during the capture of an image frame (i.e., an image frame for compositing into a single still image, an image frame for use as part of a video sequence) can create various distortions within the image frame. For example, movement of one or more objects within the image frame can blur and / or blend these objects together or can leave motion artifacts within the captured image frame. Summary of the Invention

[0004] Some aspects of the present disclosure are summarized below to provide a basic understanding of the technologies discussed. This summary is not an exhaustive overview of all the expected features of the present disclosure, and is neither intended to identify the key or important elements of all aspects of the present disclosure, nor to depict the scope of any or all aspects of the present disclosure. The sole purpose of this summary of the invention is to present some concepts of one or more aspects of the present disclosure in a generalized form as a prelude to the more detailed embodiments given later.

[0005] An image frame can be received to detect and correct motion distortions and motion artifacts within the image frame. The image frame can be analyzed (e.g., sequentially analyzed) to correct these errors. An earlier first image frame can be compared with a later second image frame to generate a motion map that can indicate the motion within the first image frame relative to the second image frame. For example, the motion map can indicate the local movement of an object depicted in the second image frame within the first image frame. Additionally or alternatively, the motion map can reflect the global motion of the device used to capture the image frame. A gain map can be calculated for the first image frame. For example, the gain map can be calculated to correct or balance the luminance values within the first image frame. One or more parameters for a temporal filtering process can be calculated based on the motion map and / or the gain map. For example, the dynamic motion blend intensity can be determined based on the motion map, and / or the dynamic spatial blend intensity can be determined based on the gain map.

[0006] A temporal filtering process may be applied to a first image frame according to one or more parameters to generate a corrected image frame of the first image frame. The temporal filtering process may blend multiple versions of the first image frame, including an original version of the first image frame, an aligned version of the first image frame, and a denoised version of the first image frame. A dynamic motion blend intensity may adjust the intensity at which the aligned version of the first image frame is blended to generate the corrected image frame. A dynamic spatial blend intensity may adjust the intensity at which the denoised version of the first image frame is blended to generate the corrected image frame. The temporal filtering process may be applied on a per-pixel basis to generate the corrected image frame. The corrected image frame may then be added to an output file. The process may be repeated to process multiple image frames. For example, the process may be sequentially repeated across all of the image frames in the received image frames (e.g., in the order in which the image frames are captured and / or received).

[0007] Temporal filtering blends pixels from adjacent images to reduce temporal noise. In some embodiments, a difference threshold informs the blend intensity. Additionally, frames may be aligned based on local gain and adjusted to compensate for motion and / or luminance changes between the frames to be blended. These adjustments may result in alignment distortion and / or unbalanced noise levels due to local gain compensation. This disclosure describes in part techniques for using a motion magnitude and a gain map to adjust the blend intensity within an image frame (such as on a per-pixel basis). More specifically, the motion magnitude may be used to adjust the temporal filtering threshold and blend intensity because motion-distorted regions may have a closer gap between noise and the true signal. For example, the temporal filtering threshold may be decreased in regions with higher motion to preserve the benefit of small local motion distortion while avoiding artifacts introduced by blended pixels with a large amount of motion. Additionally, the gain map may be used to adjust the temporal filtering intensity because both noise and the signal are amplified by local gain.

[0008] In one aspect of the present disclosure, a method for image processing includes: receiving image data including a first image frame; determining at least one of a gain map of the first image frame or a motion map of the first image frame; determining at least one parameter for a temporal filtering process based on at least one of the gain map and the motion map; and determining a corrected image frame by temporally filtering the first image frame based on the at least one parameter.

[0009] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to: receive image data including a first image frame; determine at least one of a gain map of the first image frame or a motion map of the first image frame; determine at least one parameter for a temporal filtering process based on at least one of the gain map and the motion map; and determine a corrected image frame by temporally filtering the first image frame based on the at least one parameter.

[0010] In one aspect of the present disclosure, a method for image processing includes: receiving image data including a first image frame; determining a motion map of the first image frame, the motion map identifying movement within the first image frame; determining an aligned frame based on the motion map and the first image frame, wherein the aligned frame corrects movement distortion within the first image frame; determining a dynamic motion blending intensity based on the motion map; and determining a corrected image frame by temporally filtering the first image frame and the aligned frame based on the dynamic motion blending intensity.

[0011] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to execute processor-readable code to cause the at least one processor to perform operations that include: receiving image data including a first image frame; determining a motion map of the first image frame, the motion map identifying movement within the first image frame; determining an aligned frame based on the motion map and the first image frame, wherein the aligned frame corrects movement distortion within the first image frame; determining a dynamic motion blending intensity based on the motion map; and determining a corrected image frame by temporally filtering the first image frame and the aligned frame based on the dynamic motion blending intensity.

[0012] The image processing methods described herein may be performed by an image capture device and / or on image data captured by one or more image capture devices. An image capture device (a device that can capture one or more digital images, whether a still image photograph or a sequence of images of a video) may be incorporated into a variety of devices. By way of example, an image capture device may include a standalone digital camera or digital video camera, a wireless communication device phone equipped with a camera (such as a mobile phone, cellular or satellite radiotelephone), a personal digital assistant (PDA), a panel or tablet device, a gaming device, a computing device (such as a webcam, a video surveillance camera), or other devices having digital imaging or video capabilities.

[0013] The image processing techniques described herein may relate to a digital camera having an image sensor and processing circuitry (e.g., an application specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), or a central processing unit (CPU)). An image signal processor (ISP) may include one or more of these processing circuits and may be configured to perform operations to obtain image data for processing according to the image processing techniques described herein and / or the image processing techniques involved in the image processing techniques described herein. The ISP may be configured to control the capture of image frames from one or more image sensors and determine one or more image frames from the one or more image sensors to generate a view of a scene in an output image frame. The output image frame may be part of a sequence of image frames forming a video sequence. The video sequence may include other image frames received from the image sensor or other image sensors.

[0014] In an example application, an Image Signal Processor (ISP) may receive instructions for capturing a sequence of image frames in response to the loading of software, such as a camera application, to generate a preview display from an image capture device. The image signal processor may be configured to generate a single output image frame stream based on the image frames received from one or more image sensors. The single output image frame stream may include raw image data from the image sensors, merged image data from the image sensors, or corrected image data processed by one or more algorithms within the image signal processor. For example, the image frames may be processed by an Image Post-Processing Engine (IPE) and / or other image processing circuitry to process the image frames obtained from the image sensors in the image signal processor (the image frames may have had some processing performed on the data before being output to the image signal processor), thereby performing one or more of tone mapping, portrait illumination, contrast enhancement, gamma correction, etc. The output image frames from the ISP may be stored in memory and retrieved by an application processor executing the camera application, which may perform further processing on the output image frames to adjust the appearance of the output image frames and reproduce the output image frames on a display for user viewing.

[0015] After an output frame representing a scene is determined by the image signal processor and / or the application processor (such as through the image processing techniques described in various embodiments herein), the output image frames may be displayed on a device display as a single still image and / or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network, and / or printed to an output medium. For example, an Image Signal Processor (ISP) may be configured to obtain an input frame of image data (e.g., pixel values) from one or more image sensors and, in turn, generate a corresponding output image frame (e.g., a preview display frame, a still image capture, a frame for video, a frame for object tracking, etc.). In other examples, the image signal processor may output the image frames to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization (e.g., Auto Focus (AF), Auto White Balance (AWB), and Auto Exposure Control (AEC)), generating a video file via the output frames, configuring the frames for display, configuring the frames for storage, sending the frames over a network connection, etc. Generally, an Image Signal Processor (ISP) may obtain incoming frames from one or more image sensors, as well as generate an output frame stream and output the output frame stream to various output destinations.

[0016] In some aspects, an output image frame can be generated by combining aspects of the image correction of the present disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR). In the case of HDR photography, a first image frame and a second image frame are captured using different exposure times, different apertures, different lenses, and / or other characteristics that can result in an improved dynamic range of the fused image when combining the two image frames. In some aspects, the method can be performed for MFNR photography, where the first image frame and the second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to generate a corrected first image frame that has reduced noise compared to the captured first image frame.

[0017] In some aspects, the device can include an image signal processor or a processor (e.g., an application processor) that includes specific functionality for camera control and / or processing, such as enabling or disabling a merging module or otherwise controlling aspects of the image correction. The methods and techniques described herein can be performed entirely by the image signal processor or the processor, or the various operations can be split between the image signal processor and the processor and, in some aspects, across additional processors.

[0018] The device can include one, two, or more image sensors, such as a first image sensor. When there are multiple image sensors, the configurations of these image sensors can be different. For example, the first image sensor can have a larger field of view (FOV) than the second image sensor, or the first image sensor can have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor can be a wide-angle image sensor, and the second image sensor can be a telephoto image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens can have a first magnification, and the second lens can have a second magnification different from the first magnification. Any of these or other configurations can be part of a lens cluster on a mobile device, such as where multiple image sensors and associated lenses are located at offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view can be included. The image processing techniques described herein can be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0019] In additional aspects of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes components for capturing image frames. The device also includes one or more components for capturing data representative of a scene, such as image sensors (including charge-coupled devices (CCDs), Bayer filter sensors, infrared (IR) detectors, ultraviolet (UV) detectors, complementary metal-oxide semiconductor (CMOS) sensors) and time-of-flight detectors. The device may also include one or more components for focusing and / or concentrating light onto one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components can be controlled to capture a first image frame and / or a second image frame input to the image processing techniques described herein.

[0020] For those of ordinary skill in the art, other aspects, features, and specific implementations will become apparent upon reviewing the following description of specific exemplary aspects in conjunction with the accompanying drawings. Although the features may be discussed below with respect to certain aspects and drawings, each aspect may include one or more of the advantageous features discussed herein. In other words, while one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with each aspect. In a similar manner, although the exemplary aspects may be discussed below as device, system, or method aspects, the exemplary aspects may be implemented in various devices, systems, and methods.

[0021] The method may be embedded in a computer-readable medium as computer program code, the computer program code including instructions that cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device that includes: a first network adapter configured to send data, such as recorded images or videos or streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and a memory. The processor may cause the output image frames described herein to be sent over a wireless communication network, such as a 5G NR communication network.

[0022] The features and technical advantages of examples in accordance with the present disclosure have been outlined above rather broadly so that the detailed description that follows may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein, both their organization and method of operation, as well as associated advantages, will be better understood when considered in conjunction with the accompanying drawings. Each of the drawings provided is for the purpose of illustration and description and not as a definition of the limits of the claims.

[0023] Although aspects and specific implementations are described by way of some examples in this application, those skilled in the art will understand that additional specific implementations and use cases may arise in many different arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses can be implemented via integrated chips and other devices based on non-module components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchase devices, medical devices, artificial intelligence (AI)-enabled devices, etc.). Although some examples may or may not specifically point to use cases or applications, applicability of various types of the described innovations may occur. The scope of specific implementations can range from chip-level or module components to non-module, non-chip-level implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more aspects of the described innovations. In some practical environments, devices incorporating the described aspects and features may also necessarily include additional components and features for implementing and practicing the claimed and described aspects. For example, the transmission and reception of wireless signals necessarily includes multiple components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc. having different sizes, shapes, and configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A further understanding of the nature and advantages of the present disclosure can be realized by referring to the following drawings. In the drawings, like components or features may have the same reference numeral. Additionally, various components of the same type can be distinguished by adding a dash and a second label used to differentiate between like components after the reference numeral. If only the first reference numeral is used in the specification, the description applies to any one of the like components having the same first reference numeral, regardless of the second reference numeral.

[0025] Figure 1 A block diagram illustrating an example device for performing image capture from one or more image sensors.

[0026] Figure 2 A block diagram illustrating an example data flow path for image data processing in an image capture device in accordance with one or more embodiments of the present disclosure.

[0027] Figure 3 A block diagram of an example implementation of an image signal processor in accordance with an exemplary embodiment of the present disclosure.

[0028] Figure 4 A flowchart showing an example method for processing an image frame to correct motion distortion according to some embodiments of the present disclosure.

[0029] Figure 5 A block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure.

[0030] The same reference numerals and names in the various figures indicate the same elements. Detailed Description

[0031] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. On the contrary, the detailed description includes specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every instance and that in some instances, for the sake of clarity of presentation, well-known structures and components are shown in block diagram form.

[0032] Existing temporal filtering techniques for correcting motion distortion within an image frame can correct certain types of distortion, but typically introduce additional forms of distortion. For example, applying temporal filtering using image frames that have not been motion-aligned to align objects within consecutive image frames results in loss of detail and blurring due to the movement of objects within the image frames. However, using motion-aligned image frames also introduces artifact distortion within the resulting image frames. Additionally, applying local gain to the image frames used by the temporal filtering process may result in noise level imbalance because regions with high levels of local gain have higher noise values in the resulting image frames after the temporal filtering is complete. The disadvantages mentioned here are merely representative and are included to emphasize the problems that the inventors have identified and sought to improve in existing devices. Aspects of the devices described below may address some or all of these disadvantages as well as other disadvantages known in the art. Aspects of the improved devices described herein may present other benefits different from those described above and may be used in other applications different from those described above.

[0033] The present disclosure provides systems, devices, methods, and computer-readable media that support image processing, including techniques for improved temporal filtering techniques to correct motion distortion within an image frame. Specifically, a gain map and / or a motion map may be calculated for a received image frame. One or more parameters may be calculated based on the gain map and / or the motion map. Then, the one or more parameters may be used to dynamically change the manner in which a temporal filtering process is applied to the received image frame. For example, the one or more parameters may change the intensity at which a motion-aligned image frame is blended into a corrected image frame. As another example, the one or more parameters may change the intensity at which a denoised image frame is blended into a corrected image frame.

[0034] Certain specific implementations of the subject matter described in this disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present invention provides techniques for addressing the drawbacks of prior temporal filtering techniques. For example, due to the improved temporal filtering process applied to generate a corrected image frame, the resulting image may have better image quality. Specifically, by reducing the blending intensity of the motion-aligned image frame in regions with a large amount of motion and reducing the blending intensity in other regions of the image frame, the improved temporal filtering technique may be able to combine the improved image details achieved by the motion-aligned image frame without also combining motion-aligned artifacts in other regions of the motion-aligned frame. Additionally, by increasing the blending intensity of the denoised image frame in regions with high gain, the improved temporal filtering technique may be able to combine the improved luminance values from local gain processing while also correcting the higher noise values in high-gain regions that are more prone to spatial noise. These techniques may correspondingly improve the ability of a computing device to accurately capture images and / or videos.

[0035] Example devices for using one or more image sensors to capture image frames, such as smartphones, may include a configuration of one, two, three, four, or more cameras on the rear side (e.g., the side opposite the main user display) and / or the front side (e.g., the side same as the main user display) of the device. These devices may include one or more image signal processors (ISPs), computer vision processors (CVPs) (e.g., AI engines), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors (ISPs) may store the output image frame in a memory and / or otherwise provide the output image frame to the processing circuitry (such as via a bus). The processing circuitry may perform further processing, such as encoding, storing, transmitting, or other manipulation of the output image frame.

[0036] As used herein, an image sensor may refer to the image sensor itself and any particular other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory, whether a short-term buffer or a long-term non-volatile memory. For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing the individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which digital representation is provided to digital circuitry coupled to the image sensor.

[0037] In the description of embodiments herein, numerous specific details are set forth, such as examples of specific components, circuits, and processes, to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. Additionally, in the following description and for purposes of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, those skilled in the art will understand that practicing the teachings disclosed herein may not require these specific details. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the present disclosure.

[0038] Certain portions of the detailed description that follows are presented in terms of processes, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, the processes, logic blocks, processes, etc. are conceived of as self-consistent sequences of steps or instructions leading to a desired result. These steps are those requiring physical manipulation of physical quantities. Although not necessarily, typically, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0039] In the figures, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described below in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such specific implementation decisions should not be construed as causing a departure from the scope of the present disclosure. Moreover, an example device may include components other than those shown, including well-known components such as processors, memories, and the like.

[0040] Aspects of the present disclosure are applicable to any electronic device that includes, is coupled to, or otherwise processes data from one, two, or more image sensors capable of capturing image frames (or "frames"). The terms "output image frame" and "corrected image frame" may refer to an image frame that has been processed by any of the techniques discussed herein. Additionally, aspects of the present disclosure may be implemented in image sensors or devices coupled to the image sensors having the same or different capabilities and characteristics, such as resolution, shutter speed, sensor type, etc. Additionally, aspects of the present disclosure may be implemented in a device for processing image frames, whether or not the device includes or is coupled to an image sensor, such as a processing device that can retrieve stored images for processing, including processing devices present in a cloud computing system.

[0041] Unless otherwise specifically stated, it should be understood from the following discussion that, throughout this application, discussions using terms such as "access", "receive", "transmit", "use", "select", "determine", "normalize", "multiply", "average", "monitor", "compare", "apply", "update", "measure", "derive", "set", "generate", etc. refer to actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the registers, memories, or other such information storage, transmission, or display devices of the computer system.

[0042] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smart phone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components capable of implementing at least some portions of the present disclosure. Although the description and examples herein use the term "device" to describe aspects of the present disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus can include a device or a portion of a device for performing the described operations.

[0043] Certain components in a device or apparatus described as "components for accessing", "components for receiving", "components for transmitting", "components for using", "components for selecting", "components for determining", "components for normalizing", "components for multiplying", or other similarly named terms referring to one or more operations on data, such as image data, may refer to processing circuitry (e.g., an application specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a central processing unit (CPU)) configured to perform the described functions by a combination of hardware, software, or hardware configured by software.

[0044] Figure 1 A block diagram of an example device 100 for performing image capture from one or more image sensors is shown. Device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some particular implementations, device 100 also includes or is coupled to a processor 104 and a memory 106 storing instructions 108. Device 100 may also include or be coupled to a display 114 and input / output (I / O) components 116. The I / O components 116 can be used to interact with a user, such as a touchscreen interface and / or physical buttons.

[0045] The I / O components 116 may also include a network interface for communicating with other devices including a wide area network (WAN) adapter 152, a local area network (LAN) adapter 153, and / or a personal area network (PAN) adapter 154. An example WAN adapter is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 154 is a Bluetooth wireless network adapter. Each of the adapters 152, 153, and / or 154 may be coupled to an antenna that includes multiple antennas configured for main set reception and diversity reception and / or configured to receive a particular frequency band.

[0046] Device 100 may also include or be coupled to a power source 118 for device 100, such as a battery or a component that couples device 100 to an energy source. Device 100 may also include or be coupled to Figure 1 additional feature portions or components not shown. In one example, a wireless interface that may include multiple transceivers and a baseband processor may be coupled to or included in the WAN adapter 152 for a wireless communication device. In another example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0047] The device may include or be coupled to a sensor hub 150 that interfaces with sensors to receive data regarding the movement of the device 100, data regarding the environment surrounding the device 100, and / or other non-camera sensor data. An example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which can also be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, the gyroscope in the electronic image stabilization system (EIS) may be coupled to the sensor hub or directly coupled to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0048] The image signal processor 112 may receive image data such as for forming an image frame. In one implementation, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first camera 103 and the second camera 105, respectively. In another implementation, a wired interface couples the image signal processor 112 to an external image sensor. In yet another implementation, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0049] The first camera 103 may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may be controlled by an associated autofocus (AF) algorithm 133 executed in the ISP 112, which adjusts the lenses 131 and 132 to focus on a specific focal plane at a certain scene depth from the image sensors 101 and 102. The AF algorithm 133 may be assisted by a depth sensor 140.

[0050] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 focus light onto the image sensors 101 and 102 respectively through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements to digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of a scene. For example, the first lens 131 may be an ultra-wide (UW) lens, and the second lens 132 may be a wide (W) lens. The plurality of image sensors may include a combination of ultra-wide (high field of view (FOV)) sensors, wide sensors, tele sensors, and ultra-tele (low FOV) sensors.

[0051] That is, each image sensor can be configured through hardware configuration and / or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is configured with different lenses having different magnifications, which results in different fields of view. The sensors can be configured such that the UW sensor has a larger FOV than the W sensor, the W sensor has a larger FOV than the T sensor, and the T sensor has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV can capture a field of view in the range of 64 degrees to 84 degrees, a sensor configured for an ultra-side FOV can capture a field of view in the range of 100 degrees to 140 degrees, a sensor configured for a tele FOV can capture a field of view in the range of 10 degrees to 30 degrees, and a sensor configured for an ultra-tele FOV can capture a field of view in the range of 1 degree to 8 degrees.

[0052] The camera 103 may have a variable aperture (VA) camera, where the aperture can be controlled to a specific size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values correspond to smaller aperture sizes, and smaller aperture values correspond to larger aperture sizes. The camera 103 may have different characteristics based on the current aperture size, such as different depths of field (DOF) at different aperture sizes.

[0053] The image signal processor 112 processes the image frames captured by the image sensors 101 and 102. Although Figure 1Illustrated is that device 100 includes two image sensors 101 and 102 coupled to image signal processor 112, but any number (e.g., one, two, three, four, five, six, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor such as depth sensor 140 may be coupled to image signal processor 112, and the output from the depth sensor may be processed in a manner similar to that of image sensors 101 and 102. Example depth sensors include active sensors including one or more of indirect time of flight (iToF), direct time of flight (dToF), light detection and ranging (LiDAR), mmWave, radio detection and ranging (radar), and / or hybrid depth sensors (such as structured light). In embodiments without depth sensor 140, similar information regarding the depth or depth map of an object can be generated passively from the disparity between two image sensors (e.g., using disparity depth measurement or stereo depth measurement), phase detection autofocus (PDAF) sensors, etc. Additionally, there may be any number of additional image sensors or image signal processors for device 100.

[0054] In some embodiments, image signal processor 112 may execute instructions from a memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in image signal processor 112, or instructions provided by processor 104. Additionally or alternatively, image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in this disclosure. For example, image signal processor 112 may include one or more image front ends (IFE) 135, one or more image post - processing engines 136 (IPE), one or more automatic exposure compensation (AEC) 134 engines, and / or one or more video analysis engines (EVA). AF 133, AEC 134, IFE 135, IPE 136, and EVA 137 may each include dedicated circuitry, may be embodied as software code executed by ISP 112, and / or a combination of hardware and software code executed on ISP 112.

[0055] In some embodiments, the memory 106 may include a non-transitory or non-volatile computer-readable medium storing computer-executable instructions 108 to perform all or a portion of one or more operations described in the present disclosure. In some embodiments, the instructions 108 include a camera application (or other suitable application) for generating an image or video to be executed by the device 100. The instructions 108 may also include other applications or programs to be executed by the device 100, such as an operating system and specific applications other than those for image or video generation. A camera application, such as executed by the processor 104, may cause the device 100 to generate an image using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames or may be accessed by the processor 104 to obtain the processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to an external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, the device 100 is a system-on-chip (SoC) that combines the image signal processor 112, the processor 104, the sensor hub 150, the memory 106, and the input / output component 116 into a single package.

[0056] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform the various operations described herein, including motion distortion correction and noise balancing operations. For example, the execution of the instructions may direct the image signal processor 112 to start or end capturing an image frame or a sequence of image frames, where the capture includes motion distortion as described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing a script or instructions of one or more software programs (such as the instructions 108 stored in the memory 106). For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating an image or video) stored in the memory 106.

[0057] When executing a camera application, the processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, a camera application executed on the processor 104 may receive a user command to start a video preview display. Upon receiving the user command, the image signal processor 112 captures and processes a video including a sequence of image frames from one or more of the image sensors 101 or 102. Image processing such as for generating an “output” or “corrected” image frame according to the techniques described herein may be applied to one or more of the image frames in the sequence. Execution of instructions 108 by the processor 104 outside of the camera application may also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine 124 or other coprocessor) to offload certain tasks from the core 104A. The AI engine 124 may be used to offload tasks related to, for example, face detection and / or object recognition. In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0058] In some embodiments, the display 114 may include one or more suitable displays or screens that allow user interaction and / or present items to the user, such as a preview of image frames captured by the image sensors 101 and 102. In some embodiments, the display 114 is a touch-sensitive display. The I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and provide output to the user via the display 114. For example, the I / O component 116 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (such as a power button), a slider, a switch, etc.

[0059] Although shown as being coupled to each other via the processor 104, components such as the processor 104, the memory 106, the image signal processor 112, the display 114, and the I / O component 116 may be coupled to each other in various other arrangements, such as being coupled to each other via one or more local buses, which are not shown for simplicity. Although the image signal processor 112 is illustrated as being separate from the processor 104, the image signal processor 112 may be a core of the processor 104, which is an application processor unit (APU), included in a system-on-chip (SoC), or otherwise included in the processor 104. Although reference is made to the device 100 in the examples herein to perform aspects of the present disclosure, some device components may not be Figure 1is shown to prevent obscuring aspects of the present disclosure. Additionally, other components, the number of components, or combinations of components may be included in a suitable apparatus for performing aspects of the present disclosure. Accordingly, the present disclosure is not limited to a particular apparatus or component configuration, including apparatus 100.

[0060] Figure 1 An exemplary image capture device may be operated to obtain improved images using improved motion distortion correction techniques, such as improved temporal filtering techniques. Figure 2 An example method of operating one or more cameras, such as camera 103, is shown and described below.

[0061] Figure 2 is a block diagram illustrating an example data flow path for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 of system 200 may communicate with an image signal processor (ISP) 112 via a bi-directional bus and / or separate control and data lines. The processor 104 may control the camera 103 via the camera control 210, such as to configure the camera 103 via a driver executed on the processor 104. The camera control 210 may be managed by a camera application 204 executed on the processor 104, which provides user-accessible settings such that the user may specify individual camera settings or select a profile with corresponding camera settings. The camera control 210 communicates with the camera 103 to configure the camera 103 according to commands received from the camera application 204. The camera application 204 may be, for example, a photography application, a document scanning application, a messaging application, or other application that processes image data obtained from the camera 103.

[0062] Camera configuration may be parameters that specify, for example, frame rate, image resolution, readout duration, exposure level, aspect ratio, aperture size, etc. The camera 103 may obtain image data based on the camera configuration. For example, the processor 104 may execute the camera application 204 to instruct the camera 103 via the camera control 210 to set a first camera configuration of the camera 103, obtain first image data from the camera 103 operating in the first camera configuration, instruct the camera 103 to set a second camera configuration of the camera 103, and obtain second image data from the camera 103 operating in the second camera configuration.

[0063] In some embodiments where camera 103 is a variable aperture (VA) camera system, processor 104 may execute camera application 204 to instruct camera 103 to be configured to a first aperture size, obtain first image data from camera 103, instruct camera 103 to be configured to a second aperture size, and obtain second image data from camera 103. Reconfiguration of the aperture and acquisition of the first and second image data may occur when there is little or no change in the scene that can be captured at the first aperture size and at the second aperture size. Example aperture sizes are f / 2.0, f / 2.8, f / 3.2, f / 8.0, etc. Larger aperture values correspond to smaller aperture sizes, and smaller aperture values correspond to larger aperture sizes. That is, f / 2.0 is a larger aperture size than f / 8.0.

[0064] The image data received from camera 103 may be processed in one or more blocks of ISP 112 to form image frame 230 stored in memory 106 and / or provided to processor 104. ISP 112 may use temporal filtering to process the received image frames to correct for movement of objects within the image frames. Processor 104 may further process the image data to apply effects to image frame 230. The effects may include Bokeh, lighting, color cast, and / or high dynamic range (HDR) merging. In some embodiments, the functionality may be embedded in different components, such as ISP 112, DSP, ASIC, or other custom logic circuitry for performing additional image processing.

[0065] Figure 3FIG. 0 is a block diagram of a system 300 according to an exemplary embodiment of the present disclosure. The system 300 includes an image frame 302, an image frame 304, an ISP 112, and an output image frame 332. The ISP 112 may be configured to receive one or more image frames 302, 304 and generate a corrected image frame 320 based on the image frames 302, 304. Specifically, the ISP 112 may be configured to generate the corrected image frame 320 to correct motion distortion and / or motion artifacts located within the first image frame 302. The image frames 302, 304 may be received as image data from image sensors (such as image sensors 101, 102). In some cases, the ISP 112 may be configured to receive and sequentially process the image frames (e.g., as part of an image processing pipeline and / or a video processing pipeline). For example, the ISP 112 may initially receive the image frame 304 (e.g., after being captured by the image sensors 101, 102) and may subsequently receive the image frame 302 (e.g., after being subsequently captured by the image sensors 101, 102). In some cases, the image frame 302 may be regarded as the current image frame or the target image frame of the image processing pipeline or the video processing pipeline, and the image frame 304 may be regarded as the reference frame of the image processing pipeline. In additional or alternative embodiments, the ISP 112 may receive multiple image frames including the image frames 302, 304. For example, the multiple image frames may be pre-stored and retrieved by the ISP 112 for further processing.

[0066] To correct motion distortion, the ISP may be configured to apply a temporal filtering process 314 to the image frame 302 to generate the corrected image frame 320. The temporal filtering process 314 may mix multiple versions of the image frame 302 to generate the corrected image frame 320 that corrects the motion distortion within the image frame 302. Specifically, the temporal filtering process 314 may be configured to mix the image frame 302, a motion-aligned frame 310, and a denoised frame 312 to generate the corrected image frame.

[0067] A denoised frame 312 can be generated to reduce noise (e.g., spatial noise, image noise, granularity) within the image frame 302. The noise within the image can be reduced according to one or more spatial denoising processes (such as MFNR). A motion-aligned frame 310 can be generated to remove or reduce movement within the image frame 302. For example, the motion-aligned frame 310 can be generated to align the image frame 302 with a previously captured image frame (e.g., the image frame 304). The motion-aligned frame 310 can be configured by applying a motion alignment process to the image frame. For example, a block-by-block process can be used to find the motion vectors between corresponding blocks (such as 8×8 pixel blocks) of the image frames 302, 304. The corresponding blocks can be identified by a semi-global matching (SGM) process to identify the most similar blocks in the image frames 302, 304. Then, the pixels within the image frame 302 can be warped based on the motion vectors to form the motion-aligned frame 310.

[0068] The motion alignment process can be applied according to the motion map 306. The ISP can determine that the motion map 306 can be generated to reflect movement within the image frame 302 relative to the previous image frame 304. In some embodiments, the motion map 306 can be generated by comparing a first image frame with a second image frame to generate the motion map 306. For example, the motion map 306 can be calculated based on the difference between the second image frame 304 and the first image frame 302. In additional or alternative embodiments, the motion map 306 can be calculated based on sensor data (e.g., motion sensor data) from the device 100 (such as an image capture device) that captured the first image frame 302 and the second image frame 304. For example, the ISP 112 can be configured to calculate one or more global motion estimates that reflect the movement of the image capture device and one or more local motion estimates that reflect the movement of one or more objects depicted within the image frames 302, 304. In some aspects, the global motion estimate can be calculated based on sensor data (such as gyroscope or accelerometer data indicating the movement of the image capture device). The global motion estimate can include one or both of the magnitude and direction of the movement of the image capture device. The local motion estimate can be captured by comparing the first image frame 302 with the second image frame 304. For example, the local motion estimate can be determined by comparing the positions of one or more objects within each of the first image frame 302 and the second image frame 304. The local motion estimate can be calculated as the difference between the image frames 302, 304. As another example, the local motion estimate can be calculated based on texture processing using Harris corner detection and related techniques. Then the local motion estimate and / or the global motion estimate can be combined to generate the motion map 306.

[0069] In some aspects, the motion map 306 can be calculated by comparing a first image frame 302 with a second image frame 304 to generate an estimate of the motion vectors between the image frames 302, 304. Then the motion vectors and sensor data (e.g., motion sensor data) can be analyzed together to determine the alignment of the image capture device (e.g., to separate the global motion of the image capture device from the local motion of the objects depicted within the image frames 302, 304). Then, the ISP 112 can perform a matching process based on the alignment and the image frames 302, 304 to generate the motion map 306. In certain embodiments, the matching process can be performed as a semi-global matching (SGM) process.

[0070] The content of the motion map 306 can correspond to a specific portion of the image frame 302. For example, each pixel of the image frame 302 can have a corresponding entry in the motion map 306. As another example, each entry in the motion map 306 can correspond to multiple pixels within the image frame 302 (e.g., 4 pixels, 9 pixels, 16 pixels, or more). The content of the motion map 306 can indicate the movement within the corresponding portion of the image frame 302. For example, the entry in the motion map 306 can indicate the magnitude of the movement of the object or feature depicted within the corresponding portion of the image frame 302 (e.g., relative to the image frame 304). In various embodiments, the magnitude of the movement can be indicated as one or more of the number of pixels moved, the distance moved, the speed of movement, etc. In certain embodiments, the movement may only be indicated within the motion map 306 if the movement exceeds a predetermined threshold. For example, a modified motion map can be calculated as:

[0071] where

[0072] M OFFSET = MotionWeight * 256,

[0073] where:

[0074] M MOD (n x ,n y ) is the modified motion map value of the pixel (n x ,n y ) of the first image frame 302,

[0075] M(n x ,n y ) is the original motion map value of the pixel (n x ,n y ) of the first image frame 302 (e.g., calculated according to the above techniques), and

[0076] MotionWeight ∈ [0, 1] is a tunable parameter.

[0077] ISP 112 may also be configured to generate a gain map 308 based on the image frame 302. In some embodiments, the gain map 308 may be generated based on the luminance values within the image frame 302. In some embodiments, the gain map 308 may be generated to correct for low luminance values within the first. For example, the gain map 308 may include a gain increase (e.g., an increase in the luminance of the image frame 302) on a per-pixel basis within the image frame 302 to correct for portions of the image frame that have a lower luminance relative to other portions within the image frame 302 (e.g., neighboring pixels, neighboring regions).

[0078] The content of the gain map 308 may correspond to a specific portion of the image frame 302. For example, each pixel of the image frame 302 may have a corresponding entry in the gain map 308. As another example, each entry in the gain map 308 may correspond to multiple pixels within the image frame 302 (e.g., 4 pixels, 9 pixels, 16 pixels, or more). The content of the gain map 308 may indicate the luminance increase to be applied to the corresponding portion of the image frame 302. For example, an entry in the gain map 308 may indicate an increase in the pixel value or magnitude of the corresponding portion of the image frame 302.

[0079] The ISP may be configured to determine at least one parameter for the temporal filtering process 314 based on at least one of the gain map 308 and the motion map 306. The temporal filtering process 314 includes a dynamic motion blend intensity 316 and a dynamic spatial blend intensity 318. The dynamic motion blend intensity 316 may be applied to the motion-aligned frame 310 during the temporal filtering process 314. Specifically, when blending the frames 302, 310, 312, a higher value of the dynamic motion blend intensity 316 may result in a stronger blend of the motion-aligned frame 310. The ISP 112 may calculate the dynamic motion blend intensity 316 based on the motion map 306. For example, the dynamic motion blend intensity 316 may be calculated on a per-pixel basis for each pixel within the image frame 302 based on the corresponding value within the motion map 306. In some embodiments, the dynamic motion blend intensity 316 increases for higher values within the motion map 306. As a specific example, the dynamic motion blend intensity 316 may be calculated as:

[0080]

[0081] where:

[0082] α Motion (n x ,n y ) is the dynamic motion blend intensity value of the pixel (n x ,n y ) of the first image frame 302,

[0083] α1 is a predetermined static motion mixing value (such as a value from 0.5 to 1 in various specific embodiments), and

[0084] M(n x ,n u ) is the original motion map value of the pixel (n x ,n y ) of the first image frame 302 (the M MOD (n x ,n y ) can be used in some specific embodiments).

[0085] In some specific embodiments, the value of α1 can be determined based on the motion map 306. For example, when the value of the corresponding part of the motion map 306 is less than a predetermined threshold, the value of α1 can be set to 1 to prevent motion mixing from being applied in the corresponding area of the image frame 302 because motion correction is not required in this area.

[0086] The dynamic spatial mixing intensity 318 can be applied to the denoised frame 312 during the temporal filtering process 314. Specifically, when mixing the frames 302, 310, 312, a higher value of the dynamic spatial mixing intensity 318 can result in stronger mixing of the denoised frame 312. The ISP 112 can calculate the dynamic spatial mixing intensity 318 based on the gain map 308. For example, the spatial motion mixing intensity 318 can be calculated on a per-pixel basis for each pixel in the image frame 302 based on the corresponding value within the gain map 308. In some specific embodiments, the spatial motion mixing intensity 318 increases for higher values within the gain map 308. As a specific example, the dynamic spatial mixing intensity 318 can be calculated as:

[0087]

[0088] where:

[0089] α Spatial (n x ,n y ) is the dynamic spatial mixing intensity value of the pixel (n x ,n y ) of the first image frame 302,

[0090] α2 is a predetermined static spatial mixing value (such as a value from 0 to 1), and

[0091] G(n x ,n y ) is the gain map value of the pixel (n x ,n y ) of the first image frame 302.

[0092] In some specific implementations, the value of α2 can be determined based on motion map 306. For example, when the value of the corresponding part of motion map 306 exceeds a predetermined threshold, the value of α2 can be set to 1 to apply spatial blending in the corresponding area of image frame 302.

[0093] ISP 112 can be configured to determine the corrected image frame 320 by temporally filtering the image frame 302 according to the temporal filtering process 314 and at least one of the dynamic motion blending intensity 316 and the dynamic spatial blending intensity 318. The temporal filtering can be applied on a per-pixel basis. For example, ISP 112 can iterate over each pixel of the image frame 302 and generate the corresponding corrected pixel in the corrected image frame 320 based on the corresponding values of the motion-aligned frame 310, the denoised frame 312, the dynamic motion blending intensity, and the dynamic spatial blending intensity. As a specific example, the output pixel value of the corrected image frame 320 can be calculated as:

[0094] I Corrected (n x ,n y ) = α Motion *I Aligned (n x ,n y ) + (1 - α motion )[α spatial I Denoised (n x ,n y ) + (1 - α Spatial )I Orig (n x ,n y )],

[0095] where:

[0096] I Corrected (n x ,n y ) is the pixel value of the pixel (n x ,n y ) in the corrected image frame 320,

[0097] I Original (n x ,n y ) is the pixel value of the pixel (n x ,n y ) in the first image frame 302,

[0098] I Aligned (n x ,n y ) is the pixel value of the pixel (n x ,n y ) in the aligned frame 310,

[0099] I Denoised (n x ,n y ) is the pixel value of the pixel (n x ,n y ) of the denoised frame 312,

[0100] α Spatial (n x ,n y ) is the dynamic spatial mixing intensity value of the pixel (n x ,n y ) of the first image frame 302, and

[0101] α Motion (n x ,n y ) is the dynamic motion mixing intensity of the pixel (n x ,n y ) of the first image frame 302.

[0102] After iterating over all pixels according to the temporal filtering process 314, the corrected image frame 320 can be completed. Once generated, the ISP 112 (or another component of the device 100) can add the corrected image frame 320 to the output image frame 332 (e.g., the output image frame of a video and / or composite image). In some embodiments, the corrected image frame 320 can also be used as a reference image frame for correcting future image frames. For example, the second image frame 304 can be an image frame previously generated by the ISP 112 according to the temporal filtering process 314.

[0103] In various embodiments, the dynamic motion mixing intensity 316 and / or the dynamic spatial mixing intensity 318 may not be calculated separately for the image frame 302. For example, in some embodiments, only the dynamic motion mixing intensity 316 may be calculated, and a predetermined spatial mixing intensity (e.g., a default α2 value) may be calculated and used in place of the dynamic spatial mixing intensity 318. As another example, only the dynamic spatial mixing intensity 318 may be calculated, and a predetermined motion mixing intensity (e.g., a default α1 value) may be used in place of the dynamic motion mixing intensity 316.

[0104] Figure 2 The system 200 and / or Figure 3 The system 300 may be configured to perform the operations described in reference Figure 4 to determine the output image frames 230, 330. Figure 3 FIG. shows a flowchart of an example method for processing image data to correct motion distortion according to some embodiments of the present disclosure. Figure 3 The capture in can obtain an improved digital representation of the scene, thereby producing a photo or video with higher image quality (IQ).

[0105] At block 402, image data including a first image frame is received. For example, ISP 112 may receive image data including first image frame 302. The image data may be received from image sensors 101, 102, such as when the image sensors are configured with the camera configuration. The image data may be received at ISP 112, processed by the image front end (IFE) and / or image post-processing engine (IPE) of ISP 112, and stored in memory. In some embodiments, the capture of the image data may be initiated by a camera application executing on processor 104, which causes camera control 210 to activate camera 103 to capture the image data and causes the image data to be provided to a processor, such as processor 104 or ISP 112.

[0106] At block 404, at least one of (i) a gain map and (ii) a motion map is determined. For example, ISP 112 may determine at least one of (i) gain map 308 of first image frame 302 and / or (ii) motion map 306 of first image frame 302. For example, gain map 308 may be determined based on luminance values within first image frame 302. As another example, motion map 306 may be determined by comparing first image frame 302 with a second image frame 304 (e.g., a previously captured image frame) from the image data.

[0107] At block 406, at least one parameter is determined for a temporal filtering process. For example, ISP 112 may determine at least one parameter for temporal filtering process 314 based on at least one of gain map 308 and motion map 306. The at least one parameter may include a dynamic motion blend intensity 316 generated based on motion map 306 and / or a dynamic spatial blend intensity 318 generated based on gain map 308. In certain embodiments, the at least one parameter may be generated on a per-pixel basis for image frame 302.

[0108] At block 408, the corrected image frame is determined by temporally filtering the first image frame based on at least one parameter. For example, ISP 112 may determine the corrected image frame 320 by applying the temporal filtering process 314 to the first image frame 302 according to at least one parameter. In a particular implementation where at least one parameter includes the dynamic motion blend intensity 316, the dynamic motion blend intensity 316 may be applied to the aligned frame 310 during the temporal filtering process 314. The motion-aligned image frame 310 may be generated based on the motion map 306 and the first image frame 302 to align the position of an object moving within the first image frame 302 with the position of the object in the previous image frame 304. In a particular implementation where at least one parameter includes the dynamic spatial blend intensity 318, the dynamic spatial blend intensity 318 may be applied to the denoised frame 312 during the temporal filtering process 314. The denoised frame 312 may be generated to correct the spatial noise within the image frame 302. In some particular implementations, the corrected image frame 320 may be determined by temporally filtering the first image frame 302 on a per-pixel basis (e.g., by applying the temporal filtering process 314 to each pixel of the image frame 302 individually).

[0109] Then, the output image frames 230, 330 may be determined based on the corrected image frame 320. The image frame 230 may be determined by the processor 104 or ISP 112 and stored in the memory 106. The stored image frame may be read by the processor 104 and used to form a preview display on the display of the device 100 and / or processed to form a photo for storage in the memory 106 and / or sent to another device.

[0110] Figure 5 is a block diagram illustrating an example processor configuration for image data processing in an image capture device according to one or more embodiments of the present disclosure. The processor 104 or other processing circuitry may be configured to operate on the image data to perform Figure 4 one or more operations of the method. The image data may be processed to determine one or more output image frames 510. In Figure 5 the processor 104 implements an image map generator 502, a parameter generator 504, and a corrected image generator 506.

[0111] The processor 104 is configured to receive image data. The first image data may represent a first image frame. The image data may be captured by an image sensor. The image map generator 502 may be configured to determine at least one of a gain map of the first image frame and a motion map of the first image frame. For example, the image map generator 502 may compare the first image frame with a second image frame to calculate the motion map of the first image frame. Specifically, the motion map may be calculated to indicate the movement of one or more objects within the first image frame relative to the second image frame and / or the movement of the device 100 that captured the first image frame and the second image frame. As another example, the image map generator 502 may determine the gain map based on the luminance values within the first image frame. Specifically, the image map generator 502 may generate gain values to correct the luminance differences in portions of the first image frame and may add the gain values to the corresponding portions of the gain map.

[0112] The parameter generator 504 may determine one or more parameters for a temporal filtering process. For example, the parameter generator 504 may determine a dynamic motion blending intensity for the temporal filtering process based on the motion map. Additionally or alternatively, the parameter generator 504 may determine a dynamic spatial blending intensity for the temporal filtering process based on the gain map.

[0113] The corrected image generator 506 may be configured to determine a corrected image frame of the first image frame by applying a temporal filtering process according to one or more parameters determined by the parameter generator 504. For example, the dynamic motion blending intensity may change the intensity at which the aligned frames are blended into the corrected image frame. As another example, the dynamic spatial blending intensity may change the intensity at which the denoised frames are blended into the corrected image. In some cases, the corrected image generator 506 may apply the temporal filtering process to the first image frame on a per-pixel basis. Then, the corrected image frame may be added to the output image frame 510.

[0114] In one or more aspects, techniques for supporting image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in connection with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image processing may include an apparatus configured to: receive image data including a first image frame; determine at least one of a gain map of the first image frame or a motion map of the first image frame; determine at least one parameter for a temporal filtering process based on the at least one of the gain map and the motion map; and determine a corrected image frame by temporally filtering the first image frame based on the at least one parameter.

[0115] Additionally, the apparatus may perform or operate according to one or more aspects as described below. In some specific implementations, the apparatus includes a wireless device, such as a UE. In some specific implementations, the apparatus includes a remote server (such as a cloud-based computing solution) that receives image data for processing to determine an output image frame. In some specific implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein for the apparatus. In some other specific implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, and the program code may be executable by a computer to cause the computer to perform the operations described herein with reference to the apparatus. In some specific implementations, the apparatus may include one or more components configured to perform the operations described herein. In some specific implementations, a method of wireless communication may include one or more operations described herein with reference to the apparatus.

[0116] In a second aspect according to the first aspect, determining at least one of the gain map or the motion map includes determining the motion map, wherein receiving the image data further includes a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

[0117] In a third aspect according to the second aspect, determining the motion map includes determining a position difference of at least a portion of the first image frame relative to the second image frame.

[0118] In a fourth aspect according to the third aspect, determining the motion map includes determining that non-zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

[0119] In a fifth aspect according to at least one of the second to fourth aspects, the instructions further cause the processor to determine an alignment frame based on the motion map and the first image frame, wherein the alignment frame corrects movement distortion within the first image frame.

[0120] In a sixth aspect according to the fifth aspect, the at least one parameter includes a dynamic motion blending intensity based on the motion map, and wherein during the temporal filtering process, the dynamic motion blending intensity is applied to the alignment frame relative to the first image frame.

[0121] In a seventh aspect according to any of the first to sixth aspects, the instructions further cause the processor to determine a denoised frame that removes spatial noise from the first image frame according to a spatial denoising process.

[0122] In an eighth aspect according to the seventh aspect, the at least one parameter includes a dynamic spatial mixing intensity based on the gain map, and wherein during the temporal filtering process, the dynamic spatial mixing intensity is applied to the denoised frame relative to the first image frame.

[0123] In a ninth aspect according to any one of the first to eighth aspects, determining the gain map is based on luminance values within the first image frame.

[0124] In a tenth aspect according to any one of the first to ninth aspects, the at least one parameter is determined on a per-pixel basis for the first image frame, and wherein determining the corrected image frame includes: temporally filtering the first image frame on a per-pixel basis.

[0125] In an eleventh aspect, supporting image processing may include a method that includes: receiving image data including a first image frame; determining at least one of a gain map of the first image frame or a motion map of the first image frame; determining at least one parameter for a temporal filtering process based on the at least one of the gain map and the motion map; and determining a corrected image frame by temporally filtering the first image frame based on the at least one parameter.

[0126] In a twelfth aspect according to the eleventh aspect, determining at least one of the gain map or the motion map includes determining the motion map, wherein receiving the image data further includes a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

[0127] In a thirteenth aspect according to the twelfth aspect, determining the motion map includes determining a position difference of at least a portion of the first image frame relative to the second image frame.

[0128] In a fourteenth aspect according to the thirteenth aspect, determining the motion map includes determining that non-zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

[0129] In a fifteenth aspect according to any one of the twelfth to fourteenth aspects, the instructions further cause the processor to determine an aligned frame based on the motion map and the first image frame, wherein the aligned frame corrects movement distortion within the first image frame.

[0130] In a sixteenth aspect according to the fifteenth aspect, the at least one parameter includes a dynamic motion mixing intensity based on the motion map, and wherein during the temporal filtering process, the dynamic motion mixing intensity is applied to the aligned frame relative to the first image frame.

[0131] In a seventeenth aspect according to any one of the eleventh to sixteenth aspects, the instruction further causes the processor to determine a denoised frame for removing spatial noise from the first image frame according to a spatial denoising process.

[0132] In an eighteenth aspect according to the seventeenth aspect, the at least one parameter includes a dynamic spatial mixing intensity based on the gain map, and during the temporal filtering process, the dynamic spatial mixing intensity is applied to the denoised frame with respect to the first image frame.

[0133] In a nineteenth aspect according to any one of the eleventh to eighteenth aspects, determining the gain map is based on luminance values within the first image frame.

[0134] In a twentieth aspect according to any one of the eleventh to nineteenth aspects, the at least one parameter is determined on a per-pixel basis for the first image frame, and determining the corrected image frame includes: temporally filtering the first image frame on a per-pixel basis.

[0135] In a twenty-first aspect, supporting image processing includes a method that includes: receiving image data including a first image frame; determining a motion map of the first image frame, the motion map identifying movement within the first image frame; determining an aligned frame based on the motion map and the first image frame, where the aligned frame corrects movement distortion within the first image frame; determining a dynamic motion mixing intensity based on the motion map; and determining a corrected image frame by temporally filtering the first image frame and the aligned frame based on the dynamic motion mixing intensity.

[0136] In a twenty-second aspect according to the twenty-first aspect, the image data further includes a second image frame, and determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

[0137] In a twenty-third aspect according to the twenty-second aspect, the motion map is generated to reflect movement within the first image frame relative to the second image frame.

[0138] In a twenty-fourth aspect according to the twenty-third aspect, determining the motion map includes determining that non-zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

[0139] In a twenty-fifth aspect according to any one of the twenty-first to twenty-fourth aspects, the method further includes determining a gain map of the first image frame; determining a denoised frame obtained by removing spatial noise from the first image frame according to a spatial denoising process; and determining a dynamic spatial mixing intensity based on the gain map, wherein determining the corrected image frame further includes performing temporal filtering on the first image frame and the denoised frame based on the dynamic spatial mixing intensity.

[0140] In a twenty-sixth aspect, an apparatus for supporting image processing includes a memory storing processor-readable code; and at least one processor coupled to the memory. The at least one processor may be configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving image data including a first image frame; determining a motion map of the first image frame, the motion map identifying movement within the first image frame; determining an aligned frame based on the motion map and the first image frame, wherein the aligned frame corrects movement distortion within the first image frame; determining a dynamic motion mixing intensity based on the motion map; and determining a corrected image frame by performing temporal filtering on the first image frame and the aligned frame based on the dynamic motion mixing intensity.

[0141] In a twenty-seventh aspect according to the twenty-sixth aspect, the image data further includes a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

[0142] In a twenty-eighth aspect according to the twenty-seventh aspect, the motion map is generated to reflect movement within the first image frame relative to the second image frame.

[0143] In a twenty-ninth aspect according to the twenty-eighth aspect, determining the motion map includes determining that non-zero values of a part of the motion map corresponding to the movement between the first image frame and the second image frame exceed a predetermined threshold.

[0144] In a thirtieth aspect according to the twenty-sixth to twenty-ninth aspects, the operations further include: determining a gain map of the first image frame; determining a denoised frame obtained by removing spatial noise from the first image frame according to a spatial denoising process; and determining a dynamic spatial mixing intensity based on the gain map, and wherein determining the corrected image frame further includes performing temporal filtering on the first image frame and the denoised frame based on the dynamic spatial mixing intensity.

[0145] Those skilled in the art should understand that any of a variety of different technologies and techniques can be used to represent information and signals. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may have been mentioned throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0146] As used herein Figures 1 to 5 the described components, functional blocks, and modules include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software code, firmware code, etc., or any combination thereof. Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, processes, and / or functions, etc., regardless of whether it is referred to as software, firmware, middleware, microcode, hardware description language, or other terms. Additionally, the features discussed herein can be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

[0147] Those skilled in the art should understand that: with reference to Figure 4 and Figure 5 one or more of the boxes (or operations) described can be combined with one or more of the boxes (or operations) described in another figure of the reference drawings. For example, Figure 4 one or more of the boxes (or operations) of Figures 1 to 3 can be combined with one or more of the boxes (or operations) of Figure 5 . As another example, one or more of the boxes associated with Figures 1 to 3 can be combined with one or more of the boxes (or operations) associated with

[0148] Those of ordinary skill in the art should also recognize that the various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each particular application, but such specific implementation decisions should not be construed as causing a departure from the scope of the disclosure. Those skilled in the art will also readily recognize that the order or combination of the components, methods, or interactions described herein are merely examples, and the components, methods, or interactions of the various aspects of the disclosure can be combined or performed in ways other than those illustrated and described herein.

[0149] The various illustrative logical components, logical blocks, modules, circuits, and algorithmic processes described in connection with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.

[0150] The hardware and data processing apparatus for implementing or performing the various illustrative logical components, logical blocks, modules, and circuits described in connection with the aspects disclosed herein can be realized or executed using a general-purpose single-chip or multi-chip processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic components, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some specific implementations, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuitry specific to a given function.

[0151] In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and structural equivalents thereof, or any combination thereof. The specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, a data processing apparatus.

[0152] If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The processes of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. A computer-readable medium includes both a computer storage medium and a communication medium including any medium that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable medium may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection may be properly termed a computer-readable medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, operations of a method or algorithm may reside as one set of code and instructions or any combination of sets of code and instructions on a machine-readable medium and a computer-readable medium, which may be incorporated into a computer program product.

[0153] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the specific implementations shown herein but are to be accorded the broadest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0154] Additionally, those of ordinary skill in the art will readily recognize that, for ease of description of the figures, terms such as "upper" and "lower" or "front" and "rear" or "top" and "bottom" or "forward" and "backward" may sometimes be used, and indicate relative positions corresponding to the orientation of the figure on the correctly oriented page, and may not reflect the correct orientation of any device as implemented.

[0155] Certain features that are described in the context of separate embodiments in this specification can also be implemented in combination in a single embodiment. Conversely, the various features that are described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. Additionally, although features may have been described above as acting in certain combinations and even initially claimed as such, one or more features from the claimed combination can in some cases be excluded from the combination, and the claimed combination can be directed to a sub-combination or variation of a sub-combination.

[0156] Similarly, although operations are depicted in the figures in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed to achieve the desired result. Additionally, the figures may schematically depict one or more example processes in the form of a flowchart. However, other operations not depicted can be incorporated into the example processes that are schematically illustrated. For example, one or more additional operations can be performed before, after, simultaneously with, or between any of the illustrated operations. In certain environments, multitasking and parallel processing are advantageous. Further, the separation of the various system components in the specific embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific embodiments also fall within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired result.

[0157] As used herein (including in the claims), the term "or" as used in a list of two or more items means that any one of the listed items can be employed alone, or any combination of two or more of the listed items can be employed. For example, if a composition is described as containing components A, B, or C, the composition can contain A alone; B alone; C alone; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C. Additionally, as used herein (including in the claims), "or" as used in a list of items beginning with "at least one of" indicates a disjunctive list, such that a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these items.

[0158] The term "substantially" is defined as being largely but not necessarily wholly that which is specified (and includes that which is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by a person of ordinary skill in the art. In any particular implementation disclosed, the term "substantially" may be replaced by "[percentage] within" that which is specified, where the percentage includes 0.1%, 1%, 5% or 10%.

[0159] The foregoing description of the disclosure has been provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, the method comprising: Receiving image data including a first image frame; Determining at least one of a gain map of the first image frame or a motion map of the first image frame; Determining at least one parameter for a temporal filtering process based on the at least one of the gain map and the motion map; And Determining a corrected image frame by temporally filtering the first image frame based on the at least one parameter.

2. The method according to claim 1, wherein determining the at least one of the gain map or the motion map includes determining the motion map, wherein receiving the image data further includes receiving a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

3. The method according to claim 2, wherein determining the motion map includes determining a position difference of at least a portion of the first image frame relative to the second image frame.

4. The method according to claim 3, wherein determining the motion map includes determining that non-zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

5. The method according to claim 2, the method further comprising determining an aligned frame based on the motion map and the first image frame, wherein the aligned frame corrects movement distortion within the first image frame.

6. The method according to claim 5, wherein the at least one parameter includes a dynamic motion mixing intensity based on the motion map, and wherein during the temporal filtering process, the dynamic motion mixing intensity is applied to the aligned frame relative to the first image frame.

7. The method according to claim 1, the method further comprising determining a denoised frame for removing spatial noise from the first image frame according to a spatial denoising process.

8. The method according to claim 7, wherein the at least one parameter includes a dynamic spatial mixing intensity based on the gain map, and wherein during the temporal filtering process, the dynamic spatial mixing intensity is applied to the denoised frame relative to the first image frame.

9. The method according to claim 1, wherein determining the gain map is based on luminance values within the first image frame.

10. The method according to claim 1, wherein the at least one parameter is determined on a per-pixel basis for the first image frame, and wherein determining the corrected image frame comprises: Temporally filtering the first image frame on a per-pixel basis.

11. An apparatus, the apparatus comprising: A processor; And A memory storing instructions that, when executed by the processor, cause the processor to: Receive image data including a first image frame; Determine at least one of a gain map of the first image frame or a motion map of the first image frame; Determine at least one parameter for a temporal filtering process based on the at least one of the gain map and the motion map; And Determine a corrected image frame by temporally filtering the first image frame based on the at least one parameter.

12. The apparatus according to claim 11, wherein determining at least one of the gain map or the motion map includes determining the motion map, wherein receiving the image data further includes a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

13. The apparatus according to claim 12, wherein determining the motion map includes determining a positional difference of at least a portion of the first image frame relative to the second image frame.

14. The apparatus according to claim 13, wherein determining the motion map includes determining that non - zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

15. The apparatus according to claim 12, wherein the instructions further cause the processor to determine an alignment frame based on the motion map and the first image frame, wherein the alignment frame corrects movement distortion within the first image frame.

16. The apparatus according to claim 15, wherein the at least one parameter includes a dynamic motion blending intensity based on the motion map, and wherein during the temporal filtering process, the dynamic motion blending intensity is applied to the alignment frame relative to the first image frame.

17. The apparatus according to claim 11, wherein the instructions further cause the processor to determine a denoised frame that removes spatial noise from the first image frame according to a spatial denoising process.

18. The apparatus according to claim 17, wherein the at least one parameter includes a dynamic spatial blending intensity based on the gain map, and wherein during the temporal filtering process, the dynamic spatial blending intensity is applied to the denoised frame relative to the first image frame.

19. The apparatus according to claim 11, wherein determining the gain map is based on luminance values within the first image frame.

20. The apparatus according to claim 11, wherein the at least one parameter is determined on a per-pixel basis for the first image frame, and wherein determining the corrected image frame comprises: Perform temporal filtering on the first image frame on a per - pixel basis.

21. A method, the method comprising: Receiving image data including a first image frame; Determining a motion map of the first image frame, the motion map identifying movement within the first image frame; Determining an alignment frame based on the motion map and the first image frame, wherein the alignment frame corrects movement distortion within the first image frame; Determining a dynamic motion blending intensity based on the motion map; And Determining a corrected image frame by temporally filtering the first image frame and the alignment frame based on the dynamic motion blending intensity.

22. The method according to claim 21, wherein the image data further includes a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

23. The method according to claim 22, wherein the motion map is generated to reflect movement within the first image frame relative to the second image frame.

24. The method according to claim 23, wherein determining the motion map includes determining that non - zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

25. The method according to claim 21, the method further comprising: Determining a gain map of the first image frame; Determining a denoised frame that removes spatial noise from the first image frame according to a spatial denoising process; And Determining a dynamic spatial mixing intensity based on the gain map, where determining the corrected image frame further includes performing temporal filtering on the first image frame and the denoised frame based on the dynamic spatial mixing intensity.

26. An apparatus, the apparatus comprising: A memory that stores processor-readable code; And At least one processor coupled to the memory, the at least one processor being configured to execute the processor-readable code to cause the at least one processor to perform operations, the operations including: Receiving image data including a first image frame; Determining a motion map of the first image frame, the motion map identifying movement within the first image frame; Determining an aligned frame based on the motion map and the first image frame, wherein the aligned frame corrects movement distortion within the first image frame; Determining a dynamic motion mixing intensity based on the motion map; and Determining a corrected image frame by performing temporal filtering on the first image frame and the aligned frame based on the dynamic motion mixing intensity.

27. The apparatus according to claim 26, wherein the image data further includes a second image frame, and wherein determining the motion map includes comparing the first image frame with the second image frame to generate the motion map.

28. The apparatus according to claim 27, wherein the motion map is generated to reflect movement within the first image frame relative to the second image frame.

29. The apparatus according to claim 28, wherein determining the motion map includes determining that non-zero values of a portion of the motion map corresponding to movement between the first image frame and the second image frame exceed a predetermined threshold.

30. The apparatus according to claim 26, wherein the operations further include: Determining a gain map of the first image frame; Determining a denoised frame that removes spatial noise from the first image frame according to a spatial denoising process; And Determining a dynamic spatial mixing intensity based on the gain map, and where determining the corrected image frame further includes performing temporal filtering on the first image frame and the denoised frame based on the dynamic spatial mixing intensity.

Images