Temporal filtering restart for improved scene integrity

Abstract

The temporal filtering operation may be reset for some pixels in an image frame to reduce the contribution from the previous input frame to reduce ghosting and other artifacts. The reset reduces the contribution, for example, to zero, immediately or within a predetermined time period (e.g., a certain number of frames). The decision on whether to reset the temporal filtering for a pixel of an image frame may be based on a probability assigned to that pixel. The probability may be based on a rule having one or more criteria. One exemplary factor for adjusting the probability is a confidence level for the temporal filtering decision for a pixel, in which the probability for a random reset of the pixel is based on the confidence level for the temporal filtering decision for those pixels.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. patent application Ser. No. 17 / 353,506, filed Jun. 21, 2021, entitled “TEMPORAL FILTERING RESTART FOR IMPROVED SCENE INTEGRITY,” which is expressly incorporated by reference herein in its entirety.

[0002] Aspects of the present disclosure relate generally to image processing. Several features of the present disclosure can enable and provide improved processing of images in computational photography. [Background technology]

[0003] Image capture devices, i.e., devices capable of capturing one or more digital images, whether still image photographs or a sequence of images for a video, may be incorporated into a wide variety of devices. By way of example, image capture devices may include standalone digital cameras or digital video camcorders, mobile phones, cellular or satellite radio telephones, personal digital assistants (PDAs), panels or tablets, camera-equipped wireless communication device handsets such as gaming devices, computing devices such as webcams, video surveillance cameras, or other devices with digital imaging or video capabilities.

[0004] Regardless of its form, image capture devices have some inherent limitations in the image quality of captured images. In some situations, a temporal filter may be applied to an image frame captured by an image capture device to improve image quality, especially in low light scenes. The temporal filter modifies at least one pixel value in an image frame based at least in part on a value for the pixel in a previous frame. The temporal filter uses information from two image frames captured at different times to obtain an output image frame that may be of higher quality than a single image frame. However, there are problems associated with temporal filtering that may reduce the resulting image quality. Summary of the Invention [Problem to be solved by the invention]

[0005] Temporal filtering can produce artifacts in the filtered image under some circumstances. For example, an object that moves or disappears from the scene may remain in the temporally filtered image output for a long time because the temporally filtered image is based on a previous image frame that contained the object. These residual objects in the image may be called "ghosts" due to the object appearing in the image even though it has left the scene. Ghosting may also appear under other circumstances, such as when temporally filtering low-contrast (e.g., signal-to-noise ratio (SNR) less than 1) images. The effects of ghosting may be more noticeable when more aggressive temporal filtering is applied. Any of these temporal filtering artifacts result in scene integrity problems in the output image frames. More aggressive filtering uses image frames from a longer time period, resulting in ghost objects appearing longer or more prominently. More aggressive temporal filtering may be useful in low-light situations where the associated longer time period can reduce noise in low-light scenes. [Means for solving the problem]

[0006] One technique for improving temporal filtering according to some embodiments of the present disclosure is to reset the temporal filtering for some pixels in an image frame to improve the image quality of the image generated by the temporal filtering process. Resetting the temporal filter can reduce the contribution from the previous image frame in the temporal filter so that the previous input frame has less influence on the output image frame. The resetting can reduce the contribution, for example, to zero, immediately or within a predetermined time period (e.g., a certain number of frames). The resetting can reduce the appearance of ghosting artifacts in the photograph. The decision on whether to reset the temporal filtering for a pixel of an image frame can be based on a probability assigned to that pixel. The probability can be based on a rule with one or more criteria. One exemplary factor for adjusting the probability is a confidence level for the temporal filtering decision for the pixel, in which the probability for random reset of the pixel is based on the confidence level for the temporal filtering decision for that pixel.

[0007] Temporal filtering may be applied to some pixels in an image frame and not to other pixels in the image frame, and the amount of temporal filtering applied to some pixels may be varied during the temporal filtering. The decision as to whether to apply temporal filtering to a particular pixel may be based on one or more rules, each having one or more criteria. The temporal filtering decision may be based on a determination of whether the pixel corresponds to a static or dynamic region of the image. A static region may be determined based on whether the temporal difference between the image frames is below a certain threshold, where the threshold may correspond to expected temporal noise in the image frames. A confidence level for the temporal filtering decision may be determined, and the confidence level may be used to determine whether to reset the pixel during the temporal filtering process.

[0008] The pixel reset rate in the temporal filter may be based on the confidence level of the temporal filtering decision for that pixel. For pixels with a high confidence level, the rate of pixel reset may be reduced or no reset may be performed. The rate of pixel reset may also be adjusted in proportion to the confidence level. For example, in regions where the confidence level for pixels in the region is above a high threshold, no pixel reset may be performed, in regions where the confidence level for pixels in the region is above a medium threshold, a low rate reset (e.g., once every 20 pixels) may be performed, and in regions where the confidence level is below the medium threshold, a normal rate reset (e.g., once every 10 pixels) may be performed. The confidence level of the temporal filtering decision may be based on one or more criteria. For example, a high confidence level may be associated with having a sufficiently high signal-to-noise ratio (SNR) in a region of the image. In another example, a high confidence level is associated with detecting the presence of a strong texture in a region of the image and at the same time, a low time difference between pixels in the image frame is input to the temporal filter. The pixel reset rate may be implemented using a reset weighting map indicating the probability of pixel reset and a random function modified to determine whether to reset or not reset individual pixels that would otherwise be determined to have temporal filtering performed.

[0009] The temporal filtering described herein may be implemented by processing two or more image frames directly on the output of an image sensor or on a stored sequence of image frames. In some embodiments, one image frame is a current image frame that refers to an image frame in the sequence of image frames currently being processed. In a real-time image capture device, the current image frame may be the most recent image frame captured by the image sensor. The temporal filtering may modify pixels of the current image frame based on one or more previous image frames occurring earlier in the sequence of image frames, such as an image frame captured earlier in time by the image sensor. In some embodiments, the temporal filter may be an infinite impulse response (IIR) filter, which processes the current image frame using a previous image frame that is itself temporally filtered. The IIR filter provides temporal filtering using lower bandwidth, lower computational resources, and less power. The IIR filter may therefore be beneficial in devices that operate from a limited power source, such as a battery, or in devices that operate with reduced processing capabilities due to device mobility factors.

[0010] In some embodiments, temporal filtering may be applied to image frames without motion compensation being applied prior to blending. In some embodiments, temporal filtering may be applied to image frames with motion compensation being applied prior to blending with other frames in the temporal filter. In embodiments where motion compensation is applied before temporal filtering, blending of image frames may be performed in static areas (e.g., areas where the change in pixels from one image frame to another is below a threshold amount).

[0011] This summary describes some aspects of the disclosure to provide a basic understanding of the described technology. This summary is not an extensive overview of all contemplated features of the disclosure, nor does it identify key or critical elements of all aspects of the disclosure, nor does it delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in summary form as a prelude to the more detailed description that is presented later.

[0012] Generally, this disclosure describes image processing techniques involving a digital camera having an image sensor and an image signal processor (ISP). The image signal processor may be configured to control the capture of image frames from one or more image sensors and process the image frames from the one or more image sensors to generate a view of a scene in a corrected image frame, where temporal filtering may be applied to blend two or more image frames captured from the one or more image sensors as part of the process to produce the corrected image frame. In one example, the image signal processor may receive instructions to capture a sequence of image frames in response to loading of software such as a camera application on a CPU. The image signal processor may be configured to generate a single flow of output frames based on each corrected image from the image sensor. The single flow of output frames may include image frames including image data from the image sensor, corrected by, for example, temporal filtering. The corrected image frames may result from combining the temporal filtering aspects of this disclosure with other computational photography techniques, such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR).

[0013] After the output frames representing the scene are determined by the image signal processor using the temporal filtering described in various embodiments herein, the view of the scene may be displayed on a device display, saved to a storage device as a picture or a sequence of pictures as a video, transmitted over a network, and / or printed on an output medium. For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from different image sensors and then generate corresponding output frames of image data (e.g., preview display frames, still image capture, frames of video, etc.). In other examples, the image signal processor may output frames of image data to various output devices and / or camera modules for further processing, such as 3A parameter synchronization (e.g., auto focus (AF), auto white balance (AWB), and auto exposure control (AEC)), generating a video file via the output frames, composing frames for display, composing frames for storage, transmitting frames over a network connection, etc. That is, the image signal processor may obtain incoming frames from one or more image sensors, each coupled to one or more camera lenses, and then generate a flow of output frames to output to various output destinations. In such an example, the image signal processor may be configured to produce a flow of output frames that may have reduced ghosting or other artifacts due to improved temporal filtering at pixel resets.

[0014] In one aspect of the disclosure, a method for image processing includes receiving a first image frame and a second image frame. The method may then determine a third image frame based on the first image frame and the second image frame by applying temporal filtering to combine the first image frame and the second image frame. The temporal filtering may include determining, for some pixels (e.g., the "first plurality of pixels"), a pixel of the third image frame as a combination of corresponding pixels of the first image frame and the second image frame. The temporal filtering may include resetting the temporal filter for other pixels (e.g., the "second plurality of pixels") by determining a pixel of the third image frame based on the first image frame and not based on the second image frame. The resetting may eliminate contributions from previous image frames to the third image frame, and instead, the pixel of the third image frame may be based only on the current image frame. This resetting of some pixels of the third image frame can reduce the presence of ghosting and other artifacts in the third image frame, thereby improving scene integrity and resulting in a more accurate representation of the scene, resulting in a higher quality photo or video for the user.

[0015] In some embodiments, the pixels of the third image frame that are reset are randomly selected for each frame. In some embodiments, the random pixels are weighted based on a confidence level associated with the temporal filtering (TF) decision for the pixel. In some embodiments, the temporal filtering may be an infinite impulse response (IIR) filter, where the second image frame input used for synthesis in the temporal filter is itself a temporally filtered image frame based on at least one image frame preceding the first and second image frames. In some embodiments, the temporal filtering may be a temporal filter that operates on more than two image frames, such as a temporal filter that synthesizes a first ("current") image frame with a first previous image frame and a second previous image frame. Resetting a temporal filter that synthesizes more than two input image frames may involve basing the output image frame on the current image frame and basing the output image frame on at least one of the previous image frames or all of the previous image frames.

[0016] In some embodiments, the method may be implemented for HDR photography, where the first and second image frames are captured using different exposure times, different apertures, different lenses, or other different characteristics that may improve the dynamic range of the fused image when the two image frames are combined. In some embodiments, the method may be implemented for MFNR photography, where the first and second image frames are captured using the same or different exposure times.

[0017] In an additional aspect of the present disclosure, an apparatus is disclosed that includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform any of the methods or techniques described herein. For example, the at least one processor may be configured to perform a step that includes receiving a first image frame and a second image frame. The at least one processor may also be configured to determine a third image frame based on the first image frame and the second image frame by applying temporal filtering to combine the first image frame and the second image frame. The temporal filtering may include, for some pixels (e.g., the "first plurality of pixels"), determining a pixel of the third image frame as a combination of corresponding pixels of the first image frame and the second image frame. The temporal filtering may include, for other pixels (e.g., the "second plurality of pixels"), resetting the temporal filter by determining a pixel of the third image frame based on the first image frame and not based on the second image frame. The reset may eliminate contributions from previous image frames to the third image frame, and instead the pixels of the third image frame may be based only on the current image frame. This reset of some pixels of the third image frame may reduce the presence of ghosting and other artifacts in the third image frame, thereby improving scene integrity and resulting in a more accurate representation of the scene, resulting in a higher quality photo or video for the user.

[0018] The at least one processor may include an image signal processor or a processor that includes specific functionality for camera control and / or processing, such as enabling or disabling temporal filtering and / or motion compensation. The at least one processor may also or alternatively include an application processor. The methods and techniques described herein may be implemented entirely by an image signal processor or an application processor, or various operations may be split between the image signal processor and the application processor, and in some aspects across additional processors.

[0019] The device may include one, two, or more image sensors, for example including a first image sensor. When multiple image sensors are present, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to acquire images through a first lens having a first optical axis, and the second sensor is configured to acquire images through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification and the second lens may have a second magnification different from the first magnification. This configuration may be done with a lens cluster on the mobile device, such as when multiple image sensors and associated lenses are located at offset locations on the front or back of the mobile device. Additional image sensors with larger, smaller, or same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any of the image sensors in a multi-sensor device.

[0020] In an additional aspect of the present disclosure, a device configured for image processing and / or image capture is disclosed. The device includes a means for capturing an image frame. The device further includes one or more means for capturing data representing a scene, such as an image sensor (including a charge-coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal-oxide semiconductor (CMOS) sensor), a time-of-flight detector, etc. The device may further include one or more means for accumulating and / or focusing a light beam onto one or more image sensors (including a simple lens, a compound lens, a spherical lens, and an aspherical lens). These components may be controlled to capture a first and / or a second image frame input to the image processing techniques described herein.

[0021] In an additional aspect of the disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including those described in the methods and techniques described herein. For example, the operations may include receiving a first image frame and a second image frame. The operations may also include determining a third image frame based on the first image frame and the second image frame by applying temporal filtering to combine the first image frame and the second image frame. The temporal filtering may include, for some pixels (e.g., the "first plurality of pixels"), determining a pixel of the third image frame as a combination of corresponding pixels of the first image frame and the second image frame. The temporal filtering may include, for other pixels (e.g., the "second plurality of pixels"), resetting the temporal filter by determining a pixel of the third image frame based on the first image frame and not based on the second image frame. The resetting may eliminate contributions from the previous image frame to the third image frame, and instead, the pixel of the third image frame may be based only on the current image frame. This resetting of some pixels of the third image frame can reduce the presence of ghosting and other artifacts in the third image frame, thereby improving scene integrity and resulting in a more accurate representation of the scene, resulting in a higher quality photo or video for the user.

[0022] Other aspects, features, and implementations will become apparent to those skilled in the art upon reviewing the following description of certain exemplary aspects together with the accompanying figures. Although features may be described in conjunction with certain aspects and figures below, various aspects may include one or more of the advantageous features described herein. In other words, although one or more aspects may be described as having certain advantageous features, one or more of such features may also be used in accordance with various aspects. Similarly, although exemplary aspects may be described below as device aspects, system aspects, or method aspects, the exemplary aspects may be implemented in various devices, systems, and methods.

[0023] The method may be embedded in a computer-readable medium as computer program code including instructions that cause a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device that includes a first network adapter configured to transmit data, such as an image or video, as a recording or as streaming data over a first network connection of a plurality of network connections, and a processor coupled to the first network adapter and a memory. The processor may cause transmission of the temporally filtered image frames described herein over a wireless communication network, such as a 5G NR communication network.

[0024] The foregoing has outlined rather broadly some of the features and technical advantages of embodiments of the present invention in order that the following Detailed Description may be better understood. Additional features and advantages of the invention that form the subject of the claims of the invention will be described below. Those skilled in the art will appreciate that the conception and specific embodiments disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same or similar purposes. Those skilled in the art will also realize that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. Additional features will be better understood from the following description when considered in conjunction with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended to limit the invention.

[0025] A further understanding of the nature and advantages of the present disclosure can be realized by reference to the following drawings. In the accompanying drawings, similar components or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. When only a first reference label is used herein, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram of a computing device configured to implement one or more of the example techniques described in embodiments of the present disclosure. [Diagram 2] 1 is a flowchart illustrating a method of temporally filtering image frames involving resetting the temporal filtering in accordance with some embodiments of the present disclosure. [Diagram 3]FIG. 1 is a block diagram illustrating an implementation of temporal filtering using an infinite impulse response (IIR) filter with filter resetting among some pixels in accordance with some embodiments of the present disclosure. [Figure 4] 1 is a block diagram illustrating an implementation of reset logic for an infinite impulse response (IIR) filter in accordance with some embodiments of the present disclosure. [Diagram 5] 1 is a flowchart illustrating a method for temporally filtering image frames using random reset of temporal filtering, as described in some embodiments of the present disclosure. [Figure 6] 1 is a flowchart illustrating a method for temporally filtering an image frame with a reset determined based on pixel characteristics, according to some embodiments of the present disclosure. [Figure 7] FIG. 2 is a block diagram illustrating weights for pixels of an image frame for random resetting of a temporal filter in accordance with some embodiments of the present disclosure. [Figure 8] 1 is a flowchart illustrating a method for determining a probability for resetting temporal filtering for a pixel of an image frame based on a confidence level, according to some embodiments of the present disclosure. [Figure 9] 11 is a graph illustrating a method for determining a probability for resetting temporal filtering for a pixel of an image frame based on a temporal difference metric according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Like reference numbers and designations in the various drawings indicate like elements.

[0028] The detailed description set forth below with reference to the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in all cases and that in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0029] The present disclosure provides systems, apparatus, methods, and computer-readable media that support image processing of captured image frames for photography and video, and in particular temporal filtering of image frames to reduce ghosting and improve scene integrity in the representation of the scene in the image frames. Particular implementations of the subject matter described in this disclosure may be implemented to realize potential advantages or benefits, such as, for example, improved image quality by reducing artifacts in a sequence of image frames captured in a low-light scene. The systems, apparatus, methods, and computer-readable media may be embedded within an image capture device, such as a mobile phone, tablet computing device, laptop computing device, other computing device, or digital camera.

[0030] An exemplary device for capturing image frames using one or more image sensors, such as a smartphone, may include an arrangement of two, three, four, or more cameras on the back (e.g., opposite the user display) or front (e.g., on the same side as the user display) of the device. A device with multiple image sensors includes one or more image signal processors (ISPs), computer vision processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors may provide processed image frames to a memory and / or a processor (such as an application processor, an image front end (IFE), an image processing engine (IPE), or other suitable processing circuitry) for further processing, such as for encoding, storage, transmission, or other manipulation.

[0031] As used herein, an image sensor may refer to the image sensor itself and any other components coupled to the image sensor that are used to generate an image frame for processing by an image signal processor or other logic circuitry, or for storage in memory, whether a short-term buffer or long-term non-volatile memory. For example, an image sensor may include other components of a camera, including shutters, buffers, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting analog signals into a digital representation of the image frame that is provided to digital circuitry coupled to the image sensor.

[0032] In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, to allow a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected to or connected through one or more intervening components or circuits. Also, for the purpose of explanation, specific names are set forth in the following description to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of the present disclosure.

[0033] Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc., is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0034] In the figures, a single block may be described as performing one or more functions. The one or more functions performed by the block may be implemented in a single component or across multiple components, and / or may be implemented using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various example components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Also, the example device may include components other than those shown, including well-known components such as a processor, memory, etc.

[0035] Aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Additionally, aspects of the present disclosure may be implemented in devices having or coupled to image sensors of the same or different capabilities and characteristics (such as resolution, shutter speed, sensor type, etc.). Additionally, aspects of the present disclosure may be implemented in devices for processing image frames, such as processing devices that may retrieve stored images for processing, including processing devices present in cloud computing systems, regardless of whether the device includes or is coupled to an image sensor.

[0036] Unless otherwise indicated, and as will be apparent from the description that follows, it will be appreciated that throughout this application, descriptions utilizing terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "solving," "generating," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the computer system's registers and memory, and convert that data to other data that is similarly represented as physical quantities in the computer system's registers, memory, or other such information storage, transmission, or display device.

[0037] The terms "device" and "apparatus" are not limited to one or a particular number of physical objects (e.g., one smartphone, one camera controller, one processing system, etc.). As used herein, a device may be any electronic device having one or more parts that may implement at least some portions of the present disclosure. The following description and examples use the term "device" to describe various aspects of the present disclosure, but the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0038] FIG. 1 illustrates a block diagram of an exemplary device 100 for performing image capture from one or more image sensors. The device 100 may include or be otherwise coupled to an image signal processor 112 for processing image frames from one or more image sensors, such as a first image sensor 101, a second image sensor 102, and a depth sensor 140. In some implementations, the device 100 also includes or is coupled to a processor 104 and a memory 106 that stores instructions 108. The device 100 may also include or be coupled to a display 114 and several input / output (I / O) components 116, such as a touch screen interface and / or physical buttons. The device 100 may further include or be coupled to a power source 118 for the device 100, such as a battery or a component for coupling the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown in FIG. 1. In one example, a wireless interface that may include a number of transceivers and a baseband processor may be included for a wireless communication device. In a further example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

[0039] The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data regarding the movement of the device 100, data regarding the environment surrounding the device 100, and / or other non-camera sensor data. Such non-camera sensors may be integrated into the device 100 in some embodiments. One exemplary non-camera sensor is a gyroscope, which is a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another exemplary non-camera sensor is an accelerometer, which is a device configured to measure acceleration, which may be used to determine velocity and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, velocity, and / or distance may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly coupled to the image signal processor 112. In another example, the non-camera sensor may be a global positioning system (GPS) receiver.

[0040] The image signal processor 112 may receive image data such as that used to form an image frame. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first and second cameras, respectively. In another embodiment, a wire interface couples the image signal processor 112 to an external image sensor. In yet another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

[0041] The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of the lenses 131 and 132 may have an associated autofocus (AF) system 133 and 134 that adjusts the lenses 131 and 132 to focus on a particular focal plane at a certain scene depth from the image sensors 101 and 102, respectively. The AF systems 133 and 134 may be aided by a depth sensor 140. The focal depth of the AF systems 133 and 134 may provide depth information about the image scene to other components of the device 100, such as the ISP 112, through metadata associated with image frames captured by the image sensors 101 and 102. The device 100 may perform image processing on image data from a combination of image sensors located within the device 100 or remotely from the device 100.

[0042] The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. The lenses 131 and 132 collect light onto the image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside an exposure window, one or more color filter arrays (CFAs) for filtering light outside a particular frequency range, one or more analog front ends for converting analog measurements to digital information, and / or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view for capturing different representations of a scene. For example, the first lens 131 may be an ultra-wide angle (UW) lens and the second lens 132 may be a wide angle (W) lens. The multiple image sensors may include a combination of ultra-wide angle (high field of view (FOV)), wide angle, telephoto, and super telephoto (low FOV) sensors. That is, each image sensor may be configured through hardware configuration and / or software settings to capture different but overlapping fields of view. In one configuration, the image sensors are configured with different lenses having different magnifications resulting in different fields of view. The sensors may be configured such that a UW sensor has a larger FOV than a W sensor, which has a larger FOV than a T sensor, which has a larger FOV than a UT sensor. For example, a sensor configured for a wide FOV may capture a field of view ranging from 64 to 84 degrees, a sensor configured for an ultra-wide FOV may capture a field of view ranging from 100 to 140 degrees, a sensor configured for a telephoto FOV may capture a field of view ranging from 10 to 30 degrees, and a sensor configured for a super-telephoto FOV may capture a field of view ranging from 1 to 8 degrees.

[0043] Image signal processor 112 processes image frames captured by image sensors 101 and 102. Although FIG. 1 illustrates device 100 as including two image sensors 101 and 102 coupled to image signal processor 112, any number of image sensors (e.g., one, two, three, four, five, six, etc.) may be coupled to image signal processor 112. In some aspects, a depth sensor, such as depth sensor 140, may be coupled to image signal processor 112, and output from the depth sensor may be processed similarly to that of image sensors 101 and 102. Additionally, any number of additional image sensors or image signal processors may be present for device 100. In some embodiments, image signal processor 112 may execute instructions from a memory, such as instructions 108 from memory 106, instructions stored in a separate memory coupled to or included in image signal processor 112, or instructions provided by processor 104. Additionally or alternatively, image signal processor 112 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in this disclosure. For example, image signal processor 112 may include circuitry specifically configured to temporally filter two or more image frames, circuitry specifically configured to implement an infinite impulse response (IIR) filter, and / or circuitry specifically configured to apply motion compensation to image frames.

[0044] In some implementations, memory 106 may include a non-transient or non-transitory computer-readable medium that stores computer-executable instructions 108 for performing all or a portion of one or more operations described in this disclosure. In some implementations, instructions 108 include a camera application (or other suitable application) to be executed by device 100 to generate images or videos. Instructions 108 may also include other applications or programs executed by device 100, such as an operating system or specific applications other than for image or video generation. Execution of the camera application, such as by processor 104, may cause device 100 to generate images using image sensors 101 and 102 and image signal processor 112. Memory 106 may also be accessed by image signal processor 112 to store processed frames or by processor 104 to obtain processed frames. In some embodiments, device 100 does not include memory 106. For example, device 100 may be a circuit that includes image signal processor 112 and the memory may be external to device 100. Device 100 may be coupled to the external memory and configured to access the memory to write output frames for display or long-term storage. In some embodiments, device 100 is a system-on-chip (SoC) that incorporates image signal processor 112, processor 104, sensor hub 150, memory 106, and input / output components 116 into a single package.

[0045] In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including the composition operation of temporal filtering. For example, execution of instructions can instruct the image signal processor 112 to begin or end capturing an image frame or a sequence of image frames, where the capturing includes the temporal filtering described in the embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processors capable of executing scripts or instructions of one or more software programs, such as the instructions 108 stored in the memory 106. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 106. In executing the camera application, the processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations on the image sensor 101 or 102. For example, the camera application may receive a capture command, and when this command is received, a video including a sequence of image frames is captured and processed. Temporal filtering may be applied to one or more image frames in the sequence. The camera application may enable and disable temporal filtering and / or configure parameters of the temporal filtering, such as the number of image frames for composition in the temporal filtering, parameters for determining the application of temporal filtering, and / or the absolute or relative number of random resets to perform per image frame. Execution of instructions 108 by processor 104 outside of the camera application may also cause device 100 to perform any number of functions or operations. In some embodiments, processor 104 may include ICs or other hardware in addition to the capability for executing software to cause device 100 to perform certain functions or operations, such as those described herein.In some other embodiments, the device 100 does not include the processor 104, such as when all of the described functionality is configured in the image signal processor 112.

[0046] In some embodiments, display 114 may include one or more suitable displays or screens that allow for user interaction and / or presentation of items to a user, such as a preview of image frames being captured by image sensors 101 and 102. In some embodiments, display 114 is a touch-sensitive display. I / O components 116 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from a user and for providing output to a user through display 114. For example, I / O components 116 may include, but are not limited to, a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (such as a power button), sliders, switches, and the like.

[0047] Although shown coupled together via the processor 104, the components (such as the processor 104, memory 106, image signal processor 112, display 114, and I / O components 116) may be coupled together in various other configurations, such as via one or more local buses not shown for simplicity. Although the image signal processor 112 is shown as separate from the processor 104, the image signal processor 112 may be a core of the processor 104, be an application processor unit (APU), be included in a system on chip (SoC), or otherwise included in the processor 104. The device 100 is referenced in the examples herein to implement aspects of the present disclosure, although some device components may not be shown in FIG. 1 to avoid obscuring aspects of the present disclosure. In addition, other components, multiple components, or combinations of components may be included in a device suitable for implementing aspects of the present disclosure. Thus, the present disclosure is not limited to a particular device or configuration of components, including the device 100.

[0048] When temporal filtering is enabled, conditions may exist that create artifacts in the output image frames. For example, an object that moves or disappears from the scene may remain in the temporally filtered output for a long time because the temporally filtered image is based on a previous image frame that contained the object. These residual objects in the display are sometimes referred to as "ghosts" due to the object appearing in the image even though it has left the scene. Ghosts may also appear under other circumstances, such as when temporally filtering low-contrast (e.g., SNR less than 1) images. The effects of ghosting may be more noticeable when more aggressive temporal filtering is applied. Any of these artifacts result in scene integrity issues for the output image frames that reduce the accuracy of the scene representation and degrade the user experience. The shortcomings mentioned here are merely representative and are included to highlight problems that the inventors have identified and sought to remedy with existing devices. Aspects of the device described below may address some or all of the shortcomings, as well as others known in the art. Aspects of the improved devices described herein may offer benefits and be used in applications other than those described above.

[0049] In one aspect of device 100, image frames captured from one or more of image sensors 101 and 102 may be modified, for example, with temporal filtering with reset for some pixels of the image frame. For example, a first image frame from a first image sensor may be temporally filtered by combining a portion of the first image frame with a portion of a second image frame received from the first sensor or a different second sensor. Temporal filtering with pixel reset is described in different embodiments with reference to the aspects of Figures 2, 3, 4, 5, 6, 7, and / or 8.

[0050] FIG. 2 is a flow chart illustrating a method of temporal filtering of image frames with resetting temporal filtering, as described in some embodiments of the present disclosure. The method 200 for image processing includes receiving a first image frame and a second image frame at block 202. In some embodiments, the first and second image frames may be received from a memory. In some embodiments, the first image frame may be received from an image sensor and the second image frame may be received from a memory. In some embodiments, the first and second image frames may be received from the same or different image sensors. In some embodiments, the first image frame is an image frame received from a memory or an image sensor and the second image frame is an output of the temporal filtering of block 204. Although only the first and second image frames are described as being received at block 202, additional image frames may be received at 202 and used in the temporal filtering of block 204.

[0051] At block 204, the method 200 includes determining a third image frame (a "temporally filtered" image frame) based on the first and second image frames by applying temporal filtering to combine the first and second image frames. The third image frame may be output to a memory for storage as a photograph, may be output to a memory for use in a sequence of image frames for storage as a video, may be displayed on a display device, may be transmitted to a remote viewer, may be transmitted to a remote storage device, and / or may be transmitted in real time to a remote display.

[0052] The resetting of the temporal filtering of block 204 may include performing different processing on the pixels of the third image frame. Some pixels of the third image frame (e.g., the "third plurality of pixels") may be determined based on the results of the temporal filtering using the first image frame and the second image frame of block 202. These pixels may be all or a subset (e.g., the "fourth plurality of pixels") of all pixels in the third image frame. Some of the third plurality of pixels to be determined using temporal filtering may be reset to improve image quality, for example, by reducing ghosting. This divides the determined third plurality of pixels into a first plurality of pixels determined by combining the corresponding pixels of the first image frame with the second image frame, and a second plurality of pixels determined based on the corresponding pixels of the first image frame without reference to the corresponding pixels of the second image frame based on the second plurality of pixels being reset. The corresponding pixels of the first and second image frames may refer to pixels at the same location relative to the image frames or pixels at the same location relative to the scene. The reset of the second plurality of pixels may eliminate or reduce the contribution from a previous image frame, such as the second image frame, to the third image frame. Instead, the second plurality of pixels may be based only on the first (or "current") image frame. This reset of some pixels of the third image frame may reduce the presence of ghosts and other artifacts in the third image frame, thereby improving scene integrity. One or more criteria may be used to determine the selection of pixels to be processed as the first plurality of pixels and to determine the selection of pixels to be processed as the second plurality of pixels. These one or more criteria may affect the probability that some pixels are selected for inclusion in the second plurality of pixels for which the temporal filtering is reset, such that the second plurality of pixels are selected probabilistically.For example, the pixels may be randomly determined such that at least a certain percentage or number of pixels are reset in each temporally filtered image frame.

[0053] The resetting of some pixels during temporal filtering in some embodiments is shown in FIG. 3. FIG. 3 is a block diagram illustrating an implementation of temporal filtering using an infinite impulse response (IIR) filter with filter resetting among some pixels according to some embodiments of the present disclosure. The image sensor 101 may generate a current image frame 302 that is input to the temporal filter 310. The temporal filter 310 also receives a previous image frame 304. The previous image frame 304 may be a feedback from the output of the temporal filter 310 for the infinite impulse response (IIR) filter. The previous image frame 304 may alternatively be an image frame captured by the image sensor 101 or another image sensor prior to the capture of the current image frame 302 in other embodiments of the temporal filter 310. The temporal filter 310 may determine an output image frame 306 by modifying pixels of the current image frame 302 by combining some pixels of the current image frame 302 with corresponding pixels of the previous image frame 304.

[0054] Some pixels 312 may be reset in the combination of image frames 302 and 304 in the temporal filter 310 by reducing or eliminating the contribution of image frame 304 to the output image frame 306. For example, pixels 312 may be randomly selected and reset such that pixels 312 are based on corresponding pixels of image frame 302 and not based on corresponding pixels of the previous image frame 304. The locations of pixels 312 in the output image frame 306 may be randomized within and between output image frames such that the reset pixels vary from one output image frame to another. The locations of random pixels 312 within the output image frames may be weighted based on characteristics of the pixel, the area around the pixel, and / or the image frame as a whole. Similarly, the number of random pixels 312 in the output image frames may vary from one frame to the next based on characteristics of the pixel, the area of ​​the pixel, the image frame as a whole, or other parameters measured by other sensors of the image capture device. In some embodiments, gyroscope and / or accelerometer data corresponding to an image frame (e.g., collected during capture of the image frame) may be used to determine a stationary camera (e.g., a camera mounted on a tripod) and, based on the determination, may reduce the probability of reset or reduce the amount of reset pixels in each frame.

[0055] Temporal filtering with compositing and resetting is described in more detail with respect to FIG. 4. FIG. 4 is a block diagram illustrating an implementation of reset logic for an infinite impulse response (IIR) filter according to some embodiments of the present disclosure. The temporal filter 310 receives the current image frame 302 and the previous image frame 304. The temporal filter 310 may modify the current image frame 302 pixel by pixel. The compositor 406 may composite a pixel of the current image frame 302 with a corresponding pixel of the previous image frame 304. The compositing may include, for example, averaging intensity values ​​of corresponding pixels of the current image frame 302 and the previous image frame 304. The compositor 406 may perform a weighted average of corresponding pixels using weight values ​​provided by the weight block 404. The weight block 404 may determine a weighting of the corresponding pixel in the previous image frame 304 based on a motion factor determined from the spatial motion of the scene from the current image frame 302 to the previous image frame 304. The reset logic 402 may determine a number of pixels for resetting and cause the weight block 404 to output a weight of zero to the compositor 406 to reset the pixel, thereby canceling, for that pixel, the contribution of the previous image frame 304 to the output image frame 306. Although per-pixel operations are described, the operations may alternatively be performed on groups of pixels in an image frame, regions of an image frame, or objects in an image frame.

[0056] In one embodiment, the reset logic 402 may randomly select pixels for resetting such that the temporal filter 310 operates according to the method shown in FIG. 5. FIG. 5 is a flow chart illustrating a method of temporally filtering image frames using random reset of temporal filtering, as described in some embodiments of the present disclosure. The method 500 begins at block 502, receiving a first image frame (e.g., the current image frame 302) and a second image frame (e.g., the previous image frame 304). At block 504, the method 500 includes determining a third image frame by temporally filtering the first image frame and the second image frame, where the temporal filtering for some pixels of the third image frame is randomly reset. The third image frame determined at block 504 may be displayed to a user and / or stored as a frame in a photograph or video sequence.

[0057] In another embodiment, the reset logic 402 may determine the particular pixels to reset based on a reset weighting value that makes some pixels more likely to be reset than other pixels. The reset weighting value may be determined based on characteristics of the pixels in the image frame, the area around the pixels in the image frame, or other characteristics. Some pixels may be determined to be more beneficial to reset than other pixels based on these characteristics, for example, whether the corresponding portion of the image frame is static, i.e., relatively unchanged over time, or dynamic, i.e., changing rapidly over time. In some embodiments, the random probability for a particular pixel may be reduced to zero to prevent the pixel from being reset, or increased to one to force a pixel reset. FIG. 6 is a flow chart illustrating a method of temporally filtering image frames with a reset determined based on pixel characteristics, according to some embodiments of the present disclosure. The method 600 begins at block 602, receiving a first image frame and a second image frame. In block 604, a third image frame is determined by temporally filtering the first image frame and the second image frame, and the temporal filtering for some pixels of the third image frame is reset based on characteristics of the pixels.

[0058] One exemplary characteristic of a pixel for adjusting the reset weighting value is whether the pixel is associated with an object. Object detection, for example through an object detection algorithm running in a computer vision (CV) processor, may be performed on the first or second image frame to identify objects such as faces, trees, roads, animals, plants, ground, etc. A pixel may be associated with an object and a reset weighting value for the pixel corresponding to the object that is adjusted together. For example, the reset weighting value may be zero to prevent resetting based on identification of a static object (e.g., a tree) in the pixels corresponding to that region of the image frame. Alternatively or additionally, motion vectors for a pixel may be correlated to identify an object in the image frame. Neighboring pixels with similar motion vectors may be associated such that the reset weighting values ​​for pixels with similar motion vectors are adjusted together.

[0059] A further exemplary characteristic of a pixel for adjusting the reset weighting value is a motion factor associated with the pixel. The motion factor may indicate spatial and / or temporal differences. For example, a fast moving object may have a high temporal difference because the object is at a significantly different location between the first image frame and the second image frame. A fast moving object is more likely to create ghosting or other artifacts, and the reset weighting value for a pixel associated with high temporal motion may be reset more frequently and randomly by increasing the reset weighting value. In some embodiments, the motion factor for a pixel may be determined based on a motion vector for the pixel. The magnitude of the motion vector may be used as the motion factor for the pixel, and the reset weighting value corresponding to the pixel may be adjusted based on the motion vector magnitude.

[0060] Another exemplary characteristic of a pixel for the determination of block 604 is a confidence level associated with a temporal filtering decision for the pixel. The decision as to whether to apply temporal filtering to a particular pixel may be based on a rule with one or more criteria. The temporal filtering decision may be based on a decision as to whether the pixel corresponds to a static or dynamic region of the image. A static region may be determined based on whether the temporal difference between image frames is below a certain threshold, which may correspond to expected temporal noise in the image frames. A confidence level for the temporal filtering decision for the pixel is determined, which may be used, in part, to decide whether to reset the pixel during the temporal filtering process. The confidence level reflects the likelihood that the decision that the pixel is static or dynamic is correct, or more generally, the likelihood that the decision to apply or not apply a temporal filter is correct. The confidence level may be determined based on a signal-to-noise ratio (SNR) and local contrast in the region of the pixel. For example, a high local contrast, indicating that neighboring pixels are very different, indicates a strong texture that is unlikely to be noise in the image frame. For pixels with a combination of high spatial difference and low temporal difference, a high confidence level may be set. For pixels with low spatial and high temporal differences, a low confidence level may be set.

[0061] For pixels with a high confidence level for the temporal filtering decision, the rate of pixel resets may be reduced or no resets may be performed. In some embodiments, the rate of pixel resets may be adjusted in proportion to the confidence level. For example, in regions where the confidence level for pixels in the region is above a high threshold, no pixel resets may be performed, in regions where the confidence level for pixels in the region is above a medium threshold, a low rate of resets (e.g., once every 20 pixels) may be performed, and in regions where the confidence level is below the medium threshold, a normal rate of resets (e.g., once every 10 pixels) may be performed. The confidence level of the temporal filtering decision may be based on one or more criteria. For example, a high confidence level may be associated with having a sufficiently high signal-to-noise ratio (SNR) in the region of the image. In another example, a high confidence level is associated with detecting the presence of a strong texture in the region of the image and at the same time, low temporal differences between pixels in the image frame are input to the temporal filter. Other factors for determining the confidence level include luminance, spatial difference, and / or temporal difference.

[0062] Resetting pixels may be based on a reset weighting value, and the likelihood of any particular pixel being randomly reset is based on the weighting for the particular pixel. A reset weighting map may be determined for each image frame corrected by the temporal filter, and the weighting map may be applied during the determination of random pixels for reset. Such a reset weighting map may include an array of values ​​ranging from zero to one corresponding to each pixel, with zero representing no possibility of reset for the corresponding pixel, and one representing a guaranteed reset for the corresponding pixel. The reset weighting map may be initialized with equal values ​​for all pixels, such as a value of 0.50, so that each pixel has an equal likelihood of being randomly reset by the temporal filter. The reset weighting map may be used in combination with other parameters such that the random function performed for each pixel of an image frame causes a desired average number of pixels in the image frame to be reset. An exemplary reset weighting map is shown in FIG. 7.

[0063] FIG. 7 is a block diagram illustrating weights for pixels of an image frame for random resetting of a temporal filter according to some embodiments of the present disclosure. Table 700 includes reset weighting values ​​for modifying the probability of some pixels in an input image frame being reset during application of temporal filtering. In some embodiments, table 700 may have a 1:1 correlation between values ​​and pixels of an image frame such that table 700 is the same dimension as the resolution of the input image. When a pixel is reset, the value in table 700 corresponding to that pixel may be reduced to prevent the pixel from being reset too frequently. In some embodiments, table 700 may have an N:1 correlation between values ​​and pixels of an image frame such that table 700 is smaller than the resolution of the input image. Regions 702 with zero values ​​may correspond to detected objects that are essentially static such that no temporal filtering reset is performed for pixels associated with the detected object.

[0064] FIG. 8 is a flow chart illustrating a method for determining a reset weighting value for a pixel of an image frame based on a confidence level according to some embodiments of the present disclosure. The method 800 begins at block 802 by determining whether to apply temporal filtering (TF) to a pixel. A confidence level may be associated with the TF decision. If the confidence level for the TF decision is greater than a threshold at block 804, the method 800 continues at block 808 with setting a low probability of resetting the pixel corresponding to the TF decision. If the confidence level for the TF decision is less than a threshold at block 804, the method 800 continues at block 806 with setting a high probability of resetting the pixel corresponding to the TF decision. For example, block 806 may set some pixels to have a reset probability of 0.7 (first probability value), and block 808 may set some pixels to have a reset probability of 0.3 (second probability value). In some embodiments, the probability value for a pixel may be modified based on other factors, such as whether the pixel was reset in one of the N previous frames, or how many times the pixel was reset in the M previous frames. Additionally or alternatively, the probability value may be increased or decreased over time by forming a reset weighting map for a future (e.g., next) image frame based on a reset weighting map for a current or previous image frame. Method 800 is applied to pixels of an image frame to generate a reset weighting map, as shown in FIG. 7, and may be applied during the temporal filtering method of FIG. 6.

[0065] The pixel reset decision in the temporal filtering of FIG. 6 may be based on other characteristics. FIG. 9 is a graph illustrating a method of determining a probability to reset temporal filtering for a pixel of an image frame based on a temporal difference metric according to some embodiments of the disclosure. The graph 900 shows a temporal difference metric value on the y-axis versus a luminance value on the x-axis. The temporal difference metric reflects the difference in values ​​for the pixel and / or the area surrounding the pixel from one image frame to the next, or another value that reflects the dynamics, or rate of change, of the pixel's value over time. A more dynamic pixel is more likely to create artifacts and may be reset more frequently, such as with a higher reset weighting value, to improve the quality of the temporal filtering. The temporal difference metric and luminance value for a pixel may be above a first threshold 902, below the first threshold 902 but above a second threshold 904, or below the second threshold 904. The relationship between the pixel's characteristics and the thresholds 902 and 904 may be used to adjust a reset weighting map as shown in FIG. 7. Pixels in the xy space of the graph 900 having a feature value above a first threshold 902 may be set to be reset frequently, such as by increasing a reset weighting value, perhaps to 1. Pixels having a feature value below a second threshold 904 may be set to never reset, such as by setting a reset weighting value to zero. Pixels having feature values ​​between the thresholds 902 and 904 may have a weighting applied to the composition of the image frames in the temporal filter such that pixels closer to the threshold 902 have a greater contribution from the current image frame than the previous image frame.

[0066] In one or more aspects, a technique for enhancing image processing, such as any single aspect or any combination of aspects described below or with respect to one or more other processes or devices described elsewhere herein. In one or more aspects, the image processing may include performing a method including receiving a first image frame and a second image frame that temporally precedes the first image frame. The method may also include determining a third image frame based on the first image frame and the second image frame. The method may further include determining to apply temporal filtering to the first and second plurality of pixels in the third plurality of pixels in the third image frame. The method may also include determining the first plurality of pixels with temporal filtering by combining corresponding pixels of the first image frame with the second image frame. The method may further include determining the second plurality of pixels based on corresponding pixels of the first image frame without reference to corresponding pixels of the second image frame. In some implementations, the second plurality of pixels are randomly selected pixels of the third plurality of pixels. Further, the method may be implemented by an apparatus including a wireless device such as a user equipment (UE). In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the method may be embedded in a non-transitory computer-readable medium with program code recorded thereon, the program code being executable by a computer to cause the computer to perform the operations described herein with respect to the apparatus. In some implementations, the method may be implemented by one or more means configured to perform the operations described herein. In some implementations, the method of wireless communication may include one or more operations described herein with respect to the apparatus.

[0067] In a second aspect, in combination with the first aspect, a step of determining a reset weighting map corresponding to a third image frame to determine a third image frame, the second plurality of pixels being pixels in the third image frame selected based on the reset weighting map.

[0068] In a third aspect, in combination with one or more of the first or second aspects, determining the third image frame may include determining to apply temporal filtering to the first plurality of pixels and to the second plurality of pixels, and / or determining a confidence level associated with the decision to apply temporal filtering, where the second plurality of pixels are randomly selected to reset the temporal filtering based on the confidence level.

[0069] In a fourth aspect, in combination with one or more of the first to third aspects, the method may include determining a motion factor corresponding to spatial motion of pixels in the first image frame from the second image frame to the first image frame, and the reset weighting map is based on the motion factor.

[0070] In a fifth aspect, in combination with one or more of the first to fourth aspects, the step of determining the motion factor includes a step of determining a motion vector between the second image frame and the first image frame, and the reset weighting map is based on a magnitude of the motion vector.

[0071] In a sixth aspect, in combination with one or more of the first to fifth aspects, the step of determining a reset weighting map includes a step of determining a first location of the object in the first image frame, and the reset weighting map is based on the first location of the object.

[0072] In a seventh aspect, in combination with one or more of the first to sixth aspects, the second plurality of pixels are randomly selected pixels in the third image frame.

[0073] In an eighth aspect, in combination with one or more of the first to seventh aspects, the second image frame includes a temporally filtered image frame based on a combination of a fourth image frame and a fifth image frame, each of which is temporally earlier than the first image frame, such as when the filter is an IIR filter.

[0074] In a ninth aspect, in combination with one or more of the first to eighth aspects, the method includes applying motion compensation to the second image frame before determining the third image frame.

[0075] In one or more aspects, a technique for supporting a device including a processor and a memory coupled to the processor that stores instructions, the instructions, when executed by the processor, cause the device to perform operations that may include additional aspects, such as any single aspect or any combination of aspects, described below or with respect to one or more other processes or devices described elsewhere herein. In a tenth aspect, supporting image processing may include the device being configured to receive a first image frame and a second image frame that temporally precedes the first image frame. The device may be further configured to determine a third image frame based on the first image frame and the second image frame. The device may be further configured for determining to apply temporal filtering to the first and second plurality of pixels in a third plurality of pixels in the third image frame, determining the first plurality of pixels using temporal filtering by combining corresponding pixels of the first image frame with the second image frame, and / or determining the second plurality of pixels based on corresponding pixels of the first image frame without reference to corresponding pixels of the second image frame. In some implementations, the second plurality of pixels is a randomly selected pixel of the third plurality of pixels. In addition, the apparatus may implement or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a user equipment (UE) or a base station (BS), or an infrastructure component, such as a cloud-based server. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. The processor may be configured to perform operations described herein with respect to the apparatus. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by a computer to cause the computer to perform operations described herein with respect to the apparatus.In some implementations, an apparatus may include one or more means configured to perform the operations described herein.

[0076] In an eleventh aspect, in combination with the tenth aspect, the apparatus is configured with instructions for determining a reset weighting map corresponding to the third image frame to determine a third image frame, and the second plurality of pixels are pixels in the third image frame selected based on the reset weighting map.

[0077] In a twelfth aspect, in combination with one or more of the tenth to eleventh aspects, determining the third image frame further includes determining to apply temporal filtering to the first plurality of pixels and to the second plurality of pixels, and / or determining a confidence level associated with the decision to apply temporal filtering, wherein the second plurality of pixels are randomly selected for resetting the temporal filtering based on the confidence level.

[0078] In a thirteenth aspect, in combination with one or more of the tenth to twelfth aspects, the apparatus is configured with instructions for determining a motion factor corresponding to spatial motion of pixels in a first image frame from a second image frame to the first image frame, and the reset weighting map is based on the motion factor.

[0079] In a fourteenth aspect, in combination with one or more of the tenth to thirteenth aspects, determining the motion factor includes determining a motion vector between the second image frame and the first image frame, and the reset weighting map is based on a magnitude of the motion vector.

[0080] In a fifteenth aspect, in combination with one or more of the tenth to fourteenth aspects, determining the reset weighting map includes determining a first location of the object in the first image frame, and the reset weighting map is based on the first location of the object.

[0081] In a sixteenth aspect, in combination with one or more of the tenth to fifteenth aspects, the second plurality of pixels are randomly selected pixels in the third image frame.

[0082] In one or more aspects, techniques for supporting a non-transitory computer-readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations may include additional aspects, such as any single aspect or any combination of aspects described with respect to one or more other processes or devices described below or elsewhere herein. In a seventeenth aspect, supporting image processing may include a non-transitory computer-readable medium storing instructions that, when executed by a processor of the device, cause the device to perform operations, the operations including receiving a first image frame and a second image frame that temporally precedes the first image frame, determining a third image frame based on the first image frame and the second image frame, determining to apply temporal filtering to the first and second plurality of pixels within the third plurality of pixels in the third image frame, determining the first plurality of pixels using temporal filtering by combining corresponding pixels of the first image frame with the second image frame, and / or determining the second plurality of pixels based on corresponding pixels of the first image frame without reference to corresponding pixels of the second image frame. In some implementations, the second plurality of pixels is a randomly selected pixel of the third plurality of pixels. Furthermore, the instructions may cause the apparatus to perform or operate according to one or more aspects as described below. In some implementations, the apparatus includes a wireless device, such as a base station (BS) or a user equipment (UE), or includes an infrastructure device, such as a cloud-based server. In some implementations, the apparatus may include at least one processor and a memory coupled to the processor. In some aspects, the processor is an image signal processor that further includes circuitry configured to perform other image functions described herein. The processor may be configured to perform operations described herein with respect to the apparatus.In some other implementations, a non-transitory computer readable medium having program code recorded thereon or the program code may be executable by a computer to cause the computer to perform the operations described herein with respect to the apparatus.

[0083] In an eighteenth aspect, in combination with the seventeenth aspect, the instructions, when executed by the processor, cause the device to perform further operations, the operations including determining a reset weighting map corresponding to the third image frame to determine a third image frame, and the second plurality of pixels are pixels in the third image frame selected based on the reset weighting map.

[0084] In a 19th aspect, in combination with one or more of aspects 17 to 18, determining the third image frame further includes determining to apply temporal filtering to the first plurality of pixels and to the second plurality of pixels, and / or determining a confidence level associated with the decision to apply temporal filtering, wherein the second plurality of pixels are randomly selected to reset the temporal filtering based on the confidence level.

[0085] In a twentieth aspect, in combination with one or more of the seventeenth to nineteenth aspects, the instructions, when executed by the processor, cause the device to perform further operations, the operations including determining a motion factor corresponding to spatial movement of pixels in the first image frame from the second image frame to the first image frame, and the reset weighting map is based on the motion factor.

[0086] In a 21st aspect, in combination with one or more of the 17th to 20th aspects, determining the motion factor includes determining a motion vector between the second image frame and the first image frame, and the reset weighting map is based on a magnitude of the motion vector.

[0087] In a 22nd aspect, in combination with one or more of the 17th to 21st aspects, determining the reset weighting map includes determining a first location of the object in the first image frame, and the reset weighting map is based on the first location of the object.

[0088] In a twenty-third aspect, in combination with one or more of the seventeenth to twenty-second aspects, the second plurality of pixels are randomly selected pixels in the third image frame.

[0089] In a 24th aspect, in combination with one or more of the 17th to 23rd aspects, the instructions, when executed by the processor, further cause the device to perform an operation including applying motion compensation to the first image frame before determining the third image frame.

[0090] In one or more aspects, techniques for supporting image capture and image processing may be implemented in or by a device, the device including a first image sensor configured with a first field of view, a processor coupled to the first image sensor, and a memory coupled to the processor. The processor is configured to perform steps including additional aspects, such as any single aspect or any combination of aspects, described below or with respect to one or more other processes or devices described elsewhere herein. In a twenty-fifth aspect, supporting image capture may include the device being configured to receive a first image frame captured at a first time and a second image frame captured at a second time. The device is further configured to perform a process including receiving a first image frame and a second image frame that temporally precedes the first image frame, determining a third image frame based on the first image frame and the second image frame, determining to apply temporal filtering to the first and second plurality of pixels in the third plurality of pixels in the third image frame, determining the first plurality of pixels using temporal filtering by combining corresponding pixels of the first image frame with the second image frame, and / or determining the second plurality of pixels based on corresponding pixels of the first image frame without reference to corresponding pixels of the second image frame. In some implementations, the second plurality of pixels are randomly selected pixels of the third plurality of pixels. Furthermore, the device may implement or operate according to one or more aspects as described below. In some implementations, the device includes a wireless device such as a base station (BS) or a user equipment (UE), or an infrastructure device such as a cloud-based server. In some implementations, a device may include at least one processor and a memory coupled to the processor, where the processor may be configured to perform the operations described herein with respect to the device.In some other implementations, the device may include a non-transitory computer-readable medium having program code recorded thereon, the program code being executable by the device to cause the device to perform the operations described herein with respect to the device. In some implementations, the device may include one or more means configured to perform the operations described herein.

[0091] In a 26th aspect, in combination with the 25th aspect, the processor is further configured to perform a step including determining a reset weighting map corresponding to the first image frame, and the randomly selected pixel in the third image frame is based on the reset weighting map.

[0092] In a 27th aspect, in combination with one or more of the 25th to 26th aspects, determining the third image frame further includes determining to apply temporal filtering to the first plurality of pixels and to the second plurality of pixels, and / or determining a confidence level associated with the decision to apply temporal filtering, wherein the second plurality of pixels are randomly selected to reset the temporal filtering based on the confidence level.

[0093] In a 28th aspect, in combination with aspects 25 to 27, the processor is further configured to perform a step including determining a motion factor corresponding to spatial motion of pixels in the first image frame from the second image frame to the first image frame, and the reset weighting map is based on the motion factor.

[0094] In a 29th aspect, in combination with aspects 25 to 28, the device further includes a computer vision (CV) processor, and determining the reset weighting map includes determining, by the CV processor, a first location of the object in the first image frame, and the reset weighting map is based on the first location of the object.

[0095] In a 30th aspect, in combination with aspects 25 to 29, the second image frame includes a temporally filtered image frame based on a combination of a fourth image frame and a fifth image frame, each of which is temporally earlier than the first image frame.

[0096] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0097] The components, functional blocks, and modules described herein with respect to Figures 1, 3, and 4 include, among other examples, processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software code, firmware code, or any combination thereof. In addition, features described herein may be implemented via special purpose processor circuitry, via executable instructions, or a combination thereof.

[0098] Moreover, those skilled in the art will appreciate that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosure. Those skilled in the art will also readily appreciate that the order or combination of components, methods, or interactions described herein are merely examples, and that the components, methods, or interactions of various aspects of the disclosure may be combined or performed in ways other than those shown and described herein.

[0099] The various example logic, logic blocks, modules, circuits, and algorithmic processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. Interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various example components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system.

[0100] The hardware and data processing devices used to implement the various example logic, logic blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or embodied in general purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, certain processes and methods may be implemented in circuit configurations specific to a given function.

[0101] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, or any combination thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus.

[0102] If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code. The processes of the methods or algorithms disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that may enable a computer program to be transferred from one place to another. A storage medium may be any available medium that may be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection may be properly referred to as a computer-readable medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, operations of a method or algorithm may reside on machine-readable and computer-readable media, which may be incorporated into a computer program product as one or any combination or set of code and instructions.

[0103] Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to several other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the present disclosure, the principles and novel features disclosed herein.

[0104] In addition, those skilled in the art will readily appreciate that the terms "upper" and "lower" may be used to simplify illustration of the figures, indicate relative positions that correspond to the orientation of the figure on a suitably oriented page, and may not reflect the proper orientation of any implemented device.

[0105] Also, some features described in the context of separate implementations herein may be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Furthermore, although features may be described above as working in some combinations, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0106] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. Additionally, the figures may generally depict one or more exemplary processes in the form of a flow diagram. However, other operations not shown may be incorporated into the exemplary process depicted in the schematic. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. In some circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other implementations fall within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0107] The term "or" as used herein, including in the claims, when used in a list of two or more items means that any one of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if a composition is described as including components A, B, or C, the composition may include only A, only B, only C, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. Also, as used herein, including in the claims, "or" used in a list of items ending with "at least one of" indicates a disjunctive list, such as, for example, a list of "at least one of A, B, or C" means any of A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any of these in any combination thereof. The term "substantially" is defined as the majority of, but not necessarily the entirety of, what is specified (and includes what is specified, e.g., substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by those skilled in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [a percentage] of" what is specified, where the percentage includes 0.1, 1, 5, or 10 percent.

[0108] The above description of the disclosure is intended to enable any person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [Explanation of symbols]

[0109] 100 devices 101 first image sensor, image sensor 102 second image sensor, image sensor 104 processors 106 Memory 108 Instructions 112 Image Signal Processor, ISP 114 Display 116 Input / Output (I / O) Components 118 Power supply 131 First Lens, Lens 132 Second Lens, Lens 133 Autofocus (AF) system 134 Autofocus (AF) system 140 Depth Sensor 150 Sensor Hub 302 Current Image Frame 304 Previous Image Frame 306 output image frames 310 Time Filter 312 pixels 402 Reset Logic 404 Weight Block 406 Synthesizer 902 First Threshold 904 Second Threshold

Claims

1. Receiving a first image frame and a second image frame temporally preceding the first image frame; and Determining a third image frame based on the first image frame and the second image frame, wherein the determining step comprises: Determining to apply temporal filtering to a first plurality of pixels and a second plurality of pixels within a third plurality of pixels in the third image frame; Determining the first plurality of pixels using temporal filtering by synthesizing corresponding pixels of the first image frame with the second image frame; and Determining the second plurality of pixels based on corresponding pixels of the first image frame without referring to corresponding pixels of the second image frame.

2. The method of claim 1, further comprising determining a reset weighting map, wherein the second plurality of pixels are probabilistically selected based on the reset weighting map.

3. The step of determining the third image frame further comprises: Determining a reliability level associated with the determination to apply temporal filtering to the first plurality of pixels and the second plurality of pixels; and The determination of the reset weighting map is based on the reliability level.

4. The method of claim 2, further comprising determining a motion factor corresponding to a spatial motion of pixels in the first image frame from the second image frame to the first image frame, wherein the reset weighting map is based on the motion factor.

5. The method of claim 4, wherein the step of determining the motion factor comprises determining a motion vector between the second image frame and the first image frame, and the reset weighting map is based on the motion vector.

6. The method of claim 2, wherein the step of determining the reset weighting map comprises determining a first location of an object in the first image frame, and the reset weighting map is based on the first location of the object.

7. The method according to claim 1, wherein the second plurality of pixels are randomly selected pixels among the third plurality of pixels. **Claim 8** A device comprising: a processor; a memory coupled to the processor and storing instructions, which when executed by the processor cause the device to perform operations, the operations including: receiving a first image frame and a second image frame temporally preceding the first image frame; determining a third image frame based on the first image frame and the second image frame, the determining including: determining to apply temporal filtering to a first plurality of pixels and a second plurality of pixels within a third plurality of pixels within the third image frame; determining the first plurality of pixels using temporal filtering by compositing corresponding pixels of the first image frame with the second image frame; determining the second plurality of pixels based on corresponding pixels of the first image frame without referring to corresponding pixels of the second image frame. **Claim 9** The instructions, when executed by the processor, further cause the device to perform an operation including determining a reset weighting map, and the second plurality of pixels are probabilistically selected based on the reset weighting map. The device according to claim 8. **Claim 10** The determining of the third image frame further includes: determining a reliability level associated with the determining to apply temporal filtering to the first plurality of pixels and the second plurality of pixels; The device according to claim 9, wherein the determining of the reset weighting map is based on the reliability level. **Claim 11** The instructions, when executed by the processor, further cause the device to perform an operation including determining a motion factor corresponding to a spatial motion of pixels in the first image frame from the second image frame to the first image frame, and the reset weighting map is based on the motion factor. The device according to claim 10. **Claim 12** Determining the motion factor includes determining a motion vector between the second image frame and the first image frame, and the reset weighting map is based on the motion vector, the device of claim 11.

13. Determining the reset weighting map includes determining a first location of an object in the first image frame, and the reset weighting map is based on the first location of the object, the device of claim 9.

14. The device of claim 8, further comprising a first image sensor coupled to the processor.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations, the operations including receiving a first image frame and a second image frame that temporally precedes the first image frame; and determining a third image frame based on the first image frame and the second image frame, the determining including determining to apply temporal filtering to a first plurality of pixels and a second plurality of pixels within a third plurality of pixels within the third image frame; determining the first plurality of pixels using temporal filtering by combining corresponding pixels of the first image frame with the second image frame; and determining a second plurality of pixels based on corresponding pixels of the first image frame without reference to corresponding pixels of the second image frame.

Images