Image signal processor

Abstract

The present disclosure generally relates to image processing by performing, via a machine learning component (302), a first noise reduction operation (302) based on first frame data (350) to generate first processed frame data (360, 380); generating first feedback data (352) based on the first processed frame data (360, 380); and performing, via the machine learning component (302), a second noise reduction operation based on second frame data (350) and the first feedback data (352).

Description

Technical Field

[0001] The present disclosure generally relates to image processing. For example, aspects of the present disclosure include systems and techniques for performing spatial and temporal processing on image data. Background Art

[0002] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). A camera may include a processor (such as an image signal processor (ISP)) that can receive one or more image frames and process one or more image frames. For example, the raw image frames captured by the camera sensor can be processed by the ISP to generate the final image. The camera can be configured with various image capture and image processing settings to change the appearance of the image. Some camera settings such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain are determined and applied before or during the capture of the photo. Other camera settings can configure post-processing of the photo, such as changes in contrast, brightness, saturation, sharpness, levels, curves, or color.

[0003] A conventional image signal processor (ISP) has separate discrete blocks that solve various partitions of an image-based problem space. For example, a typical ISP has discrete functional blocks that each apply a specific operation to the raw camera sensor data to create a final output image. Such functional blocks may include blocks for demosaicing, noise reduction (denoising), color processing, tone mapping, and many other image processing functions. Each of these functional blocks contains many pre-adjustment parameters, resulting in an ISP with a large number (e.g., more than 10,000) of pre-adjustment parameters that must be re-adjusted according to each customer's adjustment preferences. Manual adjustment of such parameters is very time-consuming and expensive, so it is usually performed once. Once adjusted, a conventional ISP typically uses a limited set of adjustment settings to process the image. For example, there may be a set of adjustment settings for processing low-light images, and a second set of adjustment settings for processing bright-light images. For any single image, static adjustment settings are used to process the entire image. Summary of the invention

[0004] Certain aspects provide an apparatus for processing frame data. The apparatus generally includes a memory and one or more processors coupled to the memory, the one or more processors being configured to: perform a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data; generate first feedback data based on the first processed frame data; and perform a second noise reduction operation based on second frame data and the first feedback data via the machine learning component.

[0005] Certain aspects provide a method for processing frame data. The method generally includes: performing a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data; generating first feedback data based on the first processed frame data; and performing a second noise reduction operation based on second frame data and the first feedback data via the machine learning component.

[0006] Certain aspects provide a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, causes the processor to: perform a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data; generate first feedback data based on the first processed frame data; and perform a second noise reduction operation based on second frame data and the first feedback data via the machine learning component. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Exemplary embodiments of the present application are described in detail below with reference to the following drawings, in which:

[0008] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0009] Figure 2 is a block diagram illustrating an example architecture of an imaging system having an image signal processor (ISP) according to some examples;

[0010] Figure 3A , Figure 3B , Figure 3C and Figure 3D is a diagram illustrating an example of a video processing system that uses spatial and temporal information in accordance with certain aspects of the present disclosure.

[0011] Figure 4 is a diagram illustrating an example video processing system implemented by processing downsampled input frames in accordance with certain aspects of the present disclosure.

[0012] Figure 5A , Figure 5B and Figure 5C is a diagram illustrating an example of an ML component that performs video processing using different resolution versions of an input frame in accordance with certain aspects of the present disclosure.

[0013] Figure 6 is a diagram illustrating an example of a multi-frame ML video processing system according to certain aspects of the present disclosure.

[0014] Figure 7 is a diagram illustrating an example of a video processing system that performs ML operations on patches of a frame in accordance with certain aspects of the present disclosure.

[0015] Figure 8is a diagram illustrating example techniques for video processing in accordance with certain aspects of the present disclosure.

[0016] Fig. 9 is a diagram illustrating an example technique for training a network in accordance with certain aspects of the present disclosure.

[0017] Fig.10 is a diagram illustrating an example technique using temporal unrolling of frames of a video with ineffective cropping, in accordance with certain aspects of the present disclosure.

[0018] Fig.11 is a flow chart illustrating the operation of a process for processing image data.

[0019] Fig.12 is a diagram illustrating an example neural network.

[0020] Fig.13 is a diagram illustrating an example of a computer system. DETAILED DESCRIPTION

[0021] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0022] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0023] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop (aperture number), shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to an image sensor used to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, level, curve or color. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) used to process one or more image frames captured by an image sensor.

[0024] An image capture device captures an image by receiving light from a scene using an image sensor having an array of photodiodes. An image signal processor (ISP) then processes the image data captured by the image sensor's photodiodes into an image that can be stored and viewed by a user. How the scene is depicted in the image depends in part on capture settings that control how much light the image sensor receives, such as exposure time settings and aperture size settings. How the scene is depicted in the image also depends on how the ISP is adjusted to process the photodiode data captured by the image sensor into an image.

[0025] The ISP may be designed to use one or more trained machine learning (ML) models (e.g., trained neural networks (NNs) and / or other trained ML models). For example, a fully ML-based ISP may pass image data into one or more neural networks (or other trained ML models) that may output images that may be stored and viewed by a user. An ML-based ISP may be more customizable than a pre-conditioned ISP. For example, an ML-based ISP may process different images in different ways, for example, based on different scenes depicted in different images. However, due to the large number of components required to perform fast image processing, a fully ML-based ISP may also require a larger silicon area than a pre-conditioned ISP. A fully ML-based ISP also requires heavy computing resources because many pixels need to be processed. Therefore, a fully ML-based ISP may make full use of limited battery life and computing resources. Therefore, the use of a fully ML-based ISP by devices (such as mobile devices) with limited battery life and computing resources may result in a shortening of the already limited battery life of these devices, a slowdown in the computing power of these devices, etc. Similarly, a fully ML-based ISP may take up a large amount of space in devices such as mobile devices that use limited space for internal electronics. In some cases, a fully ML-based ISP may be inefficient for certain ISP tasks, for example, a convolutional neural network (CNN) may not be as efficient as a pre-conditioning ISP component in performing tone adjustment.

[0026] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for providing temporal and spatial video processing. For example, an ML component of the system may be implemented via a feedback path, allowing a processed version of a first frame of video to be used in processing of a second, subsequent frame of video. The processed frame may be warped to align features of the processed frame with features of the second, subsequent frame, allowing the ML component to perform image processing (e.g., denoising or other processing) on ​​the second frame based on the processed frame received via the feedback path. In some aspects, the spatial and temporal processing via the ML component may be implemented in conjunction with another suitable image filter. For example, the output of the ML component may be provided to an image filter for further processing, as described in more detail herein.

[0027] Various aspects of the application will be described with respect to the accompanying drawings. Figure 11 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing an image of a scene (e.g., an image of a scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or can capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0028] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include control mechanisms other than those illustrated and additional control mechanisms, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0029] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, an additional lens may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0030] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0031] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, the focusing lens may be the lens 115) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. An afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0032] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green filters. Some image sensors may have no color filters at all, and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to light of different wavelengths. Monochrome image sensors may also lack color filters and, therefore, lack color depth.

[0033] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier to amplify the analog signal output by the photodiode and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or the analog signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed for one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0034] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processors 1310 discussed for computing system 1300. Host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM , global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0035] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in a random access memory (RAM) 140 / 5020, a read-only memory (ROM) 145 / 5025, a cache, a memory unit, another storage device, or some combination thereof.

[0036] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1335, any other input device 1345, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O devices 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0037] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0038] like Figure 1 As shown, the vertical dotted line will Figure 1 The image capture and processing system 100 is divided into two parts, representing an image capture device 105A and an image processing device 105B. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O device 160. In some cases, some components illustrated in the image capture device 105A (such as the ISP 154 and / or the host processor 152) may be included in the image capture device 105A.

[0039] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some specific implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.

[0040] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0041] Figure 2is a block diagram illustrating an example architecture of an imaging system 200 having an image signal processor (ISP) 220 that can implement ML-based spatial and temporal video processing according to some examples. The imaging system 200 includes an image capture device 202 that captures image data 205. The image capture device 202 can be an example of the image capture device 105A. The image capture device 202 can include, for example, an image sensor 130 and / or one or more control mechanisms 120. A representation of the image data 205 is depicted that includes pixels that each have a single color component (e.g., red, green, or blue), where each color component corresponds to a color component of a color filter array (CFA) of the image capture device 202.

[0042] The imaging system 200 may include an ISP 220 for video image processing. Various image and video processing techniques may be used to reduce noise and improve the quality of images and videos. In some implementations, deep learning techniques may be used to select metadata about a video (e.g., optimal exposure, tone, or color) and apply the metadata to improve video output. Deep learning may also be used to perform pixel processing in an attempt to improve video quality. Deep learning may be used to understand the context associated with a video and extract semantic information as input for improving a video using image signal processing.

[0043] Certain aspects of the present disclosure implement machine learning for improving video quality (e.g., noise reduction) based on temporal and spatial information. The ML system provided herein is trained and learns complex statistical relationships associated with spatial and temporal information of a frame. Spatial and temporal information can be processed jointly by the ML system.

[0044] The imaging system 200 may include an ISP 220 having a downsampling engine 224, a spatial and temporal processing engine 225, and an image filter 226. In some aspects, the spatial and temporal processing engine 225 may include a trained machine learning system and a feedback path to facilitate temporal and spatial processing (e.g., denoising). The trained machine learning system may include one or more trained convolutional neural networks (CNNs), one or more CNNs, one or more trained neural networks (NNs), one or more NNs, one or more trained support vector machines (SVMs), one or more SVMs, one or more trained random forests, one or more random forests, one or more trained decision trees, one or more decision trees, one or more trained gradient boosting algorithms, one or more gradient boosting algorithms, one or more trained regression algorithms, one or more regression algorithms, or a combination thereof. In some examples, the trained machine learning system includes one or more trained machine learning models. In some examples, the trained machine learning models include one or more trained neural networks (NNs). Examples of one or more trained NNs include those described herein. Fig.12 Neural network 1200.

[0045] In some aspects, ISP 220 may also include image filter 226. Video processing may be performed using both the trained machine learning model and the image filter 226. ISP may include a downsampling engine 224 that may generate a downsampled version of image data 205 to be processed via a spatial and temporal processing engine 225 and an image filter 226. In some cases, processing of frames of a video may be performed using patches. For example, ISP 220 may include a frame segmentation engine 227 that may segment a frame (e.g., associated with image data 205) into regions or patches to be processed sequentially. As shown, ISP 220 generates an output image 280.

[0046] FIG. 3A to FIG. 3D 3 illustrates a video processing system 300 that uses spatial and temporal information in accordance with certain aspects of the present disclosure. Figure 3A As shown, the video processing system 300 may include an ML component 302 that may receive an input frame (input [0]) for processing at an input 350. Input [0] may be one of a stream of frames of a video (e.g., including input [0], input [1] to input [n], where n is any positive integer), which may be processed to generate output frames (e.g., output [0], output [1] to output [n]), respectively.

[0047] The video processing system 300 may also include a feedback loop 352 from an output of the ML component 302 to another input of the ML component 302. As shown, the feedback loop 352 may include a delay element 306 and a warp component 304. Initially, input [0] may be provided to the ML component 302 for processing. The ML component 302 generates an output frame (output [0]) at output 360, which may be a processed version of input [0].

[0048] The warp component 304 may warp (e.g., shift or adjust) one or more pixels of an output frame (output [0]) at output 360 to align one or more features of the output frame with a subsequent input frame (input [1]) of the video. For example, an image capture device may capture a sequence of frames in succession. From when one frame (e.g., input [0]) is captured until a subsequent frame (e.g., input [1]) is captured, the image capture device or objects in the captured scene may move, resulting in misalignment of features in the captured frames. Such movement may result in local motion corresponding to movement of one or more objects in the scene of the frame being captured or global motion corresponding to movement of the entire scene.

[0049] In order to perform video processing using a sequence of frames, an alignment operation may be performed. For example, the warp component 304 may receive sensor data indicating movement of an image capture device when capturing a frame. Based on the sensor data, the warp component 304 may determine a shift of one or more pixels to be applied to the frame in order to align features of the frame (e.g., the processed input [0] being fed back with input [1]) (e.g., any features such as the edge of a building). In addition, the delay element 306 may synchronize the feedback of the output frame so that the output frame is received at the input of the ML component 302 (e.g., warped via the warp component 304) at approximately the same time as a subsequent input frame (e.g., input [1]). The ML component 302 then processes input [1] based on the feedback of the output frame (e.g., a processed version of output [0]) to generate a subsequent output (output [1]). This process is repeated until all outputs (output [0] to output [n]) are generated.

[0050] In some aspects, input to the ML system is allowed to direct the network capacity toward a specific criterion, such as improving sharpening of a video at the expense of less denoising. Figure 3B As shown, the ML component 302 may also receive one or more indications or preferences associated with the processing of the ML component 302. For example, the ML component 302 may receive an indication 370 to increase sharpness associated with the video at the expense of more noise. As another example, the indication or preference may indicate a tradeoff between noise reduction and preserving details in the image.

[0051] Figure 3C and Figure 3D300 is illustrated using an ML component 302 and a filtering component 308 implemented in accordance with certain aspects of the present disclosure. Figure 3C As shown, the input frame can be provided to both the ML component 302 and the filtering component 308. The filtering component 308 can filter the input frame using any suitable filtering technique. For example, the filtering component 308 can perform a denoising operation on the input frame. The filtered input frame generated by the filtering component 308 can be provided to the ML component 302. In addition, the video processing system 300 can also include a feedback loop from the output of the ML component 302 to the input of the ML component 302 through the warping component 304 and the delay element 306, as described herein.

[0052] The ML component 302 can process (e.g., perform a denoising operation) the input frame to implement spatial and temporal processing based on a filtered version of the input frame as output by the filtering component 308 and feedback data received via the feedback loop 352. In some implementations, the feedback loop 352 can be located between the output of the ML component 302 and the input of the ML component 302.

[0053] In some implementations, a feedback loop 352 can be implemented from the output of the ML component 302 to the input of the filtering component 308, as shown. In this case, the filtering component 308 can use the input frame and the feedback frame warped as via the warping component 304 to perform a filtering operation, thereby providing a filtered frame output processed using temporal and spatial filtering. The ML component 302 then generates a processed output frame at the output 360 based on the input frame and the filtered frame output generated by the filtering component 308.

[0054] like Figure 3D As shown, the input frame may be provided to the ML component 302 and the filtering component 308. A feedback loop 352 having a delay element 306 and a warping component 304 may be implemented from the output of the filtering component 308 to the input of the ML component 302, as shown. The ML component 302 performs temporal and spatial processing on the input frame and generates an output frame at output 360, which may be provided to another input of the filtering component 308, as shown. The filtering component 308 then generates an output frame at output 380 based on the input frame at input 350 and the output of the ML component 302.

[0055] Figure 4An example video processing system 400 is illustrated that is implemented by processing downsampled input frames in accordance with certain aspects of the present disclosure. The video processing system 400 may receive a series of frames of a video (e.g., input [0], input [1] to input [n]) and generate a series of output frames (output [0], output [1] to output [n]). As shown, the video processing system 400 may include a front-end component 402 that may process the series of frames (e.g., input [0], input [1] to input [n]). The video processing system 400 may generate an input frame (e.g., input [1]) at an input 450 and generate a downsampled input frame at an input 452. For example, the front-end component 402 may downsample the input frame using a ratio of 1 to X, where X is any value greater than 1. As a specific example, the input frame may be downsampled using a ratio of 1:2, and thus, the downsampled input frame may have half the resolution of the input frame. The downsampled input frame may be provided to the ML component 302 of the ML engine 404 for processing using a feedback loop including a delay element 306 and a warp component 304, as described herein. The ML component 302 may perform denoising, sharpening, adding blur effects, or any combination thereof on the downsampled input frame. In some aspects, the downsampled input frame may be further processed via a warp component 410 before being provided to the ML component 302.

[0056] The output of the ML component 302 may be provided to a processing component 414 of the processing engine 422, which also receives an input frame (e.g., input [1]) at input 450 (e.g., or receives a processed version thereof via a warp component 418). In some aspects, the processing component 414 may be an ML component (e.g., implemented similarly to the ML component 302) or any suitable filtering component. The processing component 414 may perform denoising, sharpening, adding blur effects, or any combination thereof on the input frame at input 450. A feedback loop with a delay element 460 and a warp component 412 may be implemented, thereby allowing the processing component 414 to implement temporal and spatial processing as described herein. For example, the warp component 412 may warp (e.g., shift or adjust) one or more pixels of an output frame (e.g., output [0] generated based on input [0] before processing input [1]) generated by the processing component 414 to align one or more features of the output frame (e.g., output [0]) with the input frame (e.g., input [1]). The warped output frame is then provided to processing component 414 for use in processing the input frame (e.g., input [1]). Delay element 460 can synchronize the feedback of the output frame of processing component 414 so that the output frame is received at the input of processing component 414 at approximately the same time as the input frame at input 450 (e.g., warped via warping component 304). In some cases, the input frame at input 450 can be further warped via warping component 418 before being provided to processing component 414. In some aspects, the output of processing component 414 can be provided to filtering component 416, which can perform various filtering operations, such as hue adjustment, color adjustment, or additional sharpening.

[0057] Figure 5A , Figure 5B and Figure 5C The ML component 302 is illustrated for performing video processing using different resolution versions of an input frame in accordance with certain aspects of the present disclosure. For example, the ML component 302 can receive an input frame and one or more downsampled versions of the input frame. For example, the ML component 302 can receive an input frame, an input frame downsampled at a ratio of 1:4 (e.g., having one-fourth the resolution of the input frame), an input frame 1:16 (e.g., having one-sixteenth the resolution of the input frame), and so on. The ML component 302 can generate an output frame based on the different resolution versions of the input frame. For example, Figure 5B As shown, the ML component 302 may process the input frame 1:X (e.g., the input frame downsampled at a ratio of 1:X) based on the processing results of the input frame downsampled at a ratio of 1:4X. For example, the input frame downsampled at a ratio of 1:16 may be processed via the ML component 302 and the feedback loop to achieve temporal and spatial processing. The processing results of the input frame downsampled at a ratio of 1:16 may be used to process the input frame downsampled at a ratio of 1:4. For example, Figure 5BAs shown, the processing result of the input frame downsampled at a ratio of 1:4 (eg, where X is in Figure 5B =4) downsampled input frame, thereby generating an ML processing result (e.g., ML output) of the input frame downsampled at a ratio of 1:4. The processing result of the input frame downsampled at a ratio of 1:4 can be used to process the input frame (e.g., the non-downsampled input frame). Figure 5C As shown, the output of the ML component 302 generated using input frames having different resolutions can be provided to the input of the filtering component 308. The filtering component 308 can further process the input frames based on the output of the ML component 302 as described herein.

[0058] Figure 6 A multi-frame ML video processing system 600 according to certain aspects of the present disclosure is illustrated. As shown, ML operations 6021 to 602 B (collectively referred to as ML operations 602), B is any integer greater than one. ML operations 602 may be performed via ML component 302, including a feedback loop to perform temporal and spatial video processing as described herein. Each ML operation is performed on a plurality of frames. For example, ML operation 6021 may be performed on frame n-1, frame n, and frame n+1, where n is any positive integer. ML operation 6021 may be performed to generate output n (e.g., labeled output [n]) based on frame n-1, frame n, and frame n+1. ML operation 6022 may be performed to generate output n+1 (e.g., labeled output [n+1]) based on frame n, frame n+1, and frame n+2. ML operation 602 may be performed B The ML operations 6021 to 6022 may be performed in a sequential manner to generate an output n+B (eg, labeled as output [n+B]) based on the frame n+B-1, the frame n+B, and the frame n+B+1. B Although each ML operation Figure 6 , is shown as being performed on three frames for ease of understanding, but each ML operation may be performed on any suitable number of frames (e.g., two or more frames). Figure 3A , where ML operations are performed on three frames, the ML component 302 can process three input frames (input [0], input [1], and input [2]) as described herein.

[0059] Although the ML operations 602 are described and shown as being performed on various frames, the ML operations may be performed on patches of a frame. For example, a frame may be segmented into patches (e.g., regions), such as patch n-1, patch n, patch n+1 through patch n+B. The ML operations 602 may be performed in patch n-1, patch n, patch n+1 through patch n+B in a similar manner as described with respect to the frame.

[0060] Figure 7 The video processing system 700 that performs ML operations on patches of a frame according to certain aspects of the present disclosure is illustrated. For example, a particular frame may be segmented into a plurality of patches (e.g., B patches, where B is any integer greater than 1). ML operations 7021, 7022 to 7023 may be sequentially performed on each patch. B (collectively referred to as ML operations 702), where the output of the ML operation on one patch is fed forward to facilitate the ML operation of a subsequent patch. For example, ML operation 7021 can be performed on patch n to generate output n (output [n]). The result of ML operation 7021 (e.g., output [n]) can be provided and used by ML operation 7022 to process patch n+1, and so on. By dividing the frame into patches and sequentially performing ML operations 702 on the patches, the size of ML component 302 can be reduced.

[0061] Figure 8 An example technique for video processing 800 according to certain aspects of the present disclosure is illustrated. As shown, a current frame 818 may be provided to a feature extraction component 830, which provides an indication of features associated with the current frame 818 to an ML component 802 (e.g., a convolutional long short-term memory (LSTM) component). The output of the ML component 802 is provided to a convolutional gated recurrent unit (GRU) hybrid component 804.

[0062] In general, the convolutional LSTM component determines the amount of temporal mixing allowed for each pixel, which may depend on the previous frame (e.g., on the state maintained by the convolutional LSTM) because the network can detect static areas in several consecutive frames for robust decisions (e.g., it may otherwise be difficult to distinguish noise from motion). The convolutional GRU mixing component 804 performs the actual mixing between the current frame and the previous frame based on the output of the convolutional LSTM component.

[0063] The output of the convolutional GRU hybrid component 804 is provided to the post-denoiser 808 to generate an output frame 890. As shown, the output of the post-denoiser 808, the convolutional GRU hybrid component 804 and the output of the feature extraction component 830 are provided to the future gate 820. The future gate 820 can be used to extract information from the previous stage to be used as input for the next frame processing (e.g., after warping). For example, the future gate 820 can generate feedback for the feedback loop 352. The future gate 820 generates a frame 824 (e.g., feedback provided via the feedback loop 352), which can be fed back to the feature extraction component 830 and the convolutional GRU hybrid component 804 through the delay element 822, and the corresponding outputs of the feature extraction component 830 and the convolutional GRU hybrid component 804 can be generated on this basis. As shown, in some aspects, the current frame 818 can also be provided to the front denoiser 806. Pre-denoiser 806 may be used to perform initial filtering on new data (eg, a new frame) so that the data is more similar to (or comparable to) the information in frame 824 (eg, because frame 824 is filtered over time and any current frame is noisier).

[0064] The output of the front denoiser 806 may be provided to the rear denoiser 808, which may perform a denoising operation on the output of the convolutional GRU hybrid component 804 based on the output of the front denoiser 806. As shown, the LSTM state 814 may be fed back to the convolutional LSTM 802 via a delay element 812. In some aspects, the current frame 818 may correspond to the input 350, and the frame 824 cascaded with the LSTM state 814 may correspond to the feedback loop 352. In some aspects, although not shown, a warp component may be used to generate the frame 824 and the LSTM state 814. As described, the convolutional LSTM provides an output based on several frames. The LSTM state 814 is updated and used internally by the convolutional LSTM for subsequent decisions.

[0065] Fig. 9Example techniques for training a network 910 (e.g., for implementing the ML component 302) according to certain aspects of the present disclosure are illustrated. A clean image may be used as a basis for instant data generation. For example, a clean image 904 may be provided to a noise adding component 902, which may receive noise and generate a noisy version of the clean image 904. The noisy image may be provided to a trained network 910 (e.g., for generating the ML component 302). The output of the network 910 and the clean image 904 may be used to perform loss calculations and train the network accordingly. Almost noise-free video may be captured using complex motion and occlusion, different light conditions, and different exposures to generate a clean image. Then, the video may be enhanced using artificially added motion (e.g., local motion corresponding to the movement of one or more objects in the scene where the frame is being captured or global motion corresponding to the movement of the entire scene), artificial brightness changes, and artificially added noise to train the network as described. The added noise may be a mixture of Poisson and Gaussian noise. In some aspects, the noise may be applied in the sensor domain. For example, a YUV image may be restored to a linear RAW domain image. Noise can be added to the RAW domain image and converted back to the YUV domain and used as a noisy image to train the network.

[0066] Fig.10 An example technique using temporal expansion frames of a video with invalid cropping in accordance with certain aspects of the present disclosure is illustrated. As described, a video may include a plurality of frames, labeled input [0], input [1] to input [n], where n is any integer greater than 1. As shown, input [0] may be provided to ML component 10020. ML component 10020 may process input [0] and generate an output, which may be warped (e.g., for alignment with input [1]) via warp component 10080 and provided to ML component 10021. ML component 10021 processes input [1] based on the warped output of ML component 10020. Similarly, the output of ML component 10021 may be processed via warp component 10080. n-1 twisted and provided to the ML component 1002 n-1 , the input [n] can be distorted based on this. Therefore, several specific implementations of the same ML model with different inputs are provided and trained together.

[0067] In some aspects, invalid cropping can be implemented. For example, when applying a machine learning operation to an input, a convolution is performed to provide an output that is smaller than the input to the ML component due to cropping of invalid pixels of the input frame. Thus, the size of the output of ML component 10021 may be smaller than the size of the output of ML component 10020, the size of the output of ML component 10022 may be smaller than the size of the output of ML component 10021, and so on. Due to such cropping, the size of input[0] may have to be large to be at the end of the ML component sequence (e.g., at ML component 1002 n Thus, in some aspects, invalid cropping can be implemented to reduce the amount by which the size of the output of each ML component is reduced. For example, less than all invalid pixels can be cropped for each input frame. Thus, cropping can be performed by including some invalid pixels to attempt to reduce the required size of input [0].

[0068] Fig.11 1100 is a flow chart illustrating the operation of process 1100 for processing image data. The operation of process 1100 may be performed by an imaging system. In some examples, the imaging system that performs the operation may be imaging system 200. In some examples, the imaging system that performs the operation of process 1100 may include, for example, one or more components for performing the operation, which may include image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, imaging system 200, ISP 220, or a combination thereof.

[0069] At operation 1105, the imaging system generates a first frame of data (eg, Figure 3A The input [0] shown in FIG. 1 is used to perform a first noise reduction operation to generate first processed frame data (e.g., such as Figure 3A In some aspects, the imaging system may provide an indication to the machine learning component of a tradeoff between noise reduction and one or more other frame characteristics (e.g., level of detail) (e.g., Figure 3B A first noise reduction operation may be performed based on the indication of the tradeoff.

[0070] At operation 1110, the imaging system generates first feedback data based on the first processed frame data (e.g., via feedback loop 352). In some aspects, to generate the first feedback data, the imaging system warps one or more pixels of the first processed frame data (e.g., via warping component 304). The one or more pixels may be warped to align one or more features of the first processed frame data with one or more features of the second frame data.

[0071] At operation 1115, the imaging system performs a second noise reduction operation based on the second frame data (eg, input [1]) and the first feedback data via the machine learning component.

[0072] In some aspects, the imaging system is performed via an image filter (e.g., Figure 3C The imaging system may filter the second frame data based on the first feedback data via an image filter. For example, as shown in FIG. Figure 3C As shown, a feedback loop may be implemented between the output of the ML component 302 and the input of the filtering component 308 .

[0073] In some aspects, the imaging system can be implemented via an image filter (e.g., Figure 3D The filtering component 308 is shown to generate filtered frame data based on the first frame data (eg, input [1]). The second noise reduction operation performed based on the first processed frame data includes a second noise reduction operation performed based on the filtered frame data.

[0074] In some aspects, the imaging system performs a third noise reduction operation based on the first processed frame data and the third frame data via a frame processing component (e.g., processing component 414) to generate second processed frame data. The first frame data (e.g., such as Figure 4 The imaging system may generate second feedback data (e.g., via the processing component 414) based on the second processed frame data (e.g., generated by the processing component 414). Figure 4 The imaging system may perform a fourth noise reduction operation (e.g., via processing component 414) based on the fourth frame data (e.g., input [1]) and the second feedback data. The second frame data (e.g., an input frame such as input [1] downsampled at a ratio of 1:X) may be a downsampled version of the fourth frame data. In some aspects, the imaging system may perform a fourth noise reduction operation (e.g., via processing component 414) based on the fourth frame data (e.g., input [1]) and the second feedback data. Figure 5C The filtering component 308 is shown to generate a filtered output frame based on the second processed frame data and the third frame data.

[0075] In some aspects, the imaging system may perform a second noise reduction operation (e.g., Figure 6 ML operation 6021 shown) to generate a first output frame of the video (e.g., Figure 6 The imaging system may perform a third noise reduction operation based on the second frame data (eg, Figure 6ML operation 6022 shown) to generate second processed frame data, generate second feedback data based on the second processed frame data, and perform a fourth noise reduction operation based on the third frame data and the second feedback data to generate a second output frame of the video (e.g., Figure 6 Output [n+1] shown).

[0076] In some aspects, the first frame data is associated with a first patch of the frame (e.g., Figure 7 The imaging system may perform a third noise reduction operation (e.g., Figure 7 ML operation 7022 shown).

[0077] In some examples, the processes described herein (e.g., process 1100 and / or other processes described herein) can be performed by a computing device or apparatus. In some examples, process 1100 can be performed by imaging system 200. In some examples, process 1100 can be performed by a computer having Fig.13 The computing device of the computing system 1300 shown in FIG. Fig.13 The computing device of the illustrated computing system 1300 may include at least some components of the imaging system 200 and / or may implement Fig.11 Operations of process 1100.

[0078] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR head-mounted device, an AR head-mounted device, an AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0079] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0080] The process 1100 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0081] In addition, process 1100 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0082] Fig.12 is a block diagram illustrating an example of a neural network that may be used by a trained machine learning system that generates settings for use by an image signal processor (ISP) according to some examples. The neural network 1200 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks.

[0083] The input layer 1210 of the neural network 1200 includes input data. The input data of the input layer 1210 may include data representing pixels of an input image frame. In an illustrative example, the input data of the input layer 1210 may include data representing pixels of image data and / or metadata corresponding to the image data. The image may include image data from an image sensor, including raw pixel data (including a single color per pixel based on, for example, a Bayer color filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). The neural network 1200 includes a plurality of hidden layers 1212a, 1212b to 1212n. The hidden layers 1212a, 1212b to 1212n include "n" hidden layers, where "n" is an integer greater than or equal to one. The plurality of hidden layers may include as many layers as required for a given application. The neural network 1200 also includes an output layer 1214 that provides an output resulting from the processing performed by the hidden layers 1212a, 1212b to 1212n. In some examples, output layer 1214 may provide one or more settings.

[0084] Neural network 1200 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. In some cases, neural network 1200 may include a feedforward network in which there are no feedback connections in which the output of the network is fed back to itself. In some cases, network 1200 may include a recurrent neural network that may have loops that allow information to be carried across nodes when reading in input.

[0085] In some cases, information can be exchanged between layers by node-to-node interconnection between each layer. In some cases, the network may include a convolutional neural network, which may not link each node in a layer to each other node in the next layer. In a network in which information is exchanged between layers, the nodes of the input layer 1210 may activate a set of nodes in the first hidden layer 1212a. For example, as shown in the figure, each input node of the input layer 1210 may be connected to each node of the first hidden layer 1212a. The nodes of the hidden layer may transform the information by applying an activation function (e.g., a filter) to the information of each input node. The information derived from the transformation may then be passed to the nodes of the next hidden layer 1212b and the nodes of the next hidden layer may be activated, and the nodes of the next hidden layer may perform their own specified functions. Example functions include convolution functions, reduction, enlargement, data transformation and / or any other suitable functions. Then, the output of the hidden layer 1212b may activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 1212n may activate one or more nodes of the output layer 1214, which provide the processed output image. In some cases, although a node in neural network 1200 (e.g., node 1216) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0086] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 1200. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnections may have adjustable digital weights that may be set (e.g., based on a training data set), thereby allowing neural network 1200 to adapt to the input and be able to learn as more and more data is processed.

[0087] The neural network 1200 is pre-trained to process features of data from the input layer 1210 using different hidden layers 1212a, 1212b to 1212n to provide an output through the output layer 1214.

[0088] Fig.13 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.13 An example of a computing system 1300 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1305. The connection 1305 may be a physical connection using a bus, or a direct connection into the processor 1310, such as in a chipset architecture. The connection 1305 may also be a virtual connection, a networked connection, or a logical connection.

[0089] In some embodiments, computing system 1300 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.

[0090] The example system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components including system memory 1315, such as read only memory (ROM) 1320 and random access memory (RAM) 1325, to the processor 1310. The computing system 1300 may include a cache 1312 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1310.

[0091] Processor 1310 may include any general purpose processor and hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, that are configured to control processor 1310 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0092] To enable user interaction, the computing system 1300 includes an input device 1345 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1300 may also include an output device 1335 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1300. The computing system 1300 may include a communication interface 1340, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transfer, Low Energy (BLE) wireless signal transfer, Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 1340 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1300 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0093] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0094] Storage device 1330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1310, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1310, connector 1305, output device 1335, etc.

[0095] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, category, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0096] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0097] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments can be put into practice without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these embodiments in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0098] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0099] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. These instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or function group. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0100] The equipment implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0101] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0102] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used for any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0103] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0104] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0105] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0106] Claim language or other language that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0107] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed to the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0108] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. Computer-readable data storage media can form a part of a computer program product, which can include packaging materials. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0109] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0110] Illustrative aspects of the present disclosure include:

[0111] Aspect 1. A device for processing frame data, the device comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: perform a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data; generate first feedback data based on the first processed frame data; and perform a second noise reduction operation based on second frame data and the first feedback data via the machine learning component.

[0112] Aspect 2. The apparatus according to aspect 1, wherein, to generate the first feedback data, the one or more processors are configured to distort one or more pixels of the first processed frame data.

[0113] Aspect 3. An apparatus according to Aspect 2, wherein the one or more processors are configured to warp the one or more pixels to align one or more features of the first processed frame data with one or more features of the second frame data.

[0114] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein the one or more processors are further configured to provide an indication of a tradeoff between noise reduction and one or more other frame characteristics to the machine learning component, wherein the first noise reduction operation is performed based on the indication of the tradeoff.

[0115] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the one or more processors are further configured to filter the second frame data via an image filter to generate filtered frame data, and wherein the second noise reduction operation is further performed based on the filtered frame data.

[0116] Aspect 6. The apparatus according to aspect 5, wherein the one or more processors are configured to filter the second frame data based on the first feedback data via the image filter.

[0117] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the one or more processors are further configured to generate filtered frame data based on the first frame data via an image filter, wherein the second noise reduction operation performed based on the first processed frame data includes the second noise reduction operation performed based on the filtered frame data.

[0118] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the one or more processors are further configured to perform a third noise reduction operation based on the first processed frame data and the third frame data via a frame processing component to generate second processed frame data, wherein the first frame data includes a downsampled version of the third frame data.

[0119] Aspect 9. An apparatus according to aspect 8, wherein the third noise reduction operation is performed using machine learning.

[0120] Aspect 10. An apparatus according to any one of Aspects 8 to 9, wherein the one or more processors are further configured to: generate second feedback data based on the second processed frame data; and perform a fourth noise reduction operation based on fourth frame data and the second feedback data, wherein the second frame data includes a downsampled version of the fourth frame data.

[0121] Aspect 11. An apparatus according to any one of aspects 8 to 10, wherein the one or more processors are further configured to generate a filtered output frame based on the second processed frame data and the third frame data via an image filter.

[0122] Aspect 12. An apparatus according to any one of Aspects 1 to 11, wherein the one or more processors are configured to perform the second noise reduction operation to generate a first output frame of a video, and wherein the one or more processors are further configured to: perform a third noise reduction operation based on the second frame data to generate second processed frame data; generate second feedback data based on the second processed frame data; and perform a fourth noise reduction operation based on the third frame data and the second feedback data to generate a second output frame of the video.

[0123] Aspect 13. An apparatus according to any one of aspects 1 to 12, wherein: the first frame data is associated with a first patch of a frame; and the one or more processors are further configured to perform a third noise reduction operation based on a second patch of the frame and the first processed frame data generated using the first patch of the frame. Aspect 14. A method for processing frame data, the method comprising: performing a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data; generating first feedback data based on the first processed frame data; and performing a second noise reduction operation based on second frame data and the first feedback data via the machine learning component.

[0124] Clause 15. The method according to clause 14, wherein generating the first feedback data comprises distorting one or more pixels of the first processed frame data.

[0125] Clause 16. The method of clause 15, wherein the one or more pixels are warped to align one or more features of the first processed frame data with one or more features of the second frame data.

[0126] Aspect 17. A method according to any one of Aspects 14 to 16, the method also includes providing an indication of a tradeoff between noise reduction and one or more other frame characteristics to the machine learning component, wherein the first noise reduction operation is performed based on the indication of the tradeoff.

[0127] Aspect 18. A method according to any one of Aspects 14 to 17, the method further comprising filtering the second frame data via an image filter to generate filtered frame data, and wherein the second noise reduction operation is further performed based on the filtered frame data.

[0128] Aspect 19. The method according to aspect 18, wherein the second frame data is filtered based on the first feedback data via the image filter.

[0129] Aspect 20. A method according to any one of Aspects 14 to 19, the method further comprising generating filtered frame data based on the first frame data via an image filter, wherein the second noise reduction operation performed based on the first processed frame data comprises the second noise reduction operation performed based on the filtered frame data.

[0130] Aspect 21. According to any one of Aspects 14 to 20, the method also includes performing a third noise reduction operation based on the first processed frame data and the third frame data via a frame processing component to generate second processed frame data, wherein the first frame data includes a downsampled version of the third frame data.

[0131] Aspect 22. The method according to aspect 21, wherein the third noise reduction operation is performed using machine learning.

[0132] Aspect 23. The method according to any one of aspects 21 to 22, wherein the method further comprises: generating second feedback data based on the second processed frame data; and performing a fourth noise reduction operation based on fourth frame data and the second feedback data, wherein the second frame data comprises a downsampled version of the fourth frame data. Aspect 24. The method according to any one of aspects 21 to 23, wherein the method further comprises generating a filtered output frame based on the second processed frame data and the third frame data via an image filter.

[0133] Clause 25. A method according to any one of clauses 14 to 24, wherein the second noise reduction operation is performed to generate a first output frame of video, and wherein the method further comprises:

[0134] performing a third noise reduction operation based on the second frame data to generate second processed frame data; generating second feedback data based on the second processed frame data; and performing a fourth noise reduction operation based on the third frame data and the second feedback data to generate a second output frame of the video.

[0135] Aspect 26. A method according to any one of Aspects 14 to 25, wherein: the first frame data is associated with a first patch of a frame; and the method also includes performing a third noise reduction operation based on a second patch of the frame and the first processed frame data generated using the first patch of the frame.

[0136] Aspect 27. A computer-readable medium comprising at least one instruction for causing a computer or a processor to perform the operations according to any one of Aspects 1 to 26.

[0137] Aspect 28. An apparatus for processing frame data, the apparatus comprising means for performing the operations according to any one of aspects 1 to 26.

[0138] Aspect 29. An apparatus for processing frame data. The apparatus comprises at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to perform the operations according to any one of aspects 1 to 26.

Claims

1. A device for processing frame data, the device comprising: Memory; and one or more processors coupled to the memory, the one or more processors configured to: performing a first noise reduction operation based on the first frame data via a machine learning component, to generate first processed frame data; generating first feedback data based on the first processed frame data; as well as A second noise reduction operation is performed based on the second frame data and the first feedback data via the machine learning component.

2. The device according to claim 1, wherein: To generate the first feedback data, the one or more processors are configured to warp one or more pixels of the first processed frame data.

3. The apparatus of claim 2, wherein the one or more processors are configured to warp the one or more pixels to align one or more features of the first processed frame data with one or more features of the second frame data.

4. The apparatus of claim 1, wherein the one or more processors are further configured to filter the second frame data via an image filter to generate filtered frame data, and wherein the second noise reduction operation is further performed based on the filtered frame data. 5 . The apparatus of claim 4 , wherein the one or more processors are configured to filter the second frame data based on the first feedback data via the image filter.

6. The apparatus of claim 1 , wherein the one or more processors are further configured to generate filtered frame data based on the first frame data via an image filter, wherein the second noise reduction operation performed based on the first processed frame data includes the second noise reduction operation performed based on the filtered frame data.

7. The apparatus of claim 1, wherein the one or more processors are further configured to perform a third noise reduction operation based on the first processed frame data and the third frame data via a frame processing component to generate second processed frame data, wherein the first frame data includes a downsampled version of the third frame data.

8. The apparatus of claim 7, wherein the third noise reduction operation is performed using machine learning.

9. The apparatus of claim 7, wherein the one or more processors are further configured to: generating second feedback data based on the second processed frame data; and A fourth noise reduction operation is performed based on fourth frame data and the second feedback data, wherein the second frame data includes a down-sampled version of the fourth frame data.

10. The apparatus of claim 7, wherein the one or more processors are further configured to generate a filtered output frame based on the second processed frame data and the third frame data via an image filter.

11. The apparatus of claim 1 , wherein the one or more processors are configured to perform the second noise reduction operation to generate a first output frame of video, and wherein the one or more processors are further configured to: performing a third noise reduction operation based on the second frame data to generate second processed frame data; generating second feedback data based on the second processed frame data; as well as A fourth noise reduction operation is performed based on the third frame data and the second feedback data to generate a second output frame of the video.

12. The device according to claim 1, wherein: The first frame data is associated with a first patch of a frame; and The one or more processors are further configured to perform a third noise reduction operation based on a second patch of the frame and the first processed frame data generated using the first patch of the frame.

13. A method for processing frame data, the method comprising: performing, via the machine learning component, a first noise reduction operation based on the first frame data to generate first processed frame data; generating first feedback data based on the first processed frame data; as well as A second noise reduction operation is performed based on the second frame data and the first feedback data via the machine learning component.

14. The method of claim 13, wherein generating the first feedback data comprises distorting one or more pixels of the first processed frame data.

15. The method of claim 14, wherein the one or more pixels are warped to align one or more features of the first processed frame data with one or more features of the second frame data.

16. The method of claim 13, further comprising filtering the second frame data via an image filter to generate filtered frame data, and wherein the second noise reduction operation is further performed based on the filtered frame data. 17 . The method according to claim 16 , wherein the second frame data is filtered based on the first feedback data via the image filter.

18. The method of claim 13, further comprising generating filtered frame data based on the first frame data via an image filter, wherein the second noise reduction operation performed based on the first processed frame data comprises the second noise reduction operation performed based on the filtered frame data.

19. The method of claim 13, further comprising performing, via a frame processing component, a third noise reduction operation based on the first processed frame data and third frame data to generate second processed frame data, wherein the first frame data comprises a downsampled version of the third frame data.

20. The method of claim 19, wherein the third noise reduction operation is performed using machine learning.

21. The method according to claim 19, wherein the method further comprises: generating second feedback data based on the second processed frame data; as well as A fourth noise reduction operation is performed based on fourth frame data and the second feedback data, wherein the second frame data includes a down-sampled version of the fourth frame data.

22. The method of claim 19, wherein the method further comprises generating a filtered output frame based on the second processed frame data and the third frame data via an image filter.

23. The method of claim 13, wherein the second noise reduction operation is performed to generate a first output frame of video, and wherein the method further comprises: performing a third noise reduction operation based on the second frame data to generate second processed frame data; generating second feedback data based on the second processed frame data; as well as A fourth noise reduction operation is performed based on the third frame data and the second feedback data to generate a second output frame of the video.

24. The method of claim 13, wherein: The first frame data is associated with a first patch of a frame; and The method also includes performing a third noise reduction operation based on a second patch of the frame and the first processed frame data generated using the first patch of the frame.

25. A non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to: performing, via the machine learning component, a first noise reduction operation based on the first frame data to generate first processed frame data; generating first feedback data based on the first processed frame data; as well as A second noise reduction operation is performed based on the second frame data and the first feedback data via the machine learning component.

Images