Removing spatially varying noise from images using diffusion

Abstract

An embodiment relates to a computer-implemented method for denoising a noisy image. The method includes obtaining a noisy input image and estimating a time map (214) based on the noisy input image and noise in the noisy input image, where the time map is spatially varying, and the method includes performing a diffusion process on the noisy input image over two or more time steps. At each time step, the method includes estimating a denoised image based on the noisy input image, a current time-step image, and a time step using a machine-learned model; obtaining a next time-step image based on the denoised image and the current time-step image; updating, for each pixel, the next time-step image with a corresponding value from the denoised image; setting the next time-step image as the current time-step image; and decrementing the time step. The estimating, obtaining, updating, setting, and decrementing are repeated while the time step is positive. The method further includes outputting the denoised image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 507,194, entitled "SPATIALLY VARIANT NOISE REMOVAL FROM IMAGES USING DIFFUSION," filed June 9, 2023, the contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] Photographs, videos, and other images captured using a camera can contain noise and / or lack detail due to various factors, such as limitations of the camera's image capture sensor, the lighting conditions under which the image is captured, etc. Computer-based noise removal from captured images can improve the user-perceived quality of the image and restore fine detail.

[0003] The discussion of the background art provided herein is for the purpose of generally presenting the context for the present disclosure. The work of the presently named inventors, to the extent described in this background art section, as well as aspects of the present disclosure that may not specifically qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure. Summary of the Invention

[0004] Some embodiments described herein relate to methods, devices, and computer-readable media for removing noise from images using a machine-learned model trained to perform spatially varying denoising. Some embodiments described herein relate to methods, devices, and computer-readable media for training a machine-learned model to perform spatially varying denoising.

[0005] Some embodiments relate to a computer-implemented method for denoising an image. In some embodiments, the computer-implemented method for denoising a noisy input image includes obtaining a noisy input image including a plurality of pixels. The method further includes estimating a time map based on the noisy input image and noise in the noisy input image, the time map spatially varying such that a respective time step value is associated with each pixel of the noisy input image, and at least two pixels have different time step values. The method further includes performing a diffusion process on the noisy input image, the diffusion process being performed over two or more time steps starting from the highest value in the time map. The diffusion process includes, at each time step, estimating a denoised image based on the noisy input image, the current time step image, and the time step using a machine-learned model (SVNR); obtaining a next time step image based on the denoised image and the current time step image; for each pixel, if the time step is greater than or equal to the time step value associated with the pixel, updating the next time step image with a corresponding value from the denoised image; setting the next time step image as the current time step image after updating; and decrementing the time step. The estimating, obtaining, updating, setting, and decrementing are repeated while the time step is positive. The method further includes outputting the denoised image.

[0006] In some implementations, the method may further include determining noise of the noisy input image. In these implementations, each pixel has a respective noise level, and determining the noise comprises determining lead noise and shot noise from the noisy input image. In these implementations, the time map encodes the spatially varying noise of the noisy input image.

[0007] In some embodiments, during the diffusion process, the noisy input image is used as the current time-step image at a first time-step. In some embodiments, the noisy input image serves as a condition image during the diffusion process.

[0008] Some embodiments relate to a method for training a machine learning model to denoise a noisy input image. In some implementations, the method for training a machine learning model to denoise a noisy input image includes obtaining a clean image. The method further includes generating a noisy input image by adding noise to the clean image, where the noise is determined based on a pixel-by-pixel standard deviation of the noise. The method further includes estimating a time map based on the noisy input image and the noise of the noisy input image. In some embodiments, the time map is spatially varying such that a respective time step value is associated with each pixel of the noisy input image, and at least two pixels have different time step values. The method further includes generating a new time map, where each time step value of a pixel in the new time map is less than the timestamp value of a corresponding pixel in the time map. The method further includes generating, based on the new time map, a current time step image having less noise than the noisy input image. The method further includes estimating a denoised image using the machine learning model based on the noisy input image, the current time step image, and the new time map. Additionally, the method includes calculating a loss value based on the comparison of the denoised image and the clean image. The method further includes updating one or more parameters of the machine learning model based on the loss value.

[0009] In some embodiments, the standard deviation of the noise is a combination of the read noise and the shot noise. In these embodiments, the read noise is the standard deviation for the signal-independent noise and the shot noise is the standard deviation for the signal-dependent noise. In some embodiments, the read noise and the shot noise are randomly selected.

[0010] In some implementations, generating the new time map comprises subtracting a scalar from a respective time step value for each pixel in the time map, the scalar having a value between zero and the maximum time step value in the time map.

[0011] In some implementations, generating the current timestep image based on the new time map comprises sampling noise based on the new time map, hi some implementations, the current timestep image has a noise distribution similar to the noise distribution of the noisy input image.

[0012] In some embodiments, the machine learning model includes a neural network. In these embodiments, updating one or more parameters of the machine learning model based on the loss value includes performing backpropagation to adjust weights of one or more nodes of the neural network.

[0013] In some embodiments, obtaining a clean image, generating a noisy input image, estimating a time map, generating a new time map, generating a current time step image, estimating a denoised image, calculating a loss value, and updating one or more parameters of the machine learning model are performed iteratively until a stopping criterion is met. In some embodiments, the stopping criterion includes one or more of: a computational budget for training being exhausted; or a change in the parameter value of at least one of the one or more parameters between successive iterations being below a threshold.

[0014] Some embodiments include a computing device comprising a processor and a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform any of the methods described herein.

[0015] Some embodiments include a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform any of the methods described herein. [Brief explanation of the drawings]

[0016] [Figure 1] 1 illustrates a block diagram of an exemplary network environment that may be used in some implementations described herein. [Figure 2] FIG. 1 is a flow diagram illustrating an exemplary method for denoising a noisy input image, according to some implementations. [Figure 3] FIG. 1 is a flow diagram illustrating an exemplary method for performing a diffusion process, according to some embodiments. [Figure 4A] 1 shows an exemplary image. [Figure 4B] 1 shows an exemplary image. [Figure 5] An example of a noisy image is shown along with the corresponding denoised and ground truth images. [Figure 6] FIG. 1 is a flow diagram illustrating an example method for training a machine learning model to denoise a noisy input image, according to some implementations. [Figure 7] 1 is a block diagram of an example device that may be used to implement one or more functions described herein. DETAILED DESCRIPTION OF THE INVENTION

[0017] Described herein are methods, systems, and non-transitory computer-readable media for denoising noisy input images. The described techniques provide a new diffusion-based framework for the task of image denoising. The techniques leverage natural and rich images previously trained by a generative denoising diffusion model. In some implementations, the techniques perform denoising diffusion by utilizing noisy input images as both conditions and starting points for the diffusion process. To enable the integration of realistic noisy images as samples in the diffusion process, the techniques employ a denoising diffusion formulation that utilizes spatially varying time embedding (time maps). Described herein are techniques for training machine learning models to perform denoising using spatially varying time maps, and techniques for utilizing the machine learning models to remove noise from images. The described techniques can be applied to denoising any noisy image, including images with non-uniform noise distribution.

[0018] 1 illustrates a block diagram of an exemplary network environment 100 that may be used in some implementations described herein. In some implementations, the network environment 100 includes one or more server systems, such as server system 102 in the example of FIG. 1. Server system 102 may be in communication with a network 130, for example. Server system 102 may include a server device 104 and a database 106 or other storage device. In some implementations, server device 104 may host an image application 156a.

[0019] Network environment 100 may also include one or more client devices, e.g., client devices 120, 122, 124, and 126, which may communicate with each other and / or with server system 102 via network 130. Network 130 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switched or hub connection, etc. In some implementations, network 130 may include peer-to-peer communication between devices, for example, using a peer-to-peer wireless protocol (e.g., Bluetooth, Wi-Fi Direct, etc.). An example of peer-to-peer communication between two client devices 120 and 122 is indicated by arrow 132.

[0020] For ease of explanation, FIG. 1 shows one block of server system 102, server device 104, and database 106, as well as four blocks of client devices 120, 122, 124, and 126. Server blocks 102, 104, and 106 may represent multiple systems, server devices, and network databases, and the blocks may be provided in different configurations than those shown. For example, server system 102 may represent multiple server systems that can communicate with other server systems over network 130. In some implementations, server system 102 may include, for example, a cloud-hosted server. In some examples, database 106 and / or other storage devices may be provided in server system block(s) that are separate from server device 104 and that can communicate with server device 104 and other server systems over network 130.

[0021] Additionally, there may be any number of client devices. Each client device may be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smartphone, a standalone camera, a tablet computer, a television, a TV set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a watch, a headset, an armband, a brace, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may have a local database similar to database 106 or other storage. In some implementations, network environment 100 may have less than all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to the components described herein.

[0022] In various implementations, end users U1, U2, U3, and U4 may communicate with server system 102 and / or with each other using respective client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 may interact through applications running on their respective client devices and / or server system 102 and / or through network services, such as social networking services or other types of network services, implemented on server system 102. For example, each client device 120, 122, 124, and 126 may communicate data with one or more server systems (e.g., system 102).

[0023] In some implementations, server system 102 may provide appropriate data to each client device so that the client device can receive communicated or shared content uploaded to server system 102 and / or the network service. In some examples, users U1-U4 may interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0024] The network services implemented by the server system 102 may include systems that enable users to conduct various communications, form links and associations, upload and post shared content such as images, text, video, audio, and other types of content, and / or perform other functions. For example, a client device may display received data, such as content posts, that are transmitted or streamed to the client device and that originate from different client devices (or directly from different client devices) via a server and / or network service, or that originate from a server system and / or network service. In some implementations, client devices may communicate directly with each other, for example, using peer-to-peer communication between client devices as described above. In some implementations, a "user" may include one or more programs or virtual entities as well as people interfacing with a system or network.

[0025] In some implementations, any of client devices 120, 122, 124, and / or 126 may host one or more applications. For example, as shown in Figure 1, client device 120 may host application 156b. Client devices 122-126 may also host similar applications.

[0026] In some implementations, the client device may include applications that provide various types of functionality, such as calendars, address books, email, web browsers, shopping, transportation (e.g., taxi, train, flight reservations, etc.), entertainment (e.g., music players, video players, gaming applications, etc.), social networking (e.g., messaging or chat, audio / video calls, image / video sharing, etc.), etc. In some implementations, one or more of the applications may be standalone applications that execute on the client device 120. In some implementations, one or more of the applications may access a server system, e.g., server system 102, that provides the data and / or functionality of the application.

[0027] User interfaces on client devices 120, 122, 124, and / or 126 may enable the display of content such as images, video, data, and other content, as well as user content including data such as communications, privacy settings, notifications, and the like. Such user interfaces may be displayed using software on the client device, software on the server device, and / or a combination of client and server software running on server device 104, e.g., application software or client software in communication with server system 102. The user interfaces may be displayed by a display device of the client or server device, e.g., a touchscreen or other display screen, a projector, etc. In some implementations, application programs running on the server system may communicate with the client devices to receive user input at the client devices and to output data, such as video data, audio data, etc., at the client devices.

[0028] In various implementations, any of the client devices 120-126 may be used by a user and / or a customer service agent (e.g., a human customer service agent). In some implementations, the application 156 on the server system 102 and / or one or more of the client devices (120-126) may denoise an image by using a machine-learned model that performs spatially-varying denoising. In some implementations, the application 156 on the server system 102 and / or the application 156 on one or more of the client devices (120-126) may train a machine-learning model to perform spatially-varying denoising.

[0029] In some implementations, database 106 can store content items, such as ground truth images (clean images), noisy images, or other images. In various implementations, the content items can include help articles containing text, audio, video, images, or any combination thereof. Users of client devices 120-126 can access the content items via network 130.

[0030] Other implementations of the features described herein may use any type of system and / or service. For example, other networked services (e.g., internet-connected services) may be used instead of or in addition to social networking services. Any type of electronic device may utilize the features described herein. Some implementations may provide one or more features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device may display content posts stored on a storage device local to the client device, e.g., content posts previously received over a communications network.

[0031] Images, as referred to herein, may include digital images having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). Images may include still images (e.g., still photographs, images including a single frame, etc.), dynamic images (e.g., animations, animated GIFs, cinemagraphs in which some portions of the image include movement and other portions are static, etc.), or video (e.g., a sequence of images or image frames that may include audio). While the remainder of this specification will refer to images as static images, it will be understood that the techniques described herein are applicable to dynamic images, video, etc. For example, implementations described herein may be used with still images (e.g., photographs or other images), video, or dynamic images.

[0032] FIG. 2 is a flow diagram illustrating an exemplary method 200 for denoising a noisy input image, according to some implementations. In some implementations, method 200 may be implemented, for example, in server system 102 as shown in FIG. 1. In some implementations, some or all of method 200 may be implemented in one or more client devices 120, 122, 124, or 126 shown in FIG. 1, one or more server devices, and / or both the server device(s) and the client device(s). In the described example, the implementation system includes one or more digital processors or processing circuits (“processors”) and one or more storage devices (e.g., database 106 or other storage). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 200. In some examples, a first device is described as performing blocks of method 200. Some implementations may have one or more blocks of method 200 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0033] In some embodiments, method 200, or portions thereof, may be initiated automatically by a system. In some embodiments, the implementing system is the first device. For example, the method (or portions thereof) may be performed periodically or based on one or more specific events or conditions, such as an image being captured by a camera, an image being added to an image library, an image being received (e.g., from another device, via a scanner, etc.) or viewed (e.g., in an image viewing application), a predetermined amount of time having passed since the last execution of method 200, and / or the occurrence of one or more other conditions that may be specified in settings read by the method.

[0034] Method 200 may begin at block 202. At block 202, it is checked whether user consent (e.g., user permission) has been obtained to use user data in an implementation of method 200. For example, the user data may include, for example, one or more images captured or viewed by a user using a client device (e.g., any of devices 120-126). One or more blocks of the methods described herein may use such user data in some implementations.

[0035] If user consent is obtained from an associated user whose user data may be used in method 200, then at block 204 it is determined that the blocks of the method herein may be implemented using user data, as described for those blocks, and the method proceeds to block 210. If user consent is not obtained, then at block 206 it is determined that the blocks should be implemented without using user data, and the method proceeds to block 210. In some embodiments, if user consent is not obtained, the blocks are implemented without using user data, but using synthetic data and / or general-purpose or publicly accessible and publicly usable data. In some embodiments, if user consent is not obtained, the remainder of method 200 is not performed.

[0036] At block 212, a noisy input image is obtained. The noisy input image includes a plurality of pixels, each pixel having an associated pixel value, such as values ​​for each of the red, green, and blue (RGB) channels of an RGB image, or additional values. In some implementations, the pixel values ​​of the noisy input image may include luminance values, depth values, or other values.

[0037] For example, the noisy input image may be captured by a camera on a client device (any of client devices 120-126) or a scanner on the client device, received from another device (e.g., received from another client device over a network or via a local interface if method 200 is implemented on a server or client device), received from a hardware scanner, received from a camera device (e.g., a security camera or other camera), etc. In some implementations, the noisy input image may be captured in low-light conditions. The noise of the noisy input image may include shot noise, which depends on the total amount of light captured (e.g., based on exposure time) and is therefore signal-dependent. The noise of the noisy input image may also include lead noise introduced by the sensor used to capture the noisy input image, such as a complementary metal-oxide semiconductor (CMOS) sensor, a charge-coupled device (CCD) sensor, or other type of sensor used to capture the digital image. Lead noise is sensor-dependent and therefore signal-independent, and therefore does not depend on the amount of light captured. Block 212 may be followed by block 214.

[0038] At block 214, a time map of the noisy input image is estimated based on the noisy input image and the noise of the input image. In some implementations, each pixel p of the noisy input image may be associated with a respective noise level. A corresponding time step value may result in this noise level during the diffusion process. The maximum noise level for the image (e.g., three channels R, G, and B) may define the time map. In other words, each pixel of the noisy input image may be at a respective stage of the diffusion process.

[0039] In some implementations, the time map is a spatially varying time map, and thus the time map includes a respective time step value (e.g., including a time embedding) for each pixel of the noisy input image. In some implementations, at least two pixels of the noisy input image may have different time step values. In some implementations, the time step value may be utilized to apply an affine transformation to each pixel, independent of the transformation applied to other pixels of the noisy input image. In some implementations, each pixel is associated with a respective time value that is transformed into a time embedding (vector). In these implementations, the vector of every pixel is multiplied by a matrix (which is an affine transformation). The time step value of the noisy input image may be based on noise.

[0040] In some implementations, the noise of the noisy input image may be determined before estimating the time map. In these implementations, determining the noise level may involve subtracting the lead noise (σ) from the noisy input image. r ), and shot noise (σ s In these implementations, the read noise is the standard deviation of the signal-independent noise (e.g., based on the sensor characteristics of the sensor that captured the noisy input image), and the shot noise is the standard deviation of the signal-dependent noise (e.g., affected by conditions such as the total amount of light captured by the sensor during capture of the noisy input image).

[0041] In some implementations, a clipped version of the noisy input image may be used to estimate the noise. Block 214 may be followed by block 216.

[0042] At block 216, a diffusion process is performed using a machine learning model to obtain a denoised image. In some implementations, the machine learning model may be a trained machine learning model trained to perform spatially varying noise reduction (SVNR). In some implementations, the diffusion process may be performed using method 300 described below with reference to FIG. 3. Block 216 may be followed by block 218.

[0043] At block 218, the denoised image is output. For example, the denoised image may be displayed on a display device, such as a monitor, projector, augmented reality device, or other device capable of displaying images. In some implementations, the denoised image may be output to a printer device to generate a physical printout of the denoised image. In some implementations, the denoised image may be output to a system that performs additional operations, such as editing, compression, format conversion, or other operations.

[0044] Various blocks of method 200 may be combined, divided into multiple blocks, or performed in parallel. Method 200, or portions of method 200, may be repeated any number of times using additional inputs. For example, method 200 may be repeated until all images captured by a camera have been processed to obtain corresponding denoised images.

[0045] FIG. 3 is a flow diagram illustrating an exemplary method 300 for performing a diffusion process, according to some embodiments. In some embodiments, method 300 may be implemented, for example, in server system 102 as shown in FIG. 1. In some embodiments, some or all of method 200 may be implemented in one or more client devices 120, 122, 124, or 126 shown in FIG. 1, one or more server devices, and / or both the server device(s) and the client device(s). In the described example, the embodiment system includes one or more digital processors or processing circuits (“processors”) and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients may perform different blocks or other portions of method 300. In some examples, a first device is described as performing blocks of method 300. Some embodiments may have one or more blocks of method 300 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0046] Method 300 may begin at block 302. At block 302, a denoised image is estimated using a machine-learned model (trained machine-learning model). This model is trained to perform spatially varying noise removal (SVNR) based on a time map, for example, as described with reference to block 214 of method 200, and may be referred to herein as an SVNR model. An exemplary method for training a machine-learning model to perform spatially varying noise removal (SVNR) is described with reference to FIG. 6. In some implementations, method 300 begins with the highest value in the time map.

[0047] In some implementations, the SVNR model estimates a denoised image based on the noisy input image, the current time-step image (which may be the noisy input image for the first time-step), and the time-step. Block 302 may be followed by block 304.

[0048] At block 304, the next time step image is acquired. In some implementations, the next time step image (x t_hat-1 ) is obtained by combining the denoised image (x0) obtained in block 302 and the current time step image (x t_hat ). In some implementations, obtaining the next time step image may further include sampling the Gaussian noise. In some implementations, the sampling may rely on the current time map (and time step image) to determine a respective weight for each of the components. Block 304 may be followed by block 306.

[0049] At block 306, it is determined whether at least one pixel is associated with a time step value greater than or equal to the current time step. If it is determined that at least one pixel is associated with a time step value greater than or equal to the current time step, block 306 may be followed by block 316. Otherwise, block 302 may be followed by block 308.

[0050] In block 308, the next time step image (eg, the one acquired in block 304) is set as the current time step image. Block 308 may be followed by block 310.

[0051] The time step is decremented in block 310. Block 310 may be followed by block 312.

[0052] At block 312, it is determined whether the current time step is positive. A positive time step may indicate that at least one pixel of the noisy input image has a time step value greater than zero, and therefore the pixel diffusion process is not yet complete. If the current time step is positive, block 312 may be followed by block 203. In implementations using method 300 to implement block 216 of method 200, if the current time step is not positive, block 312 may be followed by block 218 to output a denoised image.

[0053] In block 316, which is executed if it is determined that at least one pixel is associated with a time step value equal to or greater than the current time step, the next time step image (obtained in block 304) is updated with the corresponding value from the denoised image (obtained in block 302). This avoids further denoising of pixels (denoised based on the time map) that have reached a time step value of zero; instead, it ensures that their values ​​stabilize as obtained in the denoised image while continuing to perform further denoising operations on other pixels. This results in spatially varying denoising, since different pixels may reach a time step value of zero in different iterations of method 300. During the update, if the value of pixel p in the time map is less than 0, the value of the pixel is set to X_{t_hat}(p) (e.g., instead of setting the value to X_{t_hat-1}(p)). Method 300 may be performed over at least two time steps.

[0054] By replacing a single time step value (which defines the number of iterations to be performed) with a spatially varying time map, the diffusion operation is now performed on a pixel basis. The use of pixel-specific time step values ​​(time embedding) in the time map makes the denoising by the SVNR model dependent on the spatially varying noise level of the noisy input image. Therefore, denoising using the SVNR model can reduce signal-dependent shot noise as well as signal-independent lead noise.

[0055] The diffusion process described with reference to FIG. 3 is performed using the noisy input image as the condition image.

[0056] FIG. 4A shows an example image. FIG. 4A depicts a noisy input image (404) of a lizard. In some implementations, the noisy input image may be generated by adding simulated noise to an image. For example, the noisy input image may be captured in low-light conditions or other situations where the captured image may contain noise. The noise in the image (402) is shown, where the noise value (illustrated with different shading) is the spatially varying standard deviation of the noise in the (quantized) image. A corresponding clean image (406) is also shown. For example, the clean image may be captured in good lighting conditions or with a camera capable of capturing relatively noise-free images in low-light conditions.

[0057] Furthermore, in some implementations, noisy images (e.g., images for use in training a machine learning model) may be generated by applying spatially varying noise to a known clean image. In this case, the known clean image may serve as ground truth during training of a machine learning model, e.g., to train the machine learning model to remove noise from the generated noisy image. For example, in the case of the techniques described above with reference to FIGS. 2 and 3, the noisy image may serve as a condition image, and the ground truth image is used for comparison, e.g., to train the machine learning model to remove noise from the noisy image by using the ground truth image to compare with the output of a diffusion process. The techniques described herein handle (remove) spatially varying noise by applying pixel-wise time embedding, e.g., as described above with reference to FIG. 2 and spatially varying time maps.

[0058] FIG. 4A also depicts a denoised image (408) obtained by applying an exemplary prior art technique that does not utilize diffusion to the noisy input image (404). FIG. 4 further illustrates another example of a denoised image (410) obtained by applying a previous diffusion technique to the noisy input image (404). The denoised image (410) is obtained after 1,000 time steps of the diffusion process. In the denoised images (408, 410), it can be seen that while such previous denoising methods remove high levels of noise, they overly smooth fine details and / or the resulting denoised image lacks fine detail. For example, while the denoised image 410 has more detail than the denoised image 412, part of the lizard's head in the clean image (406) is missing from the denoised image 410, and the denoised image 408 lacks sharp boundaries. While image 410 shows missing parts, in certain cases, previous techniques may add details not present in the image. Additionally, denoised images 408 and 410 lack fine detail in the skin areas under the eyes and on the torso.

[0059] 4A further illustrates a denoised image (412) obtained by applying the spatially varying denoising technique described herein with reference to FIGS. 2 and 3. As can be seen, the denoised image contains fine details, e.g., accurately depicting the head area and the skin area under the eyes. It can be seen that the spatially varying denoising technique described herein can restore the texture of an image even when the texture is difficult to distinguish within a noisy input image. As can be seen, the described technique produces a clean image with high fidelity while simultaneously reducing the runtime (and therefore the computational cost) (e.g., of the diffusion process) by an order of magnitude (25 time steps versus 1,000 steps).

[0060] FIG. 4B depicts the noisy image (404) and denoised image (412) of FIG. 4A , as well as the intermediate denoised image (420) obtained after 15 time steps. Conventional diffusion techniques (e.g., generated image 410) use substantially more computational resources because they start with the complete noisy image rather than the noisy input image (e.g., because the diffusion process in this example is run for 1,000 steps to obtain denoised image 410). In contrast, the spatially varying denoising technique described herein starts the diffusion process with the noisy image (404) and can generate a denoised image at a lower computational cost. As can be seen in FIG. 4B , after 15 time steps in the diffusion process, the spatially varying denoising technique generates a denoised image (420) that is a significant improvement over the noisy input image (404).

[0061] Figure 5 illustrates several examples of noisy images (502A-510A) along with their corresponding denoised images (512A-520A) and ground truth images (522A-530A). Figure 5 also illustrates enlarged (zoomed) views (502B-510B) of portions of each noisy image (indicated by small squares within the images) and the corresponding portions of the denoised images (512B-520B) and ground truth images (522B-530B). As can be seen from Figure 5, the described technique can achieve high signal fidelity (matching the ground truth image) without significant distribution drift. In the example shown in Figure 4, the noise gain is 16. Gain refers to the amplification of the signal from the camera sensor (ISO) when capturing an image. If the image capture conditions are poor, the gain is higher, implying a larger amplitude of noise in the image. Even with relatively high noise levels, the spatially varying denoising techniques described herein can produce denoised images that contain high-definition fine details while simultaneously removing a significant amount of noise. Furthermore, these techniques have high fidelity to the ground truth image, unlike other diffusion-based methods that exhibit illusions (adding details to the denoised image that are not present in the ground truth image).

[0062] The described technique produces more realistic images while also performing a large amount of noise reduction. The technique maintains fidelity to the underlying signal (in the noisy image). Furthermore, diffusion techniques starting from full noise have a fixed computational cost (runtime) based on the number of time steps to reach the denoised image. This computational cost is independent of the noise level of the noisy input image. In contrast, the computational cost (runtime) for the techniques described herein varies based on the noise level of the noisy input image (higher noise requires more cost). Also, because the described technique can start the diffusion process from a noisy input image, the computational cost can be significantly (e.g., by orders of magnitude) lower than diffusion techniques starting from full noise.

[0063] FIG. 6 is a flow diagram illustrating an example method 600 for training a machine learning model to denoise a noisy input image, according to some implementations. In some implementations, method 600 may be implemented, for example, in server system 102 as shown in FIG. 1. In some implementations, some or all of method 600 may be implemented in one or more client devices 120, 122, 124, or 126 shown in FIG. 1, one or more server devices, and / or both the server device(s) and the client device(s). In the described example, the implementation system includes one or more digital processors or processing circuits (“processors”) and one or more storage devices (e.g., database 106 or other storage). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 600. In some examples, a first device is described as performing blocks of method 600. Some implementations may have one or more blocks of method 600 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0064] In some embodiments, method 600, or portions thereof, may be initiated automatically by a system. In some embodiments, the implementing system is the first device. For example, the method (or portions thereof) may be executed periodically or based on one or more specific events or conditions, such as the availability of new or additional training data, a change or new type of image capture hardware (which may, for example, affect noise in captured images), the passage of a predetermined amount of time since the last execution of method 600, and / or the occurrence of one or more other conditions that may be specified in settings read by the method.

[0065] The training data, e.g., one or more of ground truth images, noisy input images, etc., are obtained with the permission of a particular user. In some implementations, the training data may include images captured specifically for the purpose of training a machine learning model. In some implementations, the training data may include synthetic images or generated images (e.g., images generated using generative artificial intelligence or other suitable image generation techniques). If the training data includes user-provided images, such images are utilized only with the permission of a particular user and for a particular purpose (e.g., to train a machine learning model to perform noise removal). The user is provided with the option to exclude captured images from the training data.

[0066] In block 602, a clean image is obtained. For example, the clean image may be an image captured by a camera, with little or no image noise (e.g., under good lighting conditions). In some implementations, the clean image may be obtained from training data including, for example, a large number of training images representing real-world images. Block 602 may be followed by block 604.

[0067] At block 604, a noisy input image is generated by adding noise to the clean image. The added noise is determined by the pixel-by-pixel standard deviation of the noise (σ p In some implementations, the noise standard deviation (σ p ) is the lead noise (σ r ) and shot noise (σ s ), where the read noise is the standard deviation of the signal-independent noise and the shot noise is the standard deviation of the signal-dependent noise. In some embodiments, the read noise and shot noise are selected randomly. Block 604 may be followed by block 606.

[0068] At block 606, a time map (T*) is determined based on the noisy input image and the noise in the noisy input image. The time map is spatially varying such that a respective time step value is associated with each pixel of the noisy input image such that at least two pixels have different time step values. Block 606 may be followed by block 608.

[0069] At block 608, a new time map (t_hat) is generated such that each time step value of a pixel in the new time map is lower than the time stamp value of the corresponding pixel in the time map. For example, the new time map may be generated by subtracting a scalar from each time step value of each pixel in the time map (T*). The scalar may be selected, for example, randomly, as a value between zero and the maximum time step in the time map (T*). Block 608 may be followed by block 610.

[0070] At block 610, the current time step image (x t_hat ) is generated based on a new time map, e.g., a noise level from the new time map. The current time step image is generated to be less noisy than the noisy input image (y). In other words, the current time step image is a slightly denoised version of the noisy input image (y). In some implementations, the current time step image is generated to have a noise distribution similar to the noise distribution of the noisy input image. Block 610 may be followed by block 612.

[0071] In diffusion techniques, the timestamp is linear in the number of forward diffusion steps, which is linear in the noise variance. In such cases, determining the timestamp is equivalent to determining the noise level, which can be done once at the beginning of inference. During training, the noise variance per pixel is known, so it can be used to directly calculate the timestamp or estimate the timestamp from the image. This is advantageous because it is similar to the inference process.

[0072] At block 612, a noisy input image (y), a current time step image (x t_hat ), and the new time map are provided as inputs to a machine learning model during training. The machine learning model then calculates the denoised image (x 0_hat ). Block 612 may be followed by block 614.

[0073] At block 614, the loss value is calculated based on the denoised image (x 0_hat ) to the clean image (x0). The loss value indicates the difference between the clean image (which is the ground truth) and the denoised image produced by the machine learning model being trained. Block 614 may be followed by block 616.

[0074] At block 616, one or more parameters of the machine learning model are updated based on the loss value. In some implementations, the machine learning model may include a neural network. In these implementations, updating the one or more parameters of the machine learning model based on the loss value includes performing backpropagation to adjust weights of one or more nodes of the neural network to reduce the loss value. Block 616 may be followed by block 618.

[0075] At block 618, it is determined whether a stopping criterion has been met. In some implementations, the stopping criterion may include one or more of: the amount of computation for training has been exhausted; all clean images in the training dataset of clean images have been utilized; or when a change in the parameter value of at least one of the one or more parameters between successive iterations falls below a threshold. If the stopping criterion has not been met, block 618 may be followed by block 602. If the stopping criterion has been met, block 618 is followed by block 620.

[0076] At block 620, a machine-learned model (a trained model, which is a version of the machine-learned model output after parameter adjustment) is generated. This machine-learned model can be used to perform spatially varying noise removal from any image. For example, the machine-learned model can be used for noise removal using the methods described with reference to FIGS. 2 and 3.

[0077] Various blocks of method 600 may be combined, split into multiple blocks, or executed in parallel. For example, blocks 604-616 may be executed multiple times on the same clean image, e.g., generating a different noisy version of the clean image with each execution of block 604, and training a model using each of the generated noisy versions. In some implementations, blocks 608-616 may be executed multiple times on the same noisy input image by using different scalar values ​​(e.g., randomly sampled between 0 and the maximum value in the time map) to generate a new time map in block 608, and the corresponding estimated denoised image is used to determine the loss value. During training, the model's task is to evaluate the clean image x_0 given the time map. In some implementations, method 600 may be executed in parallel with multiple images or portions of method 600 (e.g., blocks 604-614 or 604-616 for the same clean image, blocks 608-614 or 608-616 for the same noisy input, etc.). Method 600, or portions of method 600, may be repeated any number of times using additional input, e.g., new training images received after the initial machine-learned model was obtained at block 620. For example, retraining the model may improve the quality of the denoised images produced by the machine-learned model. Images of different image sizes and / or different levels of noise may be used during model training.

[0078] 7 is a block diagram of an example device 700 that may be used to implement one or more functions described herein. In one example, device 700 may be used to implement a client device, such as any of client devices 115 shown in FIG. 1. Alternatively, device 700 may implement a server device, such as server 101. In some implementations, device 700 may be used to implement a client device, a server device, or both a client device and a server device. Device 700 may be any suitable computer system, server, or other electronic or hardware device described above.

[0079] One or more methods described herein may operate in a standalone program that may be executed on any type of computing device, a program running on a web browser, a mobile application (“app”) running on a mobile computing device (e.g., a mobile phone, a smartphone, a tablet computer, a wearable device (e.g., a wristwatch, an armband, a brace, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, a head-mounted display, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., the mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., display). In another example, all computations may be performed within the mobile app (and / or other apps) on the mobile computing device. In other examples, computations may be split between the mobile computing device and one or more server devices.

[0080] In some implementations, device 700 includes a processor 702, memory 704, and an input / output (I / O) interface 706. Processor 702 may be one or more processors and / or processing circuits that execute program code to control basic operations of device 700. A "processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) with one or more cores (e.g., a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., a multiprocessor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a system having dedicated circuitry for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some implementations, processor 702 may include one or more coprocessors that perform neural network processing. In some implementations, processor 702 may be a processor that processes data to generate a probabilistic output; for example, the output generated by processor 702 may be inaccurate or accurate within a range from an expected output. Processing need not be limited to a particular geographic location or have time limitations. For example, a processor may perform its functions in "real time," "offline," "batch mode," etc. Portions of processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0081] Memory 704 is typically provided within device 700 for access by processor 702 and may be any suitable processor-readable storage medium, located separately from and / or integral with processor 702, suitable for storing instructions for execution by the processor, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 704 may store software operated by processor 702 on server device 700, including operating system 708, machine learning application 730, other applications 712, and application data 714. Other applications 712 may include applications such as a data display engine, a web hosting engine, an image display engine, a notification engine, a social networking engine, etc. In some implementations, machine learning application 730 and other applications 712 may each include instructions that enable processor 702 to perform some or all of the functions described herein, e.g., the methods of FIGS. 2, 3, and / or 6.

[0082] Other applications 712 may include, for example, an image editing application, a media display application, a communication application, a web hosting engine or application, a mapping application, a media sharing application, etc. One or more methods disclosed herein may operate in several environments and platforms, such as, for example, as a standalone computer program that may run on any type of computing device, as a web application having a web page, as a mobile application (“app”) run on a mobile computing device, etc.

[0083] In various embodiments, the machine learning application may utilize a Bayesian classifier, a support vector machine, a neural network, or other learning techniques. In some embodiments, the machine learning application 730 may include a machine-learned model (trained model) 734, an inference engine 736, and data 732. In some embodiments, the data 732 may include training data, e.g., data used to generate the trained model 734. For example, the training data may include any type of data, such as text, images, audio, video, etc. If the trained model 734 is an image denoising model, the training data may include ground truth clean images.

[0084] The training data may be obtained from any source, e.g., a data repository marked for training only, data that has been given permission to be used as training data for machine learning, etc. In embodiments in which one or more users authorize the use of their respective user data to train a machine learning model, e.g., trained model 734, the training data may include such user data. In embodiments in which users authorize the use of their respective user data, data 732 may include authorized data such as images (e.g., photographs or other user-generated images).

[0085] In some implementations, the training data may include data that is not based on user input or activity in the context being trained, e.g., synthetic data generated for training purposes, such as data generated from simulated photographs or other computer-generated imagery. In some implementations, the machine learning application 730 excludes data 732. For example, in these implementations, the trained model 734 may be generated, e.g., on a different device, and provided as part of the machine learning application 730. In various implementations, the trained model 734 may be provided as a data file that includes a model structure or format and associated weights. The inference engine 736 may read the trained model 734 data file and implement a neural network with node connectivity, layers, and weights based on the model structure or format specified in the trained model 734.

[0086] In some implementations, the trained model 734 may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., “hidden layers” between an input layer and an output layer), a convolutional neural network (e.g., a network that divides or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results of the processing of each tile), a sequence-to-sequence neural network (e.g., a network that takes sequential data, such as words in a sentence or frames of a video, as input and a resulting sequence as output), etc. The model form or structure may specify the connectivity between various nodes and the organization of the nodes into layers.

[0087] For example, nodes in a first layer (e.g., input layer) may receive data as input data 732 or application data 714. For example, if the trained model 734 is an image denoising model, the input data may include a clean image. Such data may include, for example, one or more pixels per node if, for example, the trained model is used for image analysis or image generation. Subsequent intermediate layers may receive as input the output of nodes in the previous layer according to connectivity specified in the model format or model structure. These layers may also be referred to as hidden or latent layers.

[0088] The final layer (e.g., output layer) generates the output of the machine learning application. For example, the output may be a denoised image. In some implementations, the model format or model structure also specifies the number and / or type of nodes in each layer.

[0089] In different embodiments, the trained model 734 may include multiple nodes arranged in layers per model structure or morphology. In some embodiments, a node may be a computational node without memory, configured to process, for example, one unit of input and generate one unit of output. The computation performed by the node may include, for example, multiplying each of multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output. In some embodiments, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computation may include operations such as matrix multiplication. In some embodiments, computations by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, individual processing units of a GPU, or dedicated neural circuitry. In some embodiments, a node may include memory, e.g., be able to store and use one or more previous inputs when processing a subsequent input. For example, a node with memory may include a long short-term memory (LSTM) node. LSTM nodes may use memory to maintain "state," allowing the nodes to operate like a finite state machine (FSM). Models with such nodes may be useful in processing continuous data, such as words in a sentence or paragraph, frames in a video, speech or other audio, etc. In some implementations, models with attention mechanisms and transfer networks may be used.

[0090] In some implementations, the trained model 734 may include embeddings or weights for individual nodes. For example, the model may begin as multiple nodes organized into layers, as specified by the model format or model structure. At initialization, a respective weight may be applied to the connection between each pair of nodes connected according to the model format, e.g., nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The model may then be trained to generate results, e.g., using the data 732.

[0091] For example, training may include applying supervised learning techniques. In supervised learning, the training data may include multiple inputs (e.g., a set of noisy input images) and expected outputs corresponding to each input (e.g., a set of ground truth clean images corresponding to the noisy input images). Based on a comparison of the model's outputs and the expected outputs, the values ​​of the weights are automatically adjusted, for example, in a manner that increases the probability that the model will produce the expected outputs when provided with similar inputs.

[0092] In some implementations, training may include applying unsupervised learning techniques, in which only input data may be provided and a model may be trained to distinguish between the data, e.g., to cluster the input data into multiple groups, each group containing input data that is similar in some way.

[0093] In some implementations, unsupervised learning can be used to generate a knowledge representation that can be used, for example, by machine learning application 730. In various implementations, the trained model includes a set of weights or embeddings that correspond to a model structure. In implementations in which data 732 is omitted, machine learning application 730 may include a trained model 734 that is based on prior training, for example, by a developer of machine learning application 730, by a third party, etc. In some implementations, trained model 734 may include a set of weights that are fixed (e.g., downloaded from a server that provides the weights).

[0094] The machine learning application 730 also includes an inference engine 736. The inference engine 736 is configured to apply the trained model 734 to data, such as the application data 714, to provide an inference. In some implementations, the inference engine 736 may include software code executed by the processor 702. In some implementations, the inference engine 736 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 702 to apply the trained model. In some implementations, the inference engine 736 may include software instructions, hardware instructions, or a combination thereof. In some implementations, the inference engine 736 may provide an application programming interface (API) that is used by the operating system 708 and / or other applications 712 to invoke the inference engine 736, for example, to apply the trained model 734 to the application data 714 to generate an inference. For example, the estimate of the image denoising model may be a noisy image, while the estimate is a denoised image, as described above with reference to FIGS. 2-6.

[0095] The machine learning application 730 may provide several technical advantages. For example, if the trained model 734 is generated based on unsupervised learning, the trained model 734 can be applied by the inference engine 736 to generate knowledge representations (e.g., numerical representations) from input data, e.g., application data 714. For example, a trained model for image analysis may generate a representation of an image having a smaller data size (e.g., 1 KB) than the input image (e.g., 10 MB). In some implementations, such a representation may be useful to reduce processing costs (e.g., computational costs, memory usage, etc.) for generating output (e.g., labels, classifications, sentences describing the image, denoising images from noisy images).

[0096] In some implementations, such representations may be provided as input to a different machine learning application that generates an output from the output of inference engine 736. In some implementations, the knowledge representations generated by machine learning application 730 may be provided to a different device for further processing, for example, over a network. In such implementations, providing knowledge representations rather than images may provide technical benefits, for example, reducing costs and enabling faster data transmission.

[0097] In some implementations, the machine learning application 730 may be implemented in an offline manner. In these implementations, the trained model 734 may be generated in a first stage and provided as part of the machine learning application 730. In some implementations, the machine learning application 730 may be implemented in an online manner. For example, in such implementations, an application (e.g., the operating system 708, one or more of the other applications 712) that invokes the machine learning application 730 may utilize the inferences generated by the machine learning application 730, e.g., provide the inferences to a user, and generate, e.g., a system log (e.g., actions taken by the user based on the inferences, if permitted by the user, or results of further processing, if utilized as input for further processing). The system log may be generated periodically, e.g., hourly, monthly, quarterly, etc., and, with the user's permission, may be used to update the trained model 734, e.g., to update the embeddings of the trained model 734.

[0098] In some implementations, machine learning application 730 may be implemented in a manner that can adapt to the particular configuration of device 700 on which it executes. For example, machine learning application 730 may determine a computation graph that utilizes available computational resources, e.g., processor 702. For example, if machine learning application 730 is implemented as a distributed application on multiple devices, machine learning application 730 may determine which computations to perform on each device in a manner that optimizes the computations. In another example, machine learning application 730 may determine that processor 702 includes a GPU with a particular number of GPU cores (e.g., 1000) and implement the inference engine accordingly (e.g., as 1000 individual processes or threads).

[0099] In some implementations, the machine learning application 730 may implement an ensemble of trained models. For example, the trained models 734 may include multiple trained models, each applicable to the same input data. In these implementations, the machine learning application 730 may select a particular trained model based on, for example, available computational resources, prior inference success rates, etc. In some implementations, the machine learning application 730 may execute an inference engine 736 so that multiple trained models are applied. In these implementations, the machine learning application 730 may combine outputs from the application of individual models, for example, using a voting technique to score individual outputs from the application of each trained model, or by selecting one or more particular outputs. Furthermore, in these implementations, the machine learning application may apply a time threshold (e.g., 0.5 ms) for applying individual trained models and may utilize only those individual outputs that are available within the time threshold. Outputs not received within the time threshold may not be utilized, e.g., discarded. For example, such an approach may be appropriate when there is a time limit specified between invoking the machine learning application, for example, by the operating system 708 or one or more applications 712.

[0100] In different implementations, the machine learning application 730 can generate different types of output. For example, the machine learning application 730 may provide a representation or cluster (e.g., a numerical representation of the input data), a label (e.g., a label for the input data, including an image, a document, etc.), a phrase or sentence (e.g., a description of an image or video, suitable for use as a response to an input sentence, etc.), an image (e.g., a denoised image generated by the machine learning application in response to an input image, e.g., a noisy image), audio, or video (e.g., in response to an input video, the machine learning application 730 can generate an output video with particular effects applied, e.g., an output video rendered in the style of a comic book or a particular artist, if the trained model 734 is trained using training data from a comic book or a particular artist, etc.). In some implementations, the machine learning application 730 can generate output based on a format specified by a calling application, e.g., the operating system 708 or one or more applications 712. In some implementations, the calling application can be another machine learning application. For example, such a configuration may be used in a generative adversarial network, where output from machine learning application 730 is used to train a calling machine learning application, and vice versa.

[0101] Any of the software in memory 704 may alternatively be stored in any other suitable storage location or computer-readable medium. Additionally, memory 704 (and / or other connected storage device(s)) may include one or more messages, one or more taxonomies, electronic encyclopedias, dictionaries, thesauri, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the functionality described herein. Memory 704 and any other type of storage (such as magnetic disks, optical disks, magnetic tape, or other tangible media) may be considered "storage" or "storage device."

[0102] The I / O interface 706 may provide functionality that allows the server device 700 to interface with other systems and devices. The interfaced devices may be included as part of the device 700 or may be separate and may communicate with the device 700. For example, network communication devices, storage devices (e.g., memory and / or database 106), and input / output devices may communicate through the I / O interface 706. In some implementations, the I / O interface may connect to interface devices such as input devices (keyboards, pointing devices, touchscreens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0103] Some examples of interface devices that may be connected to I / O interface 706 include one or more display devices 720 that can be used to display content, e.g., images, video, and / or user interfaces of output applications described herein. Display device 720 may be connected to device 700 via a local connection (e.g., a display bus) and / or via a network connection and may be any suitable display device. Display device 720 may include any suitable display device, such as an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touchscreen, a 3D display, or other visual display device. For example, display device 720 may be a flat display screen provided in a mobile device, multiple display screens provided in a goggle or headset device, or a monitor screen of a computing device.

[0104] The I / O interface 706 can interface to other input and output devices. Some examples include one or more cameras that can capture images. Some implementations can provide a microphone for capturing audio (e.g., as part of a captured image, voice commands, etc.), an audio speaker device for outputting audio, or other input and output devices.

[0105] For ease of explanation, FIG. 7 shows one block for each of processor 702, memory 704, I / O interface 706, and software blocks 708, 712, and 730. These blocks may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software modules. In other embodiments, device 700 may not have all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to the components shown herein. Although some components are described as performing the blocks and operations described in some embodiments herein, any suitable component or combination of components of environment 100, device 700, a similar system, or any suitable processor or processors associated with such systems may perform the described blocks and operations.

[0106] The methods described herein can be implemented by computer program instructions or code executable by a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored in a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, and the like. The program instructions can also be contained in or provided as electronic signals, for example, in the form of Software as a Service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (e.g., logic gates) or a combination of hardware and software. Exemplary hardware can be a programmable processor (e.g., a field programmable gate array (FPGA), complex programmable logic device), general-purpose processor, graphics processor, application-specific integrated circuit (ASIC), and the like. One or more of the methods may be implemented as part of or as a component of an application executing on the system, or as an application or software running in conjunction with other applications and the operating system.

[0107] Although the description is given with respect to specific embodiments thereof, these specific embodiments are merely exemplary and not limiting, and the concepts illustrated in the examples may be applied to other examples and embodiments.

[0108] In situations where certain implementations described herein may collect or use personal information about a user (e.g., user data, information about the user's social network, the user's location and time at that location, the user's biometric information, the user's activity, and demographic information), the user is provided with one or more opportunities to control whether the information is collected, whether the personal information is stored, whether the personal information is used, and how the information about the user is collected, stored, and used. That is, the systems and methods described herein collect, store, and / or use a user's personal information, particularly upon receiving explicit approval to do so from the associated user. For example, a user is provided with the ability to control whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user about whom personal information is collected is presented with one or more options that enable control of the collection of information related to that user, providing permission or approval for whether and what portions of the information are collected. For example, a user may be provided with one or more such control options via a communications network. Additionally, certain data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed. As one example, the user's identity may be treated such that personally identifiable information cannot be determined. As another example, the geographic location of the user device may be generalized to a broader area such that the user's specific location cannot be determined.

[0109] It should be noted that the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be combined or divided into different combinations of systems, devices, and functional blocks, as would be known to one of ordinary skill in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular implementation. Different programming techniques, e.g., procedural or object-based, may be used. The routines may be executed on a single processing device or on multiple processors. While steps, operations, or computations may be presented in a particular order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown herein as sequential may be executed simultaneously.

Claims

1. 1. A computer-implemented method for denoising a noisy input image, comprising: acquiring (212) the noisy input image, the noisy input image comprising a plurality of pixels, the computer-implemented method further comprising: estimating (214) a time map based on the noisy input image and noise in the noisy input image, the time map including a time step value for each of the plurality of pixels, the time step value corresponding to a noise level of the pixel with which the time step value is associated; At least two pixels have different time step values, and the computer-implemented method further comprises: performing a diffusion process on the noisy input image (216); The diffusion process is started with a time step count value being the highest value of the time step value in the time map, and is repeated until the time step count value becomes zero, and the diffusion process is estimating a denoised image based on the noisy input image, a current time-step image, and the time-step using a machine-learned model (SVNR) (302); obtaining a next time-step image based on the denoised image and the current time-step image (304); for each pixel, if the time step value for that pixel is greater than or equal to the time step count value, updating the next time step image with the corresponding value from the denoised image (316); After the updating, setting the next time-step image as the current time-step image (308); and decrementing the time step count (310); The computer-implemented method further comprises: outputting the denoised image (218); A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , further comprising determining the noise of the noisy input image.

3. 3. The computer-implemented method of claim 2, wherein each pixel has a respective noise level, and determining the noise comprises determining lead noise and shot noise from the noisy input image, wherein the lead noise is a standard deviation for signal-independent noise and the shot noise is a standard deviation for signal-dependent noise.

4. The computer-implemented method of claim 1 , wherein at a first time step while performing the diffusion process, the noisy input image is used as the current time step image.

5. The computer-implemented method of claim 1 , wherein the noisy input image serves as a condition image during the diffusion process.

6. a processor; a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform operations, the operations including: acquiring (212) a noisy input image, the noisy input image comprising a plurality of pixels, the instructions further causing the processor to perform operations including: estimating (214) a time map based on the noisy input image and noise in the noisy input image, the time map including a time step value for each of the plurality of pixels, the time step value corresponding to a noise level of the pixel with which the time step value is associated, the operations further comprising: performing a diffusion process on the noisy input image (216), the diffusion process starting with a time step count value at the highest time step value in the time map and repeated until the time step count value reaches zero, the diffusion process: estimating a denoised image based on the noisy input image, a current time-step image, and the time-step using a machine-learned model (SVNR) (302); obtaining a next time-step image based on the denoised image and the current time-step image (304); for each pixel, if the time step value for that pixel is greater than or equal to the time step count value, updating the next time step image with the corresponding value from the denoised image (316); After the updating, setting the next time-step image as the current time-step image (308); decrementing the time step count (310); The operation is The computing device further comprises outputting (218) the denoised image.

7. The computing device of claim 6 , wherein the operation further comprises determining the noise of the noisy input image.

8. 8. The computing device of claim 7, wherein each pixel has a respective noise level, and determining the noise comprises determining lead noise and shot noise from the noisy input image, the lead noise being a standard deviation for signal-independent noise and the shot noise being a standard deviation for signal-dependent noise.

9. The computing device of claim 6 , wherein at a first time step during the diffusion process, the noisy input image is used as the current time step image.

10. The computing device of claim 6 , wherein the noisy input image serves as a condition image during the diffusion process.

11. 1. A computer-implemented method for training a machine learning model to denoise a noisy input image, comprising: Acquiring a clean image (602); generating a noisy input image by adding noise to the clean image (604), wherein the noise is determined based on a pixel-by-pixel standard deviation of the noise; and estimating (606) a time map based on the noisy input image and the noise in the noisy input image, the time map including a time step value for each of a plurality of pixels of the noisy input image, the time step value corresponding to a noise level of the pixel with which the time step value is associated, the computer-implemented method further comprising: generating (608) a new time map, wherein each time step value of a pixel in the new time map is less than the time step value of a corresponding pixel in the time map, the computer-implemented method further comprising: generating (610) a current time step image based on the new time map, the current time step image having less noise than the noisy input image, the computer-implemented method further comprising: using the machine learning model to estimate (612) a denoised image based on the noisy input image, the current time-step image, and the new time map; calculating (614) a loss value based on a comparison of the denoised image and the clean image; updating (616) one or more parameters of the machine learning model based on the loss value; The computer-implemented method comprising:

12. the standard deviation of the noise is a combination of read noise and shot noise, the read noise is the standard deviation for signal-independent noise and the shot noise is the standard deviation for signal-dependent noise; The computer-implemented method of claim 11 , wherein the read noise and the shot noise are randomly selected.

13. generating the new time map comprises, for each pixel in the time map, subtracting a scalar from a respective time step value; The computer-implemented method of claim 11 , wherein the scalar has a value between zero and a maximum time step value in the time map.

14. 12. The computer-implemented method of claim 11, wherein generating the current time step image based on the new time map comprises sampling noise based on the new time map.

15. 15. The computer-implemented method of claim 14, wherein the current timestep image has a noise distribution similar to the noise distribution of the noisy input image.

16. the machine learning model comprises a neural network; 12. The computer-implemented method of claim 11, wherein updating the one or more parameters of the machine learning model based on the loss value comprises performing backpropagation to adjust weights of one or more nodes of the neural network.

17. 12. The computer-implemented method of claim 11, wherein the obtaining the clean image, the generating the noisy input image, the estimating the time map, the generating the new time map, the generating the current time step image, the estimating the denoised image, the calculating the loss value, and the updating the one or more parameters of the machine learning model are performed iteratively until a stopping criterion is met.

18. 20. The computer-implemented method of claim 17, wherein the stopping criteria include one or more of: a computational budget for training being exhausted; or a change in a parameter value of at least one of the one or more parameters between successive iterations being below a threshold.

Images