Spatially variant noise removal from images using diffusion

Abstract

Implementations relate to a computer-implemented method for denoising a noisy image. The method includes obtaining a noisy input image, estimating (214) a temporal map based on the noisy input image and noise in the noisy input image, where the temporal map is spatially varying, and performing a diffusion process on the noisy input image over two or more time steps. At each time step, the method includes estimating a denoised image based on the noisy input image, a current time step image, and a time step using a machine-learned model, obtaining a next time step image based on the denoised image and the current time step image, for each pixel, updating the next time step image with a corresponding value from the denoised image, setting the next time step image as the current time step image, and decrementing the time step. While the time step is positive, the estimating, obtaining, updating, setting, and decrementing are repeated. The method further includes outputting the denoised image.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 507,194, filed June 9, 2023, entitled “SPATIALLY VARIANT NOISE REMOVALFROM IMAGES USING DIFFUSION”, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] Photos, videos, and other images captured using a camera can contain noise and / or lack detail due to various factors, such as limitations of the camera's image capture sensor and the lighting conditions under which the image was captured. Computationally removing noise from captured images can improve the quality of the image as perceived by the user and restore fine details.

[0004] The background description provided herein is for the purpose of presenting the overall context of this disclosure. The work of the currently named inventors (to the extent described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither expressly nor impliedly acknowledged as prior art to this disclosure. Summary of the Invention

[0005] Some embodiments described herein relate to methods, apparatuses, and computer-readable media for removing noise from images using a machine learning model trained to perform spatial variability noise removal. Some embodiments described herein relate to methods, apparatuses, and computer-readable media for training machine learning models to perform spatial variability noise removal.

[0006] Some implementations relate to a computer-implemented method for denoising an image. In some implementations, a computer-implemented method for denoising a noisy input image includes obtaining a noisy input image comprising a plurality of pixels. The method further includes estimating a temporal map based on the noisy input image and noise in the noisy input image, wherein the temporal map is spatially varied such that corresponding time step values ​​are associated with each pixel of the noisy input image, and wherein at least two pixels have different time step values. The method further includes performing a diffusion process on the noisy input image, wherein the diffusion process is performed over two or more time steps, starting from the highest value in the temporal map. The diffusion process includes: at each time step: estimating a denoised image based on the noisy input image, the current time step image, and the time step using a machine learning-based model (SVNR); obtaining a next time step image based on the denoised image and the current time step image; for each pixel, updating the next time step image using the corresponding value from the denoised image if the time step is greater than or equal to the time step value associated with that pixel; after the update, setting the next time step image as the current time step image; and decrementing the time step. While the time step is positive, the estimation, acquisition, update, setting, and decrement processes are repeated. The method further includes outputting a denoised image.

[0007] In some implementations, the method may further include determining noise in the noisy input image. In these implementations, each pixel has a corresponding noise level, and determining the noise includes determining read-noise and shot-noise from the noisy input image. In these implementations, a temporal map encodes the spatial variation noise of the noisy input image.

[0008] In some embodiments, at a first time step during the diffusion process, a noisy input image is used as the current time step image. In some embodiments, the computer-implemented method of claim 1, wherein the noisy input image serves as a conditional image during the diffusion process.

[0009] Some implementations relate to a method for training a machine learning model to denoise a noisy input image. In some implementations, the method for training the machine learning model to denoise a noisy input image includes obtaining a clean image. The method further includes generating a noisy input image by adding noise to the clean image, wherein the noise is determined based on the noise standard deviation per pixel. The method further includes estimating a temporal map based on the noisy input image and the noise in the noisy input image. In some implementations, the temporal map is spatially varied such that corresponding time step values ​​are associated with each pixel of the noisy input image, wherein at least two pixels have different time step values. The method further includes generating a new temporal map where the corresponding time step values ​​of pixels in the new temporal map are lower than the timestamp values ​​of corresponding pixels in the previous temporal map. The method further includes generating a current time step image based on the new temporal map, wherein the current time step image has lower noise than the noisy input image. The method further includes using a machine learning model to estimate a denoised image based on the noisy input image, the current time step image, and the new temporal map. The method further includes calculating a loss value based on a comparison between the denoised image and the clean image. The method further includes updating one or more parameters of the machine learning model based on the loss value.

[0010] In some implementations, the noise standard deviation is a combination of readout noise and shot noise. In these implementations, the readout noise is the standard deviation of signal-independent noise, and the shot noise is the standard deviation of signal-dependent noise. In some implementations, the readout noise and shot noise are randomly selected.

[0011] In some implementations, generating a new time map involves subtracting a scalar from the corresponding time step value of each pixel in the time map, wherein the scalar has a value between zero and the maximum time step value in the time map.

[0012] In some implementations, generating the current time-step image based on the new timemap includes sampling noise based on the new timemap. In some implementations, the current time-step image has a noise distribution similar to that of the noisy input image.

[0013] In some implementations, the machine learning model includes a neural network. In these implementations, updating one or more parameters of the machine learning model based on a loss value includes performing backpropagation to adjust the weights of one or more nodes in the neural network.

[0014] In some implementations, the process of obtaining a clean image, generating a noisy input image, estimating a timemap, generating a new timemap, generating the image at the current time step, estimating the denoised image, calculating the loss value, and updating one or more parameters of the machine learning model is performed iteratively until a stopping criterion is met. In some implementations, the stopping criterion includes one or more of the following: the computational budget used for training is exhausted, or the parameter value of at least one of the one or more parameters decreases below a threshold between successive iterations.

[0015] Some implementations include a computing device that includes a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform any of the methods described herein.

[0016] Some implementations include a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform any of the methods described herein. Attached Figure Description

[0017] Figure 1 A block diagram of an example network environment that can be used in some of the implementations described herein is shown.

[0018] Figure 2 This is a flowchart illustrating an example method for denoising a noisy input image according to some embodiments.

[0019] Figure 3 This is a flowchart illustrating an example method for performing a diffusion process according to some implementations.

[0020] Figures 4A to 4B An example image is shown.

[0021] Figure 5 Examples of noisy images, along with their corresponding denoised and ground truth images, are shown.

[0022] Figure 6 This is a flowchart illustrating an example method for training a machine learning model to denoise a noisy input image, according to some implementations.

[0023] Figure 7 This is a block diagram of an example apparatus that can be used to implement one or more features described herein. Detailed Implementation

[0024] This paper describes methods, systems, and non-transitory computer-readable media for noise removal from noisy input images. The described techniques provide a novel diffusion-based framework for image denoising tasks. These techniques leverage naturally rich image priors learned by a generative denoising diffusion model. In some implementations, the techniques perform denoising diffusion by utilizing the noisy input image as a condition and starting point for the diffusion process. To enable the integration of real noisy images as samples into the diffusion process, the techniques employ a denoising diffusion formula utilizing spatially-variant temporal embeddings (temporal maps). This paper describes techniques for training machine learning models to perform noise removal using spatially-variant temporal maps and for removing noise from images using machine learning models. The described techniques can be applied to noise removal from any noisy image, including images with non-uniform noise distributions.

[0025] Figure 1 A block diagram of an example network environment 100 that can be used in some of the embodiments described herein is shown. In some embodiments, network environment 100 includes one or more server systems, such as... Figure 1 The example is server system 102. For example, server system 102 may communicate with network 130. Server system 102 may include server device 104 and database 106 or other storage devices. In some embodiments, server device 104 may provide image application 156a.

[0026] Network environment 100 may also include one or more client devices, such as client devices 120, 122, 124, and 126, which can communicate with each other and / or with server system 102 via network 130. Network 130 can be any type of communication network, including one or more of the Internet, local area network (LAN), wireless network, switch or hub connection, etc. In some embodiments, network 130 may include peer-to-peer communication between devices, for example, using peer-to-peer wireless protocols (e.g., Bluetooth®, Wi-Fi Direct, etc.). Arrow 132 illustrates an example of peer-to-peer communication between two client devices 120 and 122.

[0027] For ease of explanation, Figure 1A box is shown for server system 102, server device 104, and database 106, and four boxes are shown for client devices 120, 122, 124, and 126. Server boxes 102, 104, and 106 may represent multiple systems, server devices, and network databases, and may be provided in different configurations than those shown. For example, server system 102 may represent multiple server systems that can communicate with other server systems via network 130. In some embodiments, server system 102 may, for example, include a cloud hosting server. In some examples, database 106 and / or other storage devices may be provided in a separate server system box from server device 104 and may communicate with server device 104 and other server systems via network 130.

[0028] Furthermore, any number of client devices can exist. Each client device can be any type of electronic device, such as a desktop computer, laptop computer, portable or mobile device, mobile phone, smartphone, standalone camera, tablet computer, television, TV set-top box or entertainment device, wearable device (e.g., display glasses or goggles, watch, head-mounted device, armband, jewelry, etc.), personal digital assistant (PDA), media player, gaming device, etc. Some client devices may also have a local database similar to database 106 or other storage. In some embodiments, network environment 100 may not have all the components shown and / or may have other elements, including other types of elements that replace or supplement those described herein.

[0029] In various implementations, end users U1, U2, U3, and U4 can communicate with server system 102 and / or with each other using corresponding client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 can interact with each other via applications running on the corresponding client devices and / or server system 102 and / or via network services (e.g., social networking services or other types of network services) implemented on server system 102. For example, corresponding client devices 120, 122, 124, and 126 can transmit data to and from one or more server systems (e.g., system 102).

[0030] In some implementations, server system 102 may provide appropriate data to client devices, enabling each client device to receive transmitted content or uploaded shared content to server system 102 and / or network services. In some examples, users U1 to U4 may interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0031] The network services implemented by server system 102 may include systems that allow users to perform various communications, form links and associations, upload and publish shared content (such as images, text, video, audio, and other types of content), and / or perform other functions. For example, client devices may display received data, such as content posts sent or streamed to or from different client devices (or directly from different client devices) via the server and / or network services, or originating from the server system and / or network services. In some implementations, client devices may communicate directly with each other, for example, using peer-to-peer communication as described above. In some implementations, a "user" may include one or more programs or virtual entities and persons interacting with the system or network.

[0032] In some implementations, any of the client devices 120, 122, 124, and / or 126 can provide one or more applications. For example, such as Figure 1 As shown, client device 120 can provide application 156b. Client devices 122 to 126 can also provide similar applications.

[0033] In some embodiments, the client device may include applications providing various types of functionality, such as calendars, address books, email, web browsers, shopping, transportation (e.g., taxi, train, flight booking), entertainment (e.g., music players, video players, game applications), social networking (e.g., messaging or chat, audio / video calls, image / video sharing, etc.). In some embodiments, one or more of the applications may be standalone applications running on the client device 120. In some embodiments, one or more of the applications may access a server system, such as server system 102, that provides the application's data and / or functionality.

[0034] The user interface on client devices 120, 122, 124, and / or 126 can display user content and other content (including images, videos, data, and other content, as well as communications, privacy settings, notifications, and other data). Such a user interface can be displayed using software on the client device, software on the server device, and / or a combination of client software and server software executing on server device 104 (e.g., application software or client software communicating with server system 102). The user interface can be displayed by a display device on the client device or server device, such as a touchscreen or other display screen, projector, etc. In some embodiments, an application running on the server system can communicate with the client device to receive user input at the client device and output data, such as visual data, audio data, etc., at the client device.

[0035] In various implementations, any of the client devices 120 to 126 can be used by a user and / or a customer service agent (e.g., a human customer service agent). In some implementations, application 156 on one or more of the server system 102 and / or client devices (120 to 126) can denoise the image using a machine learning-based model that performs spatial variation noise removal. In some implementations, application 156 on one or more of the server system 102 and / or client devices (120 to 126) can train a machine learning model to perform spatial variation noise removal.

[0036] In some implementations, database 106 may store content items, such as ground truth images (clean images), noisy images, or other images. In various implementations, content items may include help articles, which may include text, audio, video, images, or any combination thereof. Users of client devices 120 to 126 can access content items via network 130.

[0037] Other embodiments of the features described herein can be used with any type of system and / or service. For example, other networked services (e.g., connection to the Internet) may be used as an alternative to or supplement to social networking services. Any type of electronic device can utilize the features described herein. Some embodiments may provide one or more of the features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device including or connected to a display device may display content posts (e.g., previously received via a communication network) stored on local storage on the client device.

[0038] The image referred to herein can include a digital image having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). An image can be a still image (e.g., a still photograph, an image with a single frame, etc.), a moving image (e.g., an animation, an animated GIF, a motion picture where part of the image is in motion and the rest is static, etc.), or a video (e.g., an image or sequence of image frames that may include audio). While the remainder of this document refers to images as still images, it is understood that the techniques described herein are applicable to moving images, videos, etc. For example, the embodiments described herein can be used for still images (e.g., photographs or other images), videos, or moving images.

[0039] Figure 2 This is a flowchart illustrating an example method 200 for denoising a noisy input image according to some embodiments. In some embodiments, method 200 may be, for example, in... Figure 1The method is implemented on the server system 102 shown. In some implementations, some or all of method 200 can be implemented on, for example... Figure 1 The method 200 may be implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or on both server devices and client devices. In the described examples, the implemented system includes one or more digital processors or processing circuitry systems (“processors”) and one or more storage devices (e.g., database 106 or other storage). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 200. In some examples, the first device is described as the block that performs method 200. Some implementations may have one or more blocks of method 200 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0040] In some implementations, method 200 or portions thereof may be initiated automatically by the system. In some implementations, the implementing system is the first device. For example, the method (or portions thereof) may be executed periodically or based on one or more specific events or conditions, such as a camera capturing an image, an image being added to an image library, an image being received (e.g., from another device, via a scanner, etc.) or being viewed (e.g., in an image viewing application), a predetermined time period having elapsed since the last execution of method 200, and / or one or more other conditions that may occur as specified in settings read by the method.

[0041] Method 200 may begin at box 202. At box 202, it is checked whether user consent (e.g., user permission) has been obtained for the use of user data in implementing method 200. For example, user data may include one or more images, such as images captured or viewed by a user using a client device (e.g., any of devices 120 to 126). In some embodiments, one or more boxes of the method described herein may use such user data.

[0042] If user consent has been obtained from the relevant user for whom user data can be used in method 200, then in box 204, it is determined that the boxes of the method herein can be implemented where user data as described for those boxes is available, and the method proceeds to box 210. If user consent has not been obtained, then in box 206 it is determined that the boxes to be implemented without using user data are to be implemented, and the method proceeds to box 210. In some embodiments, if user consent has not been obtained, the boxes are implemented without using user data and using synthetic data and / or general or publicly accessible and publicly usable data. In some embodiments, if user consent has not been obtained, the remainder of method 200 is not performed.

[0043] At box 212, a noisy input image is obtained. The noisy input image comprises multiple pixels, each with an associated pixel value, such as the value of each of the red, green, and blue (RGB) channels of an RGB image, or an additional value. In some implementations, the pixel values ​​of the noisy input image may include luminance values, depth values, or other values.

[0044] For example, a noisy input image may be captured by a camera or scanner of a client device (any of client devices 120 to 126); received from another device (e.g., via a network or through a local interface when method 200 is implemented on a server or client device); received from a hardware scanner; received from a camera device (e.g., a security camera or other camera), etc. In some embodiments, the noisy input image may be captured under low-light conditions. The noise in the noisy input image may include shot noise, which depends on the total amount of light captured (e.g., based on exposure time) and is therefore signal-dependent. The noise in the noisy input image may also include readout noise introduced by the sensor used to capture the noisy input image (e.g., a complementary metal-oxide-semiconductor (CMOS) sensor, a charge-coupled device (CCD) sensor, or other type of sensor used to capture digital images). Readout noise depends on the sensor and is therefore signal-independent, such that readout noise does not depend on the amount of light captured. Box 212 may be followed by box 214.

[0045] At box 214, a timemap of the noisy input image is estimated based on the noisy input image and the noise within the input image. In some implementations, each pixel p in the noisy input image may be associated with a corresponding noise level. A corresponding time step value may generate this noise level during the diffusion process. The maximum noise level on the image (e.g., the three channels R, G, and B) may define the timemap. In other words, each pixel in the noisy input image may be at a corresponding stage of the diffusion process.

[0046] In some implementations, the temporal map is a spatial variation temporal map, such that the temporal map includes a corresponding time step value (e.g., including a time step value of a temporal embedding) for each pixel of the noisy input image. In some implementations, at least two pixels of the noisy input image may have different time step values. In some implementations, an affine transformation can be applied to each pixel using the time step value, independent of transformations applied to other pixels of the noisy input image. In some implementations, each pixel is associated with a corresponding time value, which is converted into a temporal embedding (vector). In these implementations, the vectors of all pixels are multiplied by a matrix (which is an affine transformation). The time step values ​​in the noisy input image may be based on the noise.

[0047] In some implementations, noise in the noisy input image can be determined before estimating the time map. In these implementations, determining the noise level may include determining the readout noise (σ) from the noisy input image. r ) and shot noise (σ s In these implementations, readout noise is the standard deviation of signal-independent noise (e.g., sensor characteristics based on the sensor capturing a noisy input image), and shot noise is the standard deviation of signal-dependent noise (e.g., influenced by conditions such as the total amount of light captured by the sensor when capturing a noisy input image).

[0048] In some implementations, noise can be estimated using a cropped version of the noisy input image. Box 214 may be followed by box 216.

[0049] At box 216, a diffusion process using a machine learning model is performed to obtain a denoised image. In some implementations, the machine learning model may be a trained machine learning model trained to perform spatial variability noise removal (SVNR). In some implementations, the following reference may be used. Figure 3 The method described is 300 to perform the diffusion process. Box 216 may be followed by box 218.

[0050] At box 218, the denoised image is output. For example, the denoised image may be displayed on a display device (e.g., a monitor, projector, augmented reality device, or other device capable of displaying images). In some embodiments, the denoised image may be output to a printer to produce a physical print of the denoised image. In some embodiments, the denoised image may be output to a system that performs additional operations (e.g., editing, compression, format conversion, or other operations).

[0051] The boxes in method 200 can be combined, split into multiple boxes, or executed in parallel. Method 200 or its parts can be repeated any number of times using additional input. For example, method 200 can be repeated until all images captured by the camera have been processed to obtain the corresponding denoised image.

[0052] Figure 3 This is a flowchart illustrating an example method 300 for performing a diffusion process according to some embodiments. In some embodiments, method 300 may be, for example, in... Figure 1 The method is implemented on the server system 102 shown. In some implementations, some or all of method 200 can be implemented on, for example... Figure 1 The method 300 may be implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or on both server devices and client devices. In the described examples, the implemented system includes one or more digital processors or processing circuitry systems (“processors”) and one or more storage devices (e.g., database 106 or other storage). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 300. In some examples, the first device is described as the block that performs method 300. Some implementations may have one or more blocks of method 300 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0053] Method 300 can begin at box 302. At box 302, a machine learning-based model (a trained machine learning model) is used to estimate the denoised image. The model is trained to perform spatial variability noise removal (SVNR) based on a temporal map, as described, for example, in box 214 of reference to method 200, and may be referred to herein as the SVNR model. Reference Figure 6 This describes an example method for training a machine learning model to perform spatial variability noise removal (SVNR). In some implementations, method 300 starts from the highest value in the time graph.

[0054] In some implementations, the SVNR model estimates the denoised image based on a noisy input image, the image at the current time step (which, for the first time step, may be the noisy input image), and the time step. Box 302 may be followed by box 304.

[0055] At box 304, the image of the next time step is obtained. In some implementations, the image of the next time step (x) is obtained. t_hat-1 () is based on the denoised image (x0) obtained in box 302 and the image at the current time step (x). t_hatIn some implementations, obtaining the next time step image may further include sampling for Gaussian noise. In some implementations, sampling may depend on the current time map (and the time step image) to determine the appropriate weight for each of the components. Box 304 may be followed by box 306.

[0056] At box 306, it is determined whether at least one pixel is associated with a time step value greater than or equal to the current time step. If it is determined that at least one pixel is associated with a time step value greater than or equal to the current time step, then box 306 may be followed by box 316. Otherwise, box 302 may be followed by box 308.

[0057] At box 308, the image of the next time step (e.g., the one obtained in box 304) is set as the image of the current time step. Box 308 may be followed by box 310.

[0058] At box 310, decrease the time step. Box 310 can be followed by box 312.

[0059] At box 312, it is determined whether the current time step is positive. A positive time step can indicate that at least one pixel in the noisy input image has a time step value greater than zero, meaning the pixel diffusion process is incomplete. If the current time step is positive, then box 312 is followed by box 203. In the implementation of box 216 where method 300 is used to implement method 200, if the current time step is not positive, then box 312 can be followed by box 218 to output the denoised image.

[0060] At box 316 (which is executed if it is determined that at least one pixel is associated with a time step value greater than or equal to the current time step), the next time step image (obtained in box 304) is updated using the corresponding value from the denoised image (obtained in box 302). This ensures that further denoising is avoided for pixels that have reached a time step value of zero (already denoised based on the time map), and instead, their values ​​are stable as obtained in the denoised image, while continuing until further denoising is performed on other pixels. This provides spatially varied noise removal, as different pixels may reach a time step value of zero in different iterations of method 300. During the update, if the value of pixel p in the time map is less than 0, the value of the pixel is set to X_{t_hat}(p) (e.g., instead of setting the value to X_{t_hat-1}(p)). Method 300 can be executed over at least two time steps.

[0061] Replacing a single time step value (which defines the number of iterations to be performed) with a spatial variation temporal map results in the diffusion operation being performed pixel-by-pixel. The use of pixel-specific time step values ​​(temporal embeddings) in the temporal map makes noise removal by the SVNR model dependent on the level of spatial variation noise in the noisy input image. Therefore, denoising using the SVNR model can reduce signal-dependent shot noise along with signal-independent readout noise.

[0062] refer to Figure 3 The diffusion process described is performed using a noisy input image as a conditional image.

[0063] Figure 4A An example image is shown. Figure 4A A noisy input image of a lizard is depicted (404). In some implementations, a noisy input image can be generated by adding simulated noise to the image. For example, the noisy input image can be captured under low-light conditions or under other conditions attributable to the noise. Noise in the image is shown (402), where the noise values ​​(shown in different shades) are the standard deviation (quantization) of the spatially varied noise in the image. A corresponding clean image is also shown (406). For example, the clean image can be captured under good lighting conditions or using a camera capable of capturing relatively noise-free images under low-light conditions.

[0064] Furthermore, in some implementations, noisy images can be generated by applying spatial variation noise to a known clean image (e.g., for use in training a machine learning model). In this case, the known clean image can be used as the ground truth during the training of the machine learning model, for example, to train the machine learning model to remove noise from the generated noisy image. For example, for the reference above... Figure 2 and Figure 3 The described technique allows noisy images to be used as conditional images and compared with ground truth images, for example, to train a machine learning model to remove noise from noisy images by comparing the ground truth images with the output of a diffusion process. The technique described in this paper addresses (removes) spatially variability noise by applying pixel-wise temporal embeddings, as referenced above. Figure 2 As shown in the spatial variation time diagram.

[0065] Figure 4AA denoised image (408) obtained by applying an example prior art that does not utilize diffusion to a noisy input image (404) is also depicted. Figure 4 further illustrates another example of a denoised image (410) obtained by applying a prior art diffusion technique to a noisy input image (404). The denoised image (410) is obtained after 1000 time steps of the diffusion process. As can be seen in the denoised images (408, 410), while such prior art denoising methods remove high levels of noise, they over-smooth fine details and / or the resulting denoised images lack high-frequency details. For example, although the denoised image 410 has more high frequencies than the denoised image 412, a portion of the lizard's head in the clean image (406) is missing in the denoised image 410, and there is no sharp boundary in the denoised image 408. Although image 410 shows the missing portion, in some cases, prior art can add details that are not present in the image. Furthermore, in the denoised images 408 and 410, the skin areas below the eyes and above the torso lack fine details.

[0066] Figure 4A Further description of applying the references in this article Figure 2 and Figure 3 The denoised image (412) obtained by the described spatial variability noise removal technique. As can be seen, the denoised image includes fine details, such as correctly depicting the head portion and the skin area below the eyes. As can be seen, the spatial variability noise removal technique described herein is able to recover such textures even when it is difficult to distinguish textures in a noisy input image. As can be seen, the described technique can produce clean images with higher fidelity while reducing the runtime (and thus computational cost) of the diffusion process by several orders of magnitude (25 time steps compared to 1000 steps).

[0067] Figure 4B Depicting Figure 4A The image consists of a noisy image (404) and a denoised image (412), and an intermediate denoised image (420) obtained after 15 time steps. Conventional diffusion techniques (e.g., the diffusion technique that generates image 410) start from the complete noisy image, rather than from the noisy input image, and therefore utilize much more computational resources (e.g., because the diffusion process in this example runs for 1000 steps to obtain the denoised image 410). In contrast, the spatial variability noise removal technique described herein initiates the diffusion process from the noisy image (404) and can generate the denoised image at a lower computational cost. Figure 4BAs can be seen, after 15 time steps in the diffusion process, the spatial variation noise removal technique generates a denoised image (420) that is significantly improved compared to the noisy input image (404).

[0068] Figure 5 Several examples of noisy images (502A to 510A) and corresponding denoised images (512A to 520A) and ground truth images (522A to 530A) are shown. Figure 5 Also shown are close-up (magnified) views (502B to 510B) of a portion of each noisy image (indicated by small squares in the image), as well as corresponding portions of the denoised images (512B to 520B) and the ground truth images (522B to 530B). (See from...) Figure 5 As can be seen, the described technique achieves high signal fidelity (matching the ground truth image) without significant distribution drift. In the example shown in Figure 4, the noise gain is 16. Gain refers to the amplification of the signal from the camera sensor (ISO) when the image is captured. Higher gain indicates poorer image capture conditions, meaning a higher noise level in the image. Even at relatively high noise levels, the spatial variability noise removal technique described herein can generate denoised images that include fine high-frequency details while eliminating a significant amount of noise. Furthermore, unlike other diffusion-based methods that exhibit illusions (adding details not present in the ground truth image to the denoised image), this technique delivers high fidelity to the ground truth image.

[0069] The described technique produces more realistic images while removing a significant amount of noise. The technique maintains fidelity to the underlying signal (in noisy images). Furthermore, diffusion techniques starting from complete noise have a fixed computational cost (runtime) based on the number of time steps to reach the denoised image. This computational cost is independent of the noise level in the noisy input image. In contrast, the computational cost (runtime) of the technique described herein varies based on the noise level in the noisy input image (higher noise requires more cost). Moreover, since the described technique can initiate the diffusion process from a noisy input image, its computational cost can be significantly (e.g., orders of magnitude) lower than that of diffusion techniques starting from complete noise.

[0070] Figure 6 This is a flowchart illustrating an example method 600 for training a machine learning model to denoise a noisy input image, according to some embodiments. In some embodiments, method 600 may, for example, be in... Figure 1 The method is implemented on the server system 102 shown. In some implementations, some or all of method 600 can be implemented on, for example... Figure 1The method 600 may be implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or on both server devices and client devices. In the described examples, the implemented system includes one or more digital processors or processing circuitry systems (“processors”) and one or more storage devices (e.g., database 106 or other storage). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 600. In some examples, the first device is described as the block that performs method 600. Some implementations may have one or more blocks of method 600 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0071] In some implementations, method 600 or portions thereof may be initiated automatically by the system. In some implementations, the implementing system is the first apparatus. For example, the method (or portions thereof) may be executed periodically or based on one or more specific events or conditions, such as new or additional training data becoming available, a change in image capture hardware (e.g., image capture hardware that can affect noise in captured images) or a new type of image capture hardware, a predetermined time period elapsed since the last execution of method 600, and / or one or more other conditions that may occur in settings read by the method.

[0072] Training data (e.g., one or more of ground truth images, noisy input images, etc.) is obtained with specific user permission. In some implementations, training data may include images captured specifically for training a machine learning model. In some implementations, training data may include synthetic or generated images, such as images generated using generative artificial intelligence or other suitable image generation techniques. If the training data includes user-provided images, such images are used only with specific user permission (for training a machine learning model to perform noise removal) and for a specific purpose. Users may be provided with the option to exclude their captured images from the training data.

[0073] At box 602, a clean image is obtained. For example, a clean image could be an image captured by a camera such that it has little or no noise (e.g., under good lighting conditions). In some implementations, a clean image can be obtained from training data that includes a large number of training images (e.g., training images representing real-world images). Box 602 may be followed by box 604.

[0074] At box 604, a noisy input image is generated by adding noise to the clean image. This can be based on the standard deviation of the noise per pixel (σ). p This is used to determine the noise to be added. In some implementations, the noise standard deviation (σ) is used. p ) is readout noise (σ) r ) and shot noise (σ s The readout noise is a combination of signal-independent noise and shot noise, where the readout noise is the standard deviation of signal-dependent noise and the shot noise is the standard deviation of signal-dependent noise. In some implementations, the readout noise and shot noise are randomly selected. Box 604 may be followed by box 606.

[0075] At box 606, the time map (T) is determined based on the noisy input image and the noise in the noisy input image. The temporal map is spatially varied, such that a corresponding time step value is associated with each pixel of the noisy input image, such that at least two pixels have different time step values. Box 606 may be followed by box 608.

[0076] At box 608, a new timemap (t_hat) is generated such that the corresponding time step value of the pixel in the new timemap is lower than the timestamp value of the corresponding pixel in the original timemap. For example, this can be achieved by... (The sentence is incomplete and requires more context to translate accurately.) A new timemap is generated by subtracting a scalar from the corresponding time step value of each pixel in the zero-sum timemap. The scalar can be (e.g., randomly) chosen as the maximum time step value (T) in the zero-sum timemap. The values ​​between ) are possible. Box 608 can be followed by box 610.

[0077] At box 610, the current time step image (x) is generated based on the new time map (e.g., the noise level from the new time map). t_hat The current time-step image is generated such that it has less noise than the noisy input image (y). In other words, the current time-step image is a slightly denoised version of the noisy input image (y). In some implementations, the current time-step image is generated such that it has a noise distribution similar to that of the noisy input image. Box 610 may be followed by box 612.

[0078] In diffusion techniques, the timestamp is linearly related to the number of forward diffusion steps, which in turn is linearly related to the noise variance. In this case, determining the timestamp is equivalent to determining the noise level and can be performed once at the start of inference. During training, since the pixel-wise noise variance is known, the timestamp can be directly computed using the pixel-wise noise variance, or evaluated from the image. This is advantageous because it is similar to the inference process.

[0079] At box 612, the noisy input image (y) and the current time step image (x) are displayed. t_hat The new time-series image and the new time-series image are provided as input to the machine learning model being trained. The machine learning model generates a denoised image (x... 0_hat Box 612 can be followed by box 614.

[0080] At box 614, based on the denoised image (x) generated by the machine learning model. 0_hat The loss value is calculated by comparing the clean image (x0) with the denoised image generated by the machine learning model being trained. The loss value indicates the difference between the clean image (which is the ground truth) and the denoised image. Box 614 may be followed by box 616.

[0081] At box 616, one or more parameters of the machine learning model are updated based on the loss value. In some implementations, the machine learning model may include a neural network. In these examples, updating one or more parameters of the machine learning model based on the loss value includes performing backpropagation to adjust the weights of one or more nodes of the neural network such that the loss value is reduced. Box 616 may be followed by box 618.

[0082] At box 618, it is determined whether a stopping criterion has been met. In some implementations, the stopping criterion may include one or more of the following: the computational budget used for training has been exhausted, all clean images in the clean image training dataset have been utilized, or the change in the parameter value of at least one of the parameters between successive iterations has fallen below a threshold. If the stopping criterion is not met, box 602 may follow box 618. If the stopping criterion is met, box 620 may follow box 618.

[0083] At box 620, the machine learning model (a trained model, which is a version of the machine learning model output after parameter tuning) can be used to perform spatial variation noise removal from any image. For example, the machine learning model can be used with reference... Figure 2 and Figure 3 The method described is used for noise removal.

[0084] The boxes in method 600 can be combined, split into multiple boxes, or executed in parallel. For example, boxes 604 to 616 can be executed multiple times for the same clean image, for example, to generate different noisy versions of the clean image each time box 604 is executed, and the model can be trained using each of the generated noisy versions. In some implementations, boxes 608 to 616 can be executed multiple times for the same noisy input image using different scalar values ​​(e.g., scalar values ​​that can be randomly sampled between 0 and the maximum value in the time map) to generate a new time map at box 608, and the loss value can be determined using the corresponding estimated denoised image. During training, the model's task is to evaluate the clean image x_0 given the time map. In some implementations, method 600 can be executed in parallel with respect to multiple images or portions of method 600 (e.g., boxes 604 to 614 or 604 to 616 for the same clean image; boxes 608 to 614 or 608 to 616 for the same noisy input image, etc.). Method 600 or its components can be repeated any number of times using additional input (e.g., a new training image received after the initial machine learning model is obtained at box 620). For example, performing model retraining can improve the quality of the denoised images generated by the machine learning model. During model training, images of different sizes and / or different noise levels can be used.

[0085] Figure 7 This is a block diagram of an example device 700 that can be used to implement one or more features described herein. In one example, device 700 can be used to implement a client device, for example, Figure 1 Any of the client devices shown in client device 115. Alternatively, device 700 may implement a server device, such as server 101. In some embodiments, device 700 may be used to implement a client device, a server device, or both a client and a server device. Device 700 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0086] One or more methods described herein can operate in the following ways: a standalone program that can execute on any type of computing device, a program that runs on a web browser, or a mobile application (“app”) that runs on a mobile computing device (e.g., a mobile phone, smartphone, tablet computer, wearable device (watch, armband, jewelry, headband, virtual reality goggles or glasses, augmented reality goggles or glasses, head-mounted display, etc.), laptop computer, etc.). In one example, a client / server architecture can be used, such as a mobile computing device (as a client device), sending user input data to a server device and receiving final output data from the server for output (e.g., for display). In another example, all computation can be performed within a mobile app (and / or other apps) on a mobile computing device. In yet another example, computation can be split between a mobile computing device and one or more server devices.

[0087] In some embodiments, device 700 includes processor 702, memory 704, and input / output (I / O) interface 706. Processor 702 may be one or more processors and / or processing circuitry for executing program code and controlling the basic operations of device 700. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. Processors may include systems having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on a neural network model, neural circuitry, a processor optimized for matrix computation (e.g., matrix multiplication), or other systems. In some embodiments, processor 702 may include one or more coprocessors implementing neural network processing. In some embodiments, processor 702 may be a processor that processes data to produce probabilistic outputs; for example, the output produced by processor 702 may be imprecise or accurate within a range from the expected output. Processing is not limited to a specific geographical location or time constraints. For example, a processor can perform its functions "in real time," "offline," or in "batch mode." Different parts of the processing can be executed by different (or the same) processing systems at different times and locations. A computer can be any processor that communicates with memory.

[0088] Memory 704 is typically disposed in device 700 for access by processor 702 and may be any suitable processor-readable storage medium suitable for storing instructions for execution by the processor and located separately from and / or integrated with processor 702, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 704 may store software operated by processor 702 on server device 700, including operating system 708, machine learning application 730, other applications 712, and application data 714. Other applications 712 may include applications such as data display engines, web hosting engines, image display engines, notification engines, social networking engines, etc. In some embodiments, machine learning application 730 and other applications 712 may each include functions that enable processor 702 to perform the functions described herein (e.g., Figure 2 , Figure 3 and / or Figure 6 (Some or all of the instructions in the method).

[0089] Other applications 712 may include, for example, image editing applications, media display applications, communication applications, web hosting engines or applications, map applications, media sharing applications, etc. One or more methods disclosed herein can operate in various environments and platforms, such as as a standalone computer program that can run on any type of computing device, as a web application with web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0090] In various implementations, the machine learning application may utilize Bayesian classifiers, support vector machines, neural networks, or other learning techniques. In some implementations, the machine learning application 730 may include a machine learning-trained model 734, an inference engine 736, and data 732. In some implementations, data 732 may include training data, such as data used to generate the trained model 734. For example, training data may include any type of data, such as text, images, audio, video, etc. When the trained model 734 is an image denoising model, the training data may include ground truth clean images.

[0091] Training data can be obtained from any source, such as a data repository specifically tagged for training, or data licensed for use as training data for machine learning. In embodiments where one or more users are licensed to use their respective user data to train a machine learning model (e.g., trained model 734), the training data may include such user data. In embodiments where users are licensed to use their respective user data, data 732 may include licensed data, such as images (e.g., photographs or other user-generated images).

[0092] In some implementations, training data may include synthetic data generated for training purposes, such as data not based on user input or activity in the training context, for example, data generated from simulated photographs or other computer-generated images. In some implementations, machine learning application 730 does not include data 732. For example, in these implementations, trained model 734 may be generated, for example, on different devices and provided as part of machine learning application 730. In various implementations, trained model 734 may be provided as a data file including model structure or form and associated weights. Inference engine 736 may read the data file for trained model 734 and implement a neural network having node connectivity, layers, and weights based on the model structure or form specified in trained model 734.

[0093] In some implementations, the trained model 734 may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., "hidden layers" between the input and output layers, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results of processing from each tile), a sequence-to-sequence neural network (e.g., a network that takes sequence data such as words in a sentence or frames in a video as input and produces a sequence of results as output), and so on. The model form or structure may specify the connectivity between various nodes and the organization of nodes to layers.

[0094] For example, nodes in the first layer (e.g., the input layer) can receive data as input data 732 or application data 714. For example, when the trained model 734 is an image denoising model, the input data may include a clean image. For example, when the trained model is used for image analysis or image generation, such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive the outputs of nodes in the previous layer as input, according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers or latent layers.

[0095] The final layer (e.g., the output layer) produces the output for machine learning applications. For example, the output could be a denoised image. In some implementations, the model form or structure also specifies the number and / or type of nodes in each layer.

[0096] In various implementations, the trained model 734 may include multiple nodes arranged in layers according to a model structure or form. In some implementations, a node may be a memoryless computational node, for example, configured to process an input unit to produce an output unit. Computations performed by a node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum using a bias or intercept value to produce a node output. In some implementations, computations performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some implementations, the step / activation function may be a nonlinear function. In various implementations, such computations may include operations such as matrix multiplication. In some implementations, computations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, individual processing units of a GPU, or a dedicated neural circuit system. In some implementations, a node may include memory, for example, the ability to store one or more earlier inputs and use one or more earlier inputs when processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. An LSTM node may use memory to maintain a “state” that allows the node to function like a finite state machine (FSM). Models with such nodes can be useful in processing sequential data (e.g., words in a sentence or paragraph, frames in a video, speech or other audio, etc.). In some implementations, models with attention mechanisms and transformer networks can be used.

[0097] In some implementations, the trained model 734 may include embeddings or weights for each node. For example, the model may be initiated as multiple nodes organized into layers as specified by the model form or structure. During initialization, appropriate weights may be applied to the connections between each pair of nodes connected according to the model form (e.g., nodes in consecutive layers of a neural network). For example, the appropriate weights may be randomly assigned or initialized to default values. The model may then be trained, for example, using data 732 to produce results.

[0098] For example, training may include applying supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., a set of noisy input images) and a corresponding expected output for each input (e.g., a set of clean images corresponding to the noisy input images). The values ​​of the weights are automatically adjusted based on a comparison between the model's output and the expected output, for example, in a way that increases the probability that the model will produce the expected output when provided with similar inputs.

[0099] In some implementations, training may include applying unsupervised learning techniques. In unsupervised learning, only input data may be provided, and a model may be trained to distinguish the data, for example, by clustering the input data into multiple groups, where each group includes input data that is similar in some way.

[0100] In some implementations, unsupervised learning can be used to generate knowledge representations, which may be used, for example, by a machine learning application 730. In various implementations, the trained model includes a set of weights or embeddings corresponding to the model structure. In implementations where data 732 is omitted, the machine learning application 730 may include a trained model 734 based on prior training, for example, by the developer of the machine learning application 730, a third party, etc. In some implementations, the trained model 734 may include a fixed set of weights (e.g., downloaded from a server providing the weights).

[0101] The machine learning application 730 also includes an inference engine 736. The inference engine 736 is configured to apply a trained model 734 to data (e.g., application data 714) to provide inference. In some embodiments, the inference engine 736 may include software code to be executed by the processor 702. In some embodiments, the inference engine 736 may specify a circuit configuration (e.g., for a programmable processor, for a field-programmable gate array (FPGA), etc.) that enables the processor 702 to apply the trained model. In some embodiments, the inference engine 736 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, the inference engine 736 may provide an application programming interface (API) that can be used by the operating system 708 and / or other applications 712 to invoke the inference engine 736, for example, to apply the trained model 734 to application data 714 to generate inference. For example, the inference of an image denoising model may be a noisy image, while the inference is a denoised image, as referenced above. Figures 2 to 6 As stated above.

[0102] Machine learning application 730 can provide several technical advantages. For example, when a trained model 734 is generated based on unsupervised learning, the inference engine 736 can apply the trained model 734 to generate knowledge representations (e.g., numerical representations) based on input data (e.g., application data 714). For example, a model trained for image analysis can generate representations of images with a smaller data size (e.g., 1 KB) than the input image (e.g., 10 MB). In some implementations, such representations can help reduce processing costs (e.g., computational costs, memory usage, etc.) to generate outputs (e.g., labels, classifications, sentences describing the image, denoised images from noisy images, etc.).

[0103] In some implementations, such representations can be provided as input to different machine learning applications that generate outputs based on the output of inference engine 736. In some implementations, the knowledge representation generated by machine learning application 730 can be provided, for example, via a network to different devices for further processing. In such implementations, providing knowledge representations instead of images can provide technical benefits, such as enabling faster data transmission at reduced cost.

[0104] In some implementations, the machine learning application 730 may be implemented offline. In these implementations, a trained model 734 may be generated in the first stage and provided as part of the machine learning application 730. In some implementations, the machine learning application 730 may be implemented online. For example, in such implementations, an application that invokes the machine learning application 730 (e.g., one or more of operating system 708, other applications 712) may utilize the inference generated by the machine learning application 730, for example, by providing the inference to a user, and may generate system logs (e.g., actions taken by the user based on the inference, if permitted by the user; or the results of further processing, if used as input for further processing). The system logs may be generated periodically, for example, hourly, monthly, quarterly, etc., and may be used, with user permission, to update the trained model 734, for example, by updating the embeddings of the trained model 734.

[0105] In some implementations, the machine learning application 730 may be implemented in a manner suitable for a specific configuration of the device 700 on which the machine learning application 730 may be executed. For example, the machine learning application 730 may determine a computation graph utilizing available computing resources (e.g., processor 702). For example, if the machine learning application 730 is implemented as a distributed application across multiple devices, the machine learning application 730 may determine the computations to be performed on each device in a computationally optimized manner. In another example, the machine learning application 730 may determine that processor 702 includes a GPU with a specific number of GPU cores (e.g., 1000) and implement an inference engine accordingly (e.g., as 1000 separate processes or threads).

[0106] In some implementations, the machine learning application 730 may implement a collection of trained models. For example, trained models 734 may include multiple trained models, each adapted to the same input data. In these implementations, the machine learning application 730 may select a particular trained model, for example, based on available computing resources, the success rate of previous inferences, etc. In some implementations, the machine learning application 730 may execute an inference engine 736, causing multiple trained models to be applied. In these implementations, the machine learning application 730 may combine the outputs from applying the various models, for example, using a voting technique that scores the individual outputs from applying each trained model, or by selecting one or more specific outputs. Further, in these implementations, the machine learning application may apply a time threshold (e.g., 0.5 ms) for applying the various trained models and utilize only those individual outputs that are available within the time threshold. Outputs not received within the time threshold may not be utilized, for example, and may be discarded. Such an approach may be suitable, for example, when there is a time limit specified when the machine learning application is invoked, for example, via operating system 708 or one or more applications 712.

[0107] In different implementations, the machine learning application 730 can produce different types of output. For example, the machine learning application 730 can provide representations or clusters (e.g., numerical representations of input data), labels (e.g., labels for input data including images, documents, etc.), phrases or sentences (e.g., phrases or sentences describing images or videos, suitable for use as responses to input sentences, etc.), images (e.g., denoised images generated by the machine learning application in response to input images, such as noisy images), audio or video (e.g., in response to input video, the machine learning application 730 can produce output videos with specific effects applied, such as output videos rendered in the style of a comic book or a specific artist when training training model 734 using training data from a comic book or a specific artist), etc. In some implementations, the machine learning application 730 can produce output based on a format specified by a calling application (e.g., operating system 708 or one or more applications 712). In some implementations, the calling application can be another machine learning application. For example, such a configuration can be used for generative adversarial networks, where the calling machine learning application trains using output from the machine learning application 730, and vice versa.

[0108] Any software in the memory 704 may alternatively be stored in any other suitable storage location or computer-readable medium. Additionally, the memory 704 (and / or other connected storage devices) may store one or more messages, one or more taxonomy, electronic encyclopedias, dictionaries, thesaurus, knowledge bases, message data, grammar, user preferences, and / or other instructions and data used in the features described herein. The memory 704 and any other type of storage (disk, optical disc, magnetic tape, or other tangible media) may be considered "storage" or a "storage device".

[0109] I / O interface 706 provides functionality that enables server device 700 to interface with other systems and devices. Devices connected via the interface may be included as part of device 700, or may be separate and communicate with device 700. For example, network communication devices, storage devices (e.g., memory and / or database 106), and input / output devices may communicate via I / O interface 706. In some embodiments, the I / O interface may connect to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, camera, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0110] Examples of interface-connected devices that can be connected to I / O interface 706 may include one or more display devices 720, which may be used to display content such as images, videos, and / or user interfaces for output applications as described herein. Display device 720 may be connected to device 700 via a local connection (e.g., a display bus) and / or via a network connection, and may be any suitable display device. Display device 720 may include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3D display, or other visual display device. For example, display device 720 may be a flat panel display on a mobile device, multiple displays in goggles or a head-mounted device, or a monitor screen for a computer device.

[0111] I / O interface 706 can be connected to other input and output devices. Some examples include one or more cameras capable of capturing images. Some implementations may provide a microphone for capturing sound (e.g., as part of a captured image, voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0112] For ease of explanation, Figure 7One block is shown for each of processor 702, memory 704, I / O interface 706, and software blocks 708, 712, and 730. These blocks may represent one or more processors or processing circuitry systems, operating systems, memory, I / O interfaces, applications, and / or software modules. In other embodiments, device 700 may not have all of the components shown and / or may have other elements, including other types of elements that serve as alternatives to or supplements to those shown herein. While some components are described as performing the blocks and operations as described in some embodiments herein, any suitable component or combination of components from one or more suitable processors associated with environment 100, device 700, similar systems, or any suitable environment 100, device 700, or similar systems may perform the described blocks and operations.

[0113] The methods described herein can be implemented using computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry systems) and can be stored on a computer program product, including non-transitory computer-readable media (e.g., storage media), such as magnetic storage media, optical storage media, electromagnetic storage media, or semiconductor storage media, including semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), flash memory, hard disks, optical disks, solid-state memory drives, etc. Program instructions can also be contained in and provided as electronic signals, for example, in the form of Software as a Service (SaaS) delivered from a server (e.g., a distributed system and / or cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or a component of an application running on a system, or as an application or software running alongside other applications and an operating system.

[0114] Although the specification has described specific implementations, these implementations are illustrative and not limiting. The concepts shown in the examples can be applied to other examples and implementations.

[0115] In some implementations discussed herein where personal information about a user (e.g., user data, information about the user's social networks, the user's location and time at that location, the user's biometric information, the user's activity, and demographic information) may be collected or used, the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information, particularly when explicit authorization to do so is received from the relevant user. For example, providing the user with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Presenting one or more options to each user whose personal information is to be collected to allow control over the collection of information related to that user, providing permission or authorization regarding whether information is collected and which parts of the information are to be collected. For example, one or more such control options may be provided to the user via a communication network. Additionally, some data may be processed in one or more ways before it is stored or used, such that personally identifiable information is removed. As an example, a user's identity may be processed so that no personally identifiable information can be determined. As another example, the geographic location of a user device can be generalized to a larger area, making it impossible to determine the user's specific location.

[0116] Note that the functional blocks, operations, features, methods, apparatuses, and systems described in this disclosure can be integrated into or divided into different combinations of systems, apparatuses, and functional blocks, as is known to those skilled in the art. Routines for a particular implementation can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented. Routines can be executed on a single processing device or multiple processors. Although steps, operations, or calculations may be presented in a specific order, the order can be changed in different specific implementations. In some implementations, multiple steps or operations shown as sequential in this specification can be performed simultaneously.

Claims

1. A computer-implemented method for denoising a noisy input image, the method comprising: Obtain the noisy input image (212), wherein the noisy input image comprises multiple pixels; (214) A time map is estimated based on the noisy input image and the noise in the noisy input image, wherein the time map is spatially varied such that a corresponding time step value is associated with each pixel of the noisy input image, and wherein at least two pixels have different time step values. A diffusion process (216) is performed on the noisy input image, wherein the diffusion process is performed at two or more time steps, starting from the highest value in the time graph, wherein at each time step: The denoised image is estimated (302) using a machine learning model based on the noisy input image, the current time step image, and the time step. (304) The next time step image is obtained based on the denoised image and the current time step image; For each pixel, if the time step is greater than or equal to the time step value associated with the pixel, the next time step image is updated (316) using the corresponding pixel value from the denoised image; After the update, the next time step image is set (308) to the current time step image; and Decrease the time step (310), When the time step is positive (312), the denoised image is repeatedly estimated (302), the next time step image is obtained (304), the next time step image is updated (316) using the corresponding pixel values ​​from the denoised image, the next time step image is set (308) to the current time step image, and the time step is decremented (310); and Output the denoised image described in (218).

2. The computer-implemented method of claim 1, further comprising determining the noise in the noisy input image.

3. The computer-implemented method of claim 2, wherein each pixel has a corresponding noise level, and wherein determining the noise includes determining readout noise and shot noise from the noisy input image, wherein the readout noise is the standard deviation of signal-independent noise, and the shot noise is the standard deviation of signal-dependent noise.

4. The computer-implemented method of claim 3, wherein the time step value of the time map is based on the spatial variation noise of the noisy input image.

5. The computer-implemented method of claim 1, wherein at a first time step during the execution of the diffusion process, the noisy input image is used as the current time step image.

6. The computer-implemented method of claim 1, wherein the noisy input image serves as a conditional image during the diffusion process.

7. A computing device, comprising: processor; as well as A memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations including: Obtain (212) a noisy input image, the noisy input image comprising multiple pixels; (214) A time map is estimated based on the noisy input image and the noise in the noisy input image, wherein the time map is spatially varied such that a corresponding time step value is associated with each pixel of the noisy input image, and wherein at least two pixels have different time step values. A diffusion process (216) is performed on the noisy input image, wherein the diffusion process is performed at two or more time steps, starting from the highest value in the time graph, wherein at each time step: The denoised image is estimated (302) using a machine learning model based on the noisy input image, the current time step image, and the time step. (304) The next time step image is obtained based on the denoised image and the current time step image; For each pixel, if the time step is greater than or equal to the time step value associated with the pixel, the next time step image is updated (316) using the corresponding pixel value from the denoised image; After the update, the next time step image is set (308) to the current time step image; and Decrease the time step (310), When the time step is positive (312), the denoised image is repeatedly estimated (302), the next time step image is obtained (304), the next time step image is updated (316) using the corresponding pixel values ​​from the denoised image, the next time step image is set (308) to the current time step image, and the time step is decremented (310); and Output the denoised image described in (218).

8. The computing device of claim 7, wherein the operation further comprises determining the noise in the noisy input image.

9. The computing device of claim 8, wherein each pixel has a corresponding noise level, and wherein determining the noise includes determining readout noise and shot noise from the noisy input image, wherein the readout noise is the standard deviation of signal-independent noise, and the shot noise is the standard deviation of signal-dependent noise.

10. The computing device of claim 9, wherein the time step value of the time map is based on the spatial variation noise of the noisy input image.

11. The computing device of claim 7, wherein at a first time step during the execution of the diffusion process, the noisy input image is used as the current time step image.

12. The computing device of claim 7, wherein the noisy input image serves as a conditional image during the diffusion process.

13. A computer-implemented method for training a machine learning model to denoise a noisy input image, the method comprising: Obtain a clean image (602); (604) A noisy input image is generated by adding noise to the clean image, wherein the noise is determined based on the noise standard deviation per pixel; (606) A time map is estimated based on the noisy input image and the noise in the noisy input image, wherein the time map is spatially varied such that a corresponding time step value is associated with each pixel of the noisy input image, wherein at least two pixels have different time step values. Generate (608) a new time map, wherein the corresponding time step value of the pixel in the new time map is lower than the time step value of the pixel in the time map; (610) Generate a current time step image based on the new time map, wherein the current time step image has lower noise than the noisy input image; The machine learning model is used to estimate (612) the denoised image based on the noisy input image, the current time step image, and the new time map; The loss value is calculated (614) based on the comparison between the denoised image and the clean image; and (616) Update one or more parameters of the machine learning model based on the loss value.

14. The computer-implemented method of claim 13, wherein the noise standard deviation is a combination of readout noise and shot noise, wherein the readout noise is the standard deviation of signal-independent noise and the shot noise is the standard deviation of signal-dependent noise, wherein the readout noise and the shot noise are randomly selected.

15. The computer-implemented method of claim 13, wherein generating the new time map comprises subtracting a scalar from the corresponding time step value of each pixel in the time map, wherein the scalar has a value between zero and the maximum time step value in the time map.

16. The computer-implemented method of claim 13, wherein generating the current time step image based on the new time map includes sampling noise based on the new time map.

17. The computer-implemented method of claim 16, wherein the current time step image has a noise distribution similar to the noise distribution of the noisy input image.

18. The computer-implemented method of claim 13, wherein the machine learning model comprises a neural network, and wherein updating the one or more parameters of the machine learning model based on the loss value comprises performing backpropagation to adjust the weights of one or more nodes of the neural network.

19. The computer-implemented method of claim 13, wherein obtaining the clean image, generating the noisy input image, estimating the time map, generating the new time map, generating the current time step image, estimating the denoised image, calculating the loss value, and updating the one or more parameters of the machine learning model are performed iteratively until a stopping criterion is met.

20. The computer-implemented method of claim 19, wherein the stopping criteria include one or more of the following: the computational budget for training is exhausted, or the parameter value of at least one of the one or more parameters decreases below a threshold between successive iterations.

Images