Spatial variation noise removal from images using diffusion

Abstract

Embodiments relate to a computer-implemented method for de-noising a noisy image. The method comprises the following steps: obtaining a noisy input image; estimating (214) a time map based on the noisy input image and noise in the noisy input image, wherein the time map is spatially varying; and performing a diffusion process on the noisy input image over two or more time steps. At each time step, the method comprises: estimating a denoised image based on the noisy input image, the current time step image, and the time step using a machine-learned model; obtaining a next time step image based on the denoised image and the current time step image; for each pixel, updating the next time step image with a corresponding value from the denoised image; setting the next time step image as the current time step image; and decreasing the time step. When the time step is positive, estimating, obtaining, updating, setting, and decreasing are repeated. The method further includes outputting the denoised image.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 507,194, filed on June 9, 2023, entitled “SPATIALLY VARIANT NOISE REMOVALFROM IMAGES USING DIFFUSION,” the contents of which are incorporated herein in their entirety. Background Art

[0002] Photos, videos, and other images captured using a camera may include noise and / or lack detail due to a variety of factors, such as limitations of the camera's image capture sensor, the lighting conditions under which the image was captured, etc. Computationally removing noise from captured images can improve the quality of the image perceived by a user and restore fine detail.

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure. Summary of the invention

[0004] Some embodiments described herein relate to methods, apparatus, and computer-readable media for removing noise from an image using a machine-learned model trained to perform spatially-variant noise removal. Some embodiments described herein relate to methods, apparatus, and computer-readable media for training a machine-learned model to perform spatially-variant noise removal.

[0005] Some embodiments relate to a computer-implemented method for denoising an image. In some embodiments, a computer-implemented method for denoising a noisy input image includes obtaining a noisy input image including a plurality of pixels. The method further includes estimating a time map based on the noisy input image and noise in the noisy input image, wherein the time map is spatially varying such that a corresponding time step value is associated with each pixel of the noisy input image, and wherein at least two pixels have different time step values. The method further includes performing a diffusion process on the noisy input image, wherein the diffusion process is performed over two or more time steps starting from a highest value in the time map. The diffusion process includes: at each time step: estimating a denoised image based on the noisy input image, a current time step image, and a time step using a machine-learned model (SVNR); obtaining a next time step image based on the denoised image and the current time step image; for each pixel, if the time step is greater than or equal to the time step value associated with the pixel, updating the next time step image with a corresponding value from the denoised image; after updating, setting the next time step image to the current time step image; and decrementing the time step. When the time step is positive, estimating, obtaining, updating, setting and decrementing are repeated. The method further includes outputting a denoised image.

[0006] In some embodiments, the method may further include determining noise in the noisy input image. In these embodiments, each pixel has a corresponding noise level, and determining the noise includes determining read-noise and shot-noise from the noisy input image. In these embodiments, the temporal map encodes spatially varying noise of the noisy input image.

[0007] In some embodiments, at a first time step when performing the diffusion process, the noisy input image is used as the current time step image. In some embodiments, the computer-implemented method of claim 1, wherein the noisy input image serves as a conditional image during the diffusion process.

[0008] Some embodiments relate to a method for training a machine learning model to denoise a noisy input image. In some embodiments, the method for training a machine learning model to denoise a noisy input image includes obtaining a clean image. The method further includes generating a noisy input image by adding noise to the clean image, wherein the noise is determined based on a per-pixel noise standard deviation. The method further includes estimating a time map based on the noisy input image and the noise in the noisy input image. In some embodiments, the time map is spatially varying so that a corresponding time step value is associated with each pixel of the noisy input image, wherein at least two pixels have different time step values. The method further includes generating a new time map, wherein the corresponding time step value of a pixel in the new time map is lower than the timestamp value of the corresponding pixel in the time map. The method further includes generating based on the new time map, wherein the current time step image has lower noise than the noisy input image. The method further includes estimating a denoised image based on the noisy input image, the current time step image, and the new time map using the machine learning model. The method further includes calculating a loss value based on a comparison of the denoised image with the clean image. The method further includes updating one or more parameters of the machine learning model based on the loss value.

[0009] In some embodiments, the noise standard deviation is a combination of read noise and shot noise. In these embodiments, the read noise is the standard deviation of signal-independent noise, and the shot noise is the standard deviation of signal-dependent noise. In some embodiments, the read noise and shot noise are randomly selected.

[0010] In some implementations, generating a new time map includes subtracting a scalar from a corresponding time step value for each pixel in the time map, where the scalar has a value between zero and a maximum time step value in the time map.

[0011] In some embodiments, generating the current time step image based on the new time map includes sampling noise based on the new time map. In some embodiments, the current time step image has a noise distribution similar to the noise distribution of the noisy input image.

[0012] In some embodiments, the machine learning model includes a neural network. In these embodiments, updating one or more parameters of the machine learning model based on the loss value includes performing back propagation to adjust the weights of one or more nodes of the neural network.

[0013] In some embodiments, obtaining a clean image, generating a noisy input image, estimating a time map, generating a new time map, generating a current time step image, estimating a denoised image, calculating a loss value, and updating one or more parameters of a machine learning model are performed iteratively until a stopping criterion is met. In some embodiments, the stopping criterion includes one or more of the following: the computational budget for training is exhausted, or the change in parameter value of at least one of the one or more parameters between consecutive iterations falls below a threshold.

[0014] Some embodiments include a computing device comprising a processor and a memory coupled to the processor, the memory having stored thereon instructions that, when executed by the processor, cause the processor to perform any of the methods described herein.

[0015] Some embodiments include a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A block diagram is shown of an example network environment that may be used in some implementations described herein.

[0017] Figure 2 is a flow chart illustrating an example method for denoising a noisy input image according to some implementations.

[0018] Figure 3 is a flow chart illustrating an example method for performing a diffusion process according to some implementations.

[0019] FIG. 4A to FIG. 4B An example image is shown.

[0020] Figure 5 Examples of noisy images and corresponding denoised and true value images are shown.

[0021] Figure 6 is a flow chart illustrating an example method for training a machine learning model to denoise a noisy input image according to some embodiments.

[0022] Figure 7 is a block diagram of an example apparatus that may be used to implement one or more features described herein. DETAILED DESCRIPTION

[0023] Methods, systems, and non-transitory computer-readable media for noise removal from noisy input images are described herein. The described techniques provide a new diffusion-based framework for the task of image denoising. The techniques utilize naturally rich image priors learned by a generative denoising diffusion model. In some embodiments, the techniques perform denoising diffusion by utilizing a noisy input image as a condition and starting point for the diffusion process. In order to be able to integrate real noisy images as samples into the diffusion process, the techniques employ a denoising diffusion formula utilizing spatially-variant temporal embedding (temporal graph). Techniques for training a machine learning model to perform noise removal using a spatially-variant temporal graph and for removing noise from an image using a machine learning model are described herein. The described techniques can be applied to remove noise from any noisy image, including images with non-uniform noise distribution.

[0024] Figure 1 1 shows a block diagram of an example network environment 100 that can be used in some embodiments described herein. In some embodiments, the network environment 100 includes one or more server systems, such as Figure 1 1. The server system 102 in the example of FIG. 102 can communicate with a network 130, for example. The server system 102 can include a server device 104 and a database 106 or other storage device. In some implementations, the server device 104 can provide an image application 156a.

[0025] The network environment 100 may also include one or more client devices, such as client devices 120, 122, 124, and 126, which may communicate with each other and / or with the server system 102 via a network 130. The network 130 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switch or hub connection, etc. In some implementations, the network 130 may include peer-to-peer communications between devices, such as using peer-to-peer wireless protocols (e.g., Bluetooth®, Wi-Fi Direct, etc.). An example of peer-to-peer communications between two client devices 120 and 122 is shown by arrow 132.

[0026] For ease of explanation, Figure 1One box for server system 102, server device 104, and database 106, and four boxes for client devices 120, 122, 124, and 126 are shown. Server boxes 102, 104, and 106 can represent multiple systems, server devices, and network databases, and the boxes can be provided in a different configuration than shown. For example, server system 102 can represent multiple server systems that can communicate with other server systems via network 130. In some embodiments, server system 102 can include a cloud hosting server, for example. In some examples, database 106 and / or other storage devices can be provided in a server system box separate from server device 104, and can communicate with server device 104 and other server systems via network 130.

[0027] Moreover, there may be any number of client devices. Each client device may be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smart phone, a stand-alone camera, a tablet computer, a television, a TV set-top box or an entertainment device, a wearable device (e.g., display glasses or goggles, a watch, a head-mounted device, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may also have a local database similar to database 106 or other storages. In some embodiments, network environment 100 may not have all the components shown and / or may have other elements, including other types of elements that replace or supplement those described herein.

[0028] In various implementations, end users U1, U2, U3, and U4 may communicate with server system 102 and / or with each other using respective client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 may interact with each other via applications running on respective client devices and / or server system 102 and / or via network services (e.g., social networking services or other types of network services) implemented on server system 102. For example, respective client devices 120, 122, 124, and 126 may transmit data to and from one or more server systems (e.g., system 102).

[0029] In some implementations, the server system 102 can provide appropriate data to the client devices so that each client device can receive the transmitted content or shared content uploaded to the server system 102 and / or network service. In some examples, the users U1 to U4 can interact via audio or video conferencing, audio, video or text chat, or other communication modes or applications.

[0030] The network services implemented by the server system 102 may include systems that allow users to perform a variety of communications, form links and associations, upload and publish shared content (such as images, text, video, audio and other types of content), and / or perform other functions. For example, a client device can display received data, such as content posts sent or streamed to a client device and originating from different client devices (or directly from different client devices) via a server and / or network service, or originating from a server system and / or network service. In some embodiments, client devices can communicate directly with each other, for example using peer-to-peer communications between client devices as described above. In some embodiments, a "user" may include one or more programs or virtual entities and people communicating with a system or network.

[0031] In some implementations, any of client devices 120, 122, 124, and / or 126 may provide one or more applications. Figure 1 As shown, client device 120 may provide application 156b. Client devices 122 to 126 may also provide similar applications.

[0032] In some implementations, the client device may include applications that provide various types of functionality, such as calendars, address books, email, web browsers, shopping, transportation (e.g., taxi, train, airline reservations, etc.), entertainment (e.g., music players, video players, gaming applications, etc.), social networking (e.g., messaging or chat, audio / video calls, sharing images / videos, etc.), etc. In some implementations, one or more of the applications may be stand-alone applications executed on the client device 120. In some implementations, one or more of the applications may access a server system, such as server system 102, that provides data and / or functionality for the application.

[0033] User interfaces on client devices 120, 122, 124, and / or 126 may enable display of user content and other content, including images, videos, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using software on a client device, software on a server device, and / or a combination of client software and server software executed on server device 104 (e.g., application software or client software in communication with server system 102). The user interface may be displayed by a display device of the client device or server device, such as a touch screen or other display screen, a projector, and the like. In some implementations, an application running on a server system may communicate with a client device to receive user input at the client device and output data, such as visual data, audio data, and the like, at the client device.

[0034] In various embodiments, any of the client devices 120-126 can be used by a user and / or a customer service agent (e.g., a human customer service agent). In some embodiments, the application 156 on the server system 102 and / or one or more of the client devices (120-126) can denoise the image by using a machine-learned model that performs spatially-variant noise removal. In some embodiments, the application 156 on the server system 102 and / or one or more of the client devices (120-126) can train a machine-learned model to perform spatially-variant noise removal.

[0035] In some embodiments, the database 106 may store content items, such as true value images (clean images), noisy images, or other images. In various embodiments, the content items may include help articles that include text, audio, video, images, or any combination thereof. Users of the client devices 120 to 126 may access the content items via the network 130.

[0036] Other implementations of the features described herein may use any type of system and / or service. For example, other networked services (e.g., connected to the Internet) may be used instead of or in addition to social networking services. Any type of electronic device may utilize the features described herein. Some implementations may provide one or more features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device may display content posts stored on a storage device local to the client device (e.g., previously received over a communication network).

[0037] The images referred to herein may include digital images having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). An image may be a still image (e.g., a still photograph, an image having a single frame, etc.), a dynamic image (e.g., an animation, an animated GIF, a dynamic picture in which a portion of the image includes motion while other portions are static, etc.), or a video (e.g., an image or sequence of image frames that may include audio). Although the remainder of this document refers to images as still images, it is understood that the techniques described herein are applicable to dynamic images, videos, etc. For example, the embodiments described herein may be used for still images (e.g., photographs or other images), videos, or dynamic images.

[0038] Figure 2 is a flow chart illustrating an example method 200 for denoising a noisy input image according to some embodiments. In some embodiments, the method 200 may be performed, for example, in Figure 1102 as shown. In some embodiments, some or all of method 200 may be implemented in a Figure 1 The method 200 is implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or on both server devices and client devices. In the described examples, the implementation system includes one or more digital processors or processing circuit systems ("processors") and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can perform different blocks or other portions of the method 200. In some examples, a first device is described as performing a block of the method 200. Some embodiments may have one or more blocks of the method 200 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0039] In some embodiments, the method 200 or parts of the method may be automatically initiated by the system. In some embodiments, the implementation system is the first device. For example, the method (or parts thereof) may be performed periodically, or based on one or more specific events or conditions, such as a camera capturing an image, an image being added to an image library, an image being received (e.g., from another device, via a scanner, etc.) or viewed (e.g., in an image viewing application), a predetermined time period has passed since the last execution of the method 200, and / or one or more other conditions specified in the settings read by the method occur.

[0040] Method 200 can begin at block 202. At block 202, a check is made to see whether user consent (e.g., user permission) has been obtained for using user data when implementing method 200. For example, the user data can include one or more images, such as images captured or viewed by a user using a client device (e.g., any of devices 120 to 126). In some implementations, one or more blocks of the methods described herein can use such user data.

[0041] If user consent has been obtained from the relevant user for whom user data may be used in method 200, then in block 204, it is determined that the blocks of the method herein may be implemented using user data as described for those blocks, and the method proceeds to block 210. If user consent has not been obtained, then in block 206 it is determined that the blocks are to be implemented without the use of user data, and the method proceeds to block 210. In some embodiments, if user consent has not been obtained, the blocks are implemented without the use of user data and using synthetic data and / or data that is generally or publicly accessible and publicly usable. In some embodiments, if user consent has not been obtained, the remainder of method 200 is not performed.

[0042] At block 212, a noisy input image is obtained. The noisy input image includes a plurality of pixels, each pixel having an associated pixel value, for example, a value for each of the red, green, and blue (RGB) channels of an RGB image, or an additional value. In some embodiments, the pixel values ​​of the noisy input image may include a luminance value, a depth value, or other values.

[0043] For example, the noisy input image may be captured by a camera of a client device (any of the client devices 120 to 126) or a scanner of the client device; received from another device (e.g., when the method 200 is implemented on a server or client device, received from another client device over a network or via a local interface); received from a hardware scanner; received from a camera device (e.g., a security camera or other camera), etc. In some embodiments, the noisy input image may be captured under low light conditions. The noise in the noisy input image may include shot noise, which depends on the total amount of light captured (e.g., based on the exposure time) and is therefore signal-dependent. The noise in the noisy input image may also include read noise introduced by a sensor used to capture the noisy input image (e.g., a complementary metal oxide semiconductor (CMOS) sensor, a charge coupled device (CCD) sensor, or other types of sensors used to capture digital images). The read noise depends on the sensor and is therefore signal-independent, so that the read noise does not depend on the amount of light captured. Block 212 may be followed by block 214.

[0044] At block 214, a temporal map of the noisy input image is estimated based on the noisy input image and the noise in the input image. In some embodiments, each pixel p in the noisy input image may be associated with a corresponding noise level. The corresponding time step value may generate the noise level during the diffusion process. The maximum noise level on the image (e.g., three channels R, G, and B) may define the temporal map. In other words, each pixel in the noisy input image may be at a corresponding stage of the diffusion process.

[0045] In some embodiments, the time map is a spatially varying time map such that the time map includes a corresponding time step value for each pixel of the noisy input image (e.g., including a time step value for the time embedding). In some embodiments, at least two pixels of the noisy input image may have different time step values. In some embodiments, an affine transformation may be applied to each pixel using the time step value independently of the transformations applied to other pixels of the noisy input image. In some embodiments, each pixel is associated with a corresponding time value, which is converted to a time embedding (a vector). In these embodiments, the vector of all pixels is multiplied by a matrix (the matrix is ​​the affine transformation). The time step value in the noisy input image may be based on the noise.

[0046] In some embodiments, the noise in the noisy input image may be determined before estimating the time map. In these embodiments, determining the noise level may include determining the readout noise (σ) from the noisy input image. r ) and shot noise (σ s In these embodiments, read noise is the standard deviation of signal-independent noise (e.g., based on sensor characteristics of the sensor capturing the noisy input image), and shot noise is the standard deviation of signal-dependent noise (e.g., affected by conditions such as the total amount of light captured by the sensor when capturing the noisy input image).

[0047] In some implementations, the noise may be estimated using a cropped version of the noisy input image. Block 214 may be followed by block 216 .

[0048] At block 216, a diffusion process using a machine learning model is performed to obtain a denoised image. In some embodiments, the machine learning model may be a trained machine learning model trained to perform spatially varying noise removal (SVNR). In some embodiments, the following reference Figure 3 The method 300 is described to perform the diffusion process. Block 216 may be followed by block 218 .

[0049] At block 218, the denoised image is output. For example, the denoised image can be displayed on a display device (e.g., a monitor, a projector, an augmented reality device, or other device capable of displaying an image). In some embodiments, the denoised image can be output to a printer device to produce a physical print of the denoised image. In some embodiments, the denoised image can be output to a system that performs additional operations (e.g., editing, compression, format conversion, or other operations).

[0050] The various blocks of method 200 may be combined, split into multiple blocks, or executed in parallel. Method 200 or its parts may be repeated any number of times using additional inputs. For example, method 200 may be repeated until all images captured by the camera are processed to obtain corresponding denoised images.

[0051] Figure 3 is a flow chart illustrating an example method 300 for performing a diffusion process according to some embodiments. In some embodiments, the method 300 may be performed, for example, in Figure 1 102 as shown. In some embodiments, some or all of method 200 may be implemented in a Figure 1 The method 300 is implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or on both server devices and client devices. In the described examples, the implementation system includes one or more digital processors or processing circuit systems ("processors") and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can perform different blocks or other portions of the method 300. In some examples, a first device is described as performing a block of the method 300. Some embodiments may have one or more blocks of the method 300 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0052] Method 300 may begin at block 302. At block 302, a denoised image is estimated using a machine-learned model (trained machine-learned model). The model is trained to perform spatially-variant noise removal (SVNR) based on a temporal map, such as described in block 214 of method 200, and may be referred to herein as a SVNR model. Figure 6 An example method for training a machine learning model to perform spatially varying noise removal (SVNR) is described. In some implementations, the method 300 initiates from the highest value in the time map.

[0053] In some implementations, the SVNR model estimates a denoised image based on the noisy input image, the current time step image (which may be the noisy input image for the first time step), and the time step. Block 302 may be followed by block 304.

[0054] At block 304, the next time step image is obtained. In some embodiments, the next time step image (x t_hat-1 ) is based on the denoised image (x0) obtained in block 302 and the current time step image (x t_hat). In some embodiments, obtaining the next time step image may further include sampling Gaussian noise. In some embodiments, the sampling may depend on the current time map (and the time step image) to determine the corresponding weights of each of the components. Block 304 may be followed by block 306.

[0055] At block 306, it is determined whether at least one pixel is associated with a time step value greater than or equal to the current time step. If it is determined that at least one pixel is associated with a time step value greater than or equal to the current time step, block 306 may be followed by block 316. Otherwise, block 302 may be followed by block 308.

[0056] At block 308, the next time step image (eg, obtained in block 304) is set as the current step image. Block 308 may be followed by block 310.

[0057] At block 310, the time step is decremented. Block 310 may be followed by block 312.

[0058] At block 312, it is determined whether the current time step is positive. A positive time step may indicate that at least one pixel of the noisy input image has a time step value greater than zero, such that the diffusion process for the pixel is not complete. If the current time step is positive, block 312 is followed by block 203. In embodiments where method 300 is used to implement block 216 of method 200, if the current time step is not positive, block 312 may be followed by block 218 to output a denoised image.

[0059] At block 316 (which is performed if it is determined that at least one pixel is associated with a time step value greater than or equal to the current time step), the next time step image (obtained in block 304) is updated with the corresponding value from the denoised image (obtained in block 302). This ensures that further denoising of pixels (which have been denoised based on the temporal map) that have reached a time step value of zero is avoided, and instead their values ​​are stable as obtained in the denoised image while further denoising operations are continued on other pixels. This provides spatially-varying noise removal because different pixels may reach a time step value of zero in different iterations of the method 300. During the update, if the value of a pixel p in the temporal map is less than 0, the value of the pixel is set to X_{t_hat}(p) (e.g., instead of setting the value to X_{t_hat-1}(p)). The method 300 may be performed over at least two time steps.

[0060] Replacing a single time step value (which defines the number of iterations to be performed) with a spatially varying time map results in the diffusion operation being performed pixel by pixel. The use of pixel-specific time step values ​​(temporal embedding) in the time map makes the noise removal performed by the SVNR model dependent on the spatially varying noise level in the noisy input image. Therefore, denoising using the SVNR model can reduce signal-dependent shot noise together with signal-independent read noise.

[0061] refer to Figure 3 The described diffusion process is performed with a noisy input image as the conditional image.

[0062] Figure 4A Example images are shown. Figure 4A A noisy input image (404) of a lizard is depicted. In some embodiments, the noisy input image can be generated by adding simulated noise to the image. For example, the noisy input image can be captured under low light conditions or captured under other conditions that include noise attributable to the captured image. The noise in the image (402) is shown, where the noise value (shown in different shades) is the standard deviation of the spatially varying noise in the image (quantized). The corresponding clean image (406) is also shown. For example, the clean image can be captured under good lighting conditions, or captured using a camera that can capture relatively noise-free images under low light conditions.

[0063] Further, in some embodiments, a noisy image can be generated by applying spatially varying noise to a known clean image (e.g., for use in training a machine learning model). In this case, the known clean image can be used as a true value during the training of the machine learning model, e.g., to train the machine learning model to remove noise from the generated noisy image. For example, for the above reference Figure 2 and Figure 3 In the techniques described herein, a noisy image can serve as a conditional image and be compared using a true value image, for example, to train a machine learning model to remove noise from a noisy image by using the true value image to compare with the output of a diffusion process. The techniques described herein treat (remove) spatially varying noise by applying a pixel-by-pixel temporal embedding, for example as described above with reference to Figure 2 and spatial variation time diagrams as described.

[0064] Figure 4AAlso depicted is a denoised image (408) obtained by applying an example prior art technique that does not utilize diffusion to a noisy input image (404). FIG. 4 further illustrates another example of a denoised image (410) obtained by applying a prior art diffusion technique to a noisy input image (404). After 1000 time steps of the diffusion process, a denoised image (410) is obtained. It can be seen in the denoised images (408, 410) that, while such prior art denoising methods remove high levels of noise, such methods over-smooth fine details and / or the resulting denoised image lacks high frequency details. For example, while the denoised image 410 has more high frequencies than the denoised image 412, a portion of the head of the lizard in the clean image (406) is missing in the denoised image 410 and has no clear boundaries in the denoised image 408. Although the image 410 shows the missing portion, in some cases, the prior art can add details that are not present in the image. Further, in the denoised images 408 and 410, the skin areas below the eyes and above the torso lack fine details.

[0065] Figure 4A Further described by applying the present invention to Figure 2 and Figure 3 A denoised image (412) obtained by the described spatially-variant noise removal technique. As can be seen, the denoised image includes fine details, such as correctly delineating the head portion and delineating the skin area under the eyes. As can be seen, even when textures in the image are difficult to distinguish in a noisy input image, the spatially-variant noise removal technique described herein is able to restore such textures. As can be seen, the described technique can produce a higher fidelity clean image while reducing the running time (and thus the computational cost) (e.g., of the diffusion process) by several orders of magnitude (25 time steps compared to 1000 steps).

[0066] Figure 4B Depicted Figure 4A The noisy image (404) and the denoised image (412), as well as the intermediate denoised image (420) obtained after 15 time steps. Conventional diffusion techniques (e.g., the diffusion technique that generates image 410) start from a complete noisy image rather than from a noisy input image, and therefore, utilize significantly more computational resources (e.g., because the diffusion process in this example runs for 1000 steps to obtain the denoised image 410). In contrast, the spatially variant noise removal technique described herein initiates the diffusion process from the noisy image (404) and can generate a denoised image at a lower computational cost. Figure 4BAs can be seen in , after 15 time steps in the diffusion process, the spatially variant noise removal technique produces a denoised image ( 420 ) that is significantly improved over the noisy input image ( 404 ).

[0067] Figure 5 Several examples of noisy images (502A-510A) and corresponding denoised images (512A-520A) and true value images (522A-530A) are shown. Figure 5 Also shown are close-up (zoomed-in) views (502B to 510B) of a portion of each noisy image (indicated by the small squares in the image), as well as corresponding portions of the denoised images (512B to 520B) and the true value images (522B to 530B). Figure 5 As can be seen, the described technique is able to achieve high signal fidelity (matching the true value image) without noticeable distribution drift. In the example shown in Figure 4, the noise gain is 16. Gain refers to the amplification of the signal from the camera sensor (ISO) when the image is captured. When the image capture conditions are poor, the gain is higher, which means a higher noise level in the image. Even at relatively high noise levels, the spatially variant noise removal technique described in this article can generate a denoised image that includes fine high-frequency details while also removing a lot of noise. Further, unlike other diffusion-based methods that exhibit hallucinations (adding details in the denoised image that are not in the true value image), the technique results in high fidelity to the true value image.

[0068] The described techniques can produce more realistic images while also removing a lot of noise. The techniques maintain fidelity to the underlying signal (in a noisy image). Further, diffusion techniques that start from complete noise have a fixed computational cost (run time) based on the number of time steps to reach the denoised image. The computational cost is independent of the noise level in the noisy input image. In contrast, the computational cost (run time) of the techniques described herein varies based on the noise level in the noisy input image (higher noise requires more cost). Moreover, because the described techniques can initiate the diffusion process from a noisy input image, the computational cost can be significantly (e.g., orders of magnitude) lower than diffusion techniques that start from complete noise.

[0069] Figure 6 6 is a flow chart illustrating an example method 600 for training a machine learning model to denoise a noisy input image according to some embodiments. In some embodiments, the method 600 may be performed, for example, in Figure 1 Implementation is performed on the server system 102 shown in FIG. In some implementations, some or all of the method 600 may be implemented in a Figure 1The method 600 is implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or on both server devices and client devices. In the described examples, the implementation system includes one or more digital processors or processing circuit systems ("processors") and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or clients can perform different blocks or other portions of the method 600. In some examples, a first device is described as performing a block of the method 600. Some embodiments may have one or more blocks of the method 600 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0070] In some embodiments, method 600 or portions thereof may be automatically initiated by a system. In some embodiments, the implementing system is a first device. For example, the method (or portions thereof) may be performed periodically, or based on one or more specific events or conditions, such as new or additional training data becoming available, changes in image capture hardware (e.g., image capture hardware that may affect noise in captured images) or new types of image capture hardware, a predetermined time period has passed since the last execution of method 600, and / or one or more other conditions that may be specified in settings read by the method occur.

[0071] The training data (e.g., one or more of true value images, noisy input images, etc.) is obtained with specific user permissions. In some embodiments, the training data may include images captured specifically for the purpose of training the machine learning model. In some embodiments, the training data may include synthetic images or generated images, such as images generated using generative artificial intelligence or other suitable image generation techniques. If the training data includes user-provided images, such images are only utilized with specific user permissions (for training the machine learning model to perform noise removal) and for a specific purpose. Users may be provided with the option to exclude their captured images from the training data.

[0072] At block 602, a clean image is obtained. For example, a clean image may be an image captured by a camera such that the image has little or no noise (e.g., under good lighting conditions). In some implementations, the clean image may be obtained from training data including a large number of training images (e.g., training images representing real-world images). Block 602 may be followed by block 604.

[0073] At block 604, a noisy input image is generated by adding noise to the clean image. The noise standard deviation per pixel (σ p ) to determine the noise to be added. In some embodiments, the noise standard deviation (σ p ) is the read noise (σ r ) and shot noise (σ s ), where the read noise is the standard deviation of the signal-independent noise and the shot noise is the standard deviation of the signal-dependent noise. In some embodiments, the read noise and the shot noise are randomly selected. Block 604 may be followed by block 606.

[0074] At block 606, a time map (T ). The time map is spatially varying such that a corresponding time step value is associated with each pixel of the noisy input image such that at least two pixels have different time step values. Block 606 may be followed by block 608.

[0075] At block 608, a new time map (t_hat) is generated such that the corresponding time step values ​​of the pixels in the new time map are lower than the timestamp values ​​of the corresponding pixels in the time map. ) to generate a new temporal map. The scalar may be chosen (e.g., randomly) to be the maximum time step value (T Block 608 may be followed by block 610.

[0076] At block 610, a current time step image (x) is generated based on the new time map (eg, the noise level from the new time map). t_hat ). The current time step image is generated such that the current time step image has lower noise than the noisy input image (y). In other words, the current time step image is a slightly denoised version of the noisy input image (y). In some embodiments, the current time step image is generated such that the current time step image has a noise distribution similar to that of the noisy input image. Block 610 may be followed by block 612.

[0077] In diffusion techniques, the timestamp scales linearly with the number of forward diffusion steps, which scales linearly with the noise variance. In such cases, determining the timestamp is equivalent to determining the noise level and can be performed once at the beginning of inference. During training, since the pixel-by-pixel noise variance is known, the timestamps can be directly calculated using the pixel-by-pixel noise variance or estimated from the image. This is advantageous because it is similar to the inference process.

[0078] At block 612, the noisy input image (y), the current time step image (x t_hat ) and the new time map are provided as input to the machine learning model being trained. The machine learning model generates a denoised image (x 0_hat ). Block 612 may be followed by block 614.

[0079] At block 614, based on the denoised image (x 0_hat ) is compared with the clean image (x0) to calculate the loss value. The loss value indicates the difference between the clean image (which is the true value) and the denoised image generated by the machine learning model being trained. Block 614 may be followed by block 616.

[0080] At block 616, one or more parameters of the machine learning model are updated based on the loss value. In some implementations, the machine learning model may include a neural network. In these examples, updating the one or more parameters of the machine learning model based on the loss value includes performing back propagation to adjust the weights of one or more nodes of the neural network so that the loss value is reduced. Block 616 may be followed by block 618.

[0081] At block 618, a determination is made as to whether a stopping criterion has been met. In some implementations, the stopping criterion may include one or more of the following: the computational budget for training is exhausted, all clean images in the training data set of clean images have been utilized, or when the change in parameter value of at least one of the one or more parameters between consecutive iterations falls below a threshold. If the stopping criterion has not been met, block 618 may be followed by block 602. If the stopping criterion has been met, block 618 may be followed by block 620.

[0082] At block 620, a machine-learned model (a trained model, which is a version of the machine-learned model output after parameter adjustment) is generated. The machine-learned model can be used to perform spatially-variant noise removal from any image. For example, the machine-learned model can be used to generate a spatially-variant noise image using a reference image. Figure 2 and Figure 3 Described method for noise removal.

[0083] The various blocks of method 600 may be combined, split into multiple blocks, or executed in parallel. For example, blocks 604 to 616 may be executed multiple times for the same clean image, for example to generate different noisy versions of the clean image each time block 604 is executed, and each of the generated noisy versions is used to train the model. In some embodiments, blocks 608 to 616 may be executed multiple times for the same noisy input image by using different scalar values ​​(e.g., scalar values ​​that can be randomly sampled between 0 and the maximum value in the time map) to generate a new time map at block 608, and the loss value is determined using the corresponding estimated denoised image. During training, the task of the model is to evaluate the clean image x_0 given the time map. In some embodiments, method 600 may be executed in parallel with respect to multiple images or parts of method 600 (e.g., blocks 604 to 614 or 604 to 616 are used for the same clean image; blocks 608 to 614 or 608 to 616 are used for the same noisy input image, etc.). Method 600 or portions thereof may be repeated any number of times using additional inputs (e.g., new training images received after obtaining the initial machine-learned model at block 620). For example, performing model retraining may improve the quality of denoised images generated by the machine-learned model. During model training, images of different image sizes and / or different noise levels may be used.

[0084] Figure 7 is a block diagram of an example apparatus 700 that can be used to implement one or more features described herein. In one example, the apparatus 700 can be used to implement a client device, e.g., Figure 1 101. Alternatively, device 700 may implement a server device, such as server 101. In some embodiments, device 700 may be used to implement a client device, a server device, or both a client and a server device. Device 700 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0085] One or more methods described herein may be run in a standalone program that can be executed on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a cell phone, a smart phone, a tablet computer, a wearable device (a watch, an armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, a head mounted display, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, such as a mobile computing device (as a client device), sending user input data to a server device and receiving final output data from the server for output (e.g., for display). In another example, all computations may be performed within a mobile app (and / or other apps) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0086] In some embodiments, the device 700 includes a processor 702, a memory 704, and an input / output (I / O) interface 706. The processor 702 may be one or more processors and / or processing circuits for executing program code and controlling the basic operation of the device 700. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. The processor may include a system having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), a plurality of processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor or other system optimized for matrix calculations (e.g., matrix multiplication). In some embodiments, the processor 702 may include one or more coprocessors that implement neural network processing. In some embodiments, the processor 702 may be a processor that processes data to generate a probabilistic output, for example, the output generated by the processor 702 may be imprecise or may be accurate within a range from the expected output. Processing need not be limited to a particular geographic location or have time limits. For example, a processor may perform the functions of a processor in "real time," "offline," in "batch mode," etc. Portions of a process may be performed by different (or the same) processing systems at different times and locations. A computer may be any processor in communication with a memory.

[0087] The memory 704 is generally provided in the device 700 for access by the processor 702, and may be any suitable processor-readable storage medium suitable for storing instructions for execution by the processor and located separately from and / or integrated with the processor 702, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc. The memory 704 may store software operated by the processor 702 on the server device 700, including an operating system 708, a machine learning application 730, other applications 712, and application data 714. The other applications 712 may include applications such as a data display engine, a web hosting engine, an image display engine, a notification engine, a social networking engine, etc. In some embodiments, the machine learning application 730 and the other applications 712 may each include an application that enables the processor 702 to perform the functions described herein (e.g., Figure 2 , Figure 3 and / or Figure 6 some or all of the methods).

[0088] Other applications 712 may include, for example, image editing applications, media display applications, communication applications, web hosting engines or applications, mapping applications, media sharing applications, etc. One or more of the methods disclosed herein may operate in a number of environments and platforms, such as as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0089] In various embodiments, the machine learning application may utilize a Bayesian classifier, a support vector machine, a neural network, or other learning techniques. In some embodiments, the machine learning application 730 may include a machine-learned model (trained model) 734, an inference engine 736, and data 732. In some embodiments, the data 732 may include training data, such as data used to generate the trained model 734. For example, the training data may include any type of data, such as text, images, audio, video, etc. When the trained model 734 is an image denoising model, the training data may include true value clean images.

[0090] The training data may be obtained from any source, such as a data repository specifically labeled for training, data that is provided with permission for use as training data for machine learning, etc. In embodiments where one or more users have permitted the use of their respective user data to train a machine learning model (e.g., trained model 734), the training data may include such user data. In embodiments where users have permitted the use of their respective user data, data 732 may include permitted data, such as images (e.g., photographs or other user-generated images).

[0091] In some embodiments, the training data may include synthetic data generated for the purpose of training, such as data that is not based on user input or activity in the context being trained, such as data generated from simulated photographs or other computer-generated images. In some embodiments, the machine learning application 730 does not include data 732. For example, in these embodiments, the trained model 734 can be generated, for example, on a different device and provided as part of the machine learning application 730. In various embodiments, the trained model 734 can be provided as a data file including a model structure or form and associated weights. The inference engine 736 can read the data file for the trained model 734 and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model 734.

[0092] In some embodiments, the trained model 734 may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that takes sequence data such as words in a sentence, frames in a video, etc. as input and produces a sequence of results as output), etc. The model form or structure may specify the connectivity between various nodes and the organization of nodes into layers.

[0093] For example, the nodes of the first layer (e.g., input layer) can receive data as input data 732 or application data 714. For example, when the trained model 734 is an image denoising model, the input data may include a clean image. For example, when the trained model is used for image analysis or image generation, such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive the output of the nodes of the previous layer as input according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers or potential layers.

[0094] The final layer (e.g., output layer) produces the output of the machine learning application. For example, the output can be a denoised image. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0095] In various embodiments, the trained model 734 may include multiple nodes arranged in layers according to the model structure or form. In some embodiments, the node may be, for example, a memoryless computing node configured to process an input unit to generate an output unit. The calculation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum using a bias or intercept value to generate a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, the calculations performed by multiple nodes may be performed in parallel using, for example, multiple processor cores of a multi-core processor, various processing units using a GPU, or a dedicated neural circuit system. In some embodiments, the node may include memory, such as being able to store one or more earlier inputs and use the one or more earlier inputs when processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. LSTM nodes may use memory to maintain a "state" that allows the node to function like a finite state machine (FSM). Models with such nodes can be useful in processing sequence data (e.g., words in a sentence or paragraph, frames in a video, speech or other audio, etc.) In some embodiments, a model with an attention mechanism and a transformer network can be used.

[0096] In some embodiments, the trained model 734 may include embeddings or weights for each node. For example, the model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to the connections between each pair of nodes connected in the model form (e.g., nodes in successive layers of a neural network). For example, corresponding weights may be randomly assigned, or initialized to default values. The model may then be trained, for example, using data 732, to produce results.

[0097] For example, training may include applying supervised learning techniques. In supervised learning, the training data may include multiple inputs (e.g., a set of noisy input images) and corresponding expected outputs for each input (e.g., a set of true value clean images corresponding to the noisy input images). Based on a comparison of the output of the model with the expected output, the value of the weights is automatically adjusted, for example, in a manner that increases the probability that the model will produce the expected output when provided with similar inputs.

[0098] In some embodiments, training can include applying unsupervised learning techniques. In unsupervised learning, only input data can be provided and a model can be trained to distinguish the data, such as clustering the input data into multiple groups, where each group includes input data that is similar in some way.

[0099] In some embodiments, unsupervised learning can be used to generate a knowledge representation that can be used, for example, by a machine learning application 730. In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments in which data 732 is omitted, the machine learning application 730 can include a trained model 734 based on previous training performed, for example, by a developer of the machine learning application 730, by a third party, etc. In some embodiments, the trained model 734 can include a fixed set of weights (e.g., downloaded from a server that provides weights).

[0100] The machine learning application 730 also includes an inference engine 736. The inference engine 736 is configured to apply the trained model 734 to data (e.g., application data 714) to provide inference. In some embodiments, the inference engine 736 may include software code to be executed by the processor 702. In some embodiments, the inference engine 736 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 702 to apply the trained model. In some embodiments, the inference engine 736 may include software instructions, hardware instructions, or a combination. In some embodiments, the inference engine 736 may provide an application programming interface (API) that can be used by the operating system 708 and / or other applications 712 to call the inference engine 736, such as applying the trained model 734 to the application data 714 to generate inferences. For example, the inference of an image denoising model may be a noisy image, while the inference is a denoised image, as described above with reference to Figures 2 to 6 described.

[0101] Machine learning application 730 may provide several technical advantages. For example, when trained model 734 is generated based on unsupervised learning, trained model 734 may be applied by inference engine 736 to generate a knowledge representation (e.g., a digital representation) based on input data (e.g., application data 714). For example, a model trained for image analysis may generate a representation of an image having a smaller data size (e.g., 1 KB) than the input image (e.g., 10 MB). In some embodiments, such representations may help reduce processing costs (e.g., computational costs, memory usage, etc.) to generate outputs (e.g., labels, classifications, sentences describing images, denoised images from noisy images, etc.).

[0102] In some embodiments, such representations may be provided as input to different machine learning applications that generate outputs based on the output of the inference engine 736. In some embodiments, the knowledge representation generated by the machine learning application 730 may be provided, for example, over a network, to different devices for further processing. In such embodiments, providing knowledge representations rather than images may provide technical benefits, such as faster data transfer at reduced cost.

[0103] In some embodiments, the machine learning application 730 can be implemented in an offline manner. In these embodiments, the trained model 734 can be generated in the first stage and provided as part of the machine learning application 730. In some embodiments, the machine learning application 730 can be implemented in an online manner. For example, in such embodiments, an application that calls the machine learning application 730 (e.g., one or more of the operating system 708, other applications 712) can utilize the reasoning generated by the machine learning application 730, such as providing the reasoning to the user, and can generate a system log (e.g., if permitted by the user, the action taken by the user based on the reasoning; or if used as input for further processing, the result of further processing). The system log can be generated periodically, such as hourly, monthly, quarterly, etc., and can be used to update the trained model 734 with user permission, such as updating the embedding of the trained model 734.

[0104] In some embodiments, the machine learning application 730 may be implemented in a manner that may be suitable for a particular configuration of the device 700 on which the machine learning application 730 is executed. For example, the machine learning application 730 may determine a computational graph that utilizes available computing resources (e.g., the processor 702). For example, if the machine learning application 730 is implemented as a distributed application on multiple devices, the machine learning application 730 may determine the computations to be performed on each device in a manner that optimizes the computations. In another example, the machine learning application 730 may determine that the processor 702 includes a GPU with a specific number of GPU cores (e.g., 1000) and implement the inference engine accordingly (e.g., as 1000 separate processes or threads).

[0105] In some embodiments, the machine learning application 730 may implement a collection of trained models. For example, the trained model 734 may include multiple trained models that are each applicable to the same input data. In these embodiments, the machine learning application 730 may select a specific trained model, for example, based on available computing resources, the success rate of previous inferences, etc. In some embodiments, the machine learning application 730 may execute an inference engine 736 so that multiple trained models are applied. In these embodiments, the machine learning application 730 may combine the outputs from the application of each model, for example, using a voting technique that scores the outputs from each trained model of the application, or by selecting one or more specific outputs. Further, in these embodiments, the machine learning application may apply a time threshold (e.g., 0.5 ms) for applying each trained model, and only utilize those individual outputs that are available within the time threshold. Outputs that are not received within the time threshold may not be utilized, for example, discarded. For example, when there is a time limit specified when calling the machine learning application, for example, by the operating system 708 or one or more applications 712, such a method may be appropriate.

[0106] In different embodiments, the machine learning application 730 can generate different types of outputs. For example, the machine learning application 730 can provide representations or clusters (e.g., digital representations of input data), labels (e.g., labels for input data including images, documents, etc.), phrases or sentences (e.g., phrases or sentences that describe an image or video, are suitable for use as a response to an input sentence, etc.), images (e.g., denoised images generated by the machine learning application in response to an input image such as a noisy image), audio or video (e.g., in response to an input video, the machine learning application 730 can generate an output video with a specific effect applied, such as an output video rendered in the style of a comic book or a specific artist when the training model 734 is trained using training data from a comic book or a specific artist), etc. In some embodiments, the machine learning application 730 can generate output based on a format specified by a calling application (e.g., the operating system 708 or one or more applications 712). In some embodiments, the calling application can be another machine learning application. For example, such a configuration can be used for a generative adversarial network, in which the calling machine learning application is trained using output from the machine learning application 730, and vice versa.

[0107] Any of the software in the memory 704 may alternatively be stored in any other suitable storage location or computer-readable medium. In addition, the memory 704 (and / or other connected storage devices) may store one or more messages, one or more taxonomies, electronic encyclopedias, dictionaries, thesauruses, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. The memory 704 and any other type of storage (disk, optical disk, tape, or other tangible medium) may be considered "storage" or "storage devices".

[0108] The I / O interface 706 may provide functionality that enables the server device 700 to be interfaced with other systems and devices. The interfaced devices may be included as part of the device 700, or may be separate and in communication with the device 700. For example, a network communication device, a storage device (e.g., memory and / or database 106), and an input / output device may communicate via the I / O interface 706. In some embodiments, the I / O interface may be connected to an interface device, such as an input device (keyboard, pointing device, touch screen, microphone, camera, scanner, sensor, etc.) and / or an output device (display device, speaker device, printer, motor, etc.).

[0109] Some examples of devices connected via interfaces that can be connected to I / O interface 706 can include one or more display devices 720, which can be used to display content, such as images, videos and / or user interfaces of output applications as described herein. Display device 720 can be connected to device 700 via a local connection (e.g., display bus) and / or via a networked connection, and can be any suitable display device. Display device 720 can include any suitable display device, such as an LCD, LED, or plasma display screen, CRT, television, monitor, touch screen, 3D display screen or other visual display device. For example, display device 720 can be a flat display screen arranged on a mobile device, a plurality of display screens arranged in goggles or a head-mounted device, or a monitor screen for a computer device.

[0110] The I / O interface 706 can be connected to other input and output devices via an interface. Some examples include one or more cameras that can capture images. Some embodiments may provide a microphone for capturing sound (e.g., as part of a captured image, voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0111] For ease of explanation, Figure 7A box is shown for each of processor 702, memory 704, I / O interface 706, and software boxes 708, 712, and 730. These boxes may represent one or more processors or processing circuit systems, operating systems, memories, I / O interfaces, applications, and / or software modules. In other embodiments, device 700 may not have all the components shown and / or may have other elements, including other types of elements that replace or supplement those shown herein. Although some components are described as performing boxes and operations as described in some embodiments herein, any suitable component or combination of components of any suitable one or more processors associated with environment 100, device 700, similar systems, or such systems can perform the described boxes and operations.

[0112] The method described herein can be implemented by computer program instructions or codes that can be executed on a computer. For example, the code can be implemented by one or more digital processors (for example, a microprocessor or other processing circuit system), and can be stored on a computer program product, which includes a non-temporary computer-readable medium (for example, a storage medium), such as a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, or a semiconductor storage medium, including a semiconductor or solid-state memory, a tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a flash memory, a hard disk, an optical disk, a solid-state memory drive, etc. The program instruction can also be included in an electronic signal and provided as an electronic signal, for example in the form of software as a service (SaaS) delivered from a server (for example, a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.) or a combination of hardware and software. Example hardware can be a programmable processor (for example, a field programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be performed as a part or component of an application running on a system, or as an application or software running with other applications and an operating system.

[0113] Although the specification has been described with respect to specific embodiments thereof, these specific embodiments are merely illustrative and not restrictive. The concepts shown in the examples can be applied to other examples and embodiments.

[0114] In the case where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data, information about a user's social network, a user's location and time at that location, a user's biometric information, a user's activities and demographic information), one or more opportunities are provided to the user to control whether to collect information, whether to store personal information, whether to use personal information, and how to collect, store and use information about the user. That is, the systems and methods discussed herein collect, store and / or use user personal information, particularly when explicit authorization to do so is received from the relevant user. For example, a user is provided with control over whether a program or feature collects user information about the particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the collection of information associated with the user, providing permission or authorization regarding whether to collect information and regarding which parts of the information to be collected. For example, one or more such control options may be provided to the user via a communication network. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of the user may be processed so that no personally identifiable information can be determined. As another example, the geographic location of a user device may be generalized to a larger area such that the user's specific location cannot be determined.

[0115] Note that the functional blocks, operations, features, methods, devices and systems described in this disclosure can be integrated into or divided into different combinations of systems, devices and functional blocks, as known to those skilled in the art. Any suitable programming language and programming techniques can be used to implement the routines of specific embodiments. Different programming techniques can be adopted, such as procedural or object-oriented. The routines can be performed on a single processing device or multiple processors. Although steps, operations or calculations can be presented in a specific order, the order can be changed in different specific embodiments. In some embodiments, multiple steps or operations shown as sequential in this specification can be performed simultaneously.

Claims

1. A computer-implemented method for denoising a noisy input image, the method comprising: Obtaining (212) the noisy input image, the noisy input image comprising a plurality of pixels; estimating (214) a temporal map based on the noisy input image and noise in the noisy input image, wherein the temporal map is spatially varying such that a corresponding time step value is associated with each pixel of the noisy input image and wherein at least two pixels have different time step values; A diffusion process is performed (216) on the noisy input image, wherein the diffusion process is performed over two or more time steps starting from a highest value in the time map, wherein at each time step: estimating (302) a denoised image based on the noisy input image, the current time step image and the time step using a machine-learned model (SVNR); Obtaining (304) a next time step image based on the denoised image and the current time step image; For each pixel, if the time step is greater than or equal to the time step value associated with the pixel, updating (316) the next time step image with a corresponding value from the denoised image; After said updating, setting (308) said next time step image as said current time step image; as well as decrementing (310) the time step, wherein when the time step is positive (312), repeating the estimating, the obtaining, the updating, the setting and the decrementing; and The denoised image is output (218). 2 . The computer-implemented method of claim 1 , further comprising determining the noise in the noisy input image.

3. The computer-implemented method of claim 2, wherein each pixel has a corresponding noise level, and wherein determining the noise comprises determining read noise and shot noise from the noisy input image, wherein the read noise is a standard deviation of signal-independent noise and the shot noise is a standard deviation of signal-dependent noise.

4. The computer-implemented method of claim 3, wherein the temporal map encodes spatially varying noise of the noisy input image.

5. The computer-implemented method of claim 1, wherein at a first time step when performing the diffusion process, the noisy input image is used as the current time step image.

6. The computer-implemented method of claim 1, wherein the noisy input image serves as a conditioning image during the diffusion process.

7. A computing device comprising: processor; as well as A memory is coupled to the processor, and the memory stores instructions, which, when executed by the processor, cause the processor to perform operations including: obtaining (212) a noisy input image, the noisy input image comprising a plurality of pixels; estimating (214) a temporal map based on the noisy input image and noise in the noisy input image, wherein the temporal map is spatially varying such that a corresponding time step value is associated with each pixel of the noisy input image and wherein at least two pixels have different time step values; A diffusion process is performed (216) on the noisy input image, wherein the diffusion process is performed over two or more time steps starting from a highest value in the time map, wherein at each time step: estimating (302) a denoised image based on the noisy input image, the current time step image and the time step using a machine-learned model (SVNR); Obtaining (304) a next time step image based on the denoised image and the current time step image; For each pixel, if the time step is greater than or equal to the time step value associated with the pixel, updating (316) the next time step image with a corresponding value from the denoised image; After said updating, setting (308) said next time step image as said current time step image; as well as decrementing (310) the time step, wherein when the time step is positive (312), repeating the estimating, the obtaining, the updating, the setting and the decrementing; and The denoised image is output (218).

8. The computing device of claim 7, wherein the operations further comprise determining the noise in the noisy input image.

9. A computing device as described in claim 8, wherein each pixel has a corresponding noise level, and wherein determining the noise includes determining read noise and shot noise from the noisy input image, wherein the read noise is the standard deviation of signal-independent noise and the shot noise is the standard deviation of signal-dependent noise.

10. The computing device of claim 9, wherein the temporal map encodes spatially varying noise of the noisy input image.

11. The computing device of claim 7, wherein at a first time step when performing the diffusion process, the noisy input image is used as the current time step image.

12. The computing device of claim 7, wherein the noisy input image serves as a conditional image during the diffusion process.

13. A computer-implemented method for training a machine learning model to denoise a noisy input image, the method comprising: Obtaining (602) a clean image; generating (604) a noisy input image by adding noise to the clean image, wherein the noise is determined based on a per-pixel noise standard deviation; estimating (606) a temporal map based on the noisy input image and the noise in the noisy input image, wherein the temporal map is spatially varying such that a corresponding time step value is associated with each pixel of the noisy input image, wherein at least two pixels have different time step values; generating (608) a new temporal map, wherein corresponding time step values ​​of pixels in the new temporal map are lower than the timestamp values ​​of the corresponding pixels in the temporal map; generating (610) a current time step image based on the new time map, wherein the current time step image has lower noise than the noisy input image; estimating (612) a denoised image based on the noisy input image, the current time step image and the new time map using the machine learning model; calculating (614) a loss value based on a comparison of the denoised image and the clean image; and One or more parameters of the machine learning model are updated (616) based on the loss value.

14. The computer-implemented method of claim 13, wherein the noise standard deviation is a combination of read noise and shot noise, wherein the read noise is a standard deviation of signal-independent noise and the shot noise is a standard deviation of signal-dependent noise, wherein the read noise and the shot noise are randomly selected.

15. The computer-implemented method of claim 13, wherein generating the new temporal map comprises subtracting a scalar from a corresponding time step value for each pixel in the temporal map, wherein the scalar has a value between zero and a maximum time step value in the temporal map.

16. The computer-implemented method of claim 13, wherein generating the current time step image based on the new time map comprises sampling noise based on the new time map.

17. The computer-implemented method of claim 16, wherein the current time-step image has a noise distribution similar to a noise distribution of the noisy input image.

18. The computer-implemented method of claim 13, wherein the machine learning model comprises a neural network, and wherein updating the one or more parameters of the machine learning model based on the loss value comprises performing backpropagation to adjust weights of one or more nodes of the neural network.

19. A computer-implemented method as described in claim 13, wherein obtaining the clean image, generating the noisy input image, estimating the time map, generating the new time map, generating the current time step image, estimating the denoised image, calculating the loss value, and updating the one or more parameters of the machine learning model are performed iteratively until a stopping criterion is met.

20. The computer-implemented method of claim 19, wherein the stopping criteria include one or more of: a computational budget for training is exhausted, or a change in a parameter value of at least one of the one or more parameters between consecutive iterations falls below a threshold.

Images