Neural network training based on consistency loss

Abstract

To provide a machine learning system and a training method thereof, which remove noises of a series of images while maintaining a consistent appearance among images displayed consecutively in the sequence.SOLUTION: A machine learning system maps a first input image in the series of input images to a first output image based on a neural network algorithm and determines a first network loss based on differences between the first output image and a ground truth image. The system further maps a second input image in the series of input images to a second output image based on the neural network algorithm and determines a second network loss based on differences between the second output image and the ground truth image. The system further determines a consistency loss based on differences between the first output image and the second output image and updates the neural network algorithm based on the first network loss, the second network loss, and the consistency loss.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] This implementation generally relates to neural networks, and more particularly to the training of neural networks based on consistency loss.

Background Art

[0002] When an image capturing device captures a series of images (or video) of a scene, each image may exhibit noise. When the images are processed by a machine learning model and then presented sequentially, like in a video, the video may appear to flicker due to changes in the noise. That is, objects or light in the scene may appear unstable or uneven due to differences in noise between successive processed images.

Summary of the Invention

[0003] This summary is provided to introduce, in a concise form, a selection of concepts that are further described below in the detailed description of the implementation of the invention. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] One innovative aspect of the subject matter of this disclosure can be implemented in a method for training a neural network. The method includes the steps of: acquiring a series of input images of a scene; mapping a first input image of the series of input images to a first output image based on a neural network algorithm; determining a first network loss based on the difference between the first output image and a base genuine image; mapping a second input image of the series of input images to a second output image based on the neural network algorithm; determining a second network loss based on the difference between the second output image and a base genuine image; determining a consistency loss based on the difference between the first output image and a second output image; and updating the neural network algorithm based at least in part on the first network loss, the second network loss and the consistency loss.

[0005] Other innovative aspects of the subject matter of this disclosure can be implemented in a machine learning system comprising a processing system and a memory, the memory which, when executed by the processing system, causes the machine learning system to acquire a series of input images capturing a scene, map a first input image of the series of input images to a first output image based on a neural network algorithm, determine a first network loss based on the difference between the first output image and the base genuine image, map a second input image of the series of input images to a second output image based on the neural network algorithm, determine a second network loss based on the difference between the first output image and the base genuine image, determine a consistency loss based on the difference between the first output image and the second output image, and update the neural network algorithm based at least in part on the first network loss, the second network loss and the consistency loss.

[0006] Other innovative aspects of the subject matter of this disclosure can be implemented in a method for training a neural network. The method includes: acquiring a series of input images of a scene; mapping a first input image of the series of input images to a first output image based on a neural network algorithm; determining a first network loss based on the difference between the first output image and a first base genuine image; mapping a second input image of the series of input images to a second output image based on the neural network algorithm; determining a second network loss based on the difference between the second output image and a second base genuine image; determining a consistency loss based on the difference between the first output image and a second output image; and updating the neural network algorithm based at least in part on the first network loss, the second network loss and the consistency loss. [Brief explanation of the drawing]

[0007] This embodiment is illustrated as an example and is not intended to be limited by the figures in the attached drawings.

[0008] [Figure 1] Figure 1 illustrates a block diagram of an exemplary image capture and display system that may utilize a neural network model, based on several implementations.

[0009] [Figure 2] Figure 2 illustrates block diagrams of exemplary operation of an image processor using a neural network model, based on several implementations.

[0010] [Figure 3] Figure 3 illustrates block diagrams of exemplary machine learning systems based on several implementations.

[0011] [Figure 4] Figure 4 illustrates block diagrams of exemplary loss calculators in several implementations.

[0012] [Figure 5] Figure 5 illustrates an exemplary chart showing exemplary pyramid levels related to pyramid loss.

[0013] [Figure 6] Figure 6 illustrates exemplary machine learning systems using several implementations.

[0014] [Figure 7] Figure 7 illustrates exemplary flowcharts showing exemplary operation for training a neural network using several implementations.

[0015] [Figure 8] Figure 8 illustrates exemplary flowcharts showing exemplary operation for training a neural network using several implementations. [Modes for carrying out the invention]

[0016] The following description includes numerous specific details, such as examples of specific components, circuits, and processes, to provide a deeper understanding of the disclosure. As used herein, the term “combined” means directly connected or connected via one or more intervening components or circuits. The terms “electronic system” and “electronic device” are used interchangeably and may refer to any system capable of electronically processing information. Furthermore, certain terminology is presented in the following description for illustrative purposes to provide a deeper understanding of the aspects of the disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary to implement exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obfuscation of the disclosure. Some parts of the following detailed description are presented in the form of procedures, logic blocks, processes, and other symbolic representations of operations on data bits in computer memory.

[0017] These descriptions and expressions are means used by those skilled in the art to most efficiently disseminate the content of their deliverables in data processing techniques to others skilled in the art. In this disclosure, procedures, logical blocks, processes, etc., are devised to be a self-consistent set of steps or instructions that lead to a desired result. Such steps require the physical manipulation of physical quantities. Although not essential, these quantities usually take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise processed in a computer system. However, it should be noted that all these and similar terms should correspond to the appropriate physical quantities and are merely convenient labels applied to those quantities.

[0018] As will be clear from the discussion below, unless otherwise specified, throughout this application, discussions using terms such as "access", "receive", "send", "use", "select", "decide", "normalize", "multiply", "average", "monitor", "compare", "apply", "update", "measure", "derive" refer to the operation and processing of a computer system or similar electronic computing device that processes data represented as a physical (electronic) quantity in the registers and memories of the computer system and converts it into other data similarly represented as a physical quantity in the memory or registers of the computer system or other such information storage, transmission, or display device.

[0019] In the figures, a single block may be described as performing one or more functions. However, in actual implementation, the one or more functions performed by that block may be performed in a single component or may be performed across multiple components. And / or, the one or more functions performed by that block may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this hardware and software interchangeability, various exemplary components, blocks, modules, circuits, and processes are described below generally in terms of their functions. Whether such a function is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. One of ordinary skill in the art may implement the described functions in various ways for each individual application, but such implementation decisions should not be construed as causing a departure from the technical scope of the present invention. Also, exemplary input devices may include components other than those shown, including well-known components such as processors, memories, etc.

[0020] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described to be implemented in a particular way. Any configuration described as a module or component may be implemented together in an integrated logic device or separately and as discrete but cooperating logic devices. When implemented in software, the techniques may be realized, at least in part, by a non-transitory processor-readable recording medium that includes instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable data recording medium may form part of a computer program product, which may include packaging materials.

[0021] The non-transitory processor-readable recording medium may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, and other known recording media. The techniques may also, or instead, be realized, at least in part, by a processor-readable communication medium that transmits or communicates code in the form of instructions or data structures and is accessible, readable, and / or executable by a computer or other processor.

[0022] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors (or processing systems). As used herein, the term "processor" may refer to any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, and / or state machine that can execute scripts or instructions of one or more software programs stored in memory.

[0023] Aspects of this disclosure provide systems and methods for denoising images while reducing flicker between sequentially displayed images (e.g., as a video). In some embodiments, a neural network (e.g., a recurrent neural network) may be trained to infer a denoised representation of a series of images of a scene (also known as a "batch"). More specifically, a neural network (also known as a "neural network algorithm" or "neural network model") may be trained to reduce noise in each image while maintaining a consistent appearance between sequentially displayed images.

[0024] During training, the machine learning system may acquire a series of input images capturing a scene. The machine learning system may also acquire one or more base authentic images. A base authentic image is a reference image of the scene that contains little or no noise. In some embodiments, the base authentic image may be any image with a higher signal-to-noise ratio (SNR) than any of the series of input images. In some other embodiments, the base authentic image may be any image with a higher resolution (e.g., pixel resolution) than any of the series of input images. Furthermore, in some embodiments, the machine learning system may map a first input image of the series of input images to a first output image based on a neural network algorithm. The machine learning system may also determine a first network loss based on the difference between the first output image and the base authentic image. The machine learning system may further map a second input image of the series of input images to a second output image based on the neural network algorithm and determine a second network loss based on the difference between the second output image and the base authentic image. The first network loss and the second network loss may each represent the difference between the first and second output images and the underlying true image, respectively. For example, such a difference may reflect the amount of error or noise in each output image. The machine learning system may further determine the consistency loss based on the difference between the first and second output images. Thus, the consistency loss may represent the difference between the first and second output images. For example, such a difference may reflect the amount of noise or appearance change across a series of sequential or continuous images.

[0025] In some embodiments, the machine learning system may update the neural network algorithm based at least partially on the first network loss, the second network loss, and the consistency loss, for example, at the end of each iteration of the training process. In some embodiments, the machine learning system may repeat the training process until a certain convergence criterion is met. In some embodiments, the trained neural network model may be deployed, for example, in an image capture and display system and used to infer a denoised representation of an image in real time. By training the neural network based on the consistency loss, the neural network model can minimize the change in noise in sequentially displayed images. Thus, for example, when denoised images are displayed sequentially in a video, the video appears to have minimized or even flicker-free, smooth, and consistent.

[0026] Figure 1 illustrates a block diagram of an exemplary image capture and display system 100, which may use a neural network model in several implementations. The system 100 comprises an image capture device 110, an image processor 120, and a display device 130. The image capture device 110 (e.g., a camera) captures a series of patterns of light 101 (also referred to as "scene light 101") from a scene and converts the scene light 110 into digital image capture data 102 (also referred to as "image capture data 102"). The image capture data 102 may include, for each pattern of light in the series, an array of pixels (or pixel values) representing a digital image of the scene. The display device 130 (e.g., any other device including a television, computer monitor, smartphone, or electronic display) renders or displays the digital image by reproducing the series of light patterns on the associated screen surface.

[0027] The image processor 120 converts the image capture data 102 into image rendering data 103 that may reflect the original scene captured by the image capture device 110 as a video with little or no flicker when displayed on the display device 130. For example, the image processor 120 may be configured to correct and smooth various pixel distortions associated with the image capture data 102. In other words, the image processor 120 may be configured to reduce and minimize noise variations associated with the digital image. Although shown as a separate block in Figure 1, in actual implementations, the image processor 120 may be integrated into, or otherwise included in, the image capture device 110, the display device 130, or a combination thereof.

[0028] In some embodiments, the image processor 120 may generate a denoised representation (e.g., image rendering data 103) of the image capture data 102 based on a neural network model 123 trained by machine learning. Machine learning is a technique for improving the ability of a computer system or application to perform a particular task. During the training phase, a machine learning system may be given a set of “answers” ​​and one or more sets of raw data to be mapped to each answer. For example, a machine learning system may be trained to perform denoising operations on a set of low-light images by providing the system with a series of short-exposure images of a scene taken under low-light conditions (representing the raw data) and one or more typical images (or basic true images) of the same scene with little or no noise (representing the answers). The machine learning system may then “learn” a set of rules that can be used to analyze the raw data and reconstruct the answers. For example, a machine learning system may perform statistical analysis on the raw data to determine a set of common features (also called “rules”) associated with noise or changes in noise related to the raw data.

[0029] In some aspects, machine learning systems may use deep learning. Deep learning is a specific form of machine learning in which the model being trained is a multi-layer neural network. Deep learning architectures are often referred to as artificial neural networks because of the way they process information (similar to the nervous systems of living organisms). For example, each layer of a deep learning architecture may consist of a large number of artificial neurons. The neurons are internally connected across various layers so that input data (or raw data) can be passed from one layer to another. More specifically, each layer of neurons may perform different kinds of transformations on the input data, ultimately producing a desired output. The internally connected structure of neurons is sometimes called a neural network model. Thus, the neural network model 123 may contain a set of rules that can be used to "infer" a denoised representation of the image data 102. As a result, the image processor 120 may use the neural network model 123 to reduce and minimize the noise variations associated with the image data 102.

[0030] Figure 2 illustrates a block diagram of an exemplary operation 200 of an image processor using a neural network model, according to several implementations. Image processor 220 may be an embodiment of image processor 120 in Figure 1. Thus, image processor 220 may be configured to receive image 202 and output denoised image 203. Referring to Figure 1, image 202 may be an example of image acquisition data 102, and denoised image 203 may be an example of image rendering data 103.

[0031] In some embodiments, the image processor 220 may comprise a neural processor 221 and a neural network model 223. The neural processor 221 may comprise one or more processors configured to accelerate neural network inference. For example, the hardware architecture of the neural processor 221 may be designed to traverse the neural network faster or more efficiently than a general-purpose processor. The neural processor 221 may implement the neural network based on the neural network model 223. In some embodiments, the neural network model 223 may be an example of the neural network model 123 in Figure 1. Thus, the neural network model 223 may comprise a set of rules that can be used to infer a denoised representation of a captured image.

[0032] As illustrated in Figure 2, image 202 appears noisy or grainy. For example, the pixel distribution in image 202 includes "spikes" in brightness and color (e.g., dark pixels in bright areas of the scene and bright pixels in dark areas of the scene). On the other hand, the pixel color and brightness of the denoised image 203 follow a substantially smooth distribution. For example, in denoised image 203, dark areas of the scene appear dark, and bright areas of the scene appear bright. Although Figure 2 only illustrates a single noisy image 202 processed by the image processor 220, in actual implementations, a series of noisy images (or videos) may be processed by the image processor 220. Furthermore, the image processor 220 may use a neural processor 221 and a neural network model 223 to output a series of denoised images that appear as a smooth or stable video when displayed on a display screen.

[0033] The neural network model 223 may be trained before being incorporated into the image processor 220 to generate a series of denoised images. More specifically, the neural network model 223 may be trained by a machine learning system to infer a denoised representation of a series of noisy input images capturing a scene. In some embodiments, the training may include mapping a first input image of the series of input images to a first output image using the neural network algorithm and determining a first network loss based on the difference between the first output image and the base authentic image. The first network loss may represent the amount of noise in the first output image relative to the base authentic image. In some embodiments, the training may further include mapping a second input image of the series of input images to a second output image using the neural network algorithm and determining a second network loss based on the difference between the second output image and the base authentic image. The second network loss may represent the amount of noise in the second output image relative to the base authentic image.

[0034] In some embodiments, the first output image may be compared to the second output image to determine the consistency loss. The consistency loss may represent the difference between the first and second output images. For example, such a difference may reflect the amount of noise or change in appearance between the first and second output images. In some embodiments, the first network loss, the second network loss, and the consistency loss may be used to update the weights of the neural network algorithm. Furthermore, in some embodiments, the machine learning system may repeat the training process described above until a certain convergence criterion is met.

[0035] By updating the neural network algorithm based on the first and second network losses, the neural network algorithm may be configured to reduce noise in a series of output images. Furthermore, by updating the neural network algorithm based on consistency loss, the neural network algorithm may be configured to reduce the amount of noise variation between a series of output images so that when the output images are displayed as a video, they appear as a smooth or stable video with little or no flicker.

[0036] Figure 3 illustrates a block diagram of an exemplary machine learning system 300 in several implementations. The machine learning system 300 comprises a neural network 322, a loss calculator 326, and a network optimizer 328. In some embodiments, the machine learning system 300 is used to train the neural network 322 to generate a neural network model 323. The neural network model 323 may be embodiments of the neural network models 123 and / or 223 in Figures 1 and 2, respectively.

[0037] As illustrated in Figure 3, the machine learning system 300 may acquire a series of input images 302 (also referred to as "input images 302") and one or more base authentic images 325. In some embodiments, the signal-to-noise ratio (SNR) of each input image 302 may be lower than that of the one or more base authentic images 325. For example, in some embodiments, each of the input images 302 may represent a noisy, short-exposure image of a scene under low-light conditions, while each of the one or more base authentic images 325 may represent a less noisy or noisy, long-exposure image of the same scene under low-light conditions. Furthermore, in some embodiments, the resolution (e.g., pixel resolution) of each input image 302 may be lower than that of the one or more base authentic images 325. In such embodiments, the machine learning system 300 may upscale (or perform super-resolution processing on) each of the input images 302 using the one or more base authentic images 325.

[0038] In some embodiments, each of the input images 302 and each of the one or more base authentic images 325 may have individual luminance levels. For example, the luminance level of an image may reflect the lighting (or illuminance) of the scene depicted in the image, the exposure of the image, and / or the gain (or sensitivity) of the photosensor of the image capture device used to capture the image. In some embodiments, each of the input images 302 may be preprocessed so that the luminance level of each input image 302 is the same as the luminance level of the base authentic image 325.

[0039] As illustrated in Figure 3, the neural network 322 may be configured to receive each input image 302 and generate individual output images 303 based on the received input images 302. More specifically, the neural network 322 may generate a denoised representation of the input images 302 by attempting to regenerate one or more underlying authentic images 325. In some embodiments, the neural network 322 may be a recurrent neural network configured to learn the temporal dynamics of noise in a series of input images 302. In some embodiments, the neural network 322 may consist of a network of weighted connections across multiple layers of artificial neurons, starting with a series of input images 302 and yielding a series of denoised output images 303. In some embodiments, the neural network 322 may be configured to feed each image of the series of denoised output images 303 to a loss calculator 326.

[0040] As illustrated in Figure 3, the loss calculator 326 may be configured to receive a series of denoised output images 303 from the neural network 322. The loss calculator 326 may also be configured to receive one or more base genuine images 325. In some embodiments, the loss calculator 326 may be configured to determine the amount of network loss (or error or noise) in each image of the series of output images 303 relative to one or more base genuine images 325. For example, to determine a first network loss, the loss calculator 326 may compare a first output image 303 with a base genuine image 325. Also, to determine a second network loss, the loss calculator 326 may compare a second output image 303 with a base genuine image 325. The loss calculator 326 may also be configured to determine additional network losses by comparing each of the remaining arbitrary output images of the series of output images 303 with a base genuine image 325.

[0041] In some embodiments, the loss calculator 326 may also be configured to determine the consistency loss (or change in noise) between a series of output images 303. For example, to determine the consistency loss, the loss calculator 326 may compare a first output image 303 with a second output image 303, where the first and second output images 303 are consecutive images in the series of output images 303. In some embodiments, the loss calculator 326 may be further configured to determine the consistency loss based on any remaining consecutive images in the series of output images 303. In some embodiments, the loss calculator 326 may be configured to determine the total loss 305 by combining the consistency loss, the first network loss, the second network loss, and any additional network losses. The loss calculator 326 may further be configured to output the total loss 305 to the network optimizer 328.

[0042] In some embodiments, the network optimizer 328 may be configured to receive a total loss 305 and adjust one or more weights (also called "parameters") 307 of the neural network 322 based on the total loss 305. More specifically, the network optimizer 328 may adjust the weights 307 in a manner that reduces the total loss 305. The machine learning system 300 may repeat the above training process over one or more iterations until a certain convergence criterion is met. For example, a new series of noisy images may be provided as input to the neural network 322, the loss calculator 326 may determine the total loss 305 based on the output of the neural network 322, and the network optimizer 328 may further update the weights 307 based on the total loss 305.

[0043] Figure 4 illustrates a block diagram of an exemplary loss calculator 400 in several implementations. The loss calculator 400 may be an embodiment of the loss calculator 326 in Figure 3. As illustrated in Figure 4, the loss calculator 400 comprises a network loss calculator 432, a consistency loss calculator 434, and a total loss calculator 436.

[0044] The network loss calculator 432 may be configured to receive each image of a series of output images 403 (also referred to as "output images 403") from a neural network, for example, the neural network 322 in Figure 3. The network loss calculator 432 may also be configured to receive one or more base authentic images 425.

[0045] In some embodiments, a series of output images 403 may be characterized by the following four dimensions: (i) batch size N, (ii) height H, (iii) width W, and (iv) channel C. The batch size N may represent the total number of images in the series of output images 403. The height H may represent the height of each image in the series of output images 403 in pixels. The width W may represent the width of each image in the series of output images 403 in pixels. The channel C may represent the color (e.g., red, green, blue, and / or opacity) or the value of each pixel in each image of the series of output images 403. In some embodiments, channel C may be represented by one or more numerical values. In addition, one or more base authentic images 425 and a series of input images (e.g., a series of input images 302 in Figure 3) may also be characterized by the four dimensions N, H, W, and C, similar to the series of output images 403.

[0046] In some embodiments, the network loss calculator 432 may be configured to determine the total network loss 433 (also referred to as "network loss 433"). Here, the total network loss 433 may represent the amount of error or noise in a series of output images 403 based on one or more base authentic images 425. The network loss calculator 432 may determine the network loss 433 based on the difference between the pixel values ​​of each image in the series of output images 403 and the pixel values ​​of one or more base authentic images 425. For example, in some embodiments, the network loss calculator 432 may determine the first network loss by comparing the pixel values ​​of the first output image 403 with the pixel values ​​of the base authentic image 425. The network loss calculator 432 may also determine the second network loss by comparing the pixel values ​​of the second output image 403 with the pixel values ​​of the base authentic image 425. If the first and second output images 403 represent the respective images in a series of output images 403, the network loss calculator 432 may combine the first network loss and the second network loss to generate a total network loss 433.

[0047] In some embodiments, the network loss calculator 432 may be configured to determine the network loss 433 based on one or more pyramid losses. The pyramid loss may represent the total amount of error in the pixel values ​​of the output image 403 relative to the pixel values ​​of the underlying true image 425 at various pyramid levels (or scales). The pyramid level (or scale) may refer to the pixel granularity of an image, such as the output image 403 or the underlying true image 425. For example, a higher pyramid level may correspond to a coarser pixel granularity or fewer pixel values. As a result, an image may appear "zoomed out" (and thus show less detail) compared to the same image at a lower pyramid level. In some embodiments, the network loss 433 may include the pyramid loss for at least one output image 403. Furthermore, in some embodiments, the network loss 433 may include the pyramid loss for each image in a set of output images 403.

[0048] The consistency loss calculator 434, like the network loss calculator 432, may be configured to receive each image of a series of output images 403. In some embodiments, the consistency loss calculator 434 may be configured to determine a consistency loss 435, where the consistency loss 435 may represent the amount of change in noise between two or more consecutive images of the series of output images 403. In other words, the consistency loss 435 may represent the magnitude or degree of inconsistency between two or more consecutive images of the series of output images 403. In some embodiments, the consistency loss 435 may be based on a standard deviation and may be expressed as follows:

number

[0049] In Equation 1A, N, H, W, and C represent the four dimensions of the series of output images 403, and n, h, w, and c are indices corresponding to dimensions N, H, W, and C, respectively. cwhn This represents the pixel value in channel c, width w, height h, and batch index (also called "image index") n. cwh This represents the average value calculated over the batch dimension (for example, from image index n=0 to n=N) for pixels located in channel c, width w, and height h. Furthermore, m cwh This can be expressed as follows:

number

[0050] It should be noted that equations 1A and 1B described above are suitable for use when the batch size N of a series of output images 403 (or the corresponding series of input images) remains constant as the machine learning system (e.g., machine learning system 300 in Figure 3) undergoes multiple iterations of training.

[0051] In some other embodiments, the consistency loss 435 may represent the rank of a matrix (B) (also known as "matrix B") based on a series of output images 403. More specifically, matrix B may contain a series of vectors, where each vector corresponds to an image in the series of output images 403. Furthermore, each vector may be flattened or reduced to a single dimension representing dimensions H, W, and C, respectively, for a given image in the series of output images 403. In some embodiments, matrix B may be represented as follows:

number

[0052] The consistency loss 435 is based on matrix B or the average pixel value (as shown in equations 1A and 1B above), but in some embodiments, the consistency loss 435 may be based on some relationship between the images of a series of output images 403. For example, in some embodiments, a series of output images 430 may depict a scene with movement, such as a scene depicting a moving object. The first image of the series of output images 403 may contain one or more pixels corresponding to a first position of the object in the scene, and the second image of the series of output images 403 may contain one or more pixels corresponding to a second position of the object in the scene. In some embodiments, the motion of the object may be defined based on one or more pixels corresponding to the position of the object in the first image and one or more pixels corresponding to the position of the object in the second image. Once the motion is defined, the first and / or second images of the series of output images 403 may be modified or distorted, and the consistency loss 435 may be determined based on the defined motion and / or the modified first and second images. While relationships based on motion (or defined motion) have been described above, in some embodiments, such relationships may be based on changes in pixel values ​​(e.g., pixel intensity) between a series of output images 403.

[0053] As illustrated in Figure 4, the network loss 433 and consistency loss 435 are provided to the total loss calculator 436. The total loss calculator 436 may be configured to combine the network loss 433 and consistency loss 435 to determine the total loss 405. The total loss 405 may be expressed as follows:

number

[0054] In some embodiments where the series of output images 403 depict a static scene, y may be set to 1. In some other embodiments where the series of output images 403 depict a scene with motion, y may be set to 0. Furthermore, in some other embodiments where the series of output images 403 depict both static and motion scenes, y may be a vector of size N (the batch size of the series of output images 403). That is, y may be a vector with N elements, where each element corresponds to each image in the series of output images 403. Furthermore, an element may be set to 1 if it corresponds to an output image 403 that is part of a static scene, and to 0 if it corresponds to an output image 403 that is part of a motion scene.

[0055] Once the total loss 405 is determined, the network optimizer (e.g., network optimizer 328 in Figure 3) may use the total loss 405 to determine whether certain convergence criteria are met. For example, if the total loss 405 is less than a threshold level and / or a predetermined number of training iterations have been completed, the network optimizer may determine that the neural network (e.g., neural network 322 in Figure 3) is optimized. Therefore, the network optimizer may not update the weights of the neural network (e.g., weights 307 in Figure 3). As another example, if the total loss 405 is greater than a threshold level and / or a predetermined number of training iterations have not yet been completed, the network optimizer may determine that the neural network is not yet optimized. Therefore, the network optimizer may determine one or more weights that minimize the total loss 405 in order to update the weights of the neural network.

[0056] In some embodiments, the network optimizer may use stochastic gradient descent to determine one or more weights by minimizing the total loss 405. In addition to stochastic gradient descent, the network optimizer may use additional techniques to minimize the consistency loss component of the total loss 405. For example, if the consistency loss 435 is calculated using equations 1A and 1B above, the TensorFlow or Keras software library may be used to minimize the consistency loss 435. Another example in which the consistency loss 435 is expressed as the rank of matrix B is to replace the (non-convex) rank with its non-convex surrogate, the kernel norm, and minimize the rank by using the L1 norm sparse decomposition and the Extended Lagrange Multiplier (ALM) method.

[0057] Figure 5 illustrates an exemplary figure 500 showing exemplary pyramid levels (or scales) related to pyramid loss. In some embodiments, pyramid loss represents the difference between the output image (e.g., output image 503 or 504 in Figure 5) and the underlying authentic image (e.g., underlying authentic image 525 in Figure 5) at different pyramid levels (e.g., pyramid levels 1 and 0 in Figure 5). In some embodiments, pyramid loss may be determined by a network loss calculator (e.g., network loss calculator 432 in Figure 4).

[0058] The pyramid level (or scale) can refer to the pixel granularity of an image, such as an output image or a base authentic image. For example, a higher pyramid level may correspond to a coarser pixel granularity or fewer pixel values. Consequently, the image may appear zoomed out (or show less detail) than the same image at a lower pyramid level. For example, each pixel value corresponding to a higher pyramid level may represent the average of two or more pixel values ​​(e.g., two or more adjacent pixels) corresponding to a lower pyramid level. Conversely, a lower pyramid level may be associated with a finer pixel granularity or more pixel values. Consequently, the image may appear zoomed in (or show more detail) than the same image at a higher pyramid level. For example, the pixel values ​​corresponding to the lowest pyramid level (e.g., pyramid level 0 in Figure 5) may represent the raw (or unscaled) pixel values ​​of the output image or base authentic image.

[0059] In some embodiments, the network loss calculator may be configured to generate a scaled number of pixel values ​​corresponding to each pyramid level except the lowest pyramid level, using pooling (also called "downscaling") such as average pooling, maximum pooling, or minimum pooling (as described above). For example, the network loss calculator may pool the pixel values ​​corresponding to the lowest pyramid level (e.g., pyramid level 0 in Figure 5) to generate a scaled pixel value corresponding to the next lowest pyramid level (e.g., pyramid level 1 in Figure 5). In some embodiments, the network loss calculator may be configured to calculate the pyramid loss of an image based on a predetermined pooling size, where the pooling size includes two or more pixels. For example, referring to Figure 5, the network loss calculator may calculate the pyramid loss for the output image 503 based on a predetermined pooling size of four pixel values. That is, the network loss calculator may pool the four pixel values ​​corresponding to pyramid level 0 for the output image 503 to generate a scaled pixel value corresponding to pyramid level 1 for the output image 503. Furthermore, in some embodiments, the network loss calculator may be configured to calculate the pyramidal loss of an image based on a predetermined number of pyramidal levels (for example, two pyramidal levels such as pyramidal levels 0 and 1 in Figure 5). Furthermore, in some embodiments, the network loss calculator may be configured to calculate the pyramidal loss of an image based at least partially on a predetermined number (or a target number) of pixel values ​​for the highest pyramidal level. For example, referring to Figure 5, the network loss calculator may calculate the pyramidal loss of output image 503 based at least partially on a single pixel value corresponding to pyramidal level 1 of output image 503.

[0060] As illustrated in Figure 5, pyramid level 0 and pyramid level 1 can be used to determine the pyramid loss for output images 503 and 504, respectively. As stated above, pyramid level 0 is the lowest pyramid level shown in Figure 5. Thus, pyramid level 0 is associated with pixel values ​​that represent the raw (or unscaled) pixel values ​​of each pixel in output image 503, output image 504, and base authentic image 525. In other words, Figure 5 shows a subset of four pixels at pyramid level 0 for each of output image 503, output image 504, and base authentic image 525. That is, at pyramid level 0, Figure 5 illustrates the pixels (or pixel values) P01, P02, P03, and P04 of output image 503, the pixels (or pixel values) P01', P02', P03', and P04' of output image 504, and the pixels (or pixel values) P01'', P02'', P03'', and P04'' of base authentic image 525. Although not shown in Figure 5, in some embodiments, a subset of two, three, or five or more pixel values ​​may represent the raw pixel values ​​of each pixel in output image 503, output image 504, and base authentic image 525 at pyramid level 0.

[0061] To determine the pyramidal loss of the output image 503, the network loss calculator may first calculate the loss amount corresponding to pyramidal level 0 for the output image 503 (also called "L0 loss") and the loss amount corresponding to pyramidal level 1 for the output image 503 (also called "L1 loss"). To calculate the L0 loss of the output image 503, the network loss calculator may take the sum of the absolute differences between the corresponding pixel values ​​at pyramidal level 0 of the output image 503 and the base true image 525, as shown in Equation 3 below.

number

number

[0062] Before the L1 loss for output image 503 can be calculated, the network loss calculator may use pooling to generate pyramid level 1 pixel values ​​for output image 503 and the underlying true image 525. For example, as illustrated in Figure 5, the network loss calculator may use average pooling to generate individual pixel values ​​P1 at pyramid level 1 of output image 503 by averaging the pixel values ​​at pyramid level 0 of output image 503 across four adjacent pixels (e.g., P01, P02, P03, and P04). The value of P1 can be calculated as follows:

number

[0063] Similarly, the network loss calculator may determine the individual pixel values ​​P1'' at pyramid level 1 of the underlying authentic image 525 using average pooling, which averages the pixel values ​​at pyramid level 0 of the underlying authentic image 525. The value of P1'' can be calculated as follows:

number

[0064] To calculate the L1 loss of the output image 503, the network loss calculator may determine the absolute difference between pixel P1 of the output image 503 at pyramid level 1 and pixel P1'' of the base true image 525, as shown in Equation 4 below.

number

[0065] Substituting the pixel values ​​of the output image 503 and the base true image 525 at pyramid level 1 into equation 4, we obtain the result with an L1 loss of 0, as shown below.

number

[0066] More generally, a network loss calculator may determine the pyramidal loss over any number (i) pyramidal levels of an output image having any number (N') pixels using equation 5A below.

number

number

number

number

[0067] Using the process described above, the pyramidal loss for output image 504 can be calculated as 0.5. In some embodiments, when calculating the pyramidal loss, loss amounts corresponding to different pyramidal levels (also called "loss terms") L0, L1, ..., L z-1 Different weights may be applied to them. For example, in some applications, loss terms corresponding to lower pyramid levels may be weighted more heavily than loss terms corresponding to higher pyramid levels. In some other applications, loss terms corresponding to higher pyramid levels may be weighted more heavily than loss terms corresponding to lower pyramid levels. Furthermore, in some embodiments, the pyramid loss for one or more output images in a set of output images may be used to update the weights of the neural network algorithm.

[0068] Figure 6 illustrates exemplary machine learning systems 600 in several implementations. In some embodiments, the machine learning system 600 may be an example of the machine learning system 300 in Figure 3. Thus, the machine learning system 600 may be configured to generate a neural network model 623 based on a series of input images 602 (also referred to as input images 602) and one or more basic genuine images 625. In some embodiments, the machine learning system 600 may include a processing system 640 and memory 650.

[0069] Memory 650 may include a non-temporary computer-readable medium that may store at least the following software (SW) modules (including one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, hard drive, etc.): A neural network SW module 652 is configured to generate a denoised representation (e.g., a series of output images) of a series of input images 602 by attempting to regenerate one or more foundational genuine images 625 in order to train the neural network model 623. · A loss calculator SW module 654 configured to determine the total loss amount based on a series of output images and one or more basic genuine images 625 (wherein the loss calculator SW module 654 further comprises a network loss submodule 655 that determines the total amount of network loss based on a series of output images and one or more basic genuine images 625, and a consistency loss submodule 656 that determines the amount of consistency loss based on a series of output images), and • Network optimizer SW module 658 configured to determine one or more updated weights for the neural network SW module 652 based on the total loss amount. Each software module contains instructions that, when executed by the processing system 640, cause the machine learning system 600 to perform the corresponding function.

[0070] The processing system 640 may include one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in the machine learning system 600 (for example, in memory 650). For example, the processing system 640 may run a neural network SW module 652 to generate a denoised representation of a series of input images 602 by attempting to regenerate one or more basic authentic images 625. In other words, the processing system 640 may run a neural network SW module 652 to map a series of input images 602 to a series of output images.

[0071] The processing system 640 may further execute a loss calculator SW module 654 to determine the total loss. When executing the loss calculator SW module 654, the processing system 640 may further execute a network loss submodule 655 and a consistency loss submodule 656. For example, the processing system 640 may execute a network loss submodule 655 to determine the total amount of network loss (or noise) in a series of output images based on one or more basic true images 625. The processing system 640 may further execute a consistency loss submodule 656 to determine the amount of change in noise between a series of output images. In some embodiments, the processing system 640 may further execute a network optimizer SW module 658 to determine the updated weights of the neural network SW module 652 based on the total loss.

[0072] Figure 7 illustrates an exemplary flowchart showing exemplary operation 700 for training a neural network, with several implementations. Exemplary operation 700 may be performed by a machine learning system (e.g., machine learning systems 300 and / or 600 in Figures 3 and 6, respectively) to train a neural network to infer a denoised representation of a series of input images.

[0073] A machine learning system may acquire a series of input images capturing a scene (710). The machine learning system may also map a first input image of the series of input images to a first output image based on a neural network algorithm (720). In some embodiments, the series of input images may represent images of a still scene. Furthermore, in some embodiments, the neural network algorithm may include a recurrent neural network algorithm.

[0074] The machine learning system may further determine a first network loss based on the difference between the first output image and the underlying genuine image (730). The machine learning system may also map a second input image of a set of input images to a second output image based on the neural network algorithm (740). In some embodiments, the SNR of the underlying genuine image may be greater than the SNR of the first input image and the SNR of the second input image. In some other embodiments, the resolution of the underlying genuine image may be greater than the resolution of the first input image and the resolution of the second input image.

[0075] The machine learning system may further determine a second network loss based on the difference between the second output image and the underlying true image (750). The machine learning system may also determine a consistency loss based on the difference between the first output image and the second output image (760). In some embodiments, each of the first and second output images may contain multiple pixel values. In some embodiments, the machine learning system may determine a consistency loss based on the average pixel value corresponding to the multiple pixel values ​​of the first output image and the multiple pixel values ​​of the second output image. In some other embodiments, the machine learning system may determine a consistency loss by generating a matrix containing the multiple pixel values ​​of the first output image and the multiple pixel values ​​of the second output image and determining the rank of the matrix.

[0076] The machine learning system may further update the neural network algorithm based at least partially on the first network loss, the second network loss, and the consistency loss (770). In some embodiments, the machine learning system may further determine the total loss based at least partially on the first network loss, the second network loss, and the consistency loss. The machine learning system may further minimize the total loss and update the neural network algorithm by determining one or more parameters based on the minimized total loss.

[0077] Figure 8 illustrates an exemplary flowchart showing exemplary operation 800 for training a neural network, with several implementations. In some embodiments, exemplary operation 800 may be performed by a machine learning system (e.g., machine learning systems 300 and / or 600 in Figures 3 and 6, respectively) to train the neural network to infer a denoised representation of a series of images.

[0078] A machine learning system may acquire a series of input images capturing a scene (810). The machine learning system may also map a first input image of the series of input images to a first output image based on a neural network algorithm (820). In some embodiments, the series of input images may represent images of a stationary scene and / or a scene with movement. Furthermore, in some embodiments, the neural network algorithm may include a recurrent neural network algorithm.

[0079] The machine learning system may further determine the first network loss based on the difference between the first output image and the first base genuine image (830). In some embodiments, the SNR of the first base genuine image may be greater than the SNR of the first input image. In some other embodiments, the resolution of the first base genuine image may be higher than the resolution of the first input image.

[0080] The machine learning system may also map a second input image of a set of input images to a second output image based on the neural network algorithm (840). The machine learning system may also determine a second network loss based on the difference between the second output image and the second fundamental image (850). In some embodiments, the SNR of the second fundamental image may be greater than the SNR of the second input image. In some other embodiments, the resolution of the second fundamental image may be higher than the resolution of the second input image.

[0081] The machine learning system may also determine the consistency loss based on the difference between the first output image and the second output image (860). In some embodiments, each of the first and second output images may contain multiple pixel values. In some embodiments, the machine learning system may determine the consistency loss based on the average pixel value corresponding to the multiple pixel values ​​of the first output image and the multiple pixel values ​​of the second output image. In some other embodiments, the machine learning system may determine the consistency loss by generating a matrix containing the multiple pixel values ​​of the first output image and the multiple pixel values ​​of the second output image and determining the rank of the matrix.

[0082] The machine learning system may further update the neural network algorithm based at least partially on the first network loss, the second network loss, and the consistency loss (870).

[0083] Those skilled in the art will understand that information and signals can be represented using any of the various different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be mentioned in the above description can be represented by voltage, electric current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0084] Furthermore, those skilled in the art will understand that various exemplary logic blocks, modules, circuits, and algorithmic processes described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and processes are generally described above in terms of their function. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the functions described for each specific application in various ways. However, such implementation decisions should not be construed as resulting in a departure from the scope of this disclosure.

[0085] Methods, sequences, or algorithms described in relation to the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination of the two. The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of recording medium known in the art. An exemplary recording medium is coupled to the processor so that the processor can read information from and write information to the recording medium. Alternatively, the recording medium may be integrated into the processor.

[0086] In the above specification, embodiments are described with reference to specific examples. However, it will be apparent that various modifications and changes can be made without departing from the broader scope of disclosure presented in the appended claims. Therefore, the specification and drawings should be evaluated in an illustrative rather than restrictive sense.

Claims

1. 1. A method for training a neural network, comprising: obtaining a sequence of input images of a scene; mapping a first input image of the series of input images to a first output image based on a neural network algorithm; determining a first network loss based on a difference between the first output image and an underlying authentic image; mapping a second input image of the series of input images to a second output image based on the neural network algorithm; determining a second network loss based on a difference between the second output image and the underlying authentic image; determining a consistency loss based on a difference between the first output image and the second output image; updating the neural network algorithm based at least in part on the first network loss, the second network loss, and the consistency loss; Contains method.

2. each of the first output image and the second output image includes a plurality of pixel values; The consistency loss is determined based on an average pixel value corresponding to the plurality of pixel values ​​of the first output image and the plurality of pixel values ​​of the second output image. The method of claim 1.

3. each of the first output image and the second output image includes a plurality of pixel values; Determining the consistency loss includes generating a matrix including the plurality of pixel values ​​of the first output image and the plurality of pixel values ​​of the second output image. The method of claim 1.

4. The consistency loss is determined based on the rank of the matrix. The method of claim 3.

5. The base authentic image has a higher signal-to-noise ratio (SNR) than either the first input image or the second input image. The method of claim 1.

6. Furthermore, determining a total loss based at least in part on the first network loss, the second network loss, and the consistency loss. The method of claim 1.

7. Furthermore, minimizing the total loss; and determining one or more parameters based on the minimized total loss; and Including, The neural network algorithm is updated based on the one or more parameters. The method of claim 6.

8. Furthermore, mapping a third input image of the series of input images to a third output image based on the neural network algorithm; determining a third network loss based on a difference between the third output image and the underlying authentic image; Including, The neural network algorithm is further updated based on the third network loss. The method of claim 1.

9. a processing system; a memory for storing instructions; A machine learning system comprising: The instructions, when executed by the processing system, cause the machine learning system to: Take a series of input images of a scene, mapping a first input image of the series of input images to a first output image based on a neural network algorithm; determining a first network loss based on a difference between the first output image and an underlying authentic image; mapping a second input image of the series of input images to a second output image based on the neural network algorithm; determining a second network loss based on a difference between the second output image and the underlying authentic image; determining a consistency loss based on a difference between the first output image and the second output image; updating the neural network algorithm based at least in part on the first network loss, the second network loss, and the consistency loss. Machine learning systems.

10. 1. A method for training a neural network, comprising: obtaining a sequence of input images of a scene; mapping a first input image of the series of input images to a first output image based on a neural network algorithm; determining a first network loss based on a difference between the first output image and a first underlying authentic image; mapping a second input image of the series of input images to a second output image based on the neural network algorithm; determining a second network loss based on a difference between the second output image and a second underlying authentic image; determining a consistency loss based on a difference between the first output image and the second output image; updating the neural network algorithm based at least in part on the first network loss, the second network loss, and the consistency loss; Contains method.

Images