Learning method, learning system, learning program, noise removal method, noise removal system, and noise removal program

Abstract

To properly remove noise regardless of characteristics of data.SOLUTION: A learning method for generating a learning model for removing noise from data having a plurality of values includes: a learning acquisition step (S01) for acquiring a plurality of pieces of learning data including noise and having a plurality of values; and a learning step (S02) of performing first machine learning configured to input information based on the learning data to a learning model, and performing second machine learning configured to input, to a learning model, information based on new learning data generated by synthesizing noise-removed data of the learning data based on an output from the learning model with a difference between another piece of learning data and noise-removed data of the other learning data, and to output, from the learning model, information based on a difference of the noise-removed data of the learning data or the other learning data, to generate a learning model.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a learning method, a learning system, and a learning program for generating a learning model for removing noise from data having a plurality of values, and a noise removal method, a noise removal system, and a noise removal program for removing noise from data having a plurality of values using the generated learning model.

Background Art

[0002] Conventionally, methods for removing noise from images by machine learning have been proposed. Examples of such methods include Noise2Clean and Noise2Noise (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In noise removal by machine learning, depending on the method, the noise to be removed may be different from the original noise according to the characteristics of the image including the noise. For example, at a location where there are irregularities in a signal such as a contour in an image, the noise to be removed may be different from the original noise according to the irregularities. That is, with conventional methods, there is a possibility that appropriate noise removal cannot be performed depending on the characteristics of the image.

[0005] The present invention has been made in view of the above, and an object thereof is to provide a learning method, a learning system, a learning program, a noise removal method, a noise removal system, and a noise removal program that enable appropriate noise removal regardless of the characteristics of data.

Means for Solving the Problems

[0006] In order to achieve the above object, a learning method according to the present invention is a learning method for generating a learning model for removing noise from data having a plurality of values, including: a learning acquisition step of acquiring a plurality of learning data having a plurality of values and including noise; performing first machine learning using information based on the learning data acquired in the learning acquisition step as an input to the learning model, and synthesizing the data after noise removal of the learning data based on the output from the learning model, the difference between another piece of learning data and the data after noise removal of the another piece of learning data based on the output from the learning model, to generate new learning data; using information based on the generated new learning data as an input to the learning model; and performing second machine learning using information based on the difference between the data after noise removal of the learning data or the another piece of learning data as an output from the learning model to generate a learning model.

[0007] In the learning method according to the present invention, machine learning is performed using new learning data obtained by synthesizing the data after noise removal of the learning data and the difference between another piece of learning data corresponding to the noise to be removed. Thus, according to the learning method of the present invention, appropriate noise removal can be achieved regardless of the characteristics of the data.

[0008] The data having a plurality of values may be an image. According to this configuration, a learning model for removing noise from an image including noise can be generated.

[0009] In the learning acquisition step, instead of obtaining the original data of the learning data and obtaining each of a plurality of portions of the original data as learning data, it is also possible to obtain the original data of the learning data and obtain each of a plurality of portions of the original data as learning data. According to this configuration, it is possible to obtain the learning data necessary for machine learning without preparing a large number of learning data in advance, and to appropriately perform machine learning.

[0010] Incidentally, the present invention can be described not only as an invention of a learning method as described above, but also as an invention of a learning system and a learning program as follows. These are only different in category, but are substantially the same invention and exhibit the same operations and effects.

[0011] That is, the learning system according to the present invention is a learning system that generates a learning model for removing noise from data having a plurality of values, and includes a learning acquisition unit that acquires a plurality of learning data having a plurality of values and including noise, and performs first machine learning using information based on the learning data acquired by the learning acquisition unit as an input to the learning model, and synthesizes data after noise removal of the learning data based on the output from the learning model, a difference between another learning data and data after noise removal of the another learning data based on the output from the learning model, to generate new learning data, and uses information based on the generated new learning data as an input to the learning model, and performs second machine learning using information based on the difference between the data after noise removal of the learning data or the another learning data as an output from the learning model to generate a learning model.

[0012] Also, the learning program according to the present invention is a learning program that operates a computer as a learning system for generating a learning model for removing noise from data having a plurality of values. The computer is caused to function as learning acquisition means for acquiring a plurality of learning data having a plurality of values and including noise, performing first machine learning with information based on the learning data acquired by the learning acquisition means as an input to the learning model, and synthesizing the data after noise removal of the learning data based on the output from the learning model and the difference between another learning data and the data after noise removal of the another learning data based on the output from the learning model to generate new learning data, using the information based on the generated new learning data as an input to the learning model, and performing second machine learning with information based on the data after noise removal of the learning data or the difference of the another learning data as an output from the learning model to generate a learning model as learning means.

[0013] The noise removal method according to the present invention is a noise removal method for removing noise from data having a plurality of values using the learning model generated by the above learning method, including a removal acquisition step of acquiring noise removal target data having a plurality of values and being a noise removal target, and a noise removal step of inputting the noise removal target data acquired in the removal acquisition step into the learning model to remove noise from the noise removal target data.

[0014] In the noise removal method according to the present invention, the learning model generated by the learning method according to the present invention is used to remove noise from data. Therefore, appropriate noise removal can be enabled.

[0015] By the way, the present invention can be described as an invention of a noise removal system and a noise removal program as follows in addition to being described as an invention of a noise removal method as described above. These are the same invention substantially, differing only in category, and exhibit the same actions and effects.

[0016] That is, the noise removal system according to the present invention is a noise removal system that removes noise from data having a plurality of values using the learning model generated by the above learning method, and includes a removal acquisition means for acquiring noise removal target data that has a plurality of values and is a target for noise removal, and a noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data.

[0017] Further, the noise removal program according to the present invention is a noise removal program that causes a computer to operate as a noise removal system that removes noise from data having a plurality of values using the learning model generated by the above learning method, and causes the computer to function as a removal acquisition means for acquiring noise removal target data that has a plurality of values and is a target for noise removal, and a noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data.

Effects of the Invention

[0018] According to the present invention, appropriate noise removal can be enabled regardless of the characteristics of the data.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments of a learning method, a learning system, a learning program, a noise removal method, a noise removal system, and a noise removal program according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same reference numerals are given to the same elements, and redundant descriptions are omitted.

[0021] FIG. 1 shows a learning system 20 and a noise removal system 30 according to the present embodiment. The learning system 20 and the noise removal system 30 are realized by a computer 10. The learning system 20 is a system (device) that generates a noise removal model (denoising model), which is a learning model (trained model) for removing noise from data having a plurality of values. The noise removal system 30 is a system (device) that removes noise from data having a plurality of values using the noise removal model generated by the learning system 20.

[0022] In the present embodiment, the data having a plurality of values to be the noise removal target is an image. However, the data having a plurality of values to be the noise removal target does not necessarily have to be an image, and may be, for example, waveform data or the like.

[0023] The computer 10 is a conventional computer including hardware such as a processor such as a CPU (Central Processing Unit), a memory, and a communication module. Also, the computer 10 may be a computer system including a plurality of computers. Further, the computer 10 may be configured by cloud computing or edge computing. Each function of the computer 10 described later is exhibited by these components operating according to a program or the like.

[0024] Subsequently, the functions of the learning system 20 and the noise removal system 30 according to the present embodiment will be described. As shown in FIG. 1, the learning system 20 includes a learning acquisition unit 21 and a learning unit 22.

[0025] Before explaining each function of the learning system 20, a noise removal model generated by the learning system 20 will be explained. In the present embodiment, the noise removal model is a model that inputs information based on an image and outputs (infers) an image obtained by removing noise included in the input image from the input image. Note that the noise removal model does not necessarily have to be a model that outputs an image after noise removal, and may be a model that outputs information capable of generating an image after noise removal. For example, the noise removal model may be a model that outputs noise included in the input image.

[0026] The noise removal model includes, for example, a neural network. The neural network may be a multi-layer one. That is, the noise removal model may be generated by deep learning. Also, the neural network may be a convolutional neural network (CNN). Further, as will be described later, the neural network may have a specific structure.

[0027] The noise removal model is provided with neurons for inputting information based on an image to the input layer. For example, the information input to the noise removal model is the pixel value (luminance value) of each pixel of the image. In this case, the input layer is provided with the same number of neurons as the number of pixels of the image, and the pixel value of the corresponding pixel is input to each neuron. Note that the information input to the noise removal model may be other than the pixel value of each pixel as long as it is based on the image.

[0028] The noise removal model is provided with neurons for outputting an image with noise removed to the output layer. For example, the information output from the noise removal model is the pixel value of each pixel of the image after noise removal. In this case, the output layer is provided with the same number of neurons as the number of pixels of the image, and the pixel value of the corresponding pixel is output from each neuron.

[0029] Note that the noise removal model may be configured by means other than a neural network.

[0030] The noise removal model is assumed to be used as a program module that is part of artificial intelligence software. The noise removal model is used, for example, in a computer equipped with a CPU and memory, and the CPU of the computer operates according to instructions from the model stored in the memory. For example, the CPU of the computer operates to input information to the model according to the instruction, perform an operation according to the model, and output a result from the model. Specifically, the CPU of the computer operates to input information to the input layer of the neural network according to the instruction, perform an operation based on parameters such as learning weight coefficients in the neural network, and output a result from the output layer of the neural network. The above is the noise removal model generated by the learning system 20.

[0031] The learning acquisition unit 21 is a learning acquisition means for acquiring learning data having a plurality of values and including noise. The data having a plurality of values may be an image. The learning acquisition unit 21 may acquire a plurality of learning data. The learning acquisition unit 21 may acquire the original data of the learning data and acquire each of a plurality of parts of the original data as learning data.

[0032] The learning acquisition unit 21 acquires learning data as follows, for example. The learning acquisition unit 21 acquires an image including noise as the original image (original data) of the learning data. The acquisition of the original image is performed, for example, by receiving from an imaging device that has imaged the original image or by accepting an input operation of the original image by the user to the computer 10.

[0033] As shown in FIG. 2, the learning acquisition unit 21 cuts out and acquires, as learning data, an image of a region (part) having a preset size that is smaller than the original image from the original image. The image of the region is called a patch. The learning acquisition unit 21 acquires, for example, a plurality of patches at different random positions from one original image.

[0034] Note that the learning data acquisition unit 21 may acquire learning data by a method other than cutting out from the original image described above. The learning data acquisition unit 21 acquires a number of learning data sufficient to appropriately perform machine learning described later. The learning data acquisition unit 21 outputs the acquired learning data to the learning unit 22.

[0035] In this embodiment, the noise removed by the noise removal model is noise having specific characteristics. Therefore, the learning data shall include noise having the specific characteristics. For example, patches obtained from images captured under the same conditions as the conditions under which the image to be denoised is captured (for example, the imaging device and the imaging environment) are used as the learning data.

[0036] The learning unit 22 performs first machine learning using the information based on the learning data acquired by the learning data acquisition unit 21 as the input to the learning model, and synthesizes the data after noise removal of the learning data based on the output from the learning model, the difference between another piece of learning data and the data after noise removal of the another piece of learning data based on the output from the learning model, to generate new learning data. Then, the information based on the generated new learning data is used as the input to the learning model, and the second machine learning is performed using the information based on the difference between the data after noise removal of the learning data or the another piece of learning data as the output from the learning model to generate a learning model.

[0037] The learning unit 22 may perform an operation with the information based on the learning data acquired by the learning acquisition unit 21 as an input to the learning model, calculate the difference between the learning data and the data after noise removal of the learning data based on the output from the learning model for each of a plurality of values, and perform first machine learning based on the statistical value of the difference for each of the plurality of values to generate a learning model. The learning unit 22 may use, as the statistical value of the difference for each of the plurality of values, any one of the average value for each learning data, the value indicating the difference in distribution between learning data, the value indicating the difference in correlation between learning data, and the value indicating the difference between learning data of the statistical value of the difference between adjacent values. The learning unit 22 may perform first machine learning so that the statistical value of the difference for each of the plurality of values becomes 0.

[0038] FIG. 2 schematically shows the first machine learning. The learning unit 22 performs the first machine learning as follows. The learning unit 22 inputs a patch, which is learning data, from the learning acquisition unit 21. The learning unit 22 performs an operation with each patch as an input to the noise removal model at that time to obtain an output from the noise removal model, for example, an image after noise removal (inferred signal image). The learning unit 22 calculates the noise image inferred at that time, which is the difference between the patch and the image after noise removal. The above difference is the value for each pixel, that is, the value for each of a plurality of values of the image. For example, the learning unit 22 calculates the difference for each pixel between the patch (input image) input to the noise removal model and the image after noise removal (inference result) to obtain the noise image inferred at that time. Note that the output from the noise removal model before or during machine learning is not necessarily related to accurate noise.

[0039] In the first machine learning, a noise removal model is generated without requiring a noise-free patch and an image of the noise itself as teacher images for patches containing noise. Therefore, even when these teacher images cannot be prepared, a noise removal model can be generated. In the first machine learning, assuming that the "inference result" is a "noise-removed image", it utilizes the fact that the signal to be removed can be assumed to be "noise", and focuses on the "signal to be removed" side rather than the inference result. Many methods such as Noise2Clean and Noise2Noise adopt a method of bringing the "image from which noise is removed" side closer to the expected value, but the first machine learning has the opposite idea.

[0040] In the first machine learning, the statistical values for each group (region) of a plurality of pixels are targeted for removing noise having specific characteristics. For example, the noise to be removed is Gaussian noise in which the pixel values of the noise follow a Gaussian distribution at any position in the image. The learning unit 22 performs machine learning based on the statistical values of the obtained noise image and the conditions related to the noise included in the image. The learning unit 22 performs machine learning so that the statistical values of the noise image obtained based on the output from the noise removal model meet the above conditions.

[0041] When the noise to be removed is Gaussian noise, it is considered to have the following characteristics. (1) The "average value" is the same for any region (patch) position (in the case of Gaussian noise, the average value is "0"). (2) The "spread of the distribution (dispersion)" is the same for any region (patch) position (in the case of Gaussian noise, there is no difference in the standard deviation (σ value) between patches). (3) The "correlation value" between regions is the same for any region (patch) position (it is "uncorrelated" between any regions). (4) The "total magnitude of the difference from adjacent pixels" is the same for any region (patch) position (the noise "content" is the same).

[0042] The above-described conditions regarding the noise in machine learning by the learning unit 22 are to ensure that the statistical values of the noise images obtained using the noise removal model satisfy the above characteristics. The learning unit 22 obtains noise images for each of a plurality of patches using the noise removal model. The learning unit 22 performs machine learning on the noise removal model using the following loss evaluation function (Loss function) from the obtained plurality of noise images.

[0043] [Number] x is a patch input to the noise removal model for machine learning. f(x) is the image after noise removal, which is the output of the noise removal model. z is a noise image. N is the number of noise images. z i mn is the pixel value of the noise image i of the patch whose position in the original image is mn. mean(z i mn ) is the average value of z i mn std(z i mn ) is the standard deviation of z i mn .

[0044] sim(z i mn ) is a value based on the correlation value of z i mn with other noise images. For example, sim(z i mn ) is the sum obtained by calculating the correlation values of combinations with all other noise images among the plurality of noise images obtained using the noise removal model. The correlation value of the combination with other noise images is calculated, for example, by converting the two-dimensional pixel values of each noise image into one-dimensional vectors and taking their inner product. Or, the inner product of the two-dimensional pixel value matrices may be calculated. Also, vector or matrix normalization may be performed during the calculation. sub(z i mn ) is z i mnIt is the sum for all pixels of the absolute value of the difference between a pixel and the pixels adjacent to it in a preset direction.

[0045] The first term of the loss evaluation function is the average value of the noise images of each patch (i.e., the average value for each learning data), which corresponds to the above characteristic (1). The second term of the loss evaluation function is the value obtained by subtracting the average of the standard deviations of the noise images of all patches from the standard deviation of the noise image of each patch (i.e., a value indicating the difference in distribution between learning data), which corresponds to the above characteristic (2). The third term of the loss evaluation function is the value obtained by subtracting the average of the values based on the correlation values of the noise images of all patches from the value based on the correlation value of the noise image of each patch (i.e., a value indicating the difference in correlation between learning data), which corresponds to the above characteristic (3). The fourth term of the loss evaluation function is the value obtained by subtracting the average of the values based on the differences between adjacent pixels of all patches from the value based on the differences between adjacent pixels of each patch (i.e., a value indicating the difference in the statistical value of the differences between adjacent values between learning data), which corresponds to the above characteristic (4).

[0046] α, β, and γ are coefficients indicating the weights of the respective terms of the loss evaluation function, and are preset values (hyperparameters). By increasing the value, the characteristics corresponding to each term of the noise can be considered more significantly during machine learning. α, β, and γ are set to values of 0 or more (in the case of 0, the characteristics corresponding to that term are not considered).

[0047] Machine learning itself based on the loss evaluation function, that is, the update of the parameters of the noise removal model by error backpropagation, may be performed in the same manner as the conventional machine learning method. Since machine learning based on the above loss evaluation function is performed, machine learning is performed so that the statistical values of the noise images of each patch shown in each term of the loss evaluation function become 0. That is, the learning unit 22 performs noise narrowing according to the noise conditions.

[0048] Among the items included in the loss evaluation function, the only one that uses only the value of the target patch without using the values of other patches is the average value (the above-mentioned characteristic (1)). Other items are relative values between the value of the target patch and the values of other patches. Also, as values related to the spread of the distribution, variance, skewness, kurtosis, etc. may be used instead of, or in addition to, the standard deviation.

[0049] In the first machine learning, as a noise removal model, it may be possible to use one that includes a neural network having a shortcut connection (residual path). By this, it becomes possible to perform flawless noise removal even for objects with different luminance ranges from those during machine learning. This type is advantageous for simple noise removal. However, when readability is emphasized more than signal reproducibility (involving large conversions), it may be better not to have a shortcut connection. Also, not using batch normalization results in faster learning and stable results.

[0050] As described above, in the first machine learning, machine learning is performed so that the statistical values of the noise images shown in each term of the loss evaluation function become zero. Also, it is possible to perform machine learning to make the noise zero from an image containing noise. Also, images without noise, etc. are unnecessary (zero) for machine learning. Also, the noise removal model may be one that can be learned by ordinary deep learning, and there are no restrictions inside the model (zero). Also, although Noise2Clean and Noise2Noise are pre-training methods, the first machine learning does not require pre-training. Also, the first machine learning can re-learn the noise removal model from the noise image. Also, the learning time in this embodiment is smaller than the conventional method.

[0051] Note that the noise to be removed in this embodiment does not necessarily have to be Gaussian noise. The noise to be removed in this embodiment may be one whose statistical values of noise for each of a plurality of values (for example, pixels) follow certain conditions.

[0052] Also, the loss evaluation function does not necessarily have to be the above, and any function that follows the above concept may be used. For example, in the above loss evaluation function, for the standard deviation, based on the identification of noise that the "spread of the distribution" is the same in any region (patch) position, even if the value of the standard deviation of the noise is unknown, it can be used for training the noise removal model.

[0053] However, if the value of the standard deviation of the noise is known in advance, machine learning may be performed so that the value of the standard deviation of the noise image based on the output from the noise removal model becomes that value. For example, if it is known that the standard deviation of the noise to be removed is σ = 20, machine learning may be performed so that the value of the standard deviation of the pixel values of the noise image based on the output from the noise removal model becomes 20. Also, if it is known that the average value of the noise image is a specific value other than 0, machine learning may be performed so that the value of the average value of the pixel values of the noise image based on the output from the noise removal model becomes the specific value. Specifying the value of a statistical value (for example, standard deviation, average value) obtained without using other noise images is important for converging an infinite number of combinations of inference results.

[0054] Also, when the noise to be removed is not Gaussian noise, the loss evaluation function may be adjusted according to the characteristics of the noise. For example, when the noise to be removed is not Gaussian noise, the variation in the correlation value is not used. However, even if the noise to be removed is not an ideal one such as Gaussian noise, since the noise has the above statistical quantities, depending on the statistical quantity, it can be used in the loss evaluation function. For example, the skewness is 0 for Gaussian noise, but it is not 0 for other noises due to distortion. However, since the patch is an area composed of multiple pixels rather than a single pixel, the distribution is the same. Therefore, the relative value between the value of the target patch related to the skewness and the value of other patches can be included and used in the loss evaluation function.

[0055] In the case of non-random noise, the correlation value between patches depends on how the patches are selected. Therefore, the correlation value is not constant, and the variation value related to the correlation value (for example, the value of the third term of the loss evaluation function) does not become zero. In that case, the variation value related to the correlation value is not included in the loss evaluation function.

[0056] Also, in the case of non-random noise, there may be a case where the noise is unevenly distributed with respect to the patches, such as the noise having periodicity. In that case, the variation value of the spread of the distribution (for example, the value of the second term of the loss evaluation function) is not included in the loss evaluation function.

[0057] The above is the first machine learning. In addition to the first machine learning, the learning unit 22 performs the following second machine learning to generate a noise removal model. Note that the first machine learning may be other than the first machine learning as long as it is effective when combined with the following second machine learning.

[0058] The learning unit 22 may synthesize the difference between the data after noise removal of the learning data based on the output from the learning model and another learning data to generate new learning data, use the information based on the generated new learning data as the input to the learning model, and perform the second machine learning using the information based on the data after noise removal of the learning data or the difference of the other learning data as the output from the learning model to generate a learning model.

[0059] FIG. 3 schematically shows the second machine learning. The learning unit 22 performs the second machine learning as follows. The learning unit 22 performs an operation with each patch as the input to the noise removal model at that time to obtain the output from the noise removal model, for example, the image after noise removal (inferred signal image). The learning unit 22 calculates the noise image inferred at that time, which is the difference between the patch and the image after noise removal. The processing up to the calculation of the noise image is the same as the processing up to the calculation of the noise image in the first machine learning. The processing up to the calculation of the noise image may be common in the first machine learning and the second machine learning.

[0060] The noise removed by the noise removal model generated only by the first machine learning often differs between the vicinity of the unevenness (vicinity of the contour) of the image and the vicinity of the flat surface, like the noise image shown in FIG. 3. As described above, in the present embodiment, the noise to be removed is such that the statistical values of the noise such as Gaussian noise follow certain conditions. Therefore, the noise corresponding to the contour of the image as described above is considered to be inappropriate as the noise to be removed because it is mixed with an illegal pattern. The second machine learning enables the generated noise removal model to perform appropriate noise removal regardless of the characteristics of the data such as the contour of the image.

[0061] The learning unit 22 combines the image after noise removal of a patch and the noise image of another patch to generate a new patch (new learning data). For example, the learning unit 22 randomly shuffles the combination (pair) of the image after noise removal and the noise image, and synthesizes the image after noise removal and the noise image of different combinations. The synthesis of the image after noise removal and the noise image is performed by adding the pixel values of the pixels at the same position. The new patch is one in which the noise including the illegal pattern that is the signal of the contour part is attached to the image after noise removal of another patch. Note that the generation of different combinations may be performed other than random shuffling.

[0062] The learning unit 22 performs machine learning using the new patch as the input to the noise removal model and the image after noise removal used for generating the new patch as the output from the noise removal model. The machine learning is performed such that when the new patch is input to the noise removal model, the output from the noise removal model becomes the image after noise removal used for generating the new patch, that is, the error between the output from the noise removal model and the image after noise removal used for generating the new patch is minimized. In this way, in the second machine learning, Noise2Clean is performed using the new patch obtained by synthesis and the image after noise removal. The machine learning itself, that is, the update of the parameters of the noise removal model by error backpropagation may be performed in the same manner as the conventional machine learning method.

[0063] The above machine learning is an example in the case where an image after noise removal is output from the noise removal model. When noise (noise image) is output from the noise removal model, the learning unit 22 may perform machine learning using the noise image used for generating the new patch as the output from the noise removal model.

[0064] The learning unit 22 generates a noise removal model by repeating the machine learning process until convergence based on preset conditions for generating the noise removal model, or for a preset number of times, in the same way as conventional machine learning. The first machine learning and the second machine learning may be performed together for each repetition. Alternatively, the second machine learning may be repeated after the first machine learning has been repeated. The learning unit 22 outputs the generated noise removal model to the noise removal system 30.

[0065] Also, the machine learning may be performed, for example, by the gradient descent method using the entire obtained sample patch, or the stochastic gradient descent method using some of the samples. The number of mini - batches, which are the samples used in the stochastic gradient descent method, is usually 64 or the like. Alternatively, for the stability of learning, the number of mini - batches may be 128. The above are the functions of the learning system 20 according to the present embodiment.

[0066] Subsequently, the functions of the noise removal system 30 according to the present embodiment will be described. As shown in FIG. 1, the noise removal system 30 includes a removal acquisition unit 31 and a noise removal unit 32.

[0067] The removal acquisition unit 31 is a removal acquisition means for acquiring noise removal target data that has a plurality of values and is a target for noise removal. The removal acquisition unit 31 acquires an image including noise as the noise removal target data. The acquisition of the image is performed, for example, by receiving the image from an imaging device that has captured the image, or by accepting an input operation of the image by the user to the computer 10. The removal acquisition unit 31 outputs the acquired noise removal target data to the noise removal unit 32.

[0068] The noise removal unit 32 is a noise removal means that inputs the noise removal target data acquired by the acquisition unit 31 for removal into a learning model to remove noise from the noise removal target data. The noise removal unit 32 inputs and stores a noise removal model, which is a learning model generated by the learning system 20, and uses it for noise removal.

[0069] Fig. 4 schematically shows the removal of noise by the noise removal unit 32. The noise removal unit 32 removes noise from the image to be denoised as follows. The noise removal unit 32 divides the image to be denoised into patches by tiling. This patch may have a different size from the patch that is the learning data.

[0070] The noise removal unit 32 inputs the divided patches into the noise removal model and obtains patches with noise removed (inferred "signal images"), which are the outputs from the noise removal model. Note that the generated noise removal model may be configured to input an image of a size different from the patch used during machine learning and output the image with noise removed. This can be achieved by conventional methods. The noise removal unit 32 combines the patches with noise removed to generate an image with noise removed (inference result image (signal)) from the image to be denoised.

[0071] If there is sufficient computer processing hardware resources, the noise removal unit 32 may input the entire image into the noise removal model without dividing it into patches and obtain the image with noise removed.

[0072] The noise removal unit 32 outputs the generated image after noise removal. The output of the image after noise removal may be performed in the same manner as conventional methods according to the purpose of use of the image. The above is the function of the noise removal system 30 according to the present embodiment.

[0073] Next, using the flowcharts of FIGS. 5 and 6, the processing (operation method performed by the computer 10) executed by the computer 10 according to the present embodiment will be described. First, using the flowchart of FIG. 5, the processing executed when generating the noise removal model, that is, the learning method which is the processing executed by the learning system 20 according to the present embodiment will be described.

[0074] In this processing, an image (for example, the above-described patch) which is learning data is acquired by the learning acquisition unit 21 (S01, learning acquisition step). Subsequently, by the learning unit 22, based on the learning data, first machine learning and second machine learning are performed to generate a noise removal model which is a learning model (S02, learning step).

[0075] In the first machine learning, an operation is performed in which information based on the learning data is used as an input to the noise removal model, and a difference (noise image) between the learning data and the data after noise removal of the learning data based on the output from the noise removal model is calculated for each of a plurality of values (pixels), and it is performed based on the statistical value of the differences for each of the plurality of values. In the second machine learning, a difference between the data after noise removal of the learning data based on the output from the noise removal model and another learning data is combined to generate new learning data, information based on the generated new learning data is used as an input to the noise removal model, and information based on the difference between the data after noise removal of the learning data or the another learning data is used as an output from the noise removal model.

[0076] The generated noise removal model is output from the learning system 20 to the noise removal system 30 (S03). In the noise removal system 30, the noise removal model is stored and used in the following processing. The above is the learning method which is the processing executed by the learning system 20 according to the present embodiment.

[0077] Subsequently, using the flowchart of FIG. 6, the processing executed when removing noise from data, that is, the noise removal method which is the processing executed by the noise removal system 30 according to the present embodiment will be described.

[0078] In this process, an image, which is data to be denoised, is acquired by the acquisition unit 31 for removal (S11, acquisition step for removal). Subsequently, the data to be denoised is input into the denoising model by the denoising unit 32, and noise is removed from the data to be denoised (S12, denoising step). The image after denoising, which is the result of noise removal, is output from the denoising unit 32 to a predetermined output destination (S13). The above is the denoising method, which is the process executed by the denoising system 30 according to the present embodiment.

[0079] In the present embodiment, a learning model is generated by the first machine learning based on the statistical value of the difference according to the characteristics of the noise. Therefore, according to the present embodiment, even when data required for conventional noise removal methods such as an image without noise and a plurality of images including different noises for the same object cannot be prepared, appropriate noise removal can be achieved. However, the first machine learning does not necessarily have to be the first machine learning as described above, and it may be effective when combined with the second machine learning.

[0080] Also, as in the present embodiment, the data having a plurality of values to be denoised and the data having a plurality of values used for machine learning may be images. According to this configuration, a learning model for removing noise from an image including noise can be generated. However, as described above, the data having a plurality of values to be denoised and the data having a plurality of values used for machine learning do not necessarily have to be images, and may be, for example, waveform data or the like.

[0081] Also, as in the present embodiment, as the statistical value of the difference for each of the plurality of values used for machine learning, any one of the average value for each learning data, the value indicating the difference in distribution between learning data, the value indicating the difference in correlation between learning data, and the value indicating the difference in the statistical value of the difference between adjacent values between learning data may be used. According to this configuration, a learning model can be generated appropriately and reliably.

[0082] In this case, the first machine learning may be performed so that the statistical value of the difference for each of a plurality of values becomes zero. According to this configuration, a learning model can be generated more appropriately and reliably, particularly when the noise follows a Gaussian distribution. However, the statistical value used for machine learning does not necessarily have to be the above-mentioned one. For example, any statistical value that is over a plurality of pixels and useful for noise removal may be used. Also, even when the above statistical value is used, machine learning may be performed by a method other than the one that makes the statistical value zero.

[0083] Further, as in the present embodiment, if a second machine learning is further performed in addition to the first machine learning, as described above, the generated noise removal model can be made capable of appropriate noise removal regardless of the features of data such as the outline of an image. As a result, appropriate noise removal can be enabled regardless of the features of the data.

[0084] Also, as in the present embodiment, the noise removal model which is a learning model may include a neural network having a shortcut connection. According to this configuration, a more appropriate learning model can be generated.

[0085] Also, as in the present embodiment, it is also possible to obtain the original data of the learning data and obtain each of a plurality of parts of the original data as the learning data. According to this configuration, the learning data necessary for machine learning can be obtained without preparing a large number of learning data in advance, and machine learning can be performed appropriately.

[0086] Also, as in the present embodiment, by the noise removal system 30 performing noise removal using the generated noise removal model, appropriate noise removal can thus be enabled.

[0087] In the present embodiment, the learning system 20 and the noise removal system 30 are implemented (executed) together by the computer 10. However, the learning system 20 and the noise removal system 30 may be independently implemented respectively.

[0088] Subsequently, an example of the result of noise removal according to the present embodiment is shown. FIG. 7 shows an example of noise removal when only the first machine learning is performed (when the second machine learning is not performed). The example shown in FIG. 7 is an example of a part of the original image (Lenna) shown in FIG. 2 and the like. FIG. 7(a) is an example of an image of the hair part where the signal is complex. FIG. 7(b) is an example of an image of the eye part where the signal is complex.

[0089] In FIG. 7, examples of noise removal methods other than the present embodiment (proposed method) are also shown. The other noise removal methods are Bilateral, NLMeans, and Noise2Clean (N2C). The "acquired image" in the column without noise is an image without noise. The "acquired image" in the original column is an image including the noise to be removed. The "acquired image" in the column of each noise removal method is the image after noise removal. The "difference from the correct answer" in the original column is an image of only the noise of the image including the noise to be removed (correct noise). The "difference from the correct answer" in the original column is an image of the inferred noise. The noise is Gaussian noise with σ = 20.0.

[0090] PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure) are values calculated for an image without noise. As shown in the table of FIG. 7, the method of the present embodiment performs noise removal with better accuracy. Particularly for the eye part, a high SSIM of 0.856 is obtained.

[0091] In the first machine learning, since learning is performed according to the characteristics of noise as described above, there are few extreme differences in effects depending on the position of the target. Therefore, it is less affected by the conditions depending on the position on the signal side, and when there is no object of the same shape as the required signal nearby, it has higher accuracy than other methods.

[0092] Figs. 8 and 9 show examples of noise removal for images captured by a semiconductor inspection apparatus when only the first machine learning is performed (when the second machine learning is not performed). The semiconductor inspection apparatus is a PHEMOS1000 system manufactured by Hamamatsu Photonics. The camera is an InGaAs camera (exposure time: 10 seconds, objective lens: 20 times). The Bias voltage applied is 3.0 V and the current is 8.78 mA.

[0093] Fig. 8(a) is an image including noise to be removed. Fig. 8(b) is an image after noise removal according to the present embodiment. Figs. 8(c) and 8(d) are images after noise removal by Bilateral and NLMeans, respectively. Note that the display range (range of pixel values) of each pixel is the same.

[0094] In the present embodiment, even for a signal with noise close to the tone, the noise reduction effect is high and the signal is not deleted. In Bilateral, when the tone is close to noise, the smoothing effect works strongly and weak signals disappear. In NLMeans, for an object without a nearby signal, the effect of leaving the signal is low and it is almost deleted.

[0095] FIG. 9(a) is an image including noise to be removed. FIG. 9(b) is an image after noise removal according to the present embodiment. FIG. 9(c) is a graph showing the line profile of the position of the line indicated by (1) in FIG. 9(a). The horizontal axis represents the position on the line, and the vertical axis represents the pixel value. FIG. 9(d) is a graph showing the line profile of the position of the line at the same position as in FIG. 9(c) in FIG. 9(b). FIG. 9(e) is a graph showing the line profile of the position of the line indicated by (2) in FIG. 9(a). FIG. 9(f) is a graph showing the line profile of the position of the line at the same position as in FIG. 9(e) in FIG. 9(b). This result also shows that appropriate noise removal can be performed according to the present embodiment.

[0096] Examples of noise removal for images captured by a semiconductor inspection apparatus when performing first machine learning and second machine learning are shown in FIGS. 10 to 12. The camera is an InGaAs camera (exposure time: 10 seconds, objective lens: 20 times).

[0097] FIG. 10(a) is an image including noise to be removed. FIG. 10(b) is an image obtained by taking the average of the pixel values of each pixel of a total of five images, which includes the image including noise to be removed and four images captured in the same manner as the said image. FIG. 10(c) is an image after noise removal according to the present embodiment. FIG. 10(d) is an image after noise removal by the model obtained by Noise2Noise. Note that the display range (range of pixel values) of each pixel is the same. The minimum value is black (value 503), and the maximum value is white (value 520).

[0098] In this example, for both the present embodiment and Noise2Noise, a set of 128 patches randomly cut out (24 pixels × 24 pixels) from the above-mentioned total of five images was used, and a model trained 5000 times was used.

[0099] Regarding the region of the image shown in Fig. 11(a) of the four images shown in Fig. 10, the standard deviation (σ value) and the average value of the pixel values are shown in Fig. 11(b). As shown by these values, the method (proposed method) according to the present embodiment had the effect of reducing the background noise most compared to other methods.

[0100] Figs. 12(b) to 12(e) are graphs showing the line profiles of the positions of the lines in Fig. 12(a) of the four images shown in Fig. 11. The horizontal axis represents the position on the line, and the vertical axis represents the pixel value. Figs. 12(b) to 12(e) correspond to the image of Fig. 10(a) (image including noise to be removed), the image of Fig. 10(b) (image including noise to be removed), the image of Fig. 10(c) (image after noise removal according to the present embodiment), and the image of Fig. 10(d) (image after noise removal by the model obtained by Noise2Noise), respectively. As shown by these, in the method according to the present embodiment, there is little deterioration in resolution, and only noise is removed without blurring the spread of the light-emitting points.

[0101] Fig. 13 shows the number of repetitions of machine learning and the noise removal results when performing the first machine learning and the second machine learning. Fig. 14 shows the number of repetitions of machine learning and the noise removal results when performing only the first machine learning. The upper and lower images in Figs. 13 and 14 are the images of the noise removed based on the noise removal model obtained by repeating the machine learning the corresponding number of times and the images after noise removal.

[0102] Note that the size of the cutout (patch size) is 24 pixels × 24 pixels. Also, the number of mini-batches is 128. Also, the learning time (computing speed) is about 50 seconds per 1000 images in Fig. 13 and about 1 minute per 1000 images in Fig. 14 (when using Geforce RTX2080ti).

[0103] Fig. 15 shows a graph of the number of repetitions of machine learning (number of learning times) (horizontal axis) and the evaluation value based on SSIM of the image after noise removal (vertical axis). In this graph, when the first machine learning and the second machine learning are performed (Method 2), when the noise image when creating a new patch in the second machine learning is an ideal noise image (Replacement with ideal noise), when shuffling is not performed when creating a new patch in the second machine learning (No shuffle), and when only the first machine learning is performed and the second machine learning is not performed (Method 1 only) are shown.

[0104] As shown in these, the second machine learning can remove noise more accurately by repeating the machine learning.

[0105] Subsequently, a learning program and a noise removal program for executing the processing by the above-described series of learning system 20 and noise removal system 30 will be described. As shown in Fig. 16(a), the learning program 200 is stored in a program storage area 211 formed in a computer-readable recording medium 210 that is inserted into and accessed by a computer or is provided in the computer. The recording medium 210 may be a non-transitory recording medium.

[0106] The learning program 200 includes a learning acquisition module 201 and a learning module 202. The functions realized by executing the learning acquisition module 201 and the learning module 202 are the same as the functions of the learning acquisition unit 21 and the learning unit 22 of the learning system 20 described above, respectively.

[0107] As shown in Fig. 16(b), the noise removal program 300 is stored in a program storage area 311 formed in a computer-readable recording medium 310 that is inserted into and accessed by a computer or is provided in the computer. The recording medium 310 may be a non-transitory recording medium. Note that the recording medium 310 may be the same as the recording medium 210.

[0108] The noise removal program 300 includes a acquisition module 301 for removal and a noise removal module 302. The functions realized by executing the acquisition module 301 for removal and the noise removal module 302 are the same as the functions of the acquisition unit 31 for removal and the noise removal unit 32 of the noise removal system 30 described above, respectively.

[0109] Note that part or all of the learning program 200 and the noise removal program 300 may be transmitted via a transmission medium such as a communication line, received by another device, and recorded (including installation). Further, each module of the learning program 200 and the noise removal program 300 may be installed not on one computer but on any of a plurality of computers. In that case, the series of processes described above are performed by a computer system including the plurality of computers.

Description of Reference Numerals

[0110] 10… computer, 20… learning system, 21… acquisition unit for learning, 22… learning unit, 30… noise removal system, 31… acquisition unit for removal, 32… noise removal unit, 200… learning program, 201… acquisition module for learning, 202… learning module, 210… recording medium, 211… program storage area, 300… noise removal program, 301… acquisition module for removal, 302… noise removal module, 310… recording medium, 311… program storage area.

Claims

1. A learning method for generating a learning model for removing noise from data having a plurality of values, comprising: a learning acquisition step of acquiring a plurality of learning data having a plurality of values and including noise; performing first machine learning using information based on the learning data acquired in the learning acquisition step as an input to the learning model, and synthesizing data after noise removal of the learning data based on the output from the learning model, a difference between another learning data and data after noise removal of the another learning data based on the output from the learning model, to generate new learning data, using information based on the generated new learning data as an input to the learning model, and performing second machine learning using information based on data after noise removal of the learning data or a difference of the another learning data as an output from the learning model to generate a learning model; A learning method including the above.

2. The learning method according to claim 1, wherein the data having a plurality of values is an image.

3. The learning method according to claim 1 or 2, wherein in the learning acquisition step, original data of the learning data is acquired, and each of a plurality of portions of the original data is acquired as learning data.

4. A learning system for generating a learning model for removing noise from data having a plurality of values, comprising: a learning acquisition means for acquiring a plurality of learning data having a plurality of values and including noise; a learning means for performing first machine learning using information based on the learning data acquired by the learning acquisition means as an input to the learning model, and synthesizing data after noise removal of the learning data based on the output from the learning model, a difference between another learning data and data after noise removal of the another learning data based on the output from the learning model, to generate new learning data, using information based on the generated new learning data as an input to the learning model, and performing second machine learning using information based on data after noise removal of the learning data or a difference of the another learning data as an output from the learning model to generate a learning model; A learning system comprising the above.

5. A learning program for operating a computer as a learning system for generating a learning model for removing noise from data having a plurality of values, wherein the computer Learning acquisition means for acquiring a plurality of learning data having a plurality of values and including noise Performing first machine learning using information based on the learning data acquired by the learning acquisition means as an input to a learning model, and synthesizing data after noise removal of the learning data based on the output from the learning model, the difference between another learning data and the data after noise removal of the another learning data based on the output from the learning model to generate new learning data, using information based on the generated new learning data as an input to the learning model, and performing second machine learning using information based on the data after noise removal of the learning data or the difference of the another learning data as an output from the learning model to generate a learning model, the learning means A learning program that functions as

6. A noise removal method for removing noise from data having a plurality of values, using the learning model generated by the learning method according to any one of Claims 1 to 3, comprising A removal acquisition step of acquiring noise removal target data having a plurality of values and being a noise removal target A noise removal step of inputting the noise removal target data acquired in the removal acquisition step into the learning model to remove noise from the noise removal target data A noise removal method including

7. A noise removal system for removing noise from data having a plurality of values, using the learning model generated by the learning method according to any one of Claims 1 to 3, comprising Removal acquisition means for acquiring noise removal target data having a plurality of values and being a noise removal target Noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data A noise removal system comprising

8. A noise removal program for operating a computer as a noise removal system for removing noise from data having a plurality of values, using the learning model generated by the learning method according to any one of Claims 1 to 3, comprising Operating the computer as Removal acquisition means for acquiring noise removal target data having a plurality of values and being a noise removal target Noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data A noise removal program that functions as...

Images