Learning method, learning system, learning program, noise removal method, noise removal system, and noise removal program

Abstract

To properly remove noise even if necessary data cannot be prepared in a conventional noise removal method.SOLUTION: A learning method for generating a learning model for removing noise from data having a plurality of values includes: a learning acquisition step (S01) for acquiring learning data including noise and having a plurality of values; and a learning step (S02) of performing arithmetic processing configured to input information based on the learning data acquired in the learning acquisition step to a learning model, and calculating, for each of the values, a difference between the learning data and data obtained by removing noise from the learning data based on an output from the learning model, to generate a learning model by first machine learning based on a statistic of the difference for each of the values.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a learning method, a learning system, and a learning program for generating a learning model for removing noise from data having a plurality of values, and a noise removal method, a noise removal system, and a noise removal program for removing noise from data having a plurality of values using the generated learning model.

Background Art

[0002] Conventionally, methods for removing noise from images by machine learning have been proposed. Examples of such methods include Noise2Clean and Noise2Noise (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Noise2Clean, a pair of an image without noise and an image with noise is used as teacher data. Therefore, when an image without noise cannot be prepared, noise removal using Noise2Clean cannot be performed.

[0005] In Noise2Noise, a plurality of images including different noises for the same object are used as teacher data. Therefore, if such a plurality of images cannot be prepared, noise removal using Noise2Noise cannot be performed. For example, in the case of an image of a living body, it is difficult to prepare such a plurality of images because the position of the living body cannot be fixed during multiple imaging sessions.

[0006] The present invention has been made in view of the above, and provides a learning method, a learning system, a learning program, a noise removal method, a noise removal system, and a noise removal program that enable appropriate noise removal even when the data required by conventional noise removal methods cannot be prepared.

Means for Solving the Problems

[0007] In order to achieve the above object, a learning method according to the present invention is a learning method for generating a learning model for removing noise from data having a plurality of values, the method including: a learning acquisition step of acquiring learning data having a plurality of values and including noise; and a learning step of performing an operation using information based on the learning data acquired in the learning acquisition step as an input to the learning model, calculating a difference between the learning data and the data after noise removal of the learning data based on the output from the learning model for each of the plurality of values, and performing first machine learning based on the statistical value of the differences for each of the plurality of values to generate the learning model.

[0008] In the learning method according to the present invention, the learning model is generated based on the statistical value of the differences according to the characteristics of the noise. Therefore, according to the learning method of the present invention, even when the data required by conventional noise removal methods such as an image without noise and a plurality of images including different noises for the same object cannot be prepared, appropriate noise removal can be enabled.

[0009] Data having a plurality of values may be an image. According to this configuration, a learning model for removing noise from an image including noise can be generated.

[0010] In the learning step, as a statistical value of the difference for each of the plurality of values, any one of the average value for each learning data, a value indicating the difference in distribution between learning data, a value indicating the difference in correlation between learning data, and a value indicating the difference between learning data of the statistical value of the difference between adjacent values may be used. According to this configuration, an appropriate and reliable learning model can be generated.

[0011] In this case, in the learning step, first machine learning may be performed so that the statistical value of the difference for each of the plurality of values becomes 0. According to this configuration, an appropriate and reliable learning model can be generated particularly when the noise follows a Gaussian distribution.

[0012] In the learning acquisition step, a plurality of learning data are acquired, and in the learning step, the difference between the data after noise removal of the learning data based on the output from the learning model and another learning data is combined to generate new learning data, and information based on the generated new learning data is used as an input to the learning model, and second machine learning is performed using information based on the difference between the data after noise removal of the learning data or the other learning data as an output from the learning model to generate a learning model. According to this configuration, new learning data in which the difference between the data after noise removal of the learning data and another learning data corresponding to the removed noise is combined is used for machine learning. As a result, appropriate noise removal can be enabled regardless of the characteristics of the data.

[0013] In the learning acquisition step, learning noise data indicating noise is also acquired. In the learning step, data is input to a feature extraction model that outputs information indicating the feature amount of the data, and information based on the difference and information based on the learning noise data acquired in the learning acquisition step are respectively input, and the third machine learning is performed by comparing the information obtained as the output, and the feature extraction model and the learning model may be generated. According to this configuration, when noise data corresponding to the noise included in the data is obtained, the learning model can be generated more appropriately and reliably.

[0014] In the learning step, it is also possible to generate a learning model including a neural network having a shortcut connection. According to this configuration, a more appropriate learning model can be generated.

[0015] Also, in the learning acquisition step, it is possible to acquire the original data of the learning data and acquire each of a plurality of parts of the original data as the learning data. According to this configuration, it is possible to acquire the learning data necessary for machine learning without preparing a large number of learning data in advance, and to appropriately perform machine learning.

[0016] By the way, the present invention can be described as an invention of a learning method as described above, and can also be described as an invention of a learning system and a learning program as follows. These are only different in category, but are substantially the same invention and exhibit the same operations and effects.

[0017] That is, the learning system according to the present invention is a learning system that generates a learning model for removing noise from data having a plurality of values, and includes a learning acquisition means for acquiring learning data having a plurality of values and including noise, and performing an operation using information based on the learning data acquired by the learning acquisition means as an input to the learning model, calculating the difference between the learning data and the data after noise removal of the learning data based on the output from the learning model for each of the plurality of values, and performing first machine learning based on the statistical value of the difference for each of the plurality of values to generate a learning model.

[0018] Further, the learning program according to the present invention is a learning program that causes a computer to operate as a learning system for generating a learning model for removing noise from data having a plurality of values, and causes the computer to function as a learning acquisition means for acquiring learning data having a plurality of values and including noise, and performing an operation using information based on the learning data acquired by the learning acquisition means as an input to the learning model, calculating the difference between the learning data and the data after noise removal of the learning data based on the output from the learning model for each of the plurality of values, and performing first machine learning based on the statistical value of the difference for each of the plurality of values to generate a learning model.

[0019] The noise removal method according to the present invention is a noise removal method for removing noise from data having a plurality of values using the learning model generated by the above learning method, and includes a removal acquisition step of acquiring noise removal target data having a plurality of values and being a noise removal target, and a noise removal step of inputting the noise removal target data acquired in the removal acquisition step into the learning model to remove noise from the noise removal target data.

[0020] In the noise removal method according to the present invention, noise is removed from data using the learning model generated by the learning method according to the present invention. Therefore, appropriate noise removal can be enabled.

[0021] Incidentally, the present invention can be described not only as an invention of a noise removal method as described above, but also as an invention of a noise removal system and a noise removal program as follows. These are substantially the same invention except for different categories, and exhibit the same operations and effects.

[0022] That is, the noise removal system according to the present invention is a noise removal system that removes noise from data having a plurality of values using the learning model generated by the above learning method, and has a plurality of values and is a noise removal target. A removal acquisition means for acquiring noise removal target data, and a noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data.

[0023] Further, the noise removal program according to the present invention is a noise removal program that operates a computer as a noise removal system that removes noise from data having a plurality of values using the learning model generated by the above learning method, and the computer has a plurality of values and is a noise removal target. It functions as a removal acquisition means for acquiring noise removal target data, and a noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data.

Advantages of the Invention

[0024] According to the present invention, even when data required for conventional noise removal methods such as an image without noise and a plurality of images including different noises for the same object cannot be prepared, appropriate noise removal can be enabled.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

[0026] Hereinafter, embodiments of a learning method, a learning system, a learning program, a noise removal method, a noise removal system, and a noise removal program according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted.

[0027] FIG. 1 shows a learning system 20 and a noise removal system 30 according to the present embodiment. The learning system 20 and the noise removal system 30 are realized by a computer 10. The learning system 20 is a system (apparatus) that generates a noise removal model (denoising model), which is a learning model (trained model) for removing noise from data having a plurality of values. The noise removal system 30 is a system (apparatus) that removes noise from data having a plurality of values using the noise removal model generated by the learning system 20.

[0028] In the present embodiment, the data having a plurality of values to be the noise removal target is an image. However, the data having a plurality of values to be the noise removal target does not necessarily have to be an image, and may be, for example, waveform data or the like.

[0029] The computer 10 is a conventional computer including hardware such as a processor such as a CPU (Central Processing Unit), a memory, and a communication module. Further, the computer 10 may be a computer system including a plurality of computers. Further, the computer 10 may be configured by cloud computing or edge computing. Each function of the computer 10 described below is exhibited by these components operating according to a program or the like.

[0030] Subsequently, the functions of the learning system 20 and the noise removal system 30 according to the present embodiment will be described. As shown in FIG. 1, the learning system 20 includes a learning acquisition unit 21 and a learning unit 22.

[0031] Before explaining each function of the learning system 20, a noise removal model generated by the learning system 20 will be explained. In the present embodiment, the noise removal model is a model that inputs information based on an image and outputs (infers) an image obtained by removing noise included in the input image from the input image. Note that the noise removal model does not necessarily have to be a model that outputs an image after noise removal, and may be a model that outputs information capable of generating an image after noise removal. For example, the noise removal model may be a model that outputs noise included in the input image.

[0032] The noise removal model includes, for example, a neural network. The neural network may be a multi-layer one. That is, the noise removal model may be generated by deep learning. Further, the neural network may be a convolutional neural network (CNN). Further, as described later, the neural network may have a specific structure.

[0033] The noise removal model is provided with neurons for inputting image-based information to the input layer. For example, the information input to the noise removal model is the pixel value (luminance value) of each pixel of the image. In this case, the input layer is provided with neurons equal to the number of pixels of the image, and the pixel value of the corresponding pixel is input to each neuron. Note that the information input to the noise removal model may be other than the pixel value of each pixel as long as it is based on the image.

[0034] The noise removal model is provided with neurons for outputting the image with noise removed to the output layer. For example, the information output from the noise removal model is the pixel value of each pixel of the image after noise removal. In this case, the output layer is provided with neurons equal to the number of pixels of the image, and the pixel value of the corresponding pixel is output from each neuron.

[0035] Note that the noise removal model may be constituted by means other than a neural network.

[0036] The noise removal model is assumed to be used as a program module that is part of artificial intelligence software. The noise removal model, for example, is used in a computer equipped with a CPU and a memory, and the CPU of the computer operates according to instructions from the model stored in the memory. For example, the CPU of the computer operates to input information to the model according to the instruction, perform calculations according to the model, and output a result from the model. Specifically, the CPU of the computer operates to input information to the input layer of the neural network according to the instruction, perform calculations based on parameters such as the learning weight coefficients in the neural network, and output a result from the output layer of the neural network. The above is the noise removal model generated by the learning system 20.

[0037] The learning acquisition unit 21 is a learning acquisition means for acquiring learning data having a plurality of values and including noise. The data having a plurality of values may be an image. The learning acquisition unit 21 may acquire a plurality of pieces of learning data. The learning acquisition unit 21 may acquire the original data of the learning data and acquire each of a plurality of parts of the original data as learning data.

[0038] The learning acquisition unit 21 acquires learning data as follows, for example. The learning acquisition unit 21 acquires an image including noise as the original image (original data) of the learning data. The acquisition of the original image is performed, for example, by receiving from an imaging device that has imaged the original image or by accepting an input operation of the original image of the user to the computer 10.

[0039] As shown in FIG. 2, the learning acquisition unit 21 cuts out and acquires, as learning data, an image of a region (part) having a preset size that is smaller than the original image from the original image. The image of the region is called a patch. The learning acquisition unit 21 acquires, for example, a plurality of patches at different random positions from one original image.

[0040] Note that the learning acquisition unit 21 may acquire learning data by a method other than cutting from the above-described original image. The learning acquisition unit 21 acquires a number of pieces of learning data sufficient to appropriately perform machine learning described later. The learning acquisition unit 21 outputs the acquired learning data to the learning unit 22.

[0041] In the present embodiment, the noise removed by the noise removal model is noise having specific characteristics. Therefore, the learning data includes noise having the specific characteristics. For example, a patch acquired from an image captured under the same conditions (for example, an imaging device and an imaging environment) as the conditions under which an image to be denoised is captured is used as the learning data.

[0042] The learning unit 22 performs an operation with the information based on the learning data acquired by the learning acquisition unit 21 as the input to the learning model, calculates the difference between the learning data and the data after noise removal of the learning data based on the output from the learning model for each of a plurality of values, and performs first machine learning based on the statistical value of the difference for each of the plurality of values to generate a learning model. The learning unit 22 may use, as the statistical value of the difference for each of the plurality of values, any one of the average value for each learning data, the value indicating the difference in distribution between learning data, the value indicating the difference in correlation between learning data, and the value indicating the difference between learning data of the statistical value of the difference between adjacent values. The learning unit 22 may perform first machine learning so that the statistical value of the difference for each of the plurality of values becomes 0.

[0043] FIG. 2 schematically shows the first machine learning. The learning unit 22 performs the first machine learning as follows. The learning unit 22 inputs a patch, which is learning data, from the learning acquisition unit 21. The learning unit 22 performs an operation with each patch as the input to the noise removal model at that time to obtain an output from the noise removal model, for example, an image after noise removal (inferred signal image). The learning unit 22 calculates the noise image inferred at that time, which is the difference between the patch and the image after noise removal. The above difference is the value for each pixel, that is, the value for each of the plurality of values of the image. For example, the learning unit 22 calculates the difference for each pixel between the patch (input image) input to the noise removal model and the image after noise removal (inference result) to obtain the noise image inferred at that time. Note that the output from the noise removal model before or during machine learning is not necessarily related to accurate noise.

[0044] In the first machine learning, a noise removal model is generated without requiring a noise-free patch and an image of the noise itself as teacher images for patches containing noise. Therefore, even when these teacher images cannot be prepared, a noise removal model can be generated. In the first machine learning, assuming that the "inference result" is a "noise-removed image", it utilizes the fact that the signal to be removed can be assumed to be "noise" and focuses on the "signal to be removed" rather than the inference result. Many methods such as Noise2Clean and Noise2Noise adopt a method of bringing the "image where noise is removed" closer to the expected value, while the first machine learning has the opposite idea.

[0045] In the first machine learning, statistical values for each group (region) of a plurality of pixels are targeted for removing noise having specific characteristics. For example, the noise to be removed is Gaussian noise where the pixel values of the noise follow a Gaussian distribution at any position in the image. The learning unit 22 performs machine learning based on the statistical values of the obtained noise image and the conditions related to the noise included in the image. The learning unit 22 performs machine learning so that the statistical values of the noise image obtained based on the output from the noise removal model match the above conditions.

[0046] When the noise to be removed is Gaussian noise, it is considered to have the following characteristics. (1) The "average value" is the same for any region (patch) position (in the case of Gaussian noise, the average value is "0"). (2) The "spread of the distribution (dispersion)" is the same for any region (patch) position (in the case of Gaussian noise, there is no difference in the standard deviation (σ value) between patches). (3) The "correlation value" between regions is the same for any region (patch) position (it is "uncorrelated" between any regions). (4) The "total magnitude of the difference from adjacent pixels" is the same for any region (patch) position (the noise "content" is the same).

[0047] The above-described conditions regarding the noise in machine learning by the learning unit 22 are to ensure that the statistical values of the noise images obtained using the noise removal model satisfy the above characteristics. The learning unit 22 obtains noise images for each of a plurality of patches using the noise removal model. The learning unit 22 performs machine learning on the noise removal model using the following loss evaluation function (Loss function) from the obtained plurality of noise images.

[0048] [Number] x is a patch input to the noise removal model for which machine learning is to be performed. f(x) is the image after noise removal, which is the output of the noise removal model. z is a noise image. N is the number of noise images. z i mn is the pixel value of the noise image i of the patch whose position in the original image is mn. mean(z i mn ) is the average value of z i mn std(z i mn ) is the standard deviation of z i mn .

[0049] sim(z i mn ) is a value based on the correlation value of z i mn with other noise images. For example, sim(z i mn ) is obtained by calculating the correlation values of combinations with all other noise images among the plurality of noise images obtained using the noise removal model and taking their sum. The correlation value of the combination with other noise images is calculated, for example, by converting the two-dimensional pixel values of each noise image into one-dimensional vectors and calculating their inner product. Or, the inner product of the matrices of two-dimensional pixel values may be calculated. Also, normalization of the vector or matrix may be performed during the calculation. sub(z i mn ) is z i mnIt is the sum of the absolute values of the differences between each pixel and the pixels adjacent to it in a preset direction for all pixels.

[0050] The first term of the loss evaluation function is the average value of the noise images of each patch (i.e., the average value for each learning data), which corresponds to the above characteristic (1). The second term of the loss evaluation function is the value obtained by subtracting the average of the standard deviations of the noise images of all patches from the standard deviation of the noise image of each patch (i.e., the value indicating the difference in distribution between learning data), which corresponds to the above characteristic (2). The third term of the loss evaluation function is the value obtained by subtracting the average of the values based on the correlation values of the noise images of all patches from the value based on the correlation value of the noise image of each patch (i.e., the value indicating the difference in correlation between learning data), which corresponds to the above characteristic (3). The fourth term of the loss evaluation function is the value obtained by subtracting the average of the values based on the differences between adjacent pixels of all patches from the value based on the differences between adjacent pixels of each patch (i.e., the value indicating the difference in the statistical value of the differences between adjacent values between learning data), which corresponds to the above characteristic (4).

[0051] α, β, and γ are coefficients indicating the weights of the respective terms of the loss evaluation function, and are preset values (hyperparameters). By increasing the values, the characteristics corresponding to each term of the noise can be considered more greatly during machine learning. α, β, and γ are set to values of 0 or more (in the case of 0, the characteristics corresponding to that term are not considered).

[0052] Machine learning itself based on the loss evaluation function, that is, the update of the parameters of the noise removal model by error backpropagation, may be performed in the same manner as the conventional machine learning method. Since machine learning based on the above loss evaluation function is performed, machine learning is performed so that the statistical values of the noise images of each patch shown in each term of the loss evaluation function become 0. That is, the learning unit 22 performs noise narrowing down according to the noise conditions.

[0053] Among the items included in the loss evaluation function, the only one that uses only the value of the target patch without using the values of other patches is the average value (the above characteristic (1)). Other items are relative values between the value of the target patch and the values of other patches. Also, as values related to the spread of the distribution, variance, skewness, kurtosis, etc. may be used instead of, or in addition to, the standard deviation.

[0054] In the first machine learning, as the noise removal model, it may be possible to use one including a neural network having a shortcut connection (residual path). By this, it becomes possible to perform noise removal without breakdown even for targets with different luminance ranges from those during machine learning. This type is advantageous for simple noise removal. However, when emphasizing visibility over signal reproducibility (involving large conversions), it may be better not to have a shortcut connection. Also, not using batch normalization results in faster learning and stable results.

[0055] As described above, in the first machine learning, machine learning is performed so that the statistical value of the noise image shown in each term of the loss evaluation function becomes zero. Also, it is possible to perform machine learning to set the noise to zero from an image including noise. Also, an image without noise, etc. is unnecessary (zero) for machine learning. Also, the noise removal model may be one that can be learned by ordinary deep learning, and there are no restrictions inside the model (zero). Also, although Noise2Clean and Noise2Noise are pre-training methods, the first machine learning does not require pre-training. Also, the first machine learning can re-learn the noise removal model from a noise image. Also, the learning time in this embodiment is smaller than the conventional method.

[0056] Note that the noise to be removed in this embodiment does not necessarily have to be Gaussian noise. The noise to be removed in this embodiment may be one in which the statistical value of the noise for each of a plurality of values (for example, pixels) follows a certain condition.

[0057] Also, the loss evaluation function does not necessarily have to be the above, and it may be any function that follows the above-described concept. For example, in the above loss evaluation function, regarding the standard deviation, based on the identification of noise that the "spread of the distribution" is the same in any region (patch) position, even if the value of the standard deviation of the noise is unknown, it can be used for the training of the noise removal model.

[0058] However, if the value of the standard deviation of the noise is known in advance, machine learning may be performed so that the value of the standard deviation of the noise image based on the output from the noise removal model becomes that value. For example, if it is known that the standard deviation of the noise to be removed is σ = 20, machine learning may be performed so that the value of the standard deviation of the pixel values of the noise image based on the output from the noise removal model becomes 20. Also, if it is known that the average value of the noise image is a specific value other than 0, machine learning may be performed so that the value of the average value of the pixel values of the noise image based on the output from the noise removal model becomes the specific value. Specifying the value of a statistical value (for example, standard deviation, average value) obtained without using other noise images is important for converging an infinite number of combinations of inference results.

[0059] Also, when the noise to be removed is not Gaussian noise, the loss evaluation function may be adjusted according to the characteristics of the noise. For example, when the noise to be removed is not Gaussian noise, the variation in the correlation value is not used. However, even if the noise to be removed is not an ideal one such as Gaussian noise, since the noise has the above statistical quantities, depending on the statistical quantity, it can be used in the loss evaluation function. For example, the skewness is 0 for Gaussian noise, but it will not be 0 for other noises due to distortion. However, since the patch is a region composed of a plurality of pixels rather than a single pixel, the distribution is the same. Therefore, the relative value between the value of the target patch related to the skewness and the value of other patches can be included and used in the loss evaluation function.

[0060] In the case of non-random noise, the correlation value between patches depends on how the patches are selected. Therefore, the correlation value is not constant, and the variation value related to the correlation value (for example, the value of the third term of the loss evaluation function) does not become zero. In that case, the variation value related to the correlation value is not included in the loss evaluation function.

[0061] Also, in the case of non-random noise, the noise may be unevenly distributed with respect to the patches, such as having periodicity in the noise. In that case, the variation value of the spread of the distribution (for example, the value of the second term of the loss evaluation function) is not included in the loss evaluation function.

[0062] The above is the first machine learning. The learning unit 22 may generate a noise removal model only by the first machine learning. Also, in addition to the first machine learning, the learning unit 22 may perform further different machine learning to generate a noise removal model. For example, the learning unit 22 may perform the following different machine learning (at least any one of the second machine learning and the third machine learning) to generate a noise removal model.

[0063] The learning unit 22 may synthesize the difference between the data after noise removal of the learning data based on the output from the learning model and another learning data to generate new learning data, use the information based on the generated new learning data as the input to the learning model, and perform the second machine learning with the information based on the data after noise removal of the learning data or the difference of the other learning data as the output from the learning model to generate a learning model.

[0064] Fig. 3 schematically shows the second machine learning. The learning unit 22 performs the second machine learning as follows. The learning unit 22 performs an operation with each patch as an input to the noise removal model at that time to obtain an output from the noise removal model, for example, an image after noise removal (inferred signal image). The learning unit 22 calculates a noise image inferred at that time, which is the difference between the patch and the image after noise removal. The processing up to the calculation of the noise image is the same as the processing up to the calculation of the noise image in the first machine learning. The processing up to the calculation of the noise image may be common between the first machine learning and the second machine learning.

[0065] The noise removed by the noise removal model generated only by the first machine learning often differs between near the unevenness (near the contour) of the image and near the flat surface, like the noise image shown in Fig. 3. As described above, in this embodiment, the noise to be removed is such that the statistical value of the noise such as Gaussian noise follows a certain condition. Therefore, the noise corresponding to the contour of the image as described above is considered to be inappropriate as the noise to be removed because it contains an illegal pattern. The second machine learning enables the generated noise removal model to perform appropriate noise removal regardless of the characteristics of data such as the contour of the image.

[0066] The learning unit 22 combines the image after noise removal of a patch and the noise image of another patch to generate a new patch (new learning data). For example, the learning unit 22 randomly shuffles the combination (pair) of the image after noise removal and the noise image, and synthesizes the image after noise removal and the noise image of different combinations. The synthesis of the image after noise removal and the noise image is performed by adding the pixel values of the pixels at the same position. The new patch is one in which the noise containing the illegal pattern that is the signal of the contour part is attached to the image after noise removal of another patch. Note that the generation of different combinations may be performed other than random shuffling.

[0067] The learning unit 22 performs machine learning by using the newly generated patch as the input to the noise removal model and the image after noise removal used for generating the new patch as the output from the noise removal model. The machine learning is performed such that when the newly generated patch is input to the noise removal model, the output from the noise removal model becomes the image after noise removal used for generating the new patch, that is, the error between the output from the noise removal model and the image after noise removal used for generating the new patch is minimized. Thus, in the second machine learning, Noise2Clean is performed using the newly generated patch obtained by synthesis and the image after noise removal. The machine learning itself, that is, the update of the parameters of the noise removal model by error backpropagation may be performed in the same manner as the conventional machine learning method.

[0068] The above machine learning is an example in the case where an image after noise removal is output from the noise removal model. When noise (noise image) is output from the noise removal model, the learning unit 22 may perform machine learning using the noise image used for generating the new patch as the output from the noise removal model.

[0069] The learning unit 22 may perform a third machine learning to generate a learning model. In the third machine learning, a feature extraction model is generated by machine learning and used for generating the learning model. The feature extraction model is a feature extraction model that inputs data and outputs information indicating the feature amount of the data.

[0070] In the third machine learning, the acquisition unit 21 for learning also acquires learning noise data indicating noise. The learning noise data serves as the teacher for the noise to be removed by the noise removal model, as described below. The acquisition unit 21 for learning acquires learning noise data as follows, for example. An image containing only noise (noise image) is acquired as the original image (original data) of the learning noise data. An image containing only noise is acquired, for example, by imaging a white surface with an imaging device. Alternatively, assuming the noise is Gaussian noise, an image containing only noise may be artificially generated by simulation or the like by giving parameters such as the average value and standard deviation of the pixel values that become noise. The acquisition of the original image is performed, for example, by receiving it from the imaging device that captured the original image, or by accepting the input operation of the original image by the user to the computer 10.

[0071] As shown in FIG. 4, the acquisition unit 21 for learning cuts out and acquires, as learning noise data, an image of a region (portion) having the same size as the patch, which is learning data and smaller than the image, from the image containing only noise. The acquisition unit 21 for learning acquires, for example, a plurality of learning noise data at different random positions from one original image. Note that the learning noise data may be acquired by methods other than the above. For example, an image of a region at a position where nothing is shown (i.e., for example, a position where a white surface is imaged) may be cut out and acquired as learning noise data from the original image of the learning data. Even when an image without noise cannot be easily acquired, a noise image containing only noise as described above can be easily obtained.

[0072] The noise included in the learning noise data also has the characteristics of the noise to be removed by the noise removal model, similar to the noise included in the learning data. The acquisition unit 21 for learning outputs the acquired learning noise data to the learning unit 22.

[0073] The learning unit 22 performs third machine learning by inputting information based on differences and information based on the learning noise data acquired by the learning acquisition unit 21 into the feature extraction model, and comparing the respective information obtained as outputs, thereby generating the feature extraction model and the noise removal model.

[0074] FIG. 4 schematically shows the third machine learning. In the third machine learning, in addition to the noise removal model, a feature extraction model is generated. The feature extraction model inputs, for example, an image and outputs information indicating the feature amount of the image. The feature amount of the image is, for example, a vector of a preset dimension. Note that the feature extraction model is used only for generating the noise removal model and is not used for noise removal.

[0075] The learning unit 22 performs third machine learning as follows. The learning unit 22 inputs learning noise data from the learning acquisition unit 21. The learning unit 22 inputs pairs of a noise image based on the output from the noise removal model and a noise image that is the learning noise data into the feature extraction model respectively, to obtain the feature amounts of the respective noise images. The noise image that is the learning noise data and is paired with the noise image based on the output from the noise removal model may be randomly selected, for example, from those input from the learning unit 22. The learning unit 22 compares these feature amounts and updates the learning of the feature extraction model and the noise removal model, that is, updates the parameters of each model, so that there is no difference in these feature amounts.

[0076] For example, the feature extraction model and the third machine learning are based on the conventional method SimSiam. The advantage of SimSiam is that it does not require negative sample pairs and maximizes the similarity only with positive sample pairs. In this embodiment, since learning is performed only with positive pairs having the features of the noise image as common features, this method is used as the basis. The feature extraction model is composed of a base model that converts a two-dimensional input image into a one-dimensional feature amount (channel) and two fully connected layers that are projection nets.

[0077] Also, as shown in FIG. 4, when a noise image based on the output from the noise removal model is input to the feature extraction model, only one-sided error backpropagation is stopped. Also, when a noise image, which is learning noise data, is input to the feature extraction model, a predictor is added only on one side. These are elements for preventing the collapse of learning.

[0078] The differences between the third machine learning of this embodiment and SimSiam are as follows. The first point is that in this embodiment, the pair of images input to the feature extraction model are not obtained by different transformations (augmentation techniques), but are different from each other. As described above, one is a noise image based on the output from the noise removal model, and the other is a noise image that is learning noise data. In the third machine learning of this embodiment, these two different types of images are used.

[0079] The second point is that the connection (input) side where error backpropagation is stopped is fixed according to the type of input image. In SimSiam, it is common to multiply 0.5 by the two Loss values obtained as a result of inversely inputting two input images and use the total of 1.0. On the other hand, in the third machine learning of this embodiment, as described above, the connection (input) side where error backpropagation is stopped is fixed, and the side where the noise image based on the output from the noise removal model is input to the feature extraction model is used. Also, as described above, the side where the predictor layer is provided after the feature extraction model is the side where the noise image, which is learning noise data, is input to the feature extraction model.

[0080] The second point is that the feature extraction model is used in the same way as the discriminator of a Generative Adversarial Network (GAN). In GAN, the fake image created by the generator and a separately prepared correct new image are each input to the discriminator, and the learning of the discriminator and the generator is performed simultaneously so that the discriminator can distinguish between true and false, and the generator can generate a fake image that is mistaken for a true image by the discriminator.

[0081] When compared with the relationship of the GAN, the noise removal model of the present embodiment corresponds to the generator, and the feature extraction model corresponds to the discriminator. The learning unit 22 performs error backpropagation to the noise removal model in a direction such that the feature amount of the noise image based on the output from the noise removal model is similar to the feature amount of the noise image that is the learning noise data by a preset evaluation loss function (Loss function 2 shown in FIG. 4), and updates the two models. Note that the Loss function 1 shown in FIG. 4 is used in the first machine learning.

[0082] The third evaluation loss function is preset so that the above learning is performed. For example, as the evaluation loss function, an inner product, a cosine similarity, or the like that calculates the similarity of the feature amounts of two noise images is used.

[0083] The difference from the GAN is that in the present embodiment, the noise removal model is not appropriately learned only by the third machine learning. That is, the noise image based on the output from the noise removal model does not approach an appropriate noise image only by the third machine learning, and the target learning cannot be performed. It is a premise of the third machine learning that the noise image based on the output from the noise removal model is brought close to an appropriate noise image by the first machine learning. It is considered that the feature extraction model supplements the information on the features of the noise that is insufficient in the learning by the first machine learning, which is another method.

[0084] The learning unit 22 generates a noise removal model by repeating the machine learning process until the generation of the noise removal model converges based on preset conditions, or for a preset number of times, in the same manner as conventional machine learning. The first machine learning and the second machine learning or the third machine learning may be performed together for each repetition. Alternatively, after the first machine learning is repeatedly performed, the second machine learning or the third machine learning may be repeatedly performed. Also, both the second machine learning and the third machine learning may be performed. The learning unit 22 outputs the generated noise removal model to the noise removal system 30.

[0085] Also, machine learning may be performed, for example, by the gradient descent method using the entire obtained patch as a sample, or the stochastic gradient descent method performed using some samples. The number of mini-batches, which are the samples used in the stochastic gradient descent method, is usually 64 or the like. Or, for the stability of learning, the number of mini-batches may be 128. The above is the function of the learning system 20 according to the present embodiment.

[0086] Subsequently, the function of the noise removal system 30 according to the present embodiment will be described. As shown in FIG. 1, the noise removal system 30 includes a removal acquisition unit 31 and a noise removal unit 32.

[0087] The removal acquisition unit 31 is a removal acquisition means for acquiring noise removal target data that has a plurality of values and is a target for noise removal. The removal acquisition unit 31 acquires an image including noise as the noise removal target data. The acquisition of the image is performed, for example, by receiving the image from an imaging device that has captured the image, or by accepting an input operation of the image by the user to the computer 10. The removal acquisition unit 31 outputs the acquired noise removal target data to the noise removal unit 32.

[0088] The noise removal unit 32 is a noise removal means for inputting the noise removal target data acquired by the removal acquisition unit 31 into a learning model and removing noise from the noise removal target data. The noise removal unit 32 inputs and stores a noise removal model, which is a learning model generated by the learning system 20, and uses it for noise removal.

[0089] FIG. 5 schematically shows the removal of noise by the noise removal unit 32. The noise removal unit 32 removes noise from the image to be noise-removed as follows. The noise removal unit 32 divides the image to be noise-removed into patches by tiling. The size of this patch may be different from that of the patch that is the learning data.

[0090] The noise removal unit 32 inputs the divided patches into the noise removal model, and obtains patches with noise removed (inferred "signal images"), which are the outputs from the noise removal model. Note that the generated noise removal model may be configured to input an image of a size different from the patches used during machine learning and output the image with noise removed. This can be achieved by conventional methods. The noise removal unit 32 combines the patches with noise removed to generate an image with noise removed (inference result image (signal)) from the image to be noise removed.

[0091] If there is sufficient hardware resource for computer processing, the noise removal unit 32 may input the entire image into the noise removal model without dividing it into patches, and obtain the image with noise removed.

[0092] The noise removal unit 32 outputs the generated image after noise removal. The output of the image after noise removal may be performed in the same manner as conventional methods according to the purpose of using the image. The above is the function of the noise removal system 30 according to the present embodiment.

[0093] Subsequently, the processing executed by the computer 10 according to the present embodiment (the operation method performed by the computer 10) will be described using the flowcharts of FIGS. 6 and 7. First, the processing executed when generating the noise removal model, that is, the learning method which is the processing executed by the learning system 20 according to the present embodiment, will be described using the flowchart of FIG. 6.

[0094] In this processing, an image (for example, the above-described patches) which is learning data is acquired by the learning acquisition unit 21 (S01, learning acquisition step). Subsequently, based on the learning data, first machine learning is performed by the learning unit 22 to generate a noise removal model which is a learning model (S02, learning step).

[0095] In the first machine learning, an operation is performed in which information based on learning data is used as an input to a noise removal model, and a difference (noise image) between the learning data and the data after noise removal of the learning data based on the output from the noise removal model is calculated for each of a plurality of values (pixels), and is performed based on the statistical value of the difference for each of the plurality of values. Further, when generating the noise removal model, at least one of the second machine learning and the third machine learning may be performed together with the first machine learning.

[0096] The generated noise removal model is output from the learning system 20 to the noise removal system 30 (S03). In the noise removal system 30, the noise removal model is stored and used in the following processing. The above is the learning method which is the processing executed by the learning system 20 according to the present embodiment.

[0097] Subsequently, with reference to the flowchart of FIG. 7, the processing executed when removing noise from data, that is, the noise removal method which is the processing executed by the noise removal system 30 according to the present embodiment will be described.

[0098] In this processing, an image which is data to be subjected to noise removal is acquired by the acquisition unit 31 for removal (S11, acquisition step for removal). Subsequently, the data to be subjected to noise removal is input to the noise removal model by the noise removal unit 32, and noise is removed from the data to be subjected to noise removal (S12, noise removal step). The image after noise removal which is the result of noise removal is output from the noise removal unit 32 to a predetermined output destination (S13). The above is the noise removal method which is the processing executed by the noise removal system 30 according to the present embodiment.

[0099] In the present embodiment, a learning model is generated by the first machine learning based on the statistical value of the difference according to the characteristics of the noise. Therefore, according to the present embodiment, even when data necessary for conventional noise removal methods such as an image not including noise and a plurality of images including different noises for the same object cannot be prepared, appropriate noise removal can be enabled.

[0100] Also, as in this embodiment, the data having a plurality of values to be denoised and the data having a plurality of values used for machine learning may be images. According to this configuration, a learning model for removing noise from an image including noise can be generated. However, as described above, the data having a plurality of values to be denoised and the data having a plurality of values used for machine learning do not necessarily have to be images, and may be, for example, waveform data or the like.

[0101] Also, as in this embodiment, as the statistical value of the difference for each of the plurality of values used for machine learning, any one of the average value for each learning data, the value indicating the difference in distribution between learning data, the value indicating the difference in correlation between learning data, and the value indicating the difference between learning data of the statistical value of the difference between adjacent values may be used. According to this configuration, an appropriate and reliable learning model can be generated.

[0102] In this case, the first machine learning may be performed so that the statistical value of the difference for each of the plurality of values becomes 0. According to this configuration, especially when the noise follows a Gaussian distribution, an appropriate and reliable learning model can be generated. However, as the statistical value used for machine learning, it is not necessarily the above-mentioned one, and for example, any statistical value that is over a plurality of pixels and useful for noise removal may be used. Also, when the above-mentioned statistical value is used, machine learning may be performed by a method other than the method that makes the statistical value become 0.

[0103] Also, as in this embodiment, a second machine learning may be further performed in addition to the first machine learning. According to this configuration, as described above, the generated noise removal model can be made capable of appropriate noise removal regardless of the characteristics of data such as the outline of an image. Thereby, appropriate noise removal can be enabled regardless of the characteristics of the data.

[0104] Also, as in this embodiment, in addition to the first machine learning, further third machine learning may be performed. According to this configuration, when learning noise data, which is noise data corresponding to the noise included in the data, is obtained, the learning model can be generated more appropriately and reliably. Note that if only the third machine learning is performed without performing the first machine learning, the noise removal model will not be able to appropriately remove noise. The third machine learning becomes effective only when used in combination with the first machine learning.

[0105] However, the first machine learning may be performed without being combined with the second machine learning and the third machine learning. Also, in addition to the first machine learning, both the second machine learning and the third machine learning may be performed.

[0106] Also, as in this embodiment, the noise removal model, which is a learning model, may include a neural network having a shortcut connection. According to this configuration, a more appropriate learning model can be generated.

[0107] Also, as in this embodiment, it is also possible to obtain the original data of the learning data and obtain each of a plurality of parts of the original data as learning data. According to this configuration, it is possible to obtain the learning data required for machine learning without preparing a large number of learning data in advance, and to appropriately perform machine learning.

[0108] Also, as in this embodiment, by using the generated noise removal model, the noise removal system 30 performs noise removal, and thus appropriate noise removal can be enabled.

[0109] In this embodiment, the learning system 20 and the noise removal system 30 are implemented (performed) together by the computer 10, but the learning system 20 and the noise removal system 30 may be implemented independently.

[0110] Next, an example of the result of noise removal according to this embodiment is shown. FIG. 8 shows an example of noise removal when only the first machine learning is performed (when the second and third machine learnings are not performed). The example shown in FIG. 8 is an example of a part of the original image (Lenna) shown in FIG. 2 and the like. FIG. 8(a) is an example of an image of a hair part where the signal is complex. FIG. 8(b) is an example of an image of an eye part where the signal is complex.

[0111] In FIG. 8, examples of noise removal methods other than this embodiment (proposed method) are also shown. The other noise removal methods are Bilateral, NLMeans, and Noise2Clean (N2C). The "acquired image" in the column without noise is an image without noise. The "acquired image" in the original column is an image including noise to be removed. The "acquired image" in the column of each noise removal method is an image after noise removal. The "difference from the correct answer" in the original column is an image of only the noise of the image including noise to be removed (correct noise). The "difference from the correct answer" in the original column is an image of the removed (inferred) noise. The noise is Gaussian noise with σ = 20.0.

[0112] PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure) are values calculated for an image without noise. As shown in the table of FIG. 8, the method of this embodiment performs noise removal with better accuracy. In particular, for the eye part, a high SSIM of 0.856 is obtained.

[0113] In the first machine learning, since learning is performed according to the characteristics of the noise as described above, there is little difference in the extreme effects depending on the position of the target. Therefore, it is less affected by the conditions depending on the position on the signal side, and when there is nothing close to the shape of the signal to be obtained, the accuracy is higher than other methods.

[0114] Examples of noise removal for images captured by a semiconductor inspection apparatus when only the first machine learning is performed (when the second and third machine learning are not performed) are shown in FIGS. 9 and 10. The semiconductor inspection apparatus is a PHEMOS1000 system manufactured by Hamamatsu Photonics. The camera is an InGaAs camera (exposure time: 10 seconds, objective lens: 20 times). The Bias voltage applied is 3.0V and the current is 8.78mA.

[0115] FIG. 9(a) is an image including noise to be removed. FIG. 9(b) is an image after noise removal according to the present embodiment. FIGS. 9(c) and (d) are images after noise removal by Bilateral and NLMeans, respectively. Note that the display range (range of pixel values) of each pixel is the same.

[0116] In the present embodiment, even for a signal with a tone close to that of noise, the noise reduction effect is high and the signal is not deleted. In Bilateral, when the tone is close to that of noise, the smoothing effect works strongly and weak signals disappear. In NLMeans, for an object without a nearby signal, the effect of leaving the signal is low and it is almost deleted.

[0117] FIG. 10(a) is an image including noise to be removed. FIG. 10(b) is an image after noise removal according to the present embodiment. FIG. 10(c) is a graph showing the line profile at the position of the line indicated by (1) in FIG. 10(a). The horizontal axis represents the position on the line, and the vertical axis represents the pixel value. FIG. 10(d) is a graph showing the line profile at the position of the line at the same position as FIG. 10(c) in FIG. 10(b). FIG. 10(e) is a graph showing the line profile at the position of the line indicated by (2) in FIG. 10(a). FIG. 10(f) is a graph showing the line profile at the position of the line at the same position as FIG. 10(e) in FIG. 10(b). This result also shows that appropriate noise removal can be performed according to the present embodiment.

[0118] Figs. 11 to 13 show examples of noise removal for images captured by a semiconductor inspection apparatus when the first machine learning and the second machine learning are performed. The camera is an InGaAs camera (exposure time: 10 seconds, objective lens: 20 times).

[0119] Fig. 11(a) is an image including noise to be removed. Fig. 11(b) is an image obtained by taking the average of the pixel values for each pixel of a total of five images, which is the image including noise to be removed and four images captured in the same manner as the said image. Fig. 11(c) is the image after noise removal according to the present embodiment. Fig. 11(d) is the image after noise removal by the model obtained by Noise2Noise. Note that the display range (range of pixel values) of each pixel is the same. The minimum value is black (value 503), and the maximum value is white (value 520).

[0120] In this example, for both the present embodiment and Noise2Noise, a set of 128 patches randomly cropped (24 pixels × 24 pixels) from the above-mentioned total of five images was used, and a model trained 5000 times was used.

[0121] Regarding the region of the image shown in Fig. 12(a) of the four images shown in Fig. 11, the standard deviation (σ value) and the average value of the pixel values are shown in Fig. 12(b). As shown by these values, the method (proposed method) according to the present embodiment had the effect of reducing the background noise most compared to other methods.

[0122] Figs. 13(b) to 13(e) are graphs showing line profiles of the positions of the lines in Fig. 13(a) of the four images shown in Fig. 11. The horizontal axis represents the position on the line, and the vertical axis represents the pixel value. Figs. 13(b) to 13(e) correspond to the image of Fig. 11(a) (the image including the noise to be removed), the image of Fig. 11(b) (the image including the noise to be removed), the image of Fig. 11(c) (the image after noise removal according to the present embodiment), and the image of Fig. 11(d) (the image after noise removal by the model obtained by Noise2Noise), respectively. As shown in these, in the method according to the present embodiment, there is little degradation in resolution, and only the noise is removed without blurring the spread of the light-emitting points.

[0123] Fig. 14 shows the number of repetitions of machine learning and the noise removal result when the first machine learning and the second machine learning are performed. Fig. 15 shows the number of repetitions of machine learning and the noise removal result when only the first machine learning is performed. The upper and lower images in Figs. 14 and 15 are the images of the noise removed and the images after noise removal based on the noise removal model obtained by repeating the machine learning the corresponding number of times.

[0124] Note that the size of the cutout (patch size) is 24 pixels × 24 pixels. Also, the number of mini-batches is 128. Also, the learning time (computation speed) is about 50 seconds per 1000 images in Fig. 14 and about 1 minute per 1000 images in Fig. 15 (when using Geforce RTX2080ti).

[0125] Fig. 16 shows a graph of the number of repetitions of machine learning (number of learning times) (horizontal axis) and the evaluation value based on SSIM of the image after noise removal (vertical axis). In this graph, the case where the first machine learning and the second machine learning are performed (Method 2), the case where the noise image when creating a new patch in the second machine learning is an ideal noise image (Replacement with ideal noise), the case where shuffling is not performed when creating a new patch in the second machine learning (No shuffle), and the case where only the first machine learning is performed and the second machine learning is not performed (Method 1 only) are shown.

[0126] As shown in these, the second machine learning can remove noise more accurately by repeating the machine learning.

[0127] Figs. 17 to 19 show examples of noise removal for images captured by a semiconductor inspection apparatus when the first machine learning and the third machine learning are performed. The camera is an InGaAs camera (exposure time 10 seconds, objective lens 5 times).

[0128] Fig. 17(a) is an image including noise to be removed. Fig. 17(b) is an image obtained by taking the average of the pixel values of each pixel of a total of 4 images, which is the image including noise to be removed and 3 images captured in the same manner as the said image. Fig. 17(c) is the image after noise removal according to the present embodiment. Fig. 17(d) is the image after noise removal by the model obtained by Noise2Noise. Note that the display range (range of pixel values) of each pixel is the same. The minimum value is black (value 502), and the maximum value is white (value 530). Figs. 18(a) to (d) are enlarged images of parts of Figs. 17(a) to (d) respectively.

[0129] In this example, in both the present embodiment and Noise2Noise, a set of 128 patches randomly cut out (24 pixels × 24 pixels) from the above-mentioned total 5 images was used, and a model trained 5000 times was used.

[0130] Regarding the image area shown in Fig. 19(a) of the four images shown in Figs. 17 and 18, the standard deviation (σ value) and the average value of the pixel values are shown in Fig. 19(b). As shown by these values, the method (proposed method) according to this embodiment had the effect of reducing the background noise most compared to other methods. In the average image in the time series direction using a total of four images, the noise reduction effect was low despite the average of the four images. That is, in this actual sample, it is expected that the noise is not simple Gaussian noise in the time series direction. Therefore, in the learning using patches randomly extracted from two of the four images and cut out at the same position by the Noise2Noise method, the result of the noise reduction effect was low. In this embodiment, since learning is performed with the information for each image, it is possible to learn a model for removing noise even for an object to which a powerful learning method such as Noise2Noise cannot be applied.

[0131] Subsequently, a learning program and a noise removal program for executing the processing by the above-described series of learning system 20 and noise removal system 30 will be described. As shown in Fig. 20(a), the learning program 200 is stored in a program storage area 211 formed in a computer-readable recording medium 210 that is inserted into and accessed by a computer or is provided in the computer. The recording medium 210 may be a non-transitory recording medium.

[0132] The learning program 200 includes a learning acquisition module 201 and a learning module 202. The functions realized by executing the learning acquisition module 201 and the learning module 202 are the same as the functions of the learning acquisition unit 21 and the learning unit 22 of the learning system 20 described above, respectively.

[0133] As shown in Fig. 20(b), the noise removal program 300 is stored in a program storage area 311 formed in a computer-readable recording medium 310 that is inserted into and accessed by a computer or is provided in the computer. The recording medium 310 may be a non-transitory recording medium. Note that the recording medium 310 may be the same as the recording medium 210.

[0134] The noise removal program 300 includes a acquisition module 301 for removal and a noise removal module 302. The functions realized by executing the acquisition module 301 for removal and the noise removal module 302 are the same as the functions of the acquisition unit 31 for removal and the noise removal unit 32 of the noise removal system 30 described above, respectively.

[0135] Note that part or all of the learning program 200 and the noise removal program 300 may be transmitted via a transmission medium such as a communication line, received by other devices, and recorded (including installation). Also, each module of the learning program 200 and the noise removal program 300 may be installed on any of a plurality of computers instead of a single computer. In that case, the above-described series of processes are performed by a computer system including the plurality of computers.

Description of Reference Numerals

[0136] 10... computer, 20... learning system, 21... acquisition unit for learning, 22... learning unit, 30... noise removal system, 31... acquisition unit for removal, 32... noise removal unit, 200... learning program, 201... acquisition module for learning, 202... learning module, 210... recording medium, 211... program storage area, 300... noise removal program, 301... acquisition module for removal, 302... noise removal module, 310... recording medium, 311... program storage area.

Claims

1. A learning method for generating a learning model for removing noise from data having a plurality of values, comprising: a learning acquisition step of acquiring learning data having a plurality of values and including noise; a learning step of performing an operation using information based on the learning data acquired in the learning acquisition step as an input to the learning model, calculating a difference between the learning data and the data after noise removal of the learning data based on the output from the learning model for each of the plurality of values, and performing first machine learning based on the statistical value of the difference for each of the plurality of values to generate a learning model; A learning method including the above.

2. The learning method according to claim 1, wherein the data having a plurality of values is an image.

3. The learning method according to claim 1 or 2, wherein, in the learning step, as the statistical value of the difference for each of the plurality of values, any one of an average value for each learning data, a value indicating a difference in distribution between learning data, a value indicating a difference in correlation between learning data, and a value indicating a difference in the statistical value of the difference between adjacent values between learning data is used.

4. The learning method according to claim 3, wherein, in the learning step, first machine learning is performed so that the statistical value of the difference for each of the plurality of values becomes 0.

5. In the learning acquisition step, a plurality of learning data are acquired, In the learning step, the difference between the data after noise removal of the learning data based on the output from the learning model and another learning data is combined to generate new learning data, information based on the generated new learning data is used as an input to the learning model, and second machine learning is performed using information based on the data after noise removal of the learning data or the difference of the other learning data as an output from the learning model to generate a learning model. The learning method according to any one of claims 1 to 4.

6. In the learning acquisition step, learning noise data indicating noise is also acquired, In the learning step, information based on the difference and information based on the learning noise data acquired in the learning acquisition step are respectively input to a feature extraction model that inputs data and outputs information indicating a feature amount of the data, and third machine learning is performed by comparing each piece of information obtained as an output to generate the feature extraction model and the learning model. The learning method according to any one of claims 1 to 5.

7. The learning method according to any one of claims 1 to 6, wherein in the learning step, a learning model including a neural network having shortcut connections is generated.

8. The learning method according to any one of claims 1 to 7, wherein in the learning acquisition step, original data of the learning data is acquired, and each of a plurality of portions of the original data is acquired as learning data.

9. A learning system for generating a learning model for removing noise from data having a plurality of values, learning acquisition means for acquiring learning data having a plurality of values and including noise; learning means for performing an operation using, as an input to the learning model, information based on the learning data acquired by the learning acquisition means, calculating, for each of the plurality of values, a difference between the learning data and data after noise removal of the learning data based on an output from the learning model, performing first machine learning based on a statistical value of the difference for each of the plurality of values, and generating a learning model; A learning system comprising:

10. A learning program for operating a computer as a learning system for generating a learning model for removing noise from data having a plurality of values, causing the computer to function as learning acquisition means for acquiring learning data having a plurality of values and including noise; function as learning means for performing an operation using, as an input to the learning model, information based on the learning data acquired by the learning acquisition means, calculating, for each of the plurality of values, a difference between the learning data and data after noise removal of the learning data based on an output from the learning model, performing first machine learning based on a statistical value of the difference for each of the plurality of values, and generating a learning model; A learning program.

11. A noise removal method for removing noise from data having a plurality of values, using a learning model generated by the learning method according to any one of claims 1 to 8, a removal acquisition step of acquiring noise removal target data having a plurality of values and being a target for noise removal; a noise removal step of inputting the noise removal target data acquired in the removal acquisition step into the learning model to remove noise from the noise removal target data; A noise removal method including:

12. A noise removal system that removes noise from data having a plurality of values, using a learning model generated by the learning method according to any one of claims 1 to 8, a removal acquisition means for acquiring noise removal target data that has a plurality of values and is a target for noise removal, a noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data, and a noise removal system comprising the same.

13. A noise removal program that causes a computer to operate as a noise removal system that removes noise from data having a plurality of values, using a learning model generated by the learning method according to any one of claims 1 to 8, the computer being caused to, function as a removal acquisition means for acquiring noise removal target data that has a plurality of values and is a target for noise removal, and a noise removal means for inputting the noise removal target data acquired by the removal acquisition means into the learning model to remove noise from the noise removal target data, and a noise removal program that causes the computer to function as such.

Images