De-noising method, de-noising program, de-noising system, and learning method

Abstract

Provided is a de-noising method capable of appropriately de-noising data having a plurality of values, such as an image, even when an AI-dedicated chip or the like is used. The denoising method is a method for removing noise from data having a plurality of values, and comprises: an acquisition step (S01) for acquiring data to be denoised, which has a plurality of values and is to be denoised; a first noise generation step (S02) for performing a first denoising process on the acquired data to be denoised, and generating first noise data relating to noise removed by the first denoising process; a second noise generation step (S03) for generating second noise data by inputting the generated first noise data into a learning model generated by machine learning in advance; and a de-noising step (S04) for generating de-noised data from the de-noised data using the generated second noise data.

Description

Technical Field

[0001] The present invention relates to a denoising method, a denoising program, and a denoising system for removing noise from data having multiple values, and a learning method for generating a learning model used in denoising. Background Art

[0002] Currently, techniques of machine learning such as neural networks have been proposed for use in image denoising (for example, refer to Patent Document 1). Prior Art Documents Patent Documents

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-101507 Summary of the Invention Technical Problem to be Solved by the Invention

[0004] In the operation of AI (Artificial Intelligence) inference using a learning model generated by machine learning, from the viewpoints of speed and the like, GPUs (Graphics Processing Units) are often used. On the other hand, in the operation using a learning model for image denoising, ASICs (Application Specific Integrated Circuits) and SoCs (System on a Chip) etc. can be considered as AI (Artificial Intelligence) - dedicated chips. By using these AI - dedicated chips in image denoising, lower costs and lower power consumption can be achieved compared to the case of using a GPU.

[0005] However, compared to the data size of the image to be denoised, the data size of the image available in the AI - dedicated chip is sometimes smaller. For example, there is a case where the image to be denoised is an image in which the luminance value (pixel value) of each pixel is an unsigned integer (uint) of 14 bits or 16 bits, while on the other hand, the image available in the AI - dedicated chip is an image in which the luminance value of each pixel is an integer (int) of 8 bits.

[0006] In such a case, the image to be denoised is quantized to reduce the data size of the image so that the image becomes an image available in the AI - dedicated chip, and thus denoising can be performed using the AI - dedicated chip. However, reducing the data size by quantization results in a decrease in the accuracy of the luminance value, and thus it is sometimes difficult to achieve appropriate denoising.

[0007] One embodiment of the present invention has been completed in view of the above - mentioned circumstances, and an object thereof is to provide a denoising method, a denoising program, a denoising system, and a learning method that can perform appropriate denoising from data having multiple values such as an image even when using an AI - dedicated chip or the like. Technical means for solving the problem

[0008] In order to achieve the above object, a denoising method according to an embodiment of the present invention is a denoising method for removing noise from data having a plurality of values, including: an acquisition step of acquiring denoising target data having a plurality of values and being a denoising target; a first noise generation step of performing a first denoising process on the denoising target data acquired in the acquisition step to generate first noise data related to the noise removed by the first denoising process; a second noise generation step of inputting the first noise data generated in the first noise generation step into a learning model previously generated by machine learning to generate second noise data, where the learning model is used to generate second noise data related to the noise to be removed from the denoising target data based on the first noise data; and a denoising step of using the second noise data generated in the second noise generation step to generate denoised data based on the denoising target data acquired in the acquisition step.

[0009] In the denoising method according to an embodiment of the present invention, the first noise data generated by the first denoising process on the denoising target data is input into the learning model to generate the second noise data, and the denoised data is generated using the second noise data. Since the first noise data is generated by the first denoising process on the denoising target data, it is possible to perform operations related to denoising based on the learning model while suppressing the degradation of data accuracy. Thus, according to the denoising method of an embodiment of the present invention, even when using an AI - specific chip or the like, it is possible to appropriately denoise data having a plurality of values such as an image.

[0010] It may be that in the first noise generation step, the data obtained by the first denoising process is quantized to generate the first noise data, and in the denoising step, the second noise data generated in the second noise generation step is inverse - quantized, and the denoised data is generated based on the inverse - quantized data. According to this structure, it is possible to make the first noise data into appropriate data corresponding to the operations using the learning model. For example, it is possible to make the first noise data into data corresponding to the performance of an AI - specific chip that performs operations using the learning model. As a result, appropriate and reliable denoising can be performed.

[0011] It may be that in the first noise generation step, quantization is performed for each of a plurality of ranges of the magnitudes of the plurality of values to generate a plurality of first noise data, and in the second noise generation step, the plurality of first noise data generated in the first noise generation step are input into the learning model to generate the second noise data. According to this structure, more appropriate denoising considering the plurality of first noise data can be performed.

[0012] It may be that the processing of the first noise generation step and the second noise generation step are performed by processing devices with different performances. According to this structure, for example, an AI - dedicated chip can be used to perform operations using a learning model. Thus, denoising can be performed at low cost and low power consumption.

[0013] It may be that in the first noise generation step, as the first denoising process, a preset filter for performing the first denoising process is applied to the data to be denoised to generate first noise data. Additionally, it may be that in the first noise generation step, as the first denoising process, the frequency components within a preset range are removed from the data to be denoised to generate first noise data. According to these structures, the first denoising process can be appropriately and reliably performed. As a result, appropriate and reliable denoising can be carried out.

[0014] It may be that in the first noise generation step, size data reflecting the magnitudes of multiple values of the data to be denoised is also generated, and in the second noise generation step, the first noise data and the size data generated in the first noise generation step are input into a learning model to generate second noise data. By generating size data and using it as an input to the learning model, the magnitudes of multiple values of the data to be denoised can be considered in the learning model. As a result, more appropriate denoising can be performed.

[0015] It may be that the data to be denoised is image data, spectral data, or time - series data. According to this structure, appropriate denoising can be performed on image data, spectral data, or time - series data.

[0016] In addition, one embodiment of the present invention can be described not only as an invention of a denoising method as described above, but also as an invention of a denoising program and a denoising system as follows. They are only different in category, but are essentially the same invention and have the same functions and effects.

[0017] That is, the denoising program of one embodiment of the present invention is a denoising program for removing noise from data having multiple values, which causes a computer to function as the following units: an acquisition unit that acquires denoising - target data having multiple values and serving as the data to be denoised; a first noise generation unit that performs a first denoising process on the denoising - target data acquired by the acquisition unit to generate first noise data related to the noise removed by this first denoising process; a second noise generation unit that inputs the first noise data generated by the first noise generation unit into a learning model pre - generated by machine learning to generate second noise data, where this learning model is used to generate second noise data related to the noise to be removed from the denoising - target data based on this first noise data; and a denoising unit that uses the second noise data generated by the second noise generation unit to generate denoised data based on the denoising - target data acquired by the acquisition unit.

[0018] In addition, a denoising system according to an embodiment of the present invention is a denoising system that removes noise from data having multiple values, and includes: an acquisition unit that acquires denoising target data having multiple values and being a denoising target; a first noise generation unit that performs a first denoising process on the denoising target data acquired by the acquisition unit, and generates first noise data related to the noise removed by the first denoising process; a second noise generation unit that inputs the first noise data generated by the first noise generation unit into a learning model previously generated by machine learning, and generates second noise data, where the learning model is used to generate second noise data related to the noise to be removed from the denoising target data according to the first noise data; and a denoising unit that uses the second noise data generated by the second noise generation unit to generate denoised data according to the denoising target data acquired by the acquisition unit.

[0019] In addition, in order to achieve the above object, a learning method according to an embodiment of the present invention is a learning method for generating a learning model used for removing noise from data having multiple values, and includes: a learning data acquisition step of acquiring learning data having multiple values and containing noise; a first learning noise generation step of performing a first denoising process on the learning data acquired in the learning data acquisition step, and generating first learning noise data related to the noise removed by the first denoising process; and a learning step of using the first learning noise data generated in the first learning noise generation step as an input of the learning model, and performing machine learning to generate the learning model.

[0020] According to the learning method of an embodiment of the present invention, a learning model used in the above denoising method can be generated.

[0021] It may be that in the first learning noise generation step, frequency components of the learning data acquired in the learning data acquisition step are generated, and a method of the first denoising process is determined based on the generated frequency components. According to this configuration, the first denoising process can be performed in an appropriate method. As a result, more appropriate denoising can be performed. Advantageous Effects of the Invention

[0022] According to an embodiment of the present invention, even when using an AI dedicated chip or the like, appropriate denoising can be performed on data having multiple values such as an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a diagram showing the configurations of the denoising system and the learning system according to an embodiment of the present invention. Figure 2 It is a diagram showing an example of an image before and after quantization. Figure 3 It is a diagram showing an example of the contrast of the quantized image. Figure 4 It is a diagram showing an example of the contrast of the structure and the contrast of the noise. Figure 5 It is a diagram showing an example of an image involved in the processing before inputting an image to a learning model. Figure 6 It is a diagram showing an example of an image involved in the overall denoising process. Figure 7 It is a flowchart showing the denoising method, which is the process executed in the denoising system according to an embodiment of the present invention. Figure 8 It is a flowchart showing the learning method, which is the process executed in the learning system according to an embodiment of the present invention. Figure 9 It is a diagram showing the correspondence between noise and the luminance value of an image. Figure 10 It is a diagram showing another example of an image involved in the processing before inputting an image to a learning model. Figure 11 It is a diagram showing another example of an image involved in the processing before inputting an image to a learning model. Figure 12 It is a diagram showing an image to be denoised and the frequency components of the image. Figure 13 It is a diagram showing a learning image and the frequency components of the image. Figure 14 It is a diagram showing the frequency components of a learning image. Figure 15 It is a curve graph regarding the frequency components of a learning image. Figure 16 It is a diagram showing a filter as a candidate used in the first denoising process and the frequency components of the filter. Figure 17 It is a curve graph regarding the frequency components of a filter as a candidate used in the first denoising process. Figure 18 It is a diagram showing an image used in an embodiment. Figure 19 It is a table showing the results of an embodiment and a comparative example. Figure 20 It is a diagram showing examples of images of an embodiment and a comparative example. Figure 21 It is a diagram showing another example of images of an embodiment and a comparative example. Figure 22 It is a diagram showing yet another example of images of an embodiment and a comparative example. Figure 23 It is a diagram showing the learning speed of the present embodiment. Figure 24This is a diagram showing the structures of the denoising program and the learning program of the embodiments of the present invention together with a recording medium. Detailed Embodiments

[0024] Hereinafter, Figure 1 the embodiments of the denoising method, the denoising program, the denoising system, and the learning method of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted.

[0025] Figure 1 Fig. (a) shows the denoising system 10 of the present embodiment. Figure 1 Fig. (b) shows the learning system 20 of the present embodiment. The denoising system 10 is a system (device) that removes noise from data having multiple values. In the denoising of the denoising system 10, a learning model (learned model, inference model) generated by machine learning is used. The learning system 20 is a system (device) that performs machine learning to generate the learning model used in the denoising system 10.

[0026] In the present embodiment, the data having multiple values to be the noise removal target is, for example, an image. In addition, the noise contained in the image is, for example, read noise and shot noise. The image to be the noise removal target is, for example, an image of a substrate manufactured on a production line, and the denoised image is used for inspecting defects of the substrate. Alternatively, the image to be the noise removal target is an image of cells, and the denoised image is used for cell analysis. In addition, the image to be the noise removal target may be any image other than the above as long as it is an image containing noise.

[0027] The data having multiple values to be the noise removal target does not have to be an image. For example, it may also be spectral data or time-series data, etc. The spectral data is, for example, data of intensity corresponding to wavenumber (wavelength, frequency). The time-series data is, for example, data of intensity at each moment. Specifically, it is output data from a photomultiplier tube, or measurement data of in-vivo information such as hemoglobin concentration, etc.

[0028] The denoising system 10 and the learning system 20 are configured to include a conventional computer, which includes hardware such as a processor, a memory, and a communication module. The denoising system 10 may also include a processor for performing operations using a learning model and a processor for performing other operations. The processor for performing operations using a learning model is, for example, an ASIC or an SOC, which is an AI-specific chip. Alternatively, the processor may be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). In addition, the processor for performing other operations is, for example, an FPGA (Field Programmable Gate Array), a CPU, or a GPU. The processor that performs operations in the learning system 20 is, for example, a CPU or a GPU. In addition, the denoising system 10 and the learning system 20 may also include an imaging device such as a camera for acquiring an image used in the processing. Further, the denoising system 10 and the learning system 20 may be included in an imaging device such as a camera for acquiring an image used in the processing. In Figure 1 this case, the denoising system 10 and the learning system 20 are shown as different systems (devices), but they may also be implemented by the same system (device).

[0029] The computer that constitutes the denoising system 10 and the learning system 20 may also be a computer system including multiple computers. In addition, the computer may be constituted by cloud computing or edge computing. Each of the functions of the denoising system 10 and the learning system 20 described below is realized by the operation of these components under the action of a program or the like.

[0030] By using the above-mentioned AI-specific chip as the processor for performing operations using a learning model in the denoising system 10, for example, compared with the case of performing operations using a learning model with a GPU, low cost and low power consumption can be achieved.

[0031] As described above, in this case, compared with the data size of the image to be denoised, the data size of the image available in the AI-specific chip is sometimes smaller. In the present embodiment, the data size of the image available in the AI-specific chip is smaller than the data size of the image.

[0032] For example, this represents a situation where, as Figure 2As shown, the image to be denoised (an image captured by an imaging device such as a camera) is an image where the luminance value of each pixel is an unsigned integer of 14 bits (uint14: 0 to 16383), and the image available in the AI - specific chip is an image where the luminance value of each pixel is an integer of 8 bits (int: - 128 to 127). The AI - specific chip cannot process an image with a luminance value of 14 bits. To process this image using the AI - specific chip, as Figure 2 shown, it is necessary to quantize the image so that it can be processed by the AI - specific chip. In addition, in the Figure 2 graph showing the image before and after quantization, the horizontal axis represents pixels and the vertical axis represents luminance values. When quantizing the image to be denoised to reduce the data size of the image, the accuracy of the luminance value will decrease, so it is sometimes difficult to achieve appropriate denoising. In this embodiment, even when performing operations using a learning model with an AI - specific chip, appropriate denoising can be performed on the image.

[0033] The reason why the accuracy of luminance decreases in the case of quantization is the contrast of the image to be quantized. Figure 3 Shows the luminance value of each part of the image (the left - hand graph in the Figure 3 four graphs) and the quantized luminance value of this part (the right - hand graph in the Figure 3 four graphs). In the part of the image corresponding to the upper - left graph, the contrast is relatively small (the range of luminance values is about 100), which is less than the resolution of the AI - specific chip, i.e., 8 bits (256). Therefore, as shown in the upper - right graph, the accuracy of the quantized luminance value does not decrease or decreases slightly. That is, if the contrast of the image is small, there is no loss or less loss of information about luminance caused by quantization. In this case, even using the quantized image, appropriate denoising can be performed.

[0034] On the other hand, in the part of the image corresponding to the lower - left graph, the contrast is relatively large (the range of luminance values is about 4500), which is greater than the resolution of the AI - specific chip, i.e., 8 bits (256). Therefore, as shown in the lower - right graph, the accuracy of the quantized luminance value drops significantly. That is, if the contrast of the image is large, the loss of information about luminance caused by quantization is large, resulting in a significant deterioration of the image. In this case, if using the quantized image as described above, appropriate denoising cannot be performed.

[0035] As shown in the Figure 4 graph, the large contrast in the image ( Figure 4The contrast represented by the long arrow (in the present embodiment, this is referred to as the structure) is caused by the change in the luminance value of the object (in the present embodiment, this is referred to as the structure) that appears in the image other than the noise. In many cases, the contrast change of this structure is greater than the resolution of the AI - dedicated chip, that is, 8 bits (256). On the other hand, the contrast of the noise contained in the image ( Figure 4 the contrast represented by the short arrow) is usually smaller than the contrast change of the above - mentioned structure, and in many cases, it is less than the resolution of the AI - dedicated chip, that is, 8 bits (256).

[0036] Based on the above idea, the present embodiment performs denoising. In the present embodiment, by using the image from which the luminance value related to the structure has been substantially removed for denoising, it is possible to prevent the accuracy from decreasing due to quantization, and even when performing operations using a learning model with an AI - dedicated chip, appropriate denoising can be performed.

[0037] Next, the functions of the denoising system 10 and the learning system 20 of the present embodiment will be described. As Figure 1 shown, the denoising system 10 is configured to include an acquisition unit 11, a first noise generation unit 12, a second noise generation unit 13, and a denoising unit 14.

[0038] The acquisition unit 11 is an acquisition unit that acquires denoising target data having a plurality of values and being a target for denoising. As described above, in the present embodiment, the denoising target data is an image. The acquisition unit 11 acquires an image, for example, by receiving an image captured by a photographing device such as a camera from the photographing device. The photographing device that acquires the image to be denoised by photographing is, for example, a visible - light camera or an X - ray camera. In addition, the photographing device may be other devices.

[0039] In addition, the acquisition unit 11 may also acquire an image by accepting an input operation of an image performed by the user on the denoising system 10. In addition, the acquisition unit 11 may acquire an image by any method other than the above.

[0040] As described above, the image acquired by the acquisition unit 11 has a larger data size compared to the image that can be input into the learning model used in the denoising system 10 (that is, the image available in the AI - dedicated chip). For example, regarding the range (resolution) of the luminance value of each pixel, the image that can be input into the learning model is 8 bits, while the image acquired by the acquisition unit 11 is 14 bits. The image acquired by the acquisition unit 11 is output to the first noise generation unit 12.

[0041] The first noise generation unit 12 is a first noise generation unit that performs a first denoising process on the denoising target data acquired by the acquisition unit 11 and generates first noise data related to the noise removed by this first denoising process. The first noise generation unit 12 can generate the first noise data by quantizing the data obtained by the first denoising process. As the first denoising process, the first noise generation unit 12 can also apply a preset filter for performing the first denoising process to the denoising target data to generate the first noise data. Further, as the first denoising process, the first noise generation unit 12 can also perform a process of removing frequency components within a preset range from the denoising target data to generate the first noise data.

[0042] The first noise data is obtained by substantially excluding, from each luminance value of the image of the denoising target, the luminance values corresponding to the structure. That is, the first noise data substantially excludes the contrast of the structure.

[0043] The first noise generation unit 12 generates the first noise data as follows, for example. The first noise generation unit 12 inputs the image of the denoising target from the acquisition unit 11. The first noise generation unit 12 prestores a preset filter for performing the first denoising process and applies the filter to the input image. The filter is, for example, a filter for performing image smoothing. Specifically, the filter is a Gaussian filter, an average filter, a median filter, a bilateral filter, or the like.

[0044] Figure 5 and Figure 6 Examples of the image 30 of the denoising target and the image 31 after the filter application are shown. Further, a graph 50 showing the luminance value of each pixel of these images 30 and 31 is shown. Among these, in the graph 50, the horizontal axis represents the pixel and the vertical axis represents the luminance value (the same applies hereinafter in Figure 5 and Figure 6 ). The line type in the graph 50 corresponds to the line type of the frame of each image. In Figure 5 and Figure 6 , an example of applying a Gaussian filter to the image 30 of the denoising target to generate the image 31 after the filter application is shown.

[0045] The first noise generation unit 12 generates a difference image between the image of the denoising target and the image after the filter application. For example, the first noise generation unit 12 generates a difference image in which the value obtained by subtracting the luminance value of the image after the filter application from the luminance value of the image of the denoising target for each corresponding pixel is used as the luminance value of that pixel. Figure 5 and Figure 6 Examples of the difference image 32 and a graph 51 showing the luminance value of each pixel of this difference image 32 are shown.

[0046] As Figure 5As shown in the curve graph 50, the brightness values of the image 30 of the object to be denoised include the structural contrast and noise. Additionally, as Figure 5 shown in the curve graph 51, (it can be considered that) the brightness values of the differential image 32 include partial structure and noise. As shown in these two curve graphs 50 and 51, the contrast of the image 30 of the object to be denoised is large, and the contrast of the differential image 32 is small. In addition, the scales of the vertical axes of the two curve graphs 50 and 51 are different from each other, and the scale of the curve graph of the differential image 32 is smaller compared to the curve graph 50 of the image 30 of the object to be denoised. For the differential image that contains noise and has a small contrast, even if quantization is performed, the reduction in accuracy can be suppressed.

[0047] As the first denoising process, the first noise generation unit 12 may also perform a process of removing frequency components within a preset range from the image of the object to be denoised without applying a filter. For example, the first noise generation unit 12 performs FFT (Fast Fourier Transform) or wavelet transform on the image of the object to be denoised to generate the frequency components of the image of the object to be denoised. The first noise generation unit 12 pre-stores a threshold value of frequency and removes the frequency components with frequencies above the stored threshold value. Usually, noise is a high-frequency component, so the high-frequency components are removed in this way. The first noise generation unit 12 performs an inverse transform on the frequency components after the above removal to generate an image after the frequency components are removed. The first noise generation unit 12 generates a differential image between the image of the object to be denoised and the image after the frequency components are removed. Additionally, the first noise generation unit 12 may also use a combination of the process related to the filter and the process related to the frequency components to generate the differential image.

[0048] The first noise generation unit 12 quantizes the differential image. Quantization is a process for making the image available in the AI - specific chip. The resolution of each pixel of the quantized image becomes the resolution of the image available in the AI - specific chip. The quantized image is the first noise data. Figure 6 An example of the quantized image 33 as the first noise data generated based on the differential image 32 and a curve graph 52 showing the brightness value of each pixel of the quantized image 33 are shown.

[0049] Quantization can be performed by a conventional method. For example, quantization is performed by calculating the brightness value of the quantized image according to the brightness value of the differential image for each pixel. Specifically, the brightness value of the quantized image is calculated by the following formula. In this example, the number and positional relationship of the pixels before and after quantization remain unchanged. The functions and variables in the above formula represent the following respectively. round(x): A function that rounds x (e.g., rounds to the nearest) to the value of the brightness value (resolution) of the AI - specific chip. DiffImage: The luminance value of the difference image. maxDiff, minDiff: The maximum and minimum values of the luminance values of the difference image to be quantified. These values are preset according to the magnitude of the assumed noise, etc., such that the difference image contains noise. maxAI, minAI: The maximum and minimum values of the luminance values (resolution) of the AI - dedicated chip.

[0050] The first noise generation unit 12 outputs the quantized image, which is the generated first noise data, to the second noise generation unit 13. Additionally, the first noise generation unit 12 may output the difference image 33 to the second noise generation unit 13 without quantization.

[0051] Furthermore, the first noise data does not have to be the above - mentioned noise data. For example, as long as it is data that contains the noise of the image to be denoised from the image to be denoised and has a lower contrast than the image to be denoised. Additionally, the first noise generation unit 12 does not have to generate (generate) the first noise data as described above. As long as it performs some first denoising process and generates the first noise data related to the noise removed by this first denoising process.

[0052] The second noise generation unit 13 is a second noise generation unit that inputs the first noise data generated by the first noise generation unit 12 into a learning model generated in advance through machine learning, and generates the second noise data. The learning model is used to generate the second noise data related to the noise to be removed from the denoising target data based on the first noise data.

[0053] The second noise data represents the noise to be removed from the denoising target data. That is, the number of pixels and the resolution of the second noise data are the same as those of the first noise data. Additionally, in this embodiment, since the first noise data used to generate the second noise data is quantized, the second noise data is the quantized data of this noise. However, in the case where an unquantized image is output from the first noise generation unit 12, the second noise data is the unquantized data of this noise.

[0054] The second noise generation unit 13 generates the second noise data as follows, for example. The second noise generation unit 13 inputs the quantized difference image as the first noise data from the first noise generation unit 12. The second noise generation unit 13 stores the learning model in advance. The learning model is a model that inputs the first noise data and outputs (infers) the second noise data. The learning model is generated through the machine learning of the learning system 20 as described later.

[0055] The learning model is composed of, for example, a neural network. The neural network can be multi - layer. That is, the learning model can be generated through deep learning.

[0056] In the learning model, neurons are provided for inputting first noise data to the input layer. For example, the information input to the learning model is the luminance value of each pixel of an image as the first noise data. In this case, neurons equal in number to the number of pixels of the image are provided in the input layer, and the luminance value of the corresponding pixel is input to each neuron.

[0057] In the learning model, neurons are provided for outputting second noise data to the output layer. For example, the information output from the learning model is the luminance value of each pixel of an image as the second noise data. In this case, neurons equal in number to the number of pixels of the image are provided in the output layer, and the luminance value of the corresponding pixel is output from each neuron.

[0058] It is assumed that the learning model is used as a program module that is part of artificial intelligence software. The learning model is used, for example, in a computer having a processor and a memory, and the processor of the computer operates according to instructions from the model stored in the memory. For example, the processor of the computer operates according to the instructions, inputs information to the model, performs an operation corresponding to the model, and outputs a result from the model. Specifically, the processor of the computer operates according to the instructions, inputs information to the input layer of the neural network, performs an operation on parameters such as the weight coefficients based on learning in the neural network, and outputs a result from the output layer of the neural network. In addition, the learning model may be composed of something other than a neural network.

[0059] The second noise generation unit 13 inputs the first noise data to the learning model and performs an operation corresponding to the learning model to generate the second noise data. The generated second noise data is data in which a part of the structure contained in the first noise data is removed from the first noise data, that is, (assumed to be) data that only involves noise and does not contain a structure. Figure 6 An example of an image 34 as the second noise data generated from an image 33 as the first noise data, and a graph 53 showing the luminance value of each pixel of the image 34 are shown. The second noise generation unit 13 outputs the generated second noise data to the denoising unit 14.

[0060] As described above, the processing of the first noise generation unit 12 and the processing of the second noise generation unit 13 may also be performed by processing devices having different performances from each other. For example, the first noise generation unit 12 is implemented by a processor capable of processing an image (an image of the resolution) of the object to be denoised, and the second noise generation unit 13 is implemented by the above-mentioned AI dedicated chip that can only process the quantized image.

[0061] In addition, the frequency components of the signals contained in the image of the denoising target correspond to the imaging device that captured the image (e.g., the magnification of the lens used in the imaging device). In addition, the frequency components of the signals correspond to the imaging target of the image (e.g., an image of a substrate or an image of cells). In order to perform denoising more appropriately, the above-described first denoising process and learning model may also correspond to the type of imaging device or the type of imaging target related to the image of the denoising target, or a combination thereof.

[0062] The denoising unit 14 is a denoising unit that uses the second noise data generated by the second noise generation unit 13 to generate denoised data based on the denoising target data acquired by the acquisition unit 11. The denoising unit 14 may also inverse-quantize the second noise data generated by the second noise generation unit 13 and generate denoised data based on the inverse-quantized data.

[0063] The denoising unit 14 generates denoised data as follows, for example. The denoising unit 14 inputs the second noise data from the second noise generation unit 13. The denoising unit 14 performs inverse quantization on the input second noise data. The inverse quantization corresponds to the quantization performed by the first noise generation unit 12 and is a process of obtaining an image with the resolution of the image that is the denoising target data. The image after inverse quantization is an image of the noise to be removed from the denoising target data.

[0064] The inverse quantization can be performed by a conventional method. For example, the inverse quantization is performed by calculating the luminance value of the image after inverse quantization based on the luminance value of the image that is the second noise data for each pixel. Specifically, the luminance value of the image after inverse quantization is calculated by the following formula. In this example, the number and positional relationship of the pixels before and after inverse quantization remain unchanged. Figure 6 An example of an image (image of noise) 35 generated by inverse quantizing the image 34 that is the second noise data and a graph 54 showing the luminance value of each pixel of the image 35 are shown. The functions and variables in the above formula represent the following, respectively. round(x): A function that rounds x (e.g., rounds up or down) to a value of the luminance value (resolution) of the denoising target. AIOutImage: The luminance value of the image that is the second noise data. maxDiff, minDiff: The maximum and minimum luminance values of the difference image that is the quantization target. These values are preset according to the assumed size of the noise, etc., so that the difference image contains noise. maxAI, minAI: The maximum and minimum luminance values (resolutions) of the AI dedicated chip.

[0065] The denoising unit 14 generates a difference image between the image of the object to be denoised and the inverse-quantized image as the denoised image. For example, the denoising unit 14 generates a denoised image in which the value obtained by subtracting the luminance value of the inverse-quantized image from the luminance value of the image of the object to be denoised for each corresponding pixel is used as the luminance value of that pixel. Figure 6 An example of the denoised image 36 and a graph 55 showing the luminance value of each pixel of the image 36 are shown.

[0066] The denoising unit 14 outputs the generated denoised image. The output of the denoised image can be performed in the same manner as in the conventional method according to the purpose of use of the image. The above are the functions of the denoising system 10 of the present embodiment.

[0067] Next, the functions of the learning system 20 of the present embodiment will be described. As Figure 1 shown, the learning system 20 is configured to include a learning acquisition unit 21, a first learning noise generation unit 22, and a learning unit 23.

[0068] The learning acquisition unit 21 is a learning acquisition unit that acquires learning data having a plurality of values and including noise. The learning acquisition unit 21 acquires an image including noise as learning data. The number of pixels and the resolution of the learning image (sample image) as learning data are the same as those of the image of the object to be denoised. The image including noise is, for example, an image captured by an imaging device. In addition, the image including noise may be an image obtained by adding noise by simulation to an image that does not include noise.

[0069] The learning image may also be captured by the same type of imaging device as the image of the object to be denoised. In addition, the learning image may capture the same type of subject (for example, an image of a substrate or an image of a cell). Thus, the learning model generated by the learning system 20 can be set for each type of imaging device or each type of imaging object, or each combination thereof. In this way, by generating a learning model by matching the learning image with the image of the object to be denoised, a learning model capable of performing more appropriate denoising can be obtained.

[0070] The learning acquisition unit 21 acquires an image, for example, in the same manner as the image of the object to be denoised, by receiving the learning image from the imaging device. In addition, the learning acquisition unit 21 may acquire the learning image by accepting an input operation of the image performed by the user on the learning system 20. In addition, the learning acquisition unit 21 may acquire an image by any method other than the above. The learning acquisition unit 21 acquires a quantity of learning images sufficient for appropriately performing machine learning described later. The learning acquisition unit 21 outputs the acquired learning images to the first learning noise generation unit 22.

[0071] The first learning noise generation unit 22 is a first learning noise generation unit that performs first denoising processing on the learning data obtained by the learning acquisition unit 21 and generates first learning noise data related to the noise removed by this first denoising processing.

[0072] The first learning noise data is obtained by roughly excluding from each luminance value of the learning image the luminance values corresponding to the objects (i.e., other than noise) appearing in the image. That is, the first learning noise data roughly excludes the contrast of the structure.

[0073] The first learning noise generation unit 22 inputs the learning image from the learning acquisition unit 21. For each learning image, the first learning noise generation unit 22 generates first learning noise data in the same manner as the first noise generation unit 12 generates first noise data based on the image to be denoised. The first denoising processing and quantization used in the generation of the first learning noise data are the same as those used in the generation of the first noise data in the denoising system 10. The first learning noise generation unit 22 outputs the quantized image as the first learning noise data to the learning unit 23. In addition, the first learning noise generation unit 22 may also output the unquantized difference image to the learning unit 23.

[0074] The learning unit 23 is a learning unit that uses the first learning noise data generated by the first learning noise generation unit 22 as the input of the learning model and performs machine learning to generate the learning model.

[0075] For example, the learning unit 23 performs machine learning to generate the learning model as follows. The learning unit 23 inputs the quantized difference image as the first learning noise data from the first learning noise data. The learning unit 23 inputs the first learning noise data into the learning model and performs operations corresponding to the learning model to generate second learning noise data. The generation of the second learning noise data is performed in the same manner as the generation of the above-mentioned second noise data. The second learning noise data, like the above-mentioned second noise data, is the quantized data of the noise. However, in the case where an unquantized image is output from the first learning noise generation unit 22, the second learning noise data is the unquantized data of the noise.

[0076] The learning unit 23 performs inverse quantization on the second learning noise data. The learning unit 23 generates a difference image between the image as the learning data and the inversely quantized image as the denoised image. The inverse quantization used in the generation of the second learning noise data and the method for generating the denoised image are the same as those used in the inverse quantization and the generation of the denoised image in the generation of the second noise data in the denoising system 10.

[0077] The learning unit 23 updates the parameters of the machine learning, i.e., the learning model, using the generated denoised image corresponding to the learning image. The machine learning using the denoised image itself can be performed in the same manner as conventional machine learning methods such as backpropagation. For example, a noise-free image corresponding to the learning image is obtained in advance as a teacher image, and learning is performed in such a way as to eliminate the error between the generated denoised image and the teacher image.

[0078] In addition, even in the absence of teacher data, learning can be performed in the same manner as conventional machine learning methods. For example, multiple learning images that differ only in noise can be used, and learning can be performed by Noise2Noise.

[0079] The learning unit 23 outputs the generated learning model to the denoising system 10. The denoising system 10 inputs and stores the learning model input from the learning system 20 for denoising the above-mentioned image.

[0080] Furthermore, generally, in the generation of the learning model, the necessity for low cost and low power consumption is not as high as in the case of denoising. Therefore, the learning system 20 can also be implemented entirely by a processor capable of processing the image (of the resolution) of the object to be denoised instead of using an AI dedicated chip. However, an AI dedicated chip can also be used for the operation of the learning model in the generation of the learning model. The above are the functions of the learning system 20 of the present embodiment.

[0081] Next, use Figure 7 and Figure 8 flowcharts to illustrate the processes executed in the denoising system 10 and the learning system 20 of the present embodiment (the operation methods performed by the denoising system 10 and the learning system 20).

[0082] First, use Figure 7 flowcharts to illustrate the process executed in the denoising system 10 of the present embodiment, i.e., the denoising method. In this process, the acquisition unit 11 acquires the data to be denoised (S01, acquisition step). Next, the first noise generation unit 12 performs a first denoising process on the data to be denoised, generating first noise data related to the noise removed by this first denoising process (S02, first noise generation step). The first noise data is generated by quantifying the data obtained by the first denoising process. In addition, the first noise data can also be data that has not been quantified for the data obtained by the first denoising process.

[0083] Next, the first noise data is input to the learning model by the second noise generation unit 13 to generate second noise data (S03, second noise generation step). Next, the denoising unit 14 uses the second noise data to generate denoised data from the data to be denoised (S04, denoising step). The second noise data is used to generate denoised data on the basis of inverse quantization. The denoised data as the result of noise removal is output from the denoising unit 14 to a specified output destination (S05). The above is the processing performed in the denoising system 10 of the present embodiment, that is, the denoising method.

[0084] Next, using Figure 8 the flowchart of, the processing performed in the learning system 20 of the present embodiment, that is, the learning method, will be described. In this processing, learning data is acquired by the learning acquisition unit 21 (S11, learning acquisition step). Next, the learning first noise generation unit 22 performs first denoising processing on the learning data to generate learning first noise data related to the noise removed by the first denoising processing (S12, learning first noise generation step). The learning first noise data is generated by quantizing the data obtained by the first denoising processing. In addition, the learning first noise data may also be data that has not been quantized for the data obtained by the first denoising processing.

[0085] Next, the learning first noise data is used as an input to the learning model by the learning unit 23, and machine learning is performed to generate a learning model (S13, learning step). The generated learning model is output from the learning system 20 to the denoising system 10 (S14). In the denoising system 10, the learning model is stored and used in the above processing. The above is the processing performed in the learning system 20 of the present embodiment, that is, the learning method.

[0086] According to the above embodiment, the first noise data generated by the first denoising processing of the image to be denoised is input to the learning model to generate second noise data, and the second noise data is used to generate the denoised image. As described above, the image 33 as the first noise data is based on the difference image 32, in which the contrast change of the structure in the image 30 of the object to be denoised is substantially removed by the first denoising processing. The range of the luminance values of the difference image 32 is generally smaller than that of the image 30 of the object to be denoised. By using such an image 33 as the first noise data, it is possible to perform operations related to denoising based on the learning model while suppressing the deterioration of the accuracy of the luminance values. Thus, according to the present embodiment, even when using an AI dedicated chip or the like, it is possible to appropriately denoise data having multiple values such as an image.

[0087] Alternatively, as in the present embodiment, quantization may be performed when generating the first noise data, and inverse quantization may be performed when generating the denoised data. According to this configuration, the first noise data can be made appropriate data corresponding to the operation using the learning model. For example, the first noise data can be made data corresponding to the performance of the AI dedicated chip that performs the operation using the learning model. As a result, appropriate and reliable denoising can be performed. However, in the denoising system 10, when an image having the data size of the image to be denoised can be used, quantization and inverse quantization are not necessarily performed, and the data before quantization may be input to the learning model as the first noise data for denoising.

[0088] Alternatively, as in the present embodiment, the processing of the first noise generation unit 12 and the processing of the second noise generation unit 13 may be performed by processing devices having different performances from each other as described above. According to this configuration, for example, the AI dedicated chip can perform the operation using the learning model as described above. As a result, denoising can be performed at low cost and low power consumption. However, the processing of the first noise generation unit 12 and the processing of the second noise generation unit 13 may also be performed by the same processing device.

[0089] Alternatively, as in the present embodiment, the first denoising process may be performed using a filter. Alternatively, the first denoising process may also be performed by removing frequency components. According to these configurations, the first denoising process can be appropriately and reliably performed. As a result, appropriate and reliable denoising can be performed. However, the first denoising process is not necessarily performed as described above, and any denoising process that can substantially remove the contrast change of the structure in the image to be denoised is acceptable.

[0090] Next, a modified example of the denoising system 10 and the learning system 20 of the present embodiment will be described. In the above-described embodiment, the information input to the learning model corresponds to the difference image 32 between the image 30 to be denoised and the image 31 after the first denoising process. In the difference image 32, most of the magnitude of the original luminance value of the image 30 to be denoised is lost, so the magnitude of the original luminance value of the image 30 to be denoised is not sufficiently considered in the operation of the learning model.

[0091] The noise contained in an image generally corresponds to the magnitude of the luminance value of the image. For example, Figure 9 (a) of shows a graph representing the relationship between the luminance value of an image and the standard deviation of the noise contained in the image. Figure 9 (b) of shows a graph of an example of the luminance value of each pixel. In Figure 9In the graph of (b), the portion where the luminance value fluctuates within the range indicated by the arrow corresponds to noise. As shown in these graphs, for example, the larger the luminance value, the greater the fluctuation of the noise represented by the standard deviation. In addition, the distribution of noise varies according to the luminance value. For example, the distribution of the luminance values of the noise at low luminance follows a Poisson distribution, and the distribution of the luminance values of the noise at high luminance follows a Gaussian distribution.

[0092] Therefore, in the operation of the learning model, by enabling the consideration of the magnitude of the original luminance value of the image to be denoised, more appropriate denoising can be performed. Therefore, the denoising system 10 and the learning system 20 can adopt the following structures.

[0093] The first noise generation unit 12 also generates magnitude data (data of magnitude) that reflects the magnitudes of multiple values of the data to be denoised. The second noise generation unit 13 inputs the first noise data and the magnitude data generated by the first noise generation unit 12 into the learning model to generate second noise data. Specifically, for example, the following structure is adopted.

[0094] As Figure 10 shown, in addition to generating the image 32 for generating the first noise data, the first noise generation unit 12 generates an image 37 (magnitude data) representing the magnitude of the luminance value of each pixel of the data to be denoised from the image 30 of the object to be denoised. The first noise generation unit 12 quantizes the image 30 of the object to be denoised to generate the image 37 as the magnitude data. For example, as Figure 10 shown in the graph, the first noise generation unit 12 performs quantization that linearly transforms the resolution of the luminance value of the image 30 of the object to be denoised (0 to 16383) into the resolution of the luminance value of the image of the AI - specific chip (-128 to 127) to generate the image 37 as the magnitude data. The first noise generation unit 12 outputs the image 33 of the first noise data generated based on the image 32 and the image 37 as the magnitude data to the second noise generation unit 13.

[0095] The second noise generation unit 13 inputs these images 33 and 37 from the first noise generation unit 12. The learning model used in this structure can input these images 33 and 37. For example, in the input layer of the learning model, there are neurons for inputting the image 33 as the first noise data and neurons for inputting the image 37 as the magnitude data. That is, in the input layer of the learning model, an image of 2 ch (channels) can be input, where 1 ch is the image 33 as the first noise data and 2 ch is the image 37 as the magnitude data. In this structure, the output of the learning model is only the second noise data.

[0096] The second noise generation unit 13 inputs the image 32 as the first noise data and the image 37 as the size data into the learning model, performs operations corresponding to the learning model, and generates the second noise data. In the denoising system 10, except for the above, denoising is performed in the same manner as in the above-described embodiment. In addition, the size data does not have to be the above data, as long as it reflects the magnitude of the luminance value of each pixel of the image to be denoised. In addition, when the data to be denoised is data other than an image, the size data only needs to be data that reflects the magnitude of each value.

[0097] For example, the size data can also be the following data. As described above, the standard deviation of the noise contained in the image corresponds to the luminance value of the image. Therefore, instead of directly using the luminance value of the pixel as the size data as described above, the standard deviation of the noise can be calculated (estimated) for each pixel based on the luminance value of the pixel, and the size data can be generated based on the standard deviation. As shown below, the standard deviation of the noise is, for example, a value obtained based on the square root of the luminance value, and thus has a smaller variation amount compared to the luminance value and is not easily affected by quantization.

[0098] The first noise generation unit 12 calculates the standard deviation of the noise for each pixel of the image 30 to be denoised based on the relationship between the pixel value and the standard deviation of the noise stored in advance, according to the luminance value of the pixel. The relationship for calculating the standard deviation of the noise based on the luminance value corresponds, for example, to the type of imaging device (imaging element) that captures the image 30 to be denoised.

[0099] For example, when using a visible light CCD (Charge Coupled Device) sensor or a visible light CMOS (Complementary Metal Oxide Semiconductor) sensor as the imaging device (imaging element), the following relational expression can be used as the above relationship. The variables and the like in the above formula represent the following respectively. Noise: (calculated) standard deviation of the noise Cf: conversion coefficient [photon / count] Signal: luminance value of the pixel D: dark current luminance value R: readout noise Among the values included in the above relational expression, Cf, D, and R are inherent values corresponding to the imaging device (imaging element), and values set in advance according to the imaging device (imaging element) are used.

[0100] In addition, a curve graph can also be used for the above relationship. This curve graph represents the relationship between the luminance value calculated through actual measurement using an imaging device (imaging element) that captures an image 30 of a denoising target and the standard deviation of the noise. The calculation of the curve graph is performed as follows, for example. First, in a state where there is no subject, imaging is performed while gradually changing the intensity of the light source in stages. Then, for multiple images captured in states where the intensity of the light source is different, the average value and standard deviation of the luminance value are calculated respectively, and these are used as the relationship between the luminance value and the standard deviation of the noise to create a curve graph. In order to calculate the standard deviation of the noise based on the luminance value, the first noise generation unit 12 may also store a lookup table generated according to this curve graph, in which the standard deviation of the noise is assigned to the luminance value.

[0101] The first noise generation unit 12 generates an image with the standard deviation of the noise calculated for each pixel of the image 30 of the denoising target as the pixel value. The first noise generation unit 12 quantizes the generated image to generate an image as size data. For example, the first noise generation unit 12 performs quantization that linearly transforms the range from the minimum value of the pixel value of the generated image (the minimum value of the standard deviation of the noise) to the maximum value (the maximum value of the standard deviation of the noise) into the resolution of the luminance value of the image of the AI dedicated chip, and generates an image as size data. The generated size data can be used in the same manner as described above.

[0102] In the learning system 20, the learning first noise generation unit 22 generates an image as size data from the image as learning data in the same manner as described above. The learning unit 23 uses the learning first noise data and the size data as the input of the learning model, and performs machine learning to generate a learning model. In the learning system 20, except for the above, the generation of the learning model is performed in the same manner as the above-described embodiment.

[0103] As described above, in this structure, by generating size data and using it as the input of the learning model, it is possible to consider the size of the original luminance value of the image of the denoising target in the learning model. As a result, more appropriate denoising can be performed.

[0104] In the above-described embodiment, when quantizing the difference image 32 to generate the first noise data, the range of the luminance value of the difference image to be quantized (for example, the above-mentioned maxDiff, minDiff) is preset. Depending on the setting of this range, the part containing noise may fall outside the range, or the accuracy may be reduced due to quantization, resulting in possible loss of information that should be considered in the learning model. For this reason, considering the above points, in order to achieve more appropriate denoising, the denoising system 10 and the learning system 20 may also adopt the following structure.

[0105] The first noise generation unit 12 quantizes according to each of multiple ranges of the magnitudes of multiple values, and generates multiple first noise data. The second noise generation unit 13 inputs the multiple first noise data generated by the first noise generation unit 12 into a learning model, and generates second noise data. Specifically, for example, the following structure is adopted.

[0106] As Figure 11 shown, the first noise generation unit 12 quantizes according to the ranges of different luminance values based on the image 32 (the difference image 32 between the image 30 to be denoised and the image 31 after the first denoising process) used for generating the first noise data, and generates multiple images 33a, 33b as the first noise data. The ranges of luminance values are set in advance for the images 33a, 33b to be generated as the first noise data. For example, in the Figure 11 example shown, the first noise generation unit 12 generates the image 33a according to the range of luminance values from -200 to 200 in the difference image 32, and generates the image 33b according to the range of luminance values from -100 to 100 in the difference image 32.

[0107] Figure 11 The curves shown on the right side of the respective images 33a, 33b are the histograms of the respective luminance values of the images 33a, 33b. The image 33a covers a relatively wide range of luminance values but has relatively low precision, and the image 33b covers a relatively narrow range of luminance values but has relatively high precision. The first noise generation unit 12 outputs the multiple images 33a, 33b generated as the first noise data to the second noise generation unit 13.

[0108] The second noise generation unit 13 inputs the multiple images 33a, 33b as the first noise data from the first noise generation unit 12. The learning model used in this structure can input these images 33a, 33b. For example, neurons for inputting the image 33a as one party of the first noise data and neurons for inputting the image 33b as the other party of the first noise data are provided in the input layer of the learning model. That is, in the input layer of the learning model, images of 2 ch (channels) can be input, where 1 ch is the image 33a as one party of the first noise data, and 2 ch is the image 33b as the other party of the first noise data. In this structure, the output of the learning model is also only one second noise data.

[0109] The second noise generation unit 13 inputs the multiple images 33a, 33b as the first noise data into the learning model, and performs operations corresponding to the learning model to generate second noise data. In the denoising system 10, except for the above, denoising is performed in the same manner as in the above-described embodiment.

[0110] In the learning system 20, the learning first noise generation unit 22 generates a plurality of images as the learning first noise data according to the image for generating the learning first noise data in the same manner as described above. The learning unit 23 uses the plurality of images as the learning first noise data as the input of the learning model and performs machine learning to generate the learning model. In the learning system 20, in addition to the above, the generation of the learning model is performed in the same manner as in the above-described embodiment.

[0111] By generating a plurality of images as the first noise data as described above, it is possible to suppress the situation where the portion including the noise falls outside the range or the accuracy is reduced due to quantization. In addition, the range of the luminance values of the plurality of images generated as the first noise data is set in such a way that the above processing can be achieved. For example, as Figure 11 shown, a quantized image 33a covering a wide range of luminance values and a quantized image 33b having a narrow range but high accuracy are generated. In addition, in Figure 11 the example shown, the range of the luminance values of one party includes the range of the luminance values of the other party, but it may be other ranges. In addition, in the above example, two images generated as the first noise data are shown, but it is also possible to set ranges of three or more luminance values to generate three or more images as the first noise data.

[0112] As described above, according to this structure, more appropriate denoising considering a plurality of images as the first noise data can be performed. In addition, both the above size data and the plurality of first noise data can be used as the input of the learning model to generate the second noise data.

[0113] In the present embodiment, in order to perform more appropriate denoising, the image 32 for generating the first noise data obtained by the first denoising process (the difference image 32 between the image 30 to be denoised and the image 31 after the first denoising process) desirably contains only noise as much as possible.

[0114] Figure 12 A spectrum image 70 showing the frequency components obtained by performing FFT on the image 30 to be denoised and a curve graph 71 corresponding to the spectrum image 70 are shown. In the spectrum image 70 and the curve graph 71, the closer to the center, the lower the frequency (the center is the frequency 0), and as it moves away from the center, it becomes higher frequency. In Figure 12 the curve graph 71, the horizontal axis represents the frequency and the vertical axis represents the frequency component. As Figure 12 schematically shown in the curve graph 71, the signal spectrum 71a of the signal portion (the portion excluding the noise) of the image 30 has frequency components of frequencies of only the cut-off frequency fc or lower as fixed frequencies. On the other hand, the noise spectrum 71b of the noise portion of the image 30 has the same frequency components regardless of the frequency.

[0115] By using the image 32 for generating the first noise data obtained through the first denoising process as an image excluding frequency components above the cut-off frequency fc, it is possible to approach the above-described ideal image. Considering this, it is also possible to determine an appropriate method for the first denoising process when generating the learning model. Therefore, the learning system 20 can also adopt the following structure.

[0116] The first noise generation unit 22 for learning can determine the method for the first denoising process based on the frequency components obtained by the acquisition unit 21 for learning. Specifically, for example, the following structure is adopted.

[0117] The first noise generation unit 22 for learning determines the method for the first denoising process as follows. When using the above filter in the first denoising process, a plurality of filters (filter bank) are prepared in advance, and the most suitable filter from these filters is selected as the filter to be used in the first denoising process from the above viewpoints. The plurality of filters can include different types of filters (average filter, Gaussian filter, median filter, bilateral filter, etc.). The plurality of filters can include filters with different filter sizes. The plurality of filters can include filters with different standard deviations. When using a Gaussian filter, the standard deviation is the standard deviation of the Gaussian distribution.

[0118] When generating frequency components by performing FFT (Fast Fourier Transform) or wavelet transform, etc. as the first denoising process, a threshold related to the removal of frequency components is determined. However, when the first noise generation unit 22 for learning determines the method for the first denoising process, it is only necessary to determine using the frequency components of the learning data, and it can also be a method other than the above.

[0119] In addition, as described above, the first denoising process can correspond to the type of imaging device related to the image to be denoised, the type of imaging object, or a combination thereof. Thus, as described in the explanation of the generation of the learning model, the learning images also used when determining the first denoising process can be set for each type of imaging device, each type of imaging object, or each combination thereof. For example, for the learning images (image group 1) of cell images and a lens magnification of 20 times of the imaging device, the learning images (image group 2) of cell images and a lens magnification of 40 times of the imaging device, the learning images (image group 3) of substrate images and a lens magnification of 20 times of the imaging device, and the learning images (image group 4) of substrate images and a lens magnification of 40 times of the imaging device, the first denoising process and the generation of the learning model are determined respectively, and when performing denoising, the first denoising process and the learning model corresponding to the image to be denoised are used.

[0120] The first noise generation unit 22 for learning determines the method of the first denoising process as follows, for example. The first noise generation unit 22 for learning performs FFT or the like on each learning image, generates the frequency components of the image, and calculates the spectral image. Figure 13 An example of the learning image 80 and an example of the spectral image 81 are shown. In the spectral image 81 calculated here, the closer to the center, the lower the frequency (the center is frequency 0), and as it moves away from the center, it becomes higher frequency. In order to reduce the influence of the image edges of the learning image, the first noise generation unit 22 for learning may apply a window function (Hanning window) to the learning image before generating the frequency components such as FFT.

[0121] The first noise generation unit 22 for learning calculates the average image of each spectral image calculated for each learning image. Figure 14 An example of the average image 82 is shown. The first noise generation unit 22 for learning averages the intensities of the frequency components in each direction from the center to the outside of the average image (from the center, 0 to 2π (360°)) for each frequency, and calculates the frequency components of each frequency that are independent of the direction in the spectral image. In Figure 14 the average image 82, one of these directions, the right direction, is shown by an arrow. Additionally, Figure 14 An example of the graph 83 of the frequency components of each calculated frequency is shown.

[0122] The first noise generation unit 22 for learning calculates the maximum frequency of the extracted signal spectrum based on the calculated frequency components. The maximum frequency of the signal spectrum is Figure 12 the target value of the cut-off frequency fc shown in the graph 71 of

[0123] For example, the first noise generation unit 22 for learning calculates the cut-off frequency fc as follows. The first noise generation unit 22 for learning calculates the rate of change of each frequency of the calculated frequency components. The rate of change at a certain frequency is a value calculated by (the intensity of the frequency component at a certain frequency - the intensity of the frequency component at the previous frequency) ÷ the intensity of the frequency component at the previous frequency. The frequency interval between a certain frequency and the previous frequency is preset. For example, this interval is the minimum interval of the frequencies in the frequency component data.

[0124] The first noise generation unit 22 for learning determines the frequency among the calculated rates of change of the frequency components that becomes below a preset threshold, and takes the smallest frequency among these frequencies as the cut-off frequency fc. The threshold is, for example, 0.002. The threshold can be changed. If the threshold is increased, the signal components contained in the difference image 32 are likely to increase, and if the threshold is decreased, the difference image 32 is less likely to contain the noise near the signal. Figure 15(a) shows a graph of an example of the rate of change of each frequency of the calculated frequency components. In this example, the cut-off frequency fc is 0.13.

[0125] In addition, the first noise generation unit 22 for learning can also calculate the cut-off frequency fc as follows. High frequencies hardly contain signal components. Therefore, the average value of the intensities of the frequency components with frequencies of 90% to 100% in the range of the frequencies of the calculated frequency components is assumed to be the intensity of the noise spectrum. The first noise generation unit 22 for learning calculates the average value of the intensities of the frequency components in the pre-set frequency range of the noise spectrum (90% to 100% in the above example). The first noise generation unit 22 for learning calculates a value obtained by multiplying the calculated average value by a pre-set multiple greater than 1, and among the frequencies at which the intensity of the frequency component is less than or equal to the calculated value, the maximum frequency is taken as the cut-off frequency fc. The above multiple is, for example, 1.008.

[0126] The pre-set frequency range of the noise spectrum can be changed. In cases where the range is expanded, such as 70% to 100% or 60% to 100%, the value of the average is likely to be stable, but the probability of the signal spectrum being mixed in increases. In addition, the above multiple can also be changed. If the multiple is increased, the signal components contained in the difference image 32 are likely to increase, and if the multiple is decreased, the noise near the signal is not easily included in the difference image 32. Figure 15 (b) shows a graph of an example of the calculated frequency components. In this example, the pre-set frequency range of the noise spectrum is 90 to 100% (the shaded range). The average value of the intensities of the noise spectrum frequency components is 413. The pre-set multiple is 1.008, and the product of the average value and the multiple is 416. Based on these, in this example, the cut-off frequency fc is 0.13.

[0127] The first noise generation unit 22 for learning determines the method of the first denoising process based on the calculated cut-off frequency. For example, when the above filter is used in the first denoising process, the filter used in the first denoising process is determined as follows. The first noise generation unit 22 for learning pre-stores a filter bank as a candidate for use in the first denoising process. As described above, the standard deviations of the weights of the filters contained in the filter bank are different from each other. Figure 16 The left side of shows an example of the filters included in the filter bank as a candidate for use in the first denoising process. The value of σ in the figure is the standard deviation corresponding to the filter.

[0128] The first noise generation unit 22 for learning performs FFT etc. on each filter in the same way as the processing for each learning image, generates the frequency components of the image, and calculates the spectral image. Figure 16Examples of spectral images corresponding to each filter are shown on the right side. In addition, at this time, in order to match the frequency resolution with the learning image before generating frequency components such as FFT, the first learning noise generation unit 22 may also perform zero-padding on each filter.

[0129] Similar to the processing of the average image of each spectral image calculated for each learning image, the first learning noise generation unit 22 calculates the frequency components of each frequency independent of direction based on the spectral images of each filter. Figure 17 Examples of curves showing the frequency components of each frequency corresponding to each filter are shown.

[0130] The first learning noise generation unit 22 calculates the filter cut-off frequency, which is the frequency at which the filter cuts off the signal, for each filter based on the calculated frequency components. For example, the first learning noise generation unit 22 calculates the rate of change of each frequency of the calculated frequency components in the same way as the processing using the frequency components of the learning image, and calculates the filter cut-off frequency.

[0131] Among the frequencies where the intensity of the frequency components is below a preset threshold, the first learning noise generation unit 22 takes the maximum frequency as the filter cut-off frequency. The threshold is, for example, 0.005. The threshold can be changed. If the threshold is increased, the signal components contained in the difference image 32 are likely to increase. If the threshold is decreased, the difference image 32 is less likely to contain noise near the signal. In addition, since the filter does not contain noise, the method of calculating the cut-off frequency using the frequency range of the assumed noise spectrum cannot be used.

[0132] Figure 17 Examples of the filter cut-off frequencies of each filter calculated by this method are shown. In this example, the filter cut-off frequencies are 0.14 for the filter with σ = 2.24, 0.13 for the filter with σ = 2.45, 0.11 for the filter with σ = 2.65, 0.1 for the filter with σ = 2.83, and 0.09 for the filter with σ = 3.00, respectively.

[0133] The first learning noise generation unit 22 compares the cut-off frequency fc calculated based on the learning image with the filter cut-off frequency of each filter, and determines (selects) the filter with the filter cut-off frequency closest to the cut-off frequency fc as the filter used in the first denoising process. For example, in Figure 15 when the shown cut-off frequency fc is 0.13, Figure 17 the filter with σ = 2.45 whose filter cut-off frequency in the shown filter is 0.13 is determined as the filter used in the first denoising process.

[0134] In addition, for example, in the case of generating frequency components by performing FFT (Fast Fourier Transform) or wavelet transform as the first denoising process, the learning first noise generation unit 22 uses the cut-off frequency fc calculated from the learning image as a threshold related to the removal of frequency components. The above is the function of the method for determining the first denoising process of the learning first noise generation unit 22.

[0135] In the case of adopting the method of determining the first denoising process, the learning first noise generation unit 22 performs the first denoising process by the determined method to generate learning first noise data. In addition, the determined method of the first denoising process is also notified to the denoising system 10 and is also used in the first denoising process in the denoising system 10.

[0136] By determining the method of the first denoising process as described above, it is possible to perform the first denoising process by an appropriate method such that the image 32 used for generating the first noise data is the above-described ideal image. As a result, more appropriate denoising can be performed.

[0137] Next, an example of denoising in the present embodiment is shown. Figure 18 An image 90 to be denoised used in the example is shown. In addition, Figure 18 Images 91, 92, and 93 input to the learning model are shown. Image 91 is a 1ch image input to the learning model and is an image obtained by quantifying the difference image between the image 90 to be denoised and the image after the first denoising process. Image 92 is a 2ch image input to the learning model and is an image obtained by quantifying the image 90 to be denoised. Image 93 is a 3ch image input to the learning model and is an image obtained by quantifying the difference image between the image 90 to be denoised and the image after the first denoising process.

[0138] The quantization for generating image 91 is quantization (ordinary normalization) within the range where the luminance values of the difference image do not overflow or underflow. The quantization for generating image 93 is quantization within the range where the luminance values of a part of the difference image are allowed to overflow. The role of the quantization for generating image 93 is to input the noise components into the learning model with good accuracy in the case where the luminance values of the difference image cannot be represented with the accuracy of the quantization for generating image 91.

[0139] Figure 19 (a) shows the standard deviation of the luminance values of each part of the image. The part of the image is the part of the rectangle indicated by parentheses in the Figure 18 image 90 to be denoised. In this example and the comparative example, the resolution of the image 90 to be denoised is 32-bit floating point (FP32), and the resolution of the image input to the learning model is 8-bit integer (Int8) after quantization.Figure 19 The numerical values in the parentheses in the table of (a) correspond to the respective parts. Figure 19 The "original image" in (a) is the image 90 of the object to be denoised. "Int8 normal signal inference" is obtained by quantifying the image 90 of the object to be denoised and inferring the denoised image through the learning model. "Int8 normal noise inference" is obtained by quantifying the image 90 of the object to be denoised and inferring the noise through the learning model. "Int8 normal signal inference" and "Int8 normal noise inference" are comparative examples for comparison with the present embodiment.

[0140] "Int8 1-ch noise inference (Gaussian σ = 1.8)" is obtained by using the 1-ch image 91 as the input of the learning model to perform denoising of the present embodiment. Additionally, in this example, a Gaussian filter with σ = 1.8 was used in the first denoising process. The same applies hereinafter. "Int8 2-ch noise inference (Gaussian σ = 1.8)" is obtained by using the 1-ch image 91 and the 2-ch image 92 as the inputs of the learning model to perform denoising of the present embodiment. "Int8 3-ch noise inference (Gaussian σ = 1.8)" is obtained by using the 1-ch image 91, the 2-ch image 92, and the 3-ch image 93 as the inputs of the learning model to perform denoising of the present embodiment.

[0141] Figure 19 The smaller the standard deviation value in the table of (a) indicates that the less noise is included. As shown in this table, compared with the case of only quantifying and denoising the image 90 of the object to be denoised, the denoising of the present embodiment can perform denoising appropriately. Additionally, by inputting more images into the learning model, denoising can be performed more appropriately. Furthermore, the denoised image obtained by "Int8 normal signal inference" has a large degradation in gray scale due to quantization, so it is unacceptable in terms of image quality.

[0142] Figure 19 (b) shows the standard deviation of the brightness values of each part of the image. In this embodiment and the comparative example, the resolution of the image 90 of the object to be denoised and the resolution of the image input to the learning model are both 32-bit floating point (FP32). That is, the image input to the learning model is not quantized. Figure 19 "FP32 normal noise inference" in (b) is obtained by inferring the noise from the image 90 of the object to be denoised through the learning model. "FP32 2-ch noise inference (Gaussian σ = 1.8)" is obtained by using the non-quantized images of the 1-ch image 91 and the 2-ch image 92 as the inputs of the learning model to perform denoising of the present embodiment. As shown in this table, even without quantization, the denoising of the present embodiment can perform denoising appropriately.Figures 20 to 22 Examples of other images corresponding to the respective cases shown in the table of Figure 19 .

[0143] In addition, in the case where an AI - dedicated chip is not assumed to be used and image quantization is not performed to generate a learning model, an improvement in speed can be observed. Using the Figure 23 shown image, denoising is performed using learning models with different numbers of machine - learning repetitions, and the standard deviation of a part of the image in this case ( Figure 23 the part represented by a rectangle in the image of Figure 23 ) is shown in the curve graph and table of Figure 23 The "normal learning" shown in Figure 23 is used to generate a learning model for inferring noise based on the image to be denoised. The "differential 2 - ch learning" is used to generate a learning model for inferring noise based on the pre - quantization images of the above - mentioned 1 - ch image 91 and 2 - ch image 92. As shown in this table, the "differential 2 - ch learning" can appropriately perform denoising with fewer learning times.

[0144] Next, a denoising program and a learning program for performing the above - described processes of the denoising system 10 and the learning system 20 will be described. As shown in Figure 24 (a) of Figure 24 , the denoising program 100 is stored in a program storage area 111 formed in a computer - readable recording medium 110, and this recording medium 110 is inserted into a computer and accessed or is possessed by the computer. The recording medium 110 may also be a non - transitory recording medium.

[0145] The denoising program 100 includes an acquisition module 101, a first noise generation module 102, a second noise generation module 103, and a denoising module 104. The functions realized by causing the acquisition module 101, the first noise generation module 102, the second noise generation module 103, and the denoising module 104 to execute are the same as the functions of the acquisition unit 11, the first noise generation unit 12, the second noise generation unit 13, and the denoising unit 14 of the above - described denoising system 10, respectively.

[0146] As shown in Figure 24 (b) of Figure 24 , the learning program 200 is stored in a program storage area 211 formed in a computer - readable recording medium 210, and this recording medium 210 is inserted into a computer and accessed or is possessed by the computer. The recording medium 210 may also be a non - transitory recording medium. In addition, when the denoising program 100 and the learning program 200 are executed by the same computer, the recording medium 210 may also be the same as the recording medium 110.

[0147] The learning program 200 is configured to include a learning acquisition module 201, a first learning noise generation module 202, and a learning module 203. The functions realized by executing the learning acquisition module 201, the first learning noise generation module 202, and the learning module 203 are respectively the same as the functions of the learning acquisition unit 21, the first learning noise generation unit 22, and the learning unit 23 of the learning system 20 described above.

[0148] In addition, the denoising program 100 and the learning program 200 may also be configured such that part or all of them are transmitted via a transmission medium such as a communication line and received and recorded (including installed) by other devices. In addition, each module of the denoising program 100 and the learning program 200 may not be installed on one computer but on any of multiple computers. In this case, the above-described series of processes are performed by a computer system constituted by the multiple computers.

[0149] The denoising method, denoising program, denoising system, and learning method of the present disclosure have the following configurations. [1] A denoising method for removing noise from data having multiple values, including: An acquisition step of acquiring denoising target data having multiple values and being a denoising target; A first noise generation step of performing a first denoising process on the denoising target data acquired in the acquisition step to generate first noise data related to the noise removed by the first denoising process; A second noise generation step of inputting the first noise data generated in the first noise generation step into a learning model generated in advance by machine learning to generate second noise data, where the learning model is used to generate second noise data related to the noise to be removed from the denoising target data based on the first noise data; and A denoising step of using the second noise data generated in the second noise generation step to generate denoised data based on the denoising target data acquired in the acquisition step. [2] The denoising method according to [1], where in the first noise generation step, the data obtained by the first denoising process is quantized to generate first noise data, in the denoising step, the second noise data generated in the second noise generation step is inverse-quantized, and denoised data is generated based on the inverse-quantized data. [3] The denoising method according to [2], where in the first noise generation step, quantization is performed for each of multiple ranges of the magnitudes of the multiple values to generate multiple first noise data, In the second noise generation step, a plurality of first noise data generated in the first noise generation step are input into the learning model to generate the second noise data. [4] The denoising method according to any one of [1] to [3], wherein the processing of the first noise generation step and the second noise generation step are performed by processing devices having different performances from each other. [5] The denoising method according to any one of [1] to [4], wherein In the first noise generation step, as the first denoising process, a preset filter for performing the first denoising process is applied to the data to be denoised to generate the first noise data. [6] The denoising method according to any one of [1] to [5], wherein In the first noise generation step, as the first denoising process, a process of removing frequency components within a preset range is performed on the data to be denoised to generate the first noise data. [7] The denoising method according to any one of [1] to [6], wherein In the first noise generation step, size data reflecting the magnitudes of a plurality of values of the data to be denoised is also generated, and in the second noise generation step, the first noise data and the size data generated in the first noise generation step are input into the learning model to generate the second noise data. [8] The denoising method according to any one of [1] to [7], wherein the data to be denoised is image data, spectral data, or time-series data. [9] A denoising program for causing a computer to remove noise from data having a plurality of values, wherein the computer functions as the following units: an acquisition unit that acquires data to be denoised having a plurality of values and serving as an object of denoising; a first noise generation unit that performs a first denoising process on the data to be denoised acquired by the acquisition unit to generate first noise data related to the noise removed by the first denoising process; a second noise generation unit that inputs the first noise data generated by the first noise generation unit into a learning model generated in advance by machine learning to generate second noise data, where the learning model is used to generate, based on the first noise data, second noise data related to the noise to be removed from the data to be denoised; and A denoising unit that uses the second noise data generated by the second noise generation unit to generate denoised data based on the data to be denoised obtained by the obtaining unit.

[10] A denoising system that removes noise from data having multiple values, comprising: An obtaining unit that obtains data to be denoised having multiple values and being an object of denoising; A first noise generation unit that performs a first denoising process on the data to be denoised obtained by the obtaining unit and generates first noise data related to the noise removed by this first denoising process; A second noise generation unit that inputs the first noise data generated by the first noise generation unit into a learning model generated in advance by machine learning to generate second noise data, where this learning model is used to generate second noise data related to the noise to be removed from the data to be denoised based on this first noise data; and A denoising unit that uses the second noise data generated by the second noise generation unit to generate denoised data based on the data to be denoised obtained by the obtaining unit.

[11] A learning method for generating a learning model used to remove noise from data having multiple values, comprising: A learning data obtaining step of obtaining learning data having multiple values and containing noise; A first learning noise generation step of performing a first denoising process on the learning data obtained in the learning data obtaining step and generating first learning noise data related to the noise removed by this first denoising process; and A learning step of using the first learning noise data generated in the first learning noise generation step as an input to the learning model and performing machine learning to generate the learning model.

[12] The learning method according to claim 11, wherein in the first learning noise generation step, the frequency components of the learning data obtained in the learning data obtaining step are generated, and the method of the first denoising process is determined based on the generated frequency components. Description of Reference Numerals

[0150] 10… Denoising system, 11… Acquisition unit, 12… First noise generation unit, 13… Second noise generation unit, 14… Denoising unit, 20… Learning system, 21… Acquisition unit for learning, 22… First noise generation unit for learning, 23… Learning unit, 100… Denoising program, 110… Recording medium, 111… Program storage area, 101… Acquisition module, 102… First noise generation module, 103… Second noise generation module, 104… Denoising module, 200… Learning program, 210… Recording medium, 211… Program storage area, 201… Acquisition module for learning, 202… First noise generation module for learning, 203… Learning module.

Claims

1. A denoising method for removing noise from data having multiple values, comprising: An acquisition step of acquiring denoising target data having multiple values and being a denoising target; A first noise generation step of performing a first denoising process on the denoising target data acquired in the acquisition step to generate first noise data related to the noise removed by the first denoising process; A second noise generation step of inputting the first noise data generated in the first noise generation step into a learning model pre-generated by machine learning to generate second noise data, where the learning model is used to generate second noise data related to the noise to be removed from the denoising target data based on the first noise data; And A denoising step of using the second noise data generated in the second noise generation step to generate denoised data based on the denoising target data acquired in the acquisition step.

2. The denoising method according to claim 1, wherein In the first noise generation step, the data obtained by the first denoising process is quantized to generate the first noise data, In the denoising step, the second noise data generated in the second noise generation step is inverse-quantized, and the denoised data is generated based on the inverse-quantized data.

3. The denoising method according to claim 2, wherein In the first noise generation step, quantization is performed for each of a plurality of ranges of the magnitudes of the multiple values to generate a plurality of first noise data, In the second noise generation step, the plurality of first noise data generated in the first noise generation step are input into the learning model to generate the second noise data.

4. The denoising method according to claim 1 or 2, wherein The processing of the first noise generation step and the second noise generation step are performed by processing devices having different performances from each other.

5. The denoising method according to claim 1 or 2, wherein In the first noise generation step, as the first denoising process, a preset filter for performing the first denoising process is applied to the denoising target data to generate the first noise data.

6. The denoising method according to claim 1 or 2, wherein In the first noise generation step, as the first denoising process, the frequency components within a preset range are removed from the denoising target data to generate the first noise data.

7. The denoising method according to claim 1 or 2, wherein In the first noise generation step, magnitude data reflecting the magnitudes of the multiple values of the denoising target data is also generated, In the second noise generation step, the first noise data and the magnitude data generated in the first noise generation step are input into the learning model to generate the second noise data.

8. The denoising method according to claim 1 or 2, wherein The denoising target data is image data, spectral data, or time-series data.

9. A denoising program for causing a computer to remove noise from data having multiple values, wherein The computer functions as the following units: An acquisition unit that acquires denoising target data having multiple values and being a denoising target; A first noise generation unit that performs a first denoising process on the denoising target data obtained by the obtaining unit and generates first noise data related to the noise removed by the first denoising process; A second noise generation unit that inputs the first noise data generated by the first noise generation unit into a learning model pre-generated by machine learning to generate second noise data, where the learning model is used to generate second noise data related to the noise to be removed from the denoising target data based on the first noise data; and A denoising unit that uses the second noise data generated by the second noise generation unit to generate denoised data based on the denoising target data obtained by the obtaining unit.

10. A denoising system for removing noise from data having multiple values, comprising: An obtaining unit that obtains denoising target data having multiple values and being a denoising target; A first noise generation unit that performs a first denoising process on the denoising target data obtained by the obtaining unit and generates first noise data related to the noise removed by the first denoising process; A second noise generation unit that inputs the first noise data generated by the first noise generation unit into a learning model pre-generated by machine learning to generate second noise data, where the learning model is used to generate second noise data related to the noise to be removed from the denoising target data based on the first noise data; and A denoising unit that uses the second noise data generated by the second noise generation unit to generate denoised data based on the denoising target data obtained by the obtaining unit.

11. A learning method for generating a learning model for use in removing noise from data having multiple values, comprising: A learning data obtaining step of obtaining learning data having multiple values and including noise; A first learning noise generation step of performing a first denoising process on the learning data obtained in the learning data obtaining step and generating first learning noise data related to the noise removed by the first denoising process; and A learning step of using the first learning noise data generated in the first learning noise generation step as an input to the learning model and performing machine learning to generate the learning model.

12. The learning method according to claim 11, wherein in the first learning noise generation step, the frequency components of the learning data obtained in the learning data obtaining step are generated, and the method of the first denoising process is determined based on the generated frequency components.

Images