Learning device, noise removal device, and program

Abstract

To acquire a learning device capable of performing learning of a noise remover even in the case where learning data with less noise than a normal photographed image cannot be acquired, and in the case where the amount of the noise is unknown in learning data having the same noise as the normal photographed image, to acquire a noise removal device and to acquire a program.SOLUTION: A learning device 30 includes: a noise removal part 31A for performing noise removal with respect to first learning data using a noise remover that can learn; a noise addition part 31B for adding predetermined addition noise with respect to the first learning data in which noise removal has been performed; and a learning part 31C for learning the noise remover so as to bring distribution of the first learning data in which addition noise has been added closer to distribution of second learning data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a learning device, a noise removal device, and a program. [Background technology]

[0002] Improving image visibility by removing noise is an important issue not only for natural images such as photographs, but also in the field of medical images.

[0003] Meanwhile, in recent years, many noise removal methods using deep learning have been proposed, and a technique is known in which a noise remover is trained to convert noisy image data into low-noise image data using image data with little or no noise as training data (for example, see Non-Patent Document 1).

[0004] However, this technology cannot be applied in situations where image data with less noise than normal imaging cannot be obtained. For example, in radiography of the human body, image data with less noise can be obtained by increasing the dose compared to normal imaging, but this increases the radiation exposure, making it extremely difficult to obtain in practice.

[0005] In order to address this problem, a technique has been proposed in which a noise remover is trained using only image data having noise similar to that of a normal photographed image as training image data (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising (Zhang+, IEEE Transactions on Image Processing) [Non-Patent Document 2] Recorrupted-to-Recorrupted: Unsupervised Deep Learning for Image Denoising (Pang+, IEEE / CVF Computer Vision and Pattern Recognition 2021) Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technique of training a noise remover using only image data having the same noise as a normal photographed image as the training image data has a problem in that the amount of noise in each piece of training image data must be known. Note that, hereinafter, the image data used to train the noise remover is referred to as "training data."

[0008] The present disclosure has been made in consideration of the above circumstances, and aims to provide a learning device, a noise removal device, and a program that are capable of training a noise remover even when learning data with less noise than normally captured images cannot be obtained, or when the amount of noise in learning data that has noise similar to that of normally captured images is unknown. [Means for solving the problem]

[0009] In order to achieve the above object, a learning device of a first aspect of the present disclosure includes at least one processor, which performs noise removal on first learning data using a trainable noise remover, adds predetermined additional noise to the first learning data from which noise has been removed, and trains the noise remover so as to bring the distribution of the first learning data to which the additional noise has been added closer to the distribution of the second learning data.

[0010] A learning device according to a second aspect of the present disclosure is the learning device according to the first aspect, wherein the second learning data is the same data as the first learning data.

[0011] A learning device according to a third aspect of the present disclosure is the learning device according to the first aspect, wherein the additional noise is noise that imitates noise contained in the first learning data.

[0012] A learning device according to a fourth aspect of the present disclosure is the learning device according to the first aspect, in which the noise remover is a deep learning model.

[0013] A learning device according to a fifth aspect of the present disclosure is the learning device according to the first aspect, wherein data from which noise is removed by the noise remover is image data.

[0014] A learning device according to a sixth aspect of the present disclosure is the learning device according to the fifth aspect, wherein the image data is a radiation image.

[0015] A seventh aspect of the present disclosure is a learning device in which, in the learning device of the second aspect, a processor trains a noise remover so that a distribution of first learning data to which additional noise has been added cannot be distinguished from a distribution of second learning data using a predetermined classifier.

[0016] The learning device of an eighth aspect of the present disclosure is the learning device of the second aspect, wherein the processor trains a noise remover so as to minimize the deviation between the distribution of the first learning data to which additional noise has been added and the distribution of the second learning data.

[0017] A learning device according to a ninth aspect of the present disclosure is the learning device according to the eighth aspect, wherein the degree of deviation is a value based on the Wasserstein distance or an amount of information.

[0018] In addition, in order to achieve the above object, a noise removal device of a tenth aspect of the present disclosure includes a noise remover trained by a learning device of any one of the first to ninth aspects, and an acquisition unit that inputs predetermined data to the noise remover to acquire data from which noise contained in the data has been removed.

[0019] Furthermore, in order to achieve the above-mentioned object, a program of an eleventh aspect of the present disclosure causes a computer to execute a process of performing noise removal on first training data using a trainable noise remover, adding a predetermined additional noise to the first training data from which noise has been removed, and training the noise remover so as to bring the distribution of the first training data to which the additional noise has been added closer to the distribution of the second training data. Effect of the Invention

[0020] According to the present disclosure, it is possible to train a noise remover even when it is not possible to obtain training data with less noise than a normally captured image, or when the amount of noise in training data that has noise similar to that of a normally captured image is unknown. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a configuration of an electrical system of the server according to the embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of the configuration of an electrical system of a learning device according to an embodiment. [Figure 4] 2 is a block diagram showing an example of a configuration of an electrical system of the noise removal device according to the embodiment; FIG. [Diagram 5] 1 is a functional block diagram showing the functional configuration of a learning device and a noise removal device in an image processing system according to an embodiment. [Figure 6] FIG. 1 is a block diagram for explaining a conventional technique. [Figure 7] FIG. 2 is a block diagram illustrating a learning process according to the first embodiment. [Figure 8]FIG. 1 is a block diagram showing an example of a usage form of a Wasserstein GAN (Generative Adversarial Network)-GP (with Gradient Penalty) (hereinafter also referred to as "WGAN-GP") according to an embodiment. [Figure 9] 1 is a graph illustrating the Wasserstein distance (hereinafter also referred to as "W distance") according to an embodiment. [Figure 10] FIG. 2 is a block diagram for more specifically explaining the learning process according to the first embodiment. [Figure 11] FIG. 2 is a schematic diagram illustrating an example of a configuration of a learning data database according to the embodiment. [Figure 12] FIG. 2 is a schematic diagram showing an example of a configuration of an image database according to the embodiment. [Figure 13] 10 is a flowchart illustrating an example of a learning process according to the embodiment. [Figure 14] 10 is a flowchart illustrating an example of a noise removal process according to the embodiment. [Figure 15] FIG. 11 is a block diagram illustrating a learning process according to a second embodiment. [Figure 16] FIG. 13 is a block diagram illustrating a learning process according to a third embodiment. [Figure 17] 11A and 11B are diagrams for explaining the effect of the learning process according to the embodiment. [Figure 18] 11A and 11B are diagrams for explaining the effect of the learning process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0023] In this embodiment, a learning device that learns a noise remover that removes noise from various medical image data will be described as the learning device of the technology disclosed herein. However, the present invention is not limited to this form, and for example, the learning device of the technology disclosed herein may be applied as a learning device that learns a noise remover that removes noise from images other than medical images, such as image data obtained by shooting with a visible light camera or image data obtained by shooting with an infrared camera. In addition, the learning device of the technology disclosed herein may be applied as a learning device that learns a noise remover that removes noise from audio data such as ultrasound data and audible sound data.

[0024] [First embodiment] First, the configuration of an image processing system 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the image processing system 1 according to this embodiment.

[0025] 1, an image processing system 1 according to the present embodiment includes a server 10 configured as a cloud server or the like, a learning device 30 as a learning device of the present disclosure, and a noise removal device 50 as a noise removal device of the present disclosure. Examples of the server 10, the learning device 30, and the noise removal device 50 include information processing devices such as a personal computer and a server computer.

[0026] The server 10 according to the present embodiment stores learning data used for learning the noise remover by the learning device 30, which will be described later in detail, medical image data to be subjected to actual noise removal by the noise removal device 50, the noise remover, and a noise generator, which will be described later in detail. In this manner, in the image processing system 1 according to the present embodiment, the noise remover and the noise generator are stored in the server 10, but the present embodiment is not limited to this form. For example, the noise remover and the noise generator may be stored in the learning device 30 or the noise removal device 50. Furthermore, a device accessible from the learning device 30 and the noise removal device 50 may be used as a device for storing the noise remover and the noise generator.

[0027] The server 10, the learning device 30, and the noise removal device 50 are connected via a network N, and the server 10, the learning device 30, and the noise removal device 50 are capable of communicating with each other via the network N.

[0028] In this embodiment, the network N is a combination of a public communication line such as the Internet or a telephone network, and a communication line within the hospital such as a LAN (Local Area Network) or a WAN (Wide Area Network), but is not limited to this form. For example, the network N may be a form in which only one of the public communication line and the communication line within the hospital is applied. In addition, in this embodiment, the network N is a combination of a wired and a wireless communication line, but is not limited to this form. The network N may be a form in which only one of a wireless communication line and a wired communication line is applied.

[0029] Next, the configuration of the server 10 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of an electrical system of the server 10 according to this embodiment.

[0030] 2, the server 10 according to this embodiment includes a CPU (Central Processing Unit) 11 as a processor, a memory 12 as a temporary storage area, a non-volatile storage unit 13, an input unit 14 such as a keyboard and a mouse, a display unit 15 such as a liquid crystal display, a medium reading and writing device (R / W) 16, and a communication interface (I / F) unit 18. The CPU 11, the memory 12, the storage unit 13, the input unit 14, the display unit 15, the medium reading and writing device 16, and the communication interface (I / F) unit 18 are connected to each other via a bus B1. The medium reading and writing device 16 reads information written in a recording medium 17 and writes information to the recording medium 17.

[0031] The storage unit 13 according to the present embodiment is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 13 as a storage medium stores a learning data database 13A and an image data database 13B. The learning data database 13A and the image data database 13B will be described in detail later.

[0032] The storage unit 13 also stores the above-mentioned noise remover 90 and noise generator 92. The noise remover 90 according to the present embodiment is configured as a deep learning model, but is not limited thereto. For example, the noise remover 90 may be configured to apply another noise remover that involves learning, such as a bilateral filter. When a bilateral filter is used as the noise remover 90, the learning device 30 described later learns the parameters of the bilateral filter.

[0033] Next, the configuration of the learning device 30 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of an electrical system of the learning device 30 according to this embodiment.

[0034] 3, the learning device 30 according to this embodiment includes a CPU 31 as a processor, a memory 32 as a temporary storage area, a non-volatile storage unit 33, an input unit 34 such as a keyboard and a mouse, a display unit 35 such as a liquid crystal display, a medium reading and writing device (R / W) 36, and a communication I / F unit 38. The CPU 31, the memory 32, the storage unit 33, the input unit 34, the display unit 35, the medium reading and writing device 36, and the communication I / F unit 38 are connected to each other via a bus B2. The medium reading and writing device 36 reads information written in a recording medium 37 and writes information to the recording medium 37.

[0035] The storage unit 33 according to this embodiment is realized by an HDD, an SSD, a flash memory, or the like. A learning processing program 33A is stored in the storage unit 33 as a storage medium. The learning processing program 33A is stored (installed) in the storage unit 33 when a recording medium 37 on which the program 33A is written is connected to the medium reading and writing device 36, and the medium reading and writing device 36 reads the program 33A from the recording medium 37. The CPU 31 appropriately reads the learning processing program 33A from the storage unit 33, expands it in the memory 32, and sequentially executes the processes of the program 33A.

[0036] Next, the configuration of the noise removal device 50 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of an electrical system of the noise removal device 50 according to this embodiment.

[0037] 4, the noise removal device 50 according to this embodiment includes a CPU 51 as a processor, a memory 52 as a temporary storage area, a non-volatile storage unit 53, an input unit 54 such as a keyboard and a mouse, a display unit 55 such as a liquid crystal display, a medium reading and writing device (R / W) 56, and a communication I / F unit 58. The CPU 51, the memory 52, the storage unit 53, the input unit 54, the display unit 55, the medium reading and writing device 56, and the communication I / F unit 58 are connected to each other via a bus B3. The medium reading and writing device 56 reads information written in a recording medium 57 and writes information to the recording medium 57.

[0038] The storage unit 53 according to the present embodiment is realized by an HDD, an SSD, a flash memory, or the like. A noise removal program 53A is stored in the storage unit 53 as a storage medium. The noise removal program 53A is stored (installed) in the storage unit 53 when a recording medium 57 in which the program 53A is written is connected to the medium reading and writing device 56, and the medium reading and writing device 56 reads the program 53A from the recording medium 57. The CPU 51 appropriately reads the noise removal program 53A from the storage unit 53, expands it in the memory 52, and sequentially executes the processes of the program 53A.

[0039] Next, the functional configurations of the learning device 30 and the noise removal device 50 in the image processing system 1 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the functional configurations of the learning device 30 and the noise removal device 50 in the image processing system 1 according to this embodiment.

[0040] 5, the learning device 30 according to this embodiment includes a noise removal unit 31A, a noise addition unit 31B, and a learning unit 31C. The CPU 31 of the learning device 30 executes a learning processing program 33A to function as the noise removal unit 31A, the noise addition unit 31B, and the learning unit 31C.

[0041] The noise removal unit 31A according to this embodiment removes noise from the first learning data using a trainable noise remover 90. Furthermore, the noise adding unit 31B according to this embodiment adds a predetermined additional noise to the first learning data from which noise has been removed by the noise remover 90. Then, the learning unit 31C according to this embodiment trains the noise remover 90 so as to bring the distribution of the first learning data to which the additional noise has been added by the noise adding unit 31B closer to the distribution of the second learning data.

[0042] In this embodiment, the same data as the first learning data is applied as the second learning data, but this is not limited to this, and learning data different from the first learning data may be applied as the second learning data.

[0043] In addition, in the present embodiment, the noise to be added is a noise that imitates the noise contained in the first learning data, but the present invention is not limited to this. For example, a noise equivalent to a radiation noise generated from a device adjacent to the image capturing device that acquired the first learning data may be applied as the additional noise, regardless of the noise contained in the first learning data.

[0044] Furthermore, the learning unit 31C according to this embodiment trains the noise remover 90 so that a distribution of the first learning data to which the additional noise has been added cannot be distinguished from a distribution of the second learning data by a predetermined discriminator. In particular, the learning unit 31C according to this embodiment trains the noise remover 90 so that the degree of deviation between the distribution of the first learning data to which the additional noise has been added and the distribution of the second learning data is minimized.

[0045] In this embodiment, the W distance is used as the divergence, but the present invention is not limited to this. For example, a value based on the amount of information, such as Kullback-Leibler Divergence, may be used as the divergence.

[0046] When W-distance is used as the deviation measure, GAN or a model similar to GAN will be used, and when Kullback-Leibler divergence is applied, a diffusion model or an auto-regressive model will be used. An example of an auto-regressive model is the PixelCNN autoencoder.

[0047] When using GANs or GAN-like models, inference can be performed quickly, but learning is somewhat unstable. In contrast, when using diffusion models or autoregressive models, inference is somewhat slow, but learning is stable.

[0048] As the deviation, it is preferable to use not only the W distance or the Kullback-Leibler divergence but also a measure suitable for a model that can be used to compare original data with data obtained by removing noise from the original data and then adding noise back.

[0049] Meanwhile, the noise removal device 50 according to the present embodiment includes an acquisition unit 51 A. The CPU 51 of the noise removal device 50 executes the noise removal program 53A to function as the acquisition unit 51A.

[0050] The acquisition unit 51A in this embodiment acquires data from which noise contained in predetermined data (in this embodiment, medical image data) has been removed by inputting the predetermined data to the noise remover 90 learned by the learning device 30.

[0051] Next, a learning method of the noise remover 90 by the learning device 30 according to this embodiment will be described with reference to Figs. 6 to 10. Fig. 6 is a block diagram for explaining the conventional technique, and Fig. 7 is a block diagram for explaining the learning process according to this embodiment. Fig. 8 is a block diagram showing an example of a usage form of the WGAN-GP according to this embodiment, and Fig. 9 is a graph for explaining the W-distance according to this embodiment. Furthermore, Fig. 10 is a block diagram for more specifically explaining the learning process according to this embodiment.

[0052] As an example, as shown in FIG. 6, when training a noise remover 90 to remove noise from input image data to obtain image data with less noise than the input image data, image data with less noise or no noise is required as ground truth data.

[0053] However, it is difficult to obtain such correct data because, when dealing with radiological images, it is necessary to expose the human body to high doses of radiation, and when dealing with infrared images, there is a lot of thermal noise, etc. For example, to reduce the noise in a radiological image to one-tenth, a 100-fold dose of radiation is required, resulting in an extremely high radiation exposure.

[0054] Therefore, as shown in FIG. 7 as an example, in the learning device 30 according to the present embodiment, first, the noise removing unit 31A removes noise from the learning data 60 using the noise remover 90 to be learned, thereby generating the noise-removed learning data 62. Next, in the learning device 30 according to the present embodiment, the noise adding unit 31B adds noise to the noise-removed learning data 62 using the noise generator 92, thereby generating the noise-removed learning data 64. Then, in the learning device 30 according to the present embodiment, the learning unit 31C trains the noise remover 90 so as to bring the distribution of the noise-removed learning data 64 closer to the distribution of the learning data 60. Note that, in the present embodiment, as the noise remover 90, U-Net, which is a model for semantic segmentation and is a type of FCN (Fully Convolution Network), is applied, but is not limited thereto. For example, a convolutional autoencoder, a PixelCNN autoencoder, or the like may be applied as the noise remover 90.

[0055] In the learning device 30 according to this embodiment, as a learning method for bringing the distribution of the training data 64 after noise removal addition closer to the distribution of the training data 60, a method is applied in which the noise remover 90 is trained so as to minimize the distance between the image distribution of the training data 64 after noise removal addition and the image distribution of the training data 60 (in FIG. 7, this is indicated as “loss”).

[0056] In the learning device 30 according to the present embodiment, the learning of the noise remover 90 is performed using WGAN-GP.

[0057] That is, as shown in FIG. 8 as an example, in the learning device 30 according to this embodiment, the image distribution of the original learning data (in FIG. 8, this is indicated as "original image distribution"), which is in a scattered state due to the influence of noise, is removed by the noise remover 90, thereby obtaining noise-removed learning data 62 in which the image distribution in the data space is thinned (sparsed).

[0058] Next, in the learning device 30 according to this embodiment, noise is added to the post-noise-removal training data 62 in which the image distribution has been made sparse by the noise generator 92. This results in post-noise-removal-and-addition training data 64 in which the image distribution has been converted back into a scattered state by adding the noise component.

[0059] When the image distribution of the post-noise-removal-added training data 64 obtained as described above is brought closest to the image distribution of the original training data 60, the image distribution of the post-noise-removal-added training data 62 becomes the most thinned state, as shown in Fig. 8 as an example. Therefore, in the training device 30 according to this embodiment, the noise remover 90 is trained so that the image distribution of the post-noise-removal-added training data 64 cannot be distinguished from the image distribution of the original training data 60.

[0060] Specifically, the learning device 30 according to this embodiment uses the WGAN-GP method to derive the W distance from f (learning data 60)-f (learning data 64 after noise removal) using a discriminator f, and trains the noise remover 90 so as to minimize the W distance.

[0061] There is a theorem (Kantorovich-Rubinstein theorem) that when f is optimized and maximized for the value of f(learning data 60)-f(learning data 64 after noise reduction is added) by providing a constraint that the absolute value of the gradient of the output with respect to the input of the classifier f is less than 1 (-1≦gradient≦1), f(learning data 60)-f(learning data 64 after noise reduction is added) becomes the W distance. Therefore, the W distance is obtained by maximizing the above value with the classifier f, and the learning unit 31C according to this embodiment trains the noise remover 90 so as to minimize the W distance.

[0062] As an example, the W-distance corresponds to the total distance traveled when transporting "mountain soil" from left to right, as shown in Figure 9, and is equal to the following equation (1) when the absolute value of the maximum slope of f(x) is limited to 1 (strong duality theorem for linear programming problems).

[0063]

number

[0064] Then, as shown in the following equation (2), if ν is optimized to minimize W, μ approaches ν.

[0065]

number

[0066] This is the gist of WGAN, but in the learning device 30 according to this embodiment, the learning data 60 is converted into noise-removed and added learning data 64 within the framework of WGAN-GP shown in FIG.

[0067] In this way, in the learning device 30 according to the present embodiment, the noise remover 90 is trained using WGAN-GP, but the present invention is not limited to this. For example, as described above, the diffusion model may be applied to the training of the noise remover 90.

[0068] In addition, in the learning device 30 according to the present embodiment, as shown in FIG. 10, the noise generator 92 generates a noise image 80 unique to the device for each type of image targeted by the noise remover 90 by simulating noise occurring in the device that captured the image or in a device of the same type as the device. Then, the noise generator 92 according to the present embodiment adds the generated noise image 80 to the post-noise-removal learning data 62. For example, when the type of image targeted by the noise remover 90 is a CT (Computed Tomography) image, the noise generator 92 generates a noise image 80 by simulating noise occurring in the CT device that captured the CT image or in a CT device of the same type as the CT device, and adds the noise image 80 to the post-noise-removal learning data 62. As a method of this simulation, a method of simulating by modeling the noise generation mechanism of the target device is exemplified.

[0069] In this manner, in this embodiment, a noise image obtained by simulation according to the type of target image is applied as the noise image 80 to be added to the post-noise-removal learning data 62, but this is not limited to this.

[0070] For example, if Gaussian noise is to be removed from a target image, Gaussian noise can be applied as the noise image 80, and if Poisson noise is to be removed, Poisson noise can be applied as the noise image 80. An example of a case where Gaussian noise is to be removed is when noise caused by thermal noise occurring in many sensors is to be removed, and an example of a case where Poisson noise is to be removed is when shot noise occurring in radiological images, etc. Furthermore, if the noise to be removed cannot be expressed by simple statistical noise, the noise may be trained in some model, such as a diffusion model, and applied as the noise generator 92.

[0071] As described above, the image processing system 1 according to this embodiment applies medical images as target images. The medical images referred to here include the above-mentioned CT images, as well as simple radiation images, infrared camera images, ultrasonic echo images, sinograms (original signals before reconstruction of CT images), endoscopic images, and the like.

[0072] The above-mentioned plain radiographic images are images from which it is difficult to obtain high-quality image data, similar to CT images, and the above-mentioned infrared camera images are images from which it is difficult to obtain high-quality image data, since infrared rays emitted by objects at room temperature are weak and high-resolution sensors tend to have a lot of noise when used. Also, the above-mentioned ultrasonic echo images are greatly affected by random electrical noise from the sensor, since ultrasonic waves are attenuated as they propagate through organs deep inside the body, making it difficult to obtain signals with little noise, and the above-mentioned sinograms are also images from which it is difficult to obtain high-quality image data, similar to CT images.

[0073] In the image processing system 1 according to the present embodiment, the noise remover 90 and the noise generator 92 are provided for each type of image to be subjected to noise removal, but the present invention is not limited to this. For example, only one noise remover 90 and one noise generator 92 may be provided corresponding to a plurality of types of images.

[0074] Next, the learning data database 13A according to this embodiment will be described with reference to Fig. 11. Fig. 11 is a schematic diagram showing an example of the configuration of the learning data database 13A according to this embodiment.

[0075] The learning data database 13A according to this embodiment is a database in which the above-mentioned learning data 60 is registered, and as an example, as shown in FIG. 11, each piece of information such as an image type, an image ID (Identification), and the learning data is stored in association with each other.

[0076] The image type is information indicating the type of the corresponding learning data, the image ID is information that is predetermined to be different for each piece of the corresponding learning data in order to individually identify the learning data, and the learning data is information indicating the above-mentioned learning data 60 itself.

[0077] Next, the image data database 13B according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a schematic diagram showing an example of the configuration of the image data database 13B according to this embodiment.

[0078] The image data database 13B of this embodiment is a database in which image data to be subjected to noise removal using the noise remover 90 learned by the learning device 30 of this embodiment is registered, and as an example, as shown in FIG. 12, each piece of information on the image type, image ID, and image data is stored in association with each other.

[0079] The image type is information indicating the type of corresponding image data, the image ID is information that is predetermined to be different for each piece of corresponding image data in order to individually identify the image data, and the image data is information indicating the image data itself that is to be subjected to noise removal.

[0080] Next, the operation of the learning device 30 according to this embodiment will be described with reference to FIG. 13. The learning process shown in FIG. 13 is executed by the CPU 31 of the learning device 30 executing the learning process program 33A. The learning process shown in FIG. 13 is executed, for example, when a user inputs an instruction to start the execution of the learning process via the input unit 34. In order to avoid confusion, a case will be described here in which a sufficient amount of learning data for training the noise remover 90 is registered in the learning data database 13A. Also, a case will be described here in which the type of image targeted by the noise remover 90 to be processed is specified in advance.

[0081] In step 100 in FIG. 13, the CPU 31 reads out one piece of learning data (hereinafter referred to as "learning data to be processed") corresponding to a pre-specified image type (hereinafter referred to as "specified type") from the learning data database 13A.

[0082] In step 102, the CPU 31 removes noise from the learning data to be processed using a noise remover 90 corresponding to the specified type (hereinafter referred to as the “specified noise remover”), thereby generating the above-mentioned noise-removed learning data 62.

[0083] In step 104, the CPU 31 adds noise to the generated noise-removed learning data 62 using a noise generator 92 corresponding to the specified type (hereinafter referred to as the “specified noise generator”), thereby generating the above-mentioned noise-removed and added learning data 64.

[0084] In step 106, the CPU 31 derives the difference in image distribution between the generated post-noise-removal-added training data 64 and the training data to be processed (W distance in this embodiment) as described above. In step 108, the CPU 31 trains the designated noise remover so as to minimize the derived difference.

[0085] In step 110, CPU 31 judges whether or not the processing of steps 100 to 108 has been completed for all learning data corresponding to the specified type registered in learning data database 13A. If the judgment is negative, the process returns to step 100, whereas if the judgment is positive, the learning process ends. When repeatedly executing the processing of steps 100 to 108, CPU 31 applies learning data that has not been the target so far as the learning data to be processed.

[0086] Next, the operation of the noise removal device 50 according to this embodiment will be described with reference to FIG. 14. The CPU 51 of the noise removal device 50 executes the noise removal program 53A, thereby executing the noise removal process shown in FIG. 14. The noise removal process shown in FIG. 14 is executed, for example, when a user inputs an instruction to start the execution of the noise removal process via the input unit 54. In order to avoid confusion, a case where the learning of the noise remover 90 to be applied has been completed will be described here. Also, a case where the image data to be processed and the type of image indicated by the image data are specified in advance will be described here.

[0087] In step 200 of FIG. 14, the CPU 51 reads out image data that has been designated in advance (hereinafter, referred to as "designated image data") from the image data database 13B.

[0088] In step 202, the CPU 51 inputs the read-out designated image data to the noise remover 90 corresponding to the type of image designated in advance. In response to this, the noise remover 90 outputs post-noise-removal image data in which noise has been removed from the designated image data.

[0089] Therefore, in step 204, the CPU 51 acquires the noise-removed image data output by the noise remover 90, stores it in the storage unit 53, and then ends this noise removal process.

[0090] As described above, the learning device according to the present embodiment uses a trainable noise remover to remove noise from first learning data, adds a predetermined additional noise to the first learning data from which noise has been removed, and trains the noise remover so as to make the distribution of the first learning data to which the additional noise has been added closer to the distribution of the second learning data. Therefore, even when learning data with less noise than a normal captured image cannot be obtained, or when the amount of noise in learning data having the same noise as a normal captured image is unknown, the noise remover can be trained.

[0091] Moreover, in the learning device according to the present embodiment, the second learning data is the same as the first learning data, so that the noise remover can be trained without newly preparing the second learning data.

[0092] Moreover, in the learning device according to the present embodiment, the additional noise is set to be a noise that imitates the noise contained in the first learning data, and therefore the noise remover can be trained with higher accuracy than when the additional noise is set to be a noise unrelated to the noise contained in the first learning data.

[0093] Furthermore, according to the learning device of the present embodiment, the data to be subjected to noise removal by the noise remover is image data, and therefore it is possible to train a noise remover that removes noise from image data.

[0094] Furthermore, according to the learning device of this embodiment, the image data is a radiological image, and therefore it is possible to train a noise remover that removes noise from image data representing a radiological image.

[0095] Furthermore, according to the learning device of this embodiment, the noise remover is trained so that the distribution of the first learning data to which the additional noise has been added cannot be distinguished from the distribution of the second learning data by a predetermined discriminator. Therefore, the noise remover can be trained using a model such as WGAN-GP that minimizes the difference between the distribution of the first learning data and the distribution of the second learning data.

[0096] Furthermore, according to the learning device of this embodiment, the noise remover is trained so as to minimize the deviation between the distribution of the first learning data to which the additional noise has been added and the distribution of the second learning data. Therefore, the noise remover can be trained using the deviation.

[0097] Furthermore, in the learning device according to the present embodiment, the deviation is the Wasserstein distance (W-distance), which allows the noise remover to be trained faster than when a value based on the amount of information is used as the deviation.

[0098] [Second embodiment] In the above embodiment, an example of the case where the noise remover 90 is trained by completely unsupervised learning has been described. In the present embodiment, an example of the case where the noise remover 90 is trained by semi-supervised learning will be described.

[0099] Since the overall configuration of the image processing system 1 according to this embodiment is substantially the same as that according to the first embodiment, a description thereof will be omitted here, and the flow of the learning process executed in the learning device 30 according to this embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram for explaining the learning process according to this embodiment, and the same components as those shown in Fig. 10 are given the same reference numerals as in Fig. 10, and their description will be omitted as much as possible.

[0100] As shown in FIG. 15, the learning device 30 of this embodiment is substantially similar to the learning device 30 of the first embodiment in that it obtains noise-removed learning data 62 by removing noise from learning data 60 using a noise remover 90, and obtains noise-removed and added learning data 64 by adding a noise image to the noise-removed learning data 62 using a noise generator 92.

[0101] Thereafter, in the learning device 30 according to the first embodiment, the W distance between the post-noise-removal-added learning data 64 and the learning data 60 is calculated, and the noise remover 90 is trained to minimize the W distance. In contrast, in the learning device 30 according to the present embodiment, the W distance between the post-noise-removal-added learning data 64 and image data 66 that has less noise than the original learning data 60 is calculated, and the noise remover 90 is trained to minimize the W distance. In this embodiment, the amount of noise added to the post-noise-removal learning data 62 by the noise generator 92 is preferably greater than the amount of noise in the image data 66.

[0102] Conventional methods of learning using noise-free or low-noise image data make it impossible to obtain an image of higher image quality than the image represented by the image data, but the learning device 30 of this embodiment makes it possible to obtain an image of higher image quality than the image represented by the image data.

[0103] In this manner, in the present embodiment, an example of the case where the first learning data and the second learning data of the technology disclosed herein are different is illustrated, but the form in this case is not limited to the technology described in the present embodiment. For example, the data obtained by photographing the same part with a different modality may be used as the second learning data. In this case, the domain conversion of the image is performed simultaneously with the noise removal.

[0104] [Third embodiment] In this embodiment, an example of a case where the noise remover 90 learns by combining unsupervised learning and supervised learning will be described.

[0105] Since the overall configuration of the image processing system 1 according to this embodiment is substantially the same as that according to the first embodiment, a description thereof will be omitted here, and the flow of the learning process executed in the learning device 30 according to this embodiment will be described with reference to Fig. 16. Fig. 16 is a block diagram for explaining the learning process according to this embodiment, and the same components as those shown in Fig. 10 are given the same reference numerals as in Fig. 10, and their description will be omitted as much as possible.

[0106] As shown in Figure 16, the learning device 30 of this embodiment differs from the learning device 30 of the first embodiment in that in addition to unsupervised learning by the learning device 30 of the first embodiment shown in the lower diagram of Figure 16, the learning device 30 of this embodiment applies supervised learning shown in the upper diagram of Figure 16.

[0107] 16, the portion related to supervised learning according to this embodiment generates noise-added learning data 72 by adding noise to learning data 70, which is image data with relatively little noise, using a noise generator 92. Also, the portion related to supervised learning according to this embodiment generates noise-removed learning data 74 by removing noise from the generated noise-added learning data 72 using a noise remover 90.

[0108] In the portion relating to supervised learning according to this embodiment, the noise remover 90 is trained so as to minimize the difference between the post-noise-removal training data 74 and the training data 70.

[0109] In this embodiment, unsupervised learning and supervised learning may be performed in parallel, or unsupervised learning may be performed first and then supervised learning may be performed.

[0110] This embodiment makes it possible to obtain a noise remover 90 with higher performance than that obtained by learning using the learning device 30 according to the first embodiment.

[0111] Next, the effect of the learning process according to the above embodiment will be described with reference to FIGS.

[0112] FIG. 17 is a diagram showing an example of image data on which noise has been removed using the noise remover 90 trained by the learning process according to the first embodiment using the MNIST (Mixed National Institute of Standards and Technology database) dataset. FIG. 18 is a diagram showing an example of image data on which noise has been removed using the noise remover 90 trained by the learning process according to the first embodiment using the CelebA (https: / / mmlab.ie.cuhk.edu.hk / projects / CelebA.html) dataset. In the example shown in FIG. 17, Gaussian noise was applied as the type of noise, and an experiment was performed on each image with a standard deviation ranging from 0.15 to 0.3, with the amount of noise unknown. The noise removal result indicated as "R2R" in FIG. 17 is the noise removal result by the method described in Non-Patent Document 2.

[0113] As shown in Fig. 17, when noise is removed from an image of a number in the MNIST dataset using the R2R technique, some noise remains in the background of the number, whereas when noise is removed using the noise remover 90 according to the first embodiment, there is very little noise in the background of the number. Also, as shown in Fig. 18, when noise is removed from a face image in the CelebA dataset using the noise remover 90 according to the first embodiment, the noise that occurred in the original image is almost completely removed.

[0114] Thus, the noise remover 90 according to the above embodiment has an extremely high noise removal effect.

[0115] In the above embodiment, the CPU 31 of the learning device 30 is applied as the processor of the technology of the present disclosure, but the present disclosure is not limited thereto. For example, the CPU 11 of the server 10 or the CPU 51 of the noise removal device 50 may be applied as the processor of the technology of the present disclosure, or a plurality of these CPUs may constitute the processor of the technology of the present disclosure.

[0116] In the above embodiment, the following various processors can be used as the hardware structure of the processing units that execute various processes, such as the noise removal unit 31A, the noise addition unit 31B, the learning unit 31C, and the acquisition unit 51A. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.

[0117] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0118] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as represented by computers such as client and server. Second, there is a form in which a processor is used to realize the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as represented by a system on chip (SoC). In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0119] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0120] In the above embodiment, the learning process program 33A is pre-stored (installed) in the storage unit 33 of the learning device 30, and the noise removal program 53A is pre-installed in the storage unit 53 of the noise removal device 50, but this is not limiting. These programs may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. These programs may also be downloaded from an external device via a network.

[0121] From the above description, the invention described in the following appendix can be understood.

[0122] [Appendix 1] at least one processor; The processor, performing noise reduction on the first training data using a trainable noise reducer; adding a predetermined additional noise to the first learning data from which the noise has been removed; training the noise remover so as to bring a distribution of the first training data to which the additional noise has been added closer to a distribution of a second training data; Learning device. [Appendix 2] The second learning data is the same data as the first learning data. 2. A learning device as described in appendix 1. [Appendix 3] The additional noise is a noise that imitates a noise contained in the first learning data. 3. A learning device according to claim 1 or 2. [Appendix 4] The denoiser is a deep learning model. 4. A learning device according to any one of claims 1 to 3. [Appendix 5] The data to be subjected to noise removal by the noise remover is image data. 5. A learning device according to any one of claims 1 to 4. [Appendix 6] The image data is a radiation image. 6. A learning device as described in appended claim 5. [Appendix 7] The processor, training the noise remover so that a distribution of the first training data to which the additional noise has been added cannot be distinguished from a distribution of the second training data by a predetermined discriminator; 3. A learning device as described in appendix 2. [Appendix 8] The processor, training the noise remover so as to minimize a deviation between a distribution of the first training data to which the additional noise has been added and a distribution of the second training data; 3. A learning device as described in appendix 2. [Appendix 9] The deviation is a value based on the Wasserstein distance or the amount of information. 9. The learning device according to claim 8. [Appendix 10] A noise remover trained by the learning device according to any one of Supplementary Note 1 to Supplementary Note 9; an acquisition unit that inputs predetermined data to the noise remover to acquire data from which noise contained in the data has been removed; A noise removal device comprising: [Appendix 11] performing noise reduction on the first training data using a trainable noise reducer; adding a predetermined additional noise to the first learning data from which the noise has been removed; training the noise remover so as to bring a distribution of the first training data to which the additional noise has been added closer to a distribution of a second training data; A program that causes a computer to carry out processing. [Explanation of symbols]

[0123] 1. Image Processing System 10 Server 11 CPU 12. Memory 13 Storage section 13A Learning Data Database 13B Image data database 14 Input section 15 Display section 16 Media Read / Write Device 17 Recording media 18 Communication I / F section 30 Learning Device 31 CPU 31A Noise Canceller 31B Noise Addition Section 31C Learning Department 32 Memory 33 Storage section 33A Learning Processing Program 34 Input section 35 Display section 36 Media Read / Write Device 37 Recording media 38 Communication I / F section 50 Noise Elimination Device 51 CPU 51A Acquisition Department 52 Memory 53 Storage section 53A Noise Reduction Program 54 Input section 55 Display section 56 Media Read / Write Device 57 Recording media 58 Communication I / F section 60 Training Data 62 Training data after noise removal 64 Training data after noise removal 66 Image Data 70 Training Data 72 Training data after adding noise 74 Noise-removed training data 80 Noise Images 90 Noise Remover 92 Noise Generator f classifier N Network

Claims

1. At least one processor; The processor, performing noise reduction on the first training data using a trainable noise reducer; adding a predetermined additional noise to the first learning data from which the noise has been removed; training the noise remover so as to bring a distribution of the first training data to which the additional noise has been added closer to a distribution of the second training data; Learning device.

2. The second learning data is the same data as the first learning data. The learning device according to claim 1 .

3. The additional noise is a noise that imitates a noise contained in the first learning data. The learning device according to claim 1 .

4. The denoiser is a deep learning model. The learning device according to claim 1 .

5. The data to be subjected to noise removal by the noise remover is image data. The learning device according to claim 1 .

6. The image data is a radiation image. The learning device according to claim 5 .

7. The processor, training the noise remover so that a distribution of the first training data to which the additional noise has been added cannot be distinguished from a distribution of the second training data by a predetermined discriminator; The learning device according to claim 2 .

8. The processor, training the noise remover so that a degree of deviation between a distribution of the first training data to which the additional noise has been added and a distribution of the second training data is minimized; The learning device according to claim 2 .

9. The deviation is a value based on the Wasserstein distance or the amount of information. The learning device according to claim 8.

10. A noise remover trained by the training device according to any one of claims 1 to 9; an acquisition unit that inputs predetermined data to the noise remover to acquire data from which noise contained in the data has been removed; A noise removal device comprising:

11. performing noise reduction on the first training data using a trainable noise reducer; adding a predetermined additional noise to the first learning data from which the noise has been removed; training the noise remover so as to bring a distribution of the first training data to which the additional noise has been added closer to a distribution of the second training data; A program that causes a computer to carry out processing.

Images