Image processing device, image processing method, and program

Abstract

To prevent influence of an excessive learning of a model on an object image while forming an accurate model to obtain a less-noise image from a small amount of learning data.SOLUTION: An image processor includes: an additional noise image generation unit for generating an additional noise image including an object image, noise, and an additional noise image with a feature common to the noise, from an original image including the object image and the noise; a model optimization unit for optimizing a neural network model to convert the additional noise image to the original image; and a noise reduction image generation unit for generating a noise reduction image with less noise from the original image by using the neural network model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] For example, a technology has been proposed for reconstructing CT (Computed Tomography) images using deep learning, as described in Non-Patent Document 1. With this technology, noise contained in CT images is selectively removed while maintaining spatial resolution, so image quality can be stably improved even in low-dose regions. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Motonori Akagi, et al., "Deep learning reconstruction improves image quality of abdominal ultra-high-resolution CT," European Radiology vol. 29, 2019, pp.6163-6171 [Non-Patent Document 2] Fuminari Tatsugami, et al., "Deep learning-based image restoration algorithm for coronary CT angiography," European Radiology vol. 29, 2019, pp. 5322-5329 [Non-Patent Document 3] Toru Higaki, et al., "Deep Learning Reconstruction at CT: Phantom Study of the Image Characteristics," Academic radiology Vol. 27, 2020, pp. 82-87 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned existing technology, model parameters are pre-trained so that an image containing noise captured in a low-dose region for the same part of the same person is converted into an image containing no noise captured in a high-dose region. Therefore, the image that can be removed from noise based on the result of learning is limited to an image similar to the image used for learning, and for example, an image of a different part, an image captured by a scanner of a different model or setting, an image created using a different reconstruction function, etc., require separate learning to remove noise. In addition, in the above-mentioned learning, the features of structures other than noise contained in the image of a specific part are recorded in the model parameters, so that when applied to removing noise from an image of another part, for example, over-learning of the model may result in the depiction of a structure that does not actually exist, or the erasure of a structure that actually exists.

[0005] Therefore, an object of the present invention is to provide an image processing device, an image processing method, and a program that can construct an accurate model for obtaining a noise-reduced image even with a small amount of training data, while preventing the effect on the subject image due to over-learning of the model. [Means for solving the problem]

[0006] [1] An image processing device comprising: an added noise image generation unit that generates an added noise image including a subject image and noise, and added noise having characteristics common to the noise, from an original image including a subject image and noise; a model optimization unit that optimizes a neural network model so as to convert the added noise image into the original image; and a noise-reduced image generation unit that generates a noise-reduced image in which noise has been reduced from the original image using the neural network model. [2] The image processing device according to [1], wherein the added noise is generated according to the principle of noise generation when the original image is captured. [3] The image processing device according to [1], wherein the added noise is generated according to characteristics of the difference between the original image and a filtered image obtained by filtering the original image. [4] The image processing device described in [1], in which the added noise is generated according to the characteristics of an image obtained by sampling from the air portion of the original image. [5] An image processing device described in any one of [1] to [4], wherein the additive noise image includes a plurality of additive noise images that result in the original image when averaged, and the neural network model is optimized to convert each of the plurality of additive noise images into the original image. [6] The image processing device according to any one of [1] to [5], wherein the neural network model is optimized so as to minimize a loss function calculated using only a portion of the pixels of the original image and the added noise image. [7] An image processing device according to any one of [1] to [6], wherein the original image is a CT image or an MRI image. [8] An image processing method including the steps of generating an added noise image from an original image including a subject image and noise, the added noise having characteristics common to the noise, optimizing a neural network model to convert the added noise image into the original image, and generating a noise-reduced image in which noise has been reduced from the original image using the neural network model. [9] A program for causing a computer to function as an image processing device having an added noise image generation unit that generates an added noise image including a subject image and noise, and added noise having characteristics common to the noise, from an original image including a subject image and noise, a model optimization unit that optimizes a neural network model so as to convert the added noise image into the original image, and a noise-reduced image generation unit that generates a noise-reduced image in which noise has been reduced from the original image using the neural network model. Effect of the Invention

[0007] According to the above configuration, a noise-reduced image is generated using a neural network model optimized using an additional noise image generated from a single original image, so that it is possible to construct an accurate model for obtaining a noise-reduced image even with a small amount of training data, while preventing the subject image from being affected by over-learning of the model. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of a system including an image processing device according to an embodiment of the present invention. [Diagram 2] 2 is a flow chart showing the general steps of an image processing method performed in the system shown in FIG. 1; [Diagram 3] Photographs showing examples of images processed in an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] FIG. 1 is a block diagram showing a schematic configuration of a system including an image processing device according to an embodiment of the present invention. The system 10 shown in FIG. 1 includes an image processing device 100, a CT scanner 200, a storage device 300, and a display device 400. Since the CT scanner 200, the storage device 300, and the display device 400 can be generally used, detailed description is omitted, and the image processing device 100 will be mainly described below. Two or more devices included in the system 10 shown in FIG. 1 may be integrated. Specifically, for example, the image processing device 100 may be incorporated in the CT scanner 200. Also, for example, the image processing device 100, the storage device 300, and the display device 400 may be implemented by a single device.

[0011] The image processing device 100 includes an image input unit 110, an added noise image generation unit 120, a model optimization unit 130, and a noise-reduced image generation unit 140. Each unit of the image processing device 100 is implemented using, for example, a hardware configuration of a computer as described below. The configuration of each unit will be described below.

[0012] The image input unit 110 is implemented, for example, by a communication device that receives data from the CT scanner 200, or a driver that reads data stored in the storage device 300 or a removable medium. In this embodiment, a CT image 111 is input to the image input unit 110. The CT image 111 includes an image of body tissue, which is a subject image, and noise.

[0013] The additional noise image generating unit 120 is implemented by a processor that processes the CT image 111 input to the image input unit 110 according to a program. The additional noise image generating unit 120 generates an additional noise image 121 from the CT image 111, which is an original image. Here, the additional noise image 121 includes the same subject image and noise as the original image, as well as additional noise. Here, the additional noise has characteristics common to the noise included in the original image. Specific examples of such additional noise will be described later.

[0014] The model optimization unit 130 is implemented by a processor that processes the CT image 111 and the added noise image 121 according to a program. The model optimization unit 130 optimizes the neural network model 131 so as to convert the added noise image 121 into the CT image 111, which is the original image. Specifically, the neural network model 131 has the added noise image 121 as input data and the CT image 111 as output data, and includes at least one layer of linear transformation or nonlinear transformation. When the layer of the neural network model 131 is deep, the model optimization unit 130 executes so-called deep learning. When the neural network model 131 includes a convolution operation, the model optimization unit 130 executes so-called CNN (convolutional neural network). The model optimization unit 130 stores parameters such as weight coefficients of the neural network model 131 optimized by learning in the storage device 300.

[0015] The noise-reduced image generating unit 140 is implemented by a processor that processes the CT image 111 according to a program. The noise-reduced image generating unit 140 generates a noise-reduced image 141 in which noise is reduced from the CT image 111, using the neural network model 131 optimized by the model optimizing unit 130. As will be described later, the neural network model 131 constructed to remove a part of the noise contained in the added noise image 121 to generate the CT image 111, which is the original image, can also remove a part of the noise to generate the noise-reduced image 141 when the CT image 111 is input. The generated noise-reduced image 141 may be stored in the storage device 300, or may be displayed on the display device 400.

[0016] In the above example, the additional noise image generating unit 120 generates additional noise according to the noise generation principle when the CT image 111, which is the original image, is captured. More specifically, the additional noise in the case of the CT image 111 may be a streak artifact or a file-like noise distributed around the irradiation line. It is known that such noise is generated by non-uniformity in the density of X-ray particles during CT imaging, and therefore noise having the same characteristics as the noise already contained in the CT image 111 can be arithmetically generated from pseudorandom numbers. For example, the additional noise can be generated by approximating the non-uniformity in the density of X-ray particles using a Poisson distribution. In another example, when an MRI (Magnetic Resonance Imaging) image is used as the original image, a Rician distribution noise can be generated by Fourier transforming Gaussian distribution noise generated by thermal noise during image reconstruction.

[0017] Alternatively, the additional noise image generating unit 120 may generate additional noise according to the characteristics of the difference between the CT image 111 and a filter image obtained by filtering the original CT image 111. For example, when the CT image 111 is filtered using a mean filter, a Gaussian filter, a median filter, or the like, the noise contained in the CT image 111 is removed. Therefore, the difference between the filter image and the CT image 111 can be specified as the noise contained in the CT image 111, and noise having the same characteristics as this noise can be arithmetically generated from pseudorandom numbers. Note that the filter image in the above case causes degradation of image quality, such as a decrease in the resolution of the subject image, and therefore cannot be used as a noise-reduced image.

[0018] Alternatively, the additional noise image generating unit 120 may generate additional noise according to the characteristics of an image obtained by sampling from the air portion of the original image, the CT image 111. For example, when the air portion of the CT image 111 is sampled by removing the living body, clothing, and imaging device portion by binarization processing, the noise portion included in the CT image 111 is extracted. Therefore, an image obtained by sampling from the air portion of the CT image 111 can be specified as noise included in the CT image 111, and noise having the same characteristics as this noise can be arithmetically generated from pseudorandom numbers.

[0019] For example, let us define the vector representation of the additive noise generated above as n add Let A and n be vector expressions of the subject image and noise contained in the original image, respectively. The construction of the neural network model 131 and the generation of the noise-reduced image 141 in this embodiment can be expressed by the following formulas (1) and (2). Here, G(x) is a converter using the neural network model 131.

[0020]

number

[0021] The above formula (1) shows the steps of constructing the neural network model 131. The converter G(x) by the neural network model 131 converts the noise n+n add Addition noise n add where the additive noise n add The removal of noise n and additive noise n has a common characteristic. add From the noise group consisting of add The converter G(x) is optimized so as not to affect the object image A.

[0022] On the other hand, equation (2) shows the step of generating a noise-reduced image 141 using the constructed neural network model 131. The converter G(x) is optimized to reduce noise without affecting the object image A, so that the additional noise n add When a CT image 111 that does not include n is input, the noise n contained in the CT image 111 is reduced, that is, a part of the noise n is removed. The reduction rate at this time can be calculated simply as n / (n+n add ) becomes.

[0023] FIG. 2 is a flowchart showing schematic steps of an image processing method executed in the system shown in FIG. 1. In the illustrated example, first, a CT image 111 is input to the image input unit 110 (step S101). Next, the added noise image generating unit 120 generates an added noise image 121 (step S102). Next, the model optimizing unit 130 optimizes the neural network model 131 (step S103). For the optimization, various methods known as optimization methods of deep learning and CNN can be used. When a predetermined termination condition is satisfied, the optimization ends (step S104). For example, as illustrated, the generation of a new added noise image 121 (step S102) and the optimization of the neural network model 131 using the new added noise image 121 (step S103) may be repeated until the termination condition is satisfied, and the neural network model 131 may be optimized so as to convert a plurality of randomly generated added noise images 121 into the same CT image 111. After the neural network model 131 is optimized, the noise-reduced image generating unit 140 generates the noise-reduced image 141 from the CT image 111 using the neural network model 131 (step S105).

[0024] The optimization of the neural network model 131 may utilize the method of Noise2Noise (Jaakko Lehtinen, et al., "Noise2Noise: Learning Image Restoration without Clean Data," Proceedings of the 35th International Conference on Machine Learning, PMLR 80, 2018). In this case, the added noise image generating unit 120 generates a plurality of added noise images 121 that become the CT image 111, which is the original image, when averaged. The model optimizing unit 130 optimizes the neural network model 131 so as to convert each of the generated plurality of added noise images 121 into the CT image 111. This can reduce the time required for optimizing the neural network model 131. In addition, the neural network model 131 may be optimized so that a loss function calculated using only some pixels of the CT image 111 and the added noise image 121 is minimized. This makes it easier for parameters to change, and can reduce the time required for optimization.

[0025] In one embodiment of the present invention as described above, the noise-reduced image 141 can be generated using the neural network model 131 optimized using the added noise image 121 generated from the CT image 111. In this embodiment, the neural network model 131 is optimized using the added noise image 121 generated from a single CT image 111, so that a highly accurate model can be constructed without preparing training data that includes the site, the model and settings of the scanner, and the reconstruction function used for imaging. In addition, since the processing from learning to the generation of the noise-reduced image 141 is completed using a single CT image 111, no influence is caused on the subject image due to over-learning of the model, and a highly accurate model specialized for the image to be processed can be constructed.

[0026] As described above, the original image to be processed in this embodiment is not limited to a CT image, but may be an MRI image. The added noise does not necessarily have to be generated according to the noise generation principle when the original image is captured, but may be generated according to the characteristics of the difference between the original image and the filtered image or the characteristics of the air part in the original image. Therefore, any type of image, the noise generation principle of which is not necessarily clear, can be processed as the original image to generate a noise-reduced image.

[0027] FIG. 3 is a photograph showing an example of an image processed in an embodiment of the present invention. In the embodiment, additive noise was randomly generated according to the characteristics of the difference between the CT image and a filtered image obtained by filtering the CT image using a median filter, and the neural network model was optimized using 10,000 additive noise images each containing a different additive noise. As a result, a noise-reduced image in which the noise was reduced and the edges of the subject image were not blurred, as shown in the right side of FIG. 3, was generated from the CT image shown on the left side of FIG. 3. For one series of CT images including about 200 two-dimensional images, the processing time for generating a noise-reduced image for each image was approximately 2 to 3 minutes.

[0028] (Computer hardware configuration) FIG. 4 is a block diagram showing an example of a hardware configuration of a computer. The present invention can be implemented, for example, as an image processing device implemented by such a computer. The present invention may also be implemented as a program configured to cause the computer to function as the image processing device by being loaded into the computer. The computer 900 may be, for example, a terminal device such as a personal computer, tablet, or smartphone, or may be a server computer installed in a data center or the like. The functions of the computer 900 do not necessarily have to be realized in a single physical device, and may be realized, for example, by integrating the resources of multiple devices connected via a network.

[0029] The computer 900 includes a processor 901, a memory 903, an input device 905, an output device 907, and a bus interface 909. The computer 900 may further include a storage 911, a driver 913, a connection port 915, and a communication device 917. The processor 901 is configured by a processing circuit such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), and / or a field-programmable gate array (FPGA). The processor 901 controls the operation of the computer 900 according to a program recorded in the memory 903, the storage 911, or a removable medium 919. The memory 903 includes, for example, a read only memory (ROM) and a random access memory (RAM). The ROM stores, for example, a program for the processor 901, and operation parameters. The RAM temporarily stores, for example, programs expanded during execution by the processor 901, and parameters during execution of the programs.

[0030] The input device 905 is a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, or various switches. The input device 905 does not necessarily need to be integrated with the computer 900, and may be, for example, a remote controller that transmits a control signal by wireless communication. The input device 905 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the processor 901. The output device 907 may include a display device, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output device 907 outputs the results obtained by the processing of the computer 900 as text or an image.

[0031] The storage 911 is configured by, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage 911 stores, for example, a program for the processor 901, various data read out during execution of the program or generated by execution of the program, and various data acquired from the outside. The driver 913 is a reader / writer for a removable medium 919 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The driver 913 may read information recorded in the attached removable medium 919 and output it to the memory 903. The driver 913 may also write various data to the attached removable medium 919. The connection port 915 is a port for connecting an external connection device 921 to the computer 900. The connection port 915 may include, for example, a universal serial bus (USB) port, an IEEE1394 port, a small computer system interface (SCSI) port, or the like. The connection port 915 may also include an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, etc. By connecting an external connection device 921 to the connection port 915, various types of data can be exchanged between the computer 900 and the external connection device 921.

[0032] The communication device 917 is connected to the network 923. The network 923 may be an open communication network to which an unspecified number of devices are connected, such as the Internet, or may be a closed communication network to which a limited number of devices, such as Bluetooth (registered trademark), are connected, such as two devices. The communication device 917 may include, for example, a communication card for a LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi, or WUSB (Wireless USB). The communication device 917 transmits and receives signals or data between other devices using a predetermined protocol corresponding to the network 923.

[0033] Each component of the computer 900 described above may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. In addition, a person skilled in the art can appropriately change the configuration of the computer 900 as described above depending on the technical level at the time of implementation.

[0034] Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art in the technical field to which the present invention pertains can come up with various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present invention. [Explanation of symbols]

[0035] 10...system, 100...image processing device, 110...image input unit, 111...CT image, 120...additional noise image generation unit, 121...additional noise image, 130...model optimization unit, 131...neural network model, 140...noise reduced image generation unit, 141...noise reduced image, 200...CT scanner, 300...storage device, 400...display device.

Claims

1. an additional noise image generating unit that generates, from an original image including a subject image and noise, an additional noise image including the subject image, the noise, and additional noise having a characteristic common to the noise; a model optimization unit that optimizes a neural network model to transform the added noise image into the original image; a noise-reduced image generating unit that generates a noise-reduced image in which the noise is reduced from the original image by using the neural network model; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the additional noise is generated in accordance with a noise generation principle at the time of capturing the original image.

3. The image processing device according to claim 1 , wherein the additional noise is generated according to a characteristic of a difference between the original image and a filtered image obtained by filtering the original image.

4. The image processing apparatus according to claim 1 , wherein the added noise is generated according to characteristics of an image obtained by sampling an air portion of the original image.

5. the additive noise image includes a plurality of additive noise images that, when averaged, result in the original image; The image processing apparatus according to claim 1 , wherein the neural network model is optimized to transform each of the plurality of additive noise images into the original image.

6. 6. The image processing device according to claim 1, wherein the neural network model is optimized so as to minimize a loss function, calculated using only some pixels of the original image and the added noise image, that calculates a difference between the original image and an image obtained by converting the added noise image using the neural network model.

7. The image processing device according to claim 1 , wherein the original image is a CT image or an MRI image.

8. generating an additional noise image from an original image including a subject image and noise, the additional noise image including the subject image, the noise, and additional noise having a characteristic common to the noise; optimizing a neural network model to transform the additive noise image into the original image; generating a noise-reduced image in which the noise is reduced from the original image using the neural network model; An image processing method comprising:

9. an additional noise image generating unit that generates, from an original image including a subject image and noise, an additional noise image including the subject image, the noise, and additional noise having a characteristic common to the noise; a model optimization unit that optimizes a neural network model to transform the added noise image into the original image; a noise-reduced image generating unit that generates a noise-reduced image in which the noise is reduced from the original image by using the neural network model; A program for causing a computer to function as an image processing device comprising the steps of:

Images