Image processing apparatus and image processing method

Abstract

An image processing device (1) is provided with a processing unit (11) and a learning unit (12), and performs noise reduction processing on a target image (23). The processing unit (11) inputs an input image (21) into the CNN and outputs an output image (22) from the CNN. The learning unit (12) uses an evaluation function based on the output image (22) and the target image (23), and learns the CNN on the basis of the value of the evaluation function. The evaluation function includes an error evaluation term representing an evaluation value regarding an error between the output image and the target image, and a regularization term representing an evaluation value regarding a difference in pixel values between adjacent pixels in the output image. The image processing device (1) repeats the processing by the processing unit (11) and the learning unit (12) a plurality of times, and uses an output image (22) as a noise-reduced image after repeating a certain number of times. This realizes an image processing device and an image processing method that are capable of suppressing image quality degradation caused by CNN over-learning in noise reduction processing using DIP technology.

Description

Technical Field

[0001] The invention relates to a device and a method for performing noise reduction processing on an object image. Background Art

[0002] As a technology for reducing the noise of an image containing noise, various technologies are known. Among various noise reduction processing technologies, the technology using the Deep Image Prior (Deep Image Prior) technology, which is a convolutional neural network using a deep neural network, has attracted particular attention. Hereinafter, the convolutional neural network (Convolutional Neural Network) is referred to as "CNN" and the Deep Image Prior technology is referred to as "DIP technology".

[0003] The DIP technique uses the property of CNN that meaningful structures in images are learned faster than random noise (i.e., random noise is not easily learned). Using the DIP technique, the noise of the target image can be reduced.

[0004] For example, since the tomographic image of the subject acquired by a radiation tomography device such as a PET (Positron Emission Tomography) device and a SPECT (Single Photon Emission Computed Tomography) device often contains noise, noise reduction processing is required. The invention disclosed in Patent Document 1 uses the tomographic image of the subject reconstructed based on the simultaneous counting information collected by the PET device as the target image, and uses the DIP technology to create the tomographic image after noise reduction processing. Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-128882 Non-patent literature

[0006] Non-patent document 1: J. Nuyts et al., "A concave prior penalizing relative differences for maximum-a-posteriori reconstruction in emission tomography", IEEE TNS, Vol. 49, Issue 1, pp. 56-60, 2002 Non-patent document 2: Kudo Hiroyuki, "Image reconstruction method for low-exposure CT - basic principles of statistical image reconstruction, successive approximate image reconstruction and compressed sensing (Okeru image reconstruction method for low-exposure CT - statistical image reconstruction, successive approximate image reconstruction, basis of compressed sensing)", Medical Imaging Technology, Vol.32, No.4, pp.239-248, 2014 Non-patent document 3: Antonin Chambolle, "An Algorithm for Total Variation Minimization and Applications", Journal of Mathematical Imaging and Vision 20, pp. 89-97, 2004 Summary of the invention Problems to be solved by the invention

[0007] Although the noise reduction processing using the DIP technology has excellent noise reduction performance, there is a problem of image quality degradation caused by over-learning of CNN. That is, as mentioned above, the DIP technology uses the property of CNN that random noise is not easy to learn, but as the number of CNN learning increases, random noise will also be restored. In this way, due to over-learning of CNN, random noise is also restored, and image quality will deteriorate.

[0008] An object of the present invention is to provide an image processing device and an image processing method that can suppress image quality degradation caused by over-learning of CNN in noise reduction processing using the DIP technique. Methods used to solve problems

[0009] The embodiment of the present invention is an image processing device. The image processing device is a device for performing noise reduction processing on a target image, and comprises (1) a processing unit that inputs an input image into a convolutional neural network and outputs an output image from the convolutional neural network, and (2) a learning unit that uses an evaluation function including an error evaluation term and a regularization term to make the convolutional neural network learn based on the value of the evaluation function, wherein the error evaluation term represents an evaluation value of the error between the output image and the target image, and the regularization term represents an evaluation value of the difference in pixel values ​​between adjacent pixels in the output image, and the output image after the processing of the processing unit and the learning unit is repeated multiple times is used as the image after noise reduction processing.

[0010] The embodiment of the present invention is an image processing method. The image processing method is a method for performing noise reduction processing on a target image, comprising: (1) a processing step of inputting an input image into a convolutional neural network and outputting an output image from the convolutional neural network, and (2) a learning step of using an evaluation function including an error evaluation term and a regularization term to make the convolutional neural network learn based on the value of the evaluation function, wherein the error evaluation term represents an evaluation value of the error between the output image and the target image, and the regularization term represents an evaluation value of the difference in pixel values ​​between adjacent pixels in the output image, and the output image after the processing of the processing unit and the learning unit is repeated multiple times is used as the image after noise reduction processing. Effects of the Invention

[0011] According to the embodiment of the present invention, it is possible to suppress image quality degradation caused by over-learning of CNN in noise reduction processing using the DIP technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a diagram showing the structure of the image processing device 1 . Figure 2 This is a diagram showing a structural example of a CNN. Figure 3 A diagram illustrating adjacent pixels in the output image. Figure 4 (a) to (c) are diagrams showing input images (MRI images). Figure 5 (a) to (c) are diagrams showing phantom images (correct images). Figure 6 (a) to (c) are diagrams showing target images. Figure 7 (a) to (c) are diagrams showing output images generated by noise reduction processing using the image processing method of the comparative example. Figure 8 (a) to (c) are diagrams showing output images generated by the noise reduction process using the image processing method of Example 1. Fig. 9 (a) to (c) are diagrams showing output images generated by noise reduction processing using the image processing method of Example 2. Fig.10 (a) to (c) are diagrams showing output images generated by noise reduction processing using the image processing method of Example 3. Fig.11 It is a graph showing the relationship between the number of CNN learning times and the PSNR of the output image for Examples 1 to 3 and the comparative example. DETAILED DESCRIPTION

[0013] Hereinafter, embodiments of the image processing device and the image processing method will be described in detail with reference to the accompanying drawings. In addition, the same elements are marked with the same reference numerals in the description of the accompanying drawings, and repeated descriptions are omitted. The present invention is not limited to these examples, but is intended to include all changes within the meaning and scope of the patent claims, which are equivalent to the scope of the patent claims.

[0014] Figure 1 is a diagram showing the configuration of the image processing device 1. The image processing device 1 performs noise reduction processing on the target image 23.

[0015] The image processing device 1 includes a GPU (Graphics Processing Unit) for performing processing using a convolutional neural network (CNN), an input unit (e.g., a keyboard, a mouse) for accepting input from an operator, a display unit (e.g., a liquid crystal display) for displaying images, etc., and a storage unit for storing programs and data for executing various processing. As the image processing device 1, for example, a computer having a CPU, a RAM, a ROM, a hard disk drive, etc. is used.

[0016] The image processing device 1 includes a processing unit 11 and a learning unit 12. In addition, the image processing method for performing noise reduction processing on the target image 23 using the image processing device 1 includes a processing step and a learning step. The processing unit 11 inputs the input image 21 into the CNN and outputs the output image 22 from the CNN (processing step). The learning unit 12 uses an evaluation function based on the output image 22 and the target image 23, and learns the CNN based on the value of the evaluation function (learning step).

[0017] The image processing device 1 repeats the processing of the processing unit 11 and the learning unit 12 a plurality of times according to the DIP technique, and uses the output image 22 after the repetition of a certain number of times as an image after the noise reduction processing.

[0018] The target image 23 to be subjected to the noise reduction process may be any image. The input image 21 may also be any image. In the figure, a PET image is exemplified as the target image 23, and an MRI image is exemplified as the input image 21. The input image 21 may also be a random noise image.

[0019] When a cross-sectional image of a subject acquired by a radiation tomography device (e.g., a PET device or a SPECT device) is used as the object image 23, the input image 21 may be an image representing morphological information of the subject, or may be an MRI image, a CT image, or a static PET image of the subject.

[0020] The target image 23 may be a 2D image or a 3D image. When the target image 23 is a 2D image, the input image 21 and the output image 22 are also 2D images. When the target image 23 is a 3D image, the input image 21 and the output image 22 are also 3D images.

[0021] Figure 2 This is a diagram showing an example of the structure of a CNN. The CNN shown in this figure is a 3D U-net structure including an encoder and a decoder. In this figure, the number of pixels of the input image 21 input to the CNN is set to N×N×64, and the size of each layer of the CNN is shown.

[0022] Next, the evaluation function used by the learning unit 12 in the learning step is described. Let the processing performed by the CNN be f, let the input image 21 input to the CNN be g, and let the weight coefficient parameter representing the learning state of the CNN be θ. As the learning of the CNN progresses, θ changes. When the input image g is input to the CNN with the weight coefficient θ, the output image 22 output from the CNN is represented by f. θ (g) Let the object image 23 be x0.

[0023] The evaluation function E can be arbitrary, for example, the L1 norm, L2 norm, negative log-likelihood in Poisson distribution, etc. can be used. Here, it is represented by the mean square error (MSE) expressed in the following formula (1). The evaluation function E only includes the expression for the output image f θ (g) An error evaluation item of an evaluation value of the error with the object image x0. [Number 1] E=||f θ (g)-x0|| 2 (1)

[0024] The evaluation function E shown in the following formula (2) contains not only the error evaluation term (the first term on the right), but also the regularization term (the second term on the right) used to suppress over-learning of CNN. The regularization term represents the output image f θ (g) is an evaluation value of the difference in pixel values ​​between adjacent pixels. The regularization term compensates for the difference in pixel values ​​between adjacent pixels in the output image. β is a hyperparameter that adjusts the degree of regularization effect. The smaller β is, the smaller the effect of regularization is. The larger β is, the greater the effect of regularization (i.e., the effect of suppressing over-learning of CNN). [Number 2] E=||f θ (g)-x0|| 2 +β·R(f θ (g)) (2)

[0025] In the case of a 2D image, the pixels adjacent to a certain pixel include pixels adjacent in two directions orthogonal to each other, and also pixels adjacent in a properly inclined direction. In the case of a 2D image, the number of pixels adjacent to a certain pixel is 8, excluding pixels located at the ends or corners of the image.

[0026] In the case of a 3D image, the pixels adjacent to a certain pixel include pixels adjacent in three mutually orthogonal directions, and also pixels adjacent in a properly inclined direction. In the case of a 3D image, the number of pixels adjacent to a certain pixel is 26, excluding pixels located at the ends or corners of the image.

[0027] Figure 3 is a diagram illustrating adjacent pixels in the output image. This diagram represents the output image as a 2D image, representing 3×3 pixels in it. Let the pixel value of the pixel in the center of the diagram be λ j , let the pixel values ​​of the 8 pixels adjacent to the central pixel be λ k When (k=1-8), the difference between the pixel values ​​of the central pixel and the adjacent pixels is |λ j -λ k | represents. The regularization term represents the evaluation value of the difference between pixel values ​​for all combinations of neighboring pixels in the output image.

[0028] The regularization term can be expressed in various mathematical expressions as long as it represents an evaluation value of the difference between the pixel values ​​of adjacent pixels in the output image. For example, the regularization term is expressed in the following equation (3). In the equation (3), N j represents the set of pixels k adjacent to pixel j. γ represents the value of the pixel relative to λ j The change in the value of the regularization term is the size of the change. The formula (3) contains a term for the difference in pixel values ​​of adjacent pixels in the numerator and a term for the sum of pixel values ​​of adjacent pixels in the denominator, representing an evaluation value table for the relative difference in pixel values ​​between adjacent pixels in the output image. [Number 3]

[0029] In addition, formula (3) is similar to the mathematical formula described in non-patent document 1. However, in non-patent document 1, the mathematical formula similar to formula (3) is used when reconstructing a tomographic image of the subject based on simultaneous counting information collected by the PET device, and is not used when performing noise reduction processing on the tomographic image using the DIP technology.

[0030] In addition, as a regularization term, for example, Gibbs prior (Non-patent Document 2), Totalvariation (Non-patent Document 3), etc. can also be used. In addition, these documents also record the technology of reconstructing the tomographic image of the subject, but do not record the technology of denoising the tomographic image using the DIP technology.

[0031] Next, the results of noise reduction processing on the target image are described using simulated data (target image) created by Monte Carlo simulation of the head PET device using digital brain phantom images. The phantom image was obtained from BrainWeb (https: / / brainweb.bic.mni.mcgill.ca / brainweb / ).

[0032] Figures 4 to 10 Indicates the image used in the simulation or the image after noise reduction. Figure 4 A diagram showing an input image (MRI image). Figure 5 A diagram showing a phantom image (correct image). Figure 6 is a graph representing an object image. Figure 7 FIG. 1 is a diagram showing an output image generated by a noise reduction process using an image processing method according to a comparative example.

[0033] Figure 8 FIG. 1 is a diagram showing an output image generated by the noise reduction process using the image processing method of Example 1. FIG. Fig. 9 FIG. 1 is a diagram showing an output image generated by the noise reduction process using the image processing method of the second embodiment. Fig.10 The figures show output images generated by the noise reduction process using the image processing method of Example 3. In these figures, (a) shows a cross-sectional image, (b) shows a coronal cross-sectional image, and (c) shows a sagittal cross-sectional image.

[0034] In the comparative example, the noise reduction process is performed using the DIP technique using the evaluation function of the above formula (1). In the examples 1 to 3, the noise reduction process is performed using the DIP technique using the evaluation functions of the above formulas (2) and (3). In the example 1, β = 1 × 10 -8 In Example 2, β = 3 × 10 -8 In Example 3, β = 5 × 10 -8 In any embodiment, γ=2.

[0035] Will Figure 7 to Figure 10 By comparison, it can be confirmed that compared with the comparative example ( Figure 7 ) compared to Examples 1 to 3 ( Figure 8 to Figure 10 ) image quality is improved, and it can be confirmed that the uniformity of the white matter part is also improved.

[0036] Fig.11 The graphs show the relationship between the number of CNN learning times and the PSNR of the output image for Examples 1 to 3 and Comparative Example. PSNR (Peak Signal to Noise Ratio) represents the quality of an image in decibels (dB), and a higher value means better image quality. As shown in the graph, in Comparative Example ( Figure 7 ), the PSNR reaches a maximum value of 27.21dB after 8 CNN learnings, and then the PSNR decreases when further learning is continued.

[0037] In Example 1 ( Figure 8 ), the PSNR reaches a maximum value of 27.48dB when CNN learning is performed 10 times, and then the PSNR decreases when further learning is continued. Fig. 9 ), the PSNR reaches a maximum value of 27.62dB when CNN learning is performed 13 times, and then the PSNR decreases when further learning is continued. Fig.10 ), the PSNR reaches a maximum value of 27.10dB when CNN learning is performed 16 times, and then the PSNR decreases when further learning is continued. In addition, the object image ( Figure 6 ) has a PSNR of 20.64dB.

[0038] The decrease in PSNR when CNN learning is continued after PSNR reaches the maximum value is more significant in the comparative example, and is gentler in Examples 1 to 3 than in the comparative example. Between Examples 1 to 3, the larger the value of β, the gentler the decrease in PSNR after PSNR reaches the maximum value. In addition, the maximum value of PSNR in Examples 1 to 3 is greater than the maximum value of PSNR in the comparative example.

[0039] In this way, it was confirmed that in the noise reduction processing using the DIP technology, by using an evaluation function including a regularization term to make CNN learn, wherein the evaluation function represents an evaluation value of the difference in pixel values ​​between adjacent pixels in the output image from the CNN, it is possible to suppress image quality degradation caused by over-learning of the CNN. In addition, it was confirmed that the noise reduction performance can be improved.

[0040] The image processing device and the image processing method are not limited to the above-described embodiments and configuration examples, and various modifications are possible.

[0041] The image processing device of the first mode of the above-mentioned implementation manner is a device for performing noise reduction processing on a target image, comprising (1) a processing unit that inputs an input image into a convolutional neural network and outputs an output image from the convolutional neural network, and (2) a learning unit that uses an evaluation function including an error evaluation term and a regularization term to make the convolutional neural network learn based on the value of the evaluation function, wherein the error evaluation term represents an evaluation value of the error between the output image and the target image, and the regularization term represents an evaluation value of the difference in pixel values ​​between adjacent pixels in the output image, and the output image after the processing by the processing unit and the learning unit is repeated multiple times is used as the image after noise reduction processing.

[0042] The image processing device of the second aspect may have the following configuration: in the configuration of the first aspect, the target image is a tomographic image of the subject created based on the coincidence counting information collected by the radiation tomography apparatus.

[0043] In the image processing device of the third aspect, the following structure may be adopted: in the structure of the second aspect, the processing unit inputs an image representing morphological information of the subject as an input image into the convolutional neural network.

[0044] In the image processing device of the fourth aspect, the following structure may be adopted: in the structure of the second aspect, the processing unit inputs the MRI image of the subject as an input image into the convolutional neural network.

[0045] In the image processing device of the fifth aspect, the following structure may be adopted: in the structure of the second aspect, the processing unit inputs the CT image of the subject as an input image into the convolutional neural network.

[0046] In the image processing device of the sixth aspect, the following structure may be adopted: in the structure of the second aspect, the processing unit inputs a static PET image of the subject as an input image to the convolutional neural network.

[0047] In the image processing device of the seventh aspect, the following structure may be adopted: in the structure of the first or second aspect, the processing unit inputs a random noise image as an input image into the convolutional neural network.

[0048] The image processing method of the first mode of the above-mentioned implementation manner is a method for denoising an object image, comprising: (1) a processing step of inputting an input image into a convolutional neural network and outputting an output image from the convolutional neural network, and (2) a learning step of using an evaluation function including an error evaluation term and a regularization term to make the convolutional neural network learn based on the value of the evaluation function, wherein the error evaluation term represents an evaluation value of the error between the output image and the object image, and the regularization term represents an evaluation value of the difference in pixel values ​​between adjacent pixels in the output image, and the output image after the processing by the processing unit and the learning unit is repeated multiple times is used as the image after denoising.

[0049] The image processing method of the second aspect may adopt a configuration in which, in the configuration of the first aspect, the target image is a tomographic image of the subject created based on the coincidence counting information collected by the radiation tomography apparatus.

[0050] In the image processing method of the third aspect, the following structure may be adopted: in the structure of the second aspect, in the processing step, an image representing morphological information of the subject is input as an input image to the convolutional neural network.

[0051] In the image processing method of the fourth aspect, the following structure may be adopted: in the structure of the second aspect, in the processing step, the MRI image of the subject is input as an input image to the convolutional neural network.

[0052] In the image processing method of the fifth aspect, the following structure may be adopted: in the structure of the second aspect, in the processing step, a CT image of the subject is input as an input image to the convolutional neural network.

[0053] In the image processing method of the sixth aspect, the following structure may be adopted: in the structure of the second aspect, in the processing step, a static PET image of the subject is input as an input image to the convolutional neural network.

[0054] In the image processing method of the seventh aspect, the following structure may be adopted: in the structure of the first or second aspect, in the processing step, a random noise image is input as an input image into the convolutional neural network. Industrial Applicability

[0055] The present invention can be used as an image processing device and an image processing method that can suppress image quality degradation caused by CNN overlearning in noise reduction processing using the DIP technology. Description of Reference Numerals

[0056] 1 ... image processing device; 11 ... processing unit; 12 ... learning unit; 21 ... input image; 22 ... output image; 23 ... object image.

Claims

1. An image processing device, wherein: It is a device that performs noise reduction processing on the object image. have: A processing unit that inputs an input image into a convolutional neural network and outputs an output image from the convolutional neural network; and a learning unit that causes the convolutional neural network to learn based on a value of an evaluation function including an error evaluation term and a regularization term, wherein the error evaluation term represents an evaluation value regarding an error between the output image and the target image, and the regularization term represents an evaluation value regarding a difference in pixel values ​​between adjacent pixels in the output image, The output image after the processing by the processing unit and the learning unit is repeated a plurality of times is used as an image after the noise reduction processing.

2. The image processing apparatus according to claim 1, wherein: The target image is a tomographic image of the subject created based on coincidence counting information collected by a radiation tomography apparatus.

3. The image processing apparatus according to claim 2, wherein: The processing unit inputs an image representing morphological information of the subject as the input image into the convolutional neural network.

4. The image processing apparatus according to claim 2, wherein: The processing unit inputs the MRI image of the subject as the input image into the convolutional neural network.

5. The image processing apparatus according to claim 2, wherein: The processing unit inputs the CT image of the subject as the input image into the convolutional neural network.

6. The image processing apparatus according to claim 2, wherein: The processing unit inputs a static PET image of the subject as the input image into the convolutional neural network.

7. The image processing apparatus according to claim 1 or 2, wherein: The processing unit inputs a random noise image as the input image into the convolutional neural network.

8. An image processing method, wherein: It is a method for denoising the object image. have: a processing step of inputting an input image into a convolutional neural network and outputting an output image from the convolutional neural network; and a learning step of causing the convolutional neural network to learn based on a value of an evaluation function including an error evaluation term and a regularization term, wherein the error evaluation term represents an evaluation value regarding an error between the output image and the object image, and the regularization term represents an evaluation value regarding a difference in pixel values ​​between adjacent pixels in the output image, The output image after the processing by the processing unit and the learning unit is repeated a plurality of times is used as an image after the noise reduction processing.

9. The image processing method according to claim 8, wherein: The target image is a tomographic image of the subject created based on coincidence counting information collected by a radiation tomography apparatus.

10. The image processing method according to claim 9, wherein: In the processing step, an image representing morphological information of the subject is input into the convolutional neural network as the input image.

11. The image processing method according to claim 9, wherein: In the processing step, the MRI image of the subject is input into the convolutional neural network as the input image.

12. The image processing method according to claim 9, wherein: In the processing step, the CT image of the subject is input into the convolutional neural network as the input image.

13. The image processing method according to claim 9, wherein: In the processing step, a static PET image of the subject is input into the convolutional neural network as the input image.

14. The image processing method according to claim 8 or 9, wherein: In the processing step, a random noise image is input into the convolutional neural network as the input image.

Images