Image processing device and image processing method

Abstract

Provided are an image processing device and an image processing method that are capable of improving the accuracy of removing noise included in an observation image.　This image processing device comprises a calculation unit that removes noise of an image, and is characterized in that the calculation unit performs machine learning using, as learning data, a correct answer image, which is a three-dimensional image that does not include noise or has an extremely small amount of noise, and a noise image, which is a three-dimensional image in which noise is added to the correct answer image, and that the image processing device comprises a noise remover that is generated by performing parameter adjustment so that the noise image approaches the correct answer image.

Description

Image processing device and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method for handling observation images obtained by a charged particle beam device or the like, and to a technique for removing noise from the observation images.

[0002] Noise contained in observation images obtained using charged particle beam devices, etc., can hinder the analysis of the observation images. In particular, low-dose images, which are observation images obtained when the electron beam irradiated on the sample is set to a low dose in order to minimize damage to the sample, contain a lot of noise, so noise removal is essential.

[0003] Non-Patent Document 1 discloses a method for removing noise using a machine learning machine that is generated by performing machine learning using clean two-dimensional images and two-dimensional images in which random noise has been added to the clean two-dimensional images as training data.

[0004] Xinlei Chen et al. "Deconstructing Denoising Diffusion Models for Self-Supervised Learning", arXiv:2401.14404v1 [cs.CV] 25 Jan 2024

[0005] However, in machine learning machines generated by machine learning using two-dimensional images as training data, as in Non-Patent Document 1, noise removal is insufficient, meaning that the detailed structure of the sample may be buried in noise, or non-existent particles may be erroneously imaged.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device and an image processing method that can improve the accuracy of removing noise contained in an observed image.

[0007] In order to achieve the above object, the present invention provides an image processing device having a calculation unit that removes noise from an image, wherein the calculation unit is characterized by having a noise remover that performs machine learning using a correct image, which is a three-dimensional image that does not contain noise or has an extremely small amount of noise, and a noisy image, which is a three-dimensional image in which noise has been added to the correct image, as learning data, and adjusts parameters so that the noisy image approaches the correct image.

[0008] The present invention also provides an image processing method executed by an image processing device having an arithmetic unit that removes noise from an image, characterized by comprising an acquisition step of acquiring a reference image, which is a three-dimensional image that does not contain noise or has an extremely small amount of noise, and a noise image, which is a three-dimensional image in which noise has been added to the reference image, and a generation step of performing machine learning on the reference image and the noise image as learning data, and generating a noise remover by adjusting parameters so that the noise image approaches the reference image.

[0009] The present invention also provides an image processing device having a calculation unit that removes noise from an image, characterized in that the calculation unit removes noise from the image using a noise remover generated by machine learning using as learning data a correct image, which is a three-dimensional image that does not contain noise or has an extremely small amount of noise, and a noise image, which is a three-dimensional image in which noise has been added to the correct image.

[0010] According to the present invention, it is possible to provide an image processing device and an image processing method that can improve the accuracy of removing noise contained in an observation image.

[0011] Overall configuration diagram of an image processing device. Figure showing an example of the processing flow of Example 1. Figure showing an example of the generation processing flow of the noise remover of Example 1. Figure showing an example of a three-dimensional image. Figure showing an example of the noise removal processing flow of Example 1. Figure showing an example of the generation processing flow of the noise remover of Example 2. Figure explaining decomposition of a three-dimensional image. Figure showing an example of a GUI for creating learning data. Figure showing an example of the noise removal processing flow of Example 2. Figure showing another example of the generation processing flow of the noise remover of Example 2.

[0012] Hereinafter, an embodiment of an image processing device and an image processing method according to the present invention will be described with reference to the accompanying drawings. In the following description and the accompanying drawings, components having the same functional configuration will be designated by the same reference numerals, and redundant description will be omitted.

[0013] 1 is a diagram showing the hardware configuration of an image processing device 1. The image processing device 1 is configured by a calculation unit 2, a memory 3, a storage device 4, and a network adapter 5, all of which are connected via a system bus 6 so as to be capable of transmitting and receiving signals. The image processing device 1 is also connected via a network 9 to a charged particle beam device 10, an image database 11, and a machine learning processing device 12 so as to be capable of transmitting and receiving signals. A display device 7 and an input device 8 are also connected to the image processing device 1. Here, "capable of transmitting and receiving signals" refers to a state in which signals can be transmitted and received between each other or from one device to the other, regardless of whether the transmission is electrically or optically wired or wireless.

[0014] The calculation unit 2 is a device that controls the operation of each component, and specifically, is a CPU (Central Processing Unit) or MPU (Micro Processor Unit). The calculation unit 2 loads programs stored in the storage device 4 and data required for program execution into the memory 3, executes them, and performs various image processing on images. The memory 3 stores programs executed by the calculation unit 2 and intermediate data of calculation processing. The storage device 4 is a device that stores programs executed by the calculation unit 2 and data required for program execution, and specifically, is a HDD (Hard Disk Drive) or SSD (Solid State Drive). The network adapter 5 connects the image processing device 1 to a network 9 such as a LAN (Local Area Network), a telephone line, or the Internet. Various data handled by the calculation unit 2 may be transmitted and received from outside the image processing device 1 via the network 9 such as a LAN.

[0015] The display device 7 is a device that displays the processing results, etc., of the image processing device 1, and is specifically a liquid crystal display, etc. The input device 8 is an operation device with which an operator issues operation instructions to the image processing device 1, and is specifically a keyboard, mouse, touch panel, etc. The mouse may be another pointing device such as a trackpad or trackball.

[0016] The charged particle beam device 10 is a device that generates an observation image of a sample by irradiating a sample with a charged particle beam, for example, an electron beam, and detecting secondary electrons, reflected electrons, transmitted electrons, and X-ray photons emitted from the sample. The image database 11 is a database system that stores the observation images generated by the charged particle beam device 10, images other than the observation images, images obtained by applying image processing to the observation images, and the like.

[0017] Charged particle beams such as electron beams irradiated onto a sample can damage the sample, so a lower dose is desirable. However, the lower the dose, the greater the amount of noise contained in the generated observation image, which can hinder analysis of the observation image. Therefore, it is essential to remove noise contained in the observation image, and a higher level of noise removal accuracy is required. Note that noise removal is also important for images other than the observation image.

[0018] An example of the flow of processing executed in the first embodiment will be described step by step with reference to FIG.

[0019] (S201) The calculation unit 2 generates a noise remover that removes noise contained in the observed image and images other than the observed image through machine learning. The noise remover is configured using, for example, a convolutional neural network (CNN). To generate the noise remover, a clean three-dimensional image that does not contain noise or a ground truth image that is a three-dimensional image with extremely low noise is used as a training image. Note that the clean three-dimensional image that does not contain noise is generated using a simulator or the like, and the three-dimensional image with extremely low noise is generated by irradiating a sample with a high dose of electron beam. Furthermore, a noise image, which is a three-dimensional image in which noise is added to the ground truth image, is used as the input image when generating the noise remover. By using three-dimensional images as the training image and input image, not only the relationship between pixels in the observed image but also the continuity in a direction perpendicular to the observed image is learned, thereby improving the accuracy of noise removal.

[0020] An example of the processing flow of S201 will be described step by step with reference to FIG.

[0021] (S301) The calculation unit 2 acquires a three-dimensional correct image. The three-dimensional correct image may be generated from an observation image captured by the charged particle beam device 10, or may be read from the storage device 4 or the image database 11.

[0022] An example of a three-dimensional image will be described using FIG. 4. FIG. 4 illustrates a three-dimensional image 401 in which observation images 400 captured by the charged particle beam device 10 are stacked in the time direction. That is, the three-dimensional image 401 is generated by arranging the observation images 400 formed in the XY plane in the T direction. The three-dimensional image 401 may be generated using a simulator or the like. The three-dimensional image 401 may also be a stereoscopic image generated by arranging the observation images 400 formed in the XY plane in the Z direction. Returning to the description of FIG. 3.

[0023] (S302) The calculation unit 2 generates a three-dimensional noise image by adding noise to the three-dimensional ground truth image acquired in S301. The noise added to the ground truth image is, for example, noise that occurs when the electron beam is at a low dose, and is generated by a simulator or the like.

[0024] (S303) The calculation unit 2 performs machine learning using the three-dimensional target image and the noise image as learning data, and generates a noise remover by adjusting the parameters of the CNN so that the noise image approaches the target image.

[0025] 3, a noise remover is generated using the three-dimensional ground truth image and the noise image as training data. The generated noise remover is stored in, for example, the storage device 4. Returning to the description of FIG. 2,

[0026] (S202) The calculation unit 2 uses the noise remover generated in S201 to remove noise contained in the observation image captured by the charged particle beam device 10, for example.

[0027] An example of the processing flow of S202 will be described step by step with reference to FIG.

[0028] (S501) The calculation unit 2 acquires a three-dimensional observation image. The three-dimensional observation image is, for example, an image in which observation images captured by the charged particle beam device 10 are stacked in the time direction.

[0029] (S502) The calculation unit 2 uses the noise remover generated in S201 to remove noise contained in the three-dimensional observation image acquired in S501.

[0030] Noise contained in a three-dimensional observation image is removed by the processing flow illustrated in Fig. 5. A noise remover generated using a three-dimensional ground truth image and a noise image as training data is used to remove noise contained in the observation image, so noise can be removed with higher accuracy. Note that noise contained in images other than the observation image may also be removed by the noise remover.

[0031] As described above, in Example 1, a noise remover is generated by machine learning using a three-dimensional ground truth image and a noise image as learning data, and noise contained in the observed image is removed using the noise remover. By removing noise, it is possible to prevent the detailed structure of the sample from being buried in noise or non-existent particles from being erroneously imaged, and analysis of the observed image is not hindered.

[0032] In Example 1, a method for removing noise contained in an observed image using a noise remover generated by machine learning using a 3D ground truth image and a noise image as training data was described. If a 3D image is used as training data, the size of the generated noise remover increases, and noise removal may take a long time. In Example 2, a method for speeding up noise removal by reducing the size of the noise remover is described. Note that some of the configurations and functions described in Example 1 can be applied to Example 2, and therefore, a description of similar configurations and functions will be omitted.

[0033] 2 illustrates the flow of the process executed in the second embodiment, similarly to the first embodiment. That is, in S201, a noise remover is generated by machine learning, and in S202, noise is removed by the noise remover.

[0034] An example of the processing flow of S201 in the second embodiment will be described step by step with reference to FIG.

[0035] (S601) The calculation unit 2 acquires a three-dimensional correct image in the same manner as in S301.

[0036] (S602) The calculation unit 2 decomposes the three-dimensional correct image acquired in S601 into three directions.

[0037] The decomposition of the three-dimensional image 401 will be described using FIG. 7 . The three-dimensional image 401 illustrated in FIG. 7 is an image in which observation images 400 formed in the XY plane are stacked in the T direction. The decomposition of the three-dimensional image 401 is performed by cutting out from the three-dimensional image 401 a plurality of two-dimensional images parallel to the XY plane, the YT plane, and the TX plane, which are planes perpendicular to the T direction, the X direction, and the Y direction, respectively. That is, a plurality of XY plane images 701, a YT plane image 702, and a TX plane image 703 are cut out from the three-dimensional image 401. The XY plane image 701 contains data related to the relationship between pixels in the observation image 400, and the YT plane image 702 and the TX plane image 703 contain data related to the continuity in the T direction. Returning to the description of FIG. 6 .

[0038] (S603) The calculation unit 2 generates two-dimensional noise images by adding noise to each of the two-dimensional ground truth images generated by the decomposition in S602. That is, noise is added to each of the multiple XY plane images 701, YT plane images 702, and TX plane images 703 cut out from the three-dimensional ground truth image.

[0039] (S604) The calculation unit 2 performs machine learning using the two-dimensional ground truth image generated in S602 and the two-dimensional noisy image generated in S603 as learning data, and generates a noise remover by adjusting the parameters of the CNN so that the two-dimensional noisy image approaches the two-dimensional ground truth image.

[0040] According to the process flow illustrated in Fig. 6, a noise remover is generated using a two-dimensional ground truth image and a noise image as training data. Because two-dimensional images are used as training data, the size of the generated noise remover is smaller than when three-dimensional images are used as training data, and the time required for noise removal can be shortened. Note that when generating the noise remover, the user may set image parameters and training parameters.

[0041] An example of a GUI (Graphical User Interface) for creating learning data used to set image parameters and learning parameters will be described with reference to Fig. 8 . The GUI illustrated in Fig. 8 includes an image parameter setting unit 810, an image display unit 820, and a learning parameter setting unit 830. Image parameters are set in the image parameter setting unit 810. A correct image and a noise image according to the settings in the image parameter setting unit 810 are displayed in the image display unit 820. Learning parameters are set in the learning parameter setting unit 830.

[0042] The image parameter setting unit 810 has a correct image setting unit 811, a time range setting unit 812, a direction setting unit 813, a noise type setting unit 814, and a noise level setting unit 815. The correct image setting unit 811 specifies which of a plurality of correct images stored in the storage device 4 will be used as learning data. The time range setting unit 812 sets the time range of the correct image specified in the correct image setting unit 811. The direction setting unit 813 sets the direction of the image displayed on the image display unit 820. The noise type setting unit 814 sets the type of noise to be added to the correct image. The noise level setting unit 815 sets the level of noise to be added to the correct image. Note that a noise remover may be generated according to the type and level of noise. Generating a noise remover according to the type and level of noise allows the user to select an appropriate one from the plurality of generated noise removers.

[0043] The learning parameter setting section 830 has a learning goal setting section 831, a validation setting section 832, and an execute button 833. In the learning goal setting section 831, the type of error that will be the learning goal is set. In the validation setting section 832, a target for completing learning is set. The execute button 833 is pressed when starting machine learning.

[0044] An example of the processing flow of S202 in the second embodiment will be described step by step with reference to FIG.

[0045] (S901) The calculation unit 2 acquires a three-dimensional observation image in the same manner as in S501.

[0046] (S902) The calculation unit 2 decomposes the three-dimensional observation image acquired in S901 into three directions. By decomposing the three-dimensional observation image, multiple two-dimensional images are generated in each of the three directions. Each of the multiple generated two-dimensional images includes noise generated during imaging.

[0047] (S903) The calculation unit 2 uses a noise remover to remove noise contained in each of the multiple two-dimensional images generated in S902. By the noise removal by the noise remover, multiple two-dimensional images after noise removal are generated.

[0048] (S904) The calculation unit 2 combines the multiple noise-removed two-dimensional images generated in S903 to generate a noise-removed three-dimensional image. The combination of the two-dimensional images is performed, for example, by averaging. That is, the pixel value at coordinates (X1, Y1, T1) is calculated as the average of the pixel value at coordinates (X1, Y1) in the XY plane image corresponding to T1, the pixel value at coordinates (Y1, T1) in the YT plane image corresponding to X1, and the pixel value at coordinates (T1, X1) in the TX plane image corresponding to Y1.

[0049] Noise contained in a three-dimensional observation image is removed by the process flow exemplified in Fig. 9. To remove noise contained in the observation image, a noise remover generated using a two-dimensional correct image and a noise image generated from a three-dimensional correct image as training data is used, thereby enabling high-speed noise removal with high removal accuracy. Note that the process flow for generating a noise remover using a two-dimensional correct image and a noise image as training data is not limited to that shown in Fig. 6.

[0050] Another example of the flow of the process of S201 in the second embodiment will be described step by step with reference to FIG.

[0051] (S1001) The calculation unit 2 acquires a three-dimensional correct image in the same manner as in S301.

[0052] (S1002) The calculation unit 2 generates a three-dimensional noise image in the same manner as in S302 by adding noise to the three-dimensional ground truth image acquired in S1001.

[0053] (S1003) The calculation unit 2 decomposes the 3D ground truth image acquired in S1001 and the 3D noise image generated in S1002 into three directions. By decomposing the 3D ground truth image and the noise image, multiple 2D images are generated in each of the three directions.

[0054] (S1004) The calculation unit 2 performs machine learning using the two-dimensional ground truth images and noise images, which are the multiple two-dimensional images generated in S1003, as learning data, and generates a noise remover by adjusting the parameters of the CNN so that the two-dimensional noise images approach the two-dimensional ground truth images.

[0055] The process flow illustrated in Fig. 10 generates a noise remover using a two-dimensional target image and a noise image as training data. Because two-dimensional images are used as training data, the size of the generated noise remover is smaller than when three-dimensional images are used as training data, and the time required for noise removal can be shortened. Note that noise removal by the noise remover generated by the process flow of Fig. 10 is performed according to Fig. 9.

[0056] The above describes the embodiments of the present invention. However, the present invention is not limited to the above embodiments, and the components can be modified and embodied without departing from the spirit of the invention. Furthermore, multiple components disclosed in the above embodiments may be combined as appropriate. Furthermore, some components may be omitted from all the components shown in the above embodiments.

[0057] 1: Image processing device, 2: Calculation unit, 3: Memory, 4: Storage device, 5: Network adapter, 6: System bus, 7: Display device, 8: Input device, 10: Charged particle beam device, 11: Image database, 400: Observation image, 401: Three-dimensional image, 701: XY plane image, 702: YT plane image, 703: TX plane image, 810: Image parameter setting unit, 811: Correct image setting unit, 812: Time range setting unit, 813: Direction setting unit, 814: Noise type setting unit, 815: Noise level setting unit, 820: Image display unit, 830: Learning parameter setting unit, 831: Learning goal setting unit, 832: Validation setting unit, 833: Execute button.

Claims

1. An image processing device having a calculation unit that removes noise from an image, wherein the calculation unit is equipped with a noise remover that performs machine learning using, as learning data, a correct image, which is a three-dimensional image that does not contain noise or has an extremely small amount of noise, and a noisy image, which is a three-dimensional image in which noise has been added to the correct image, and adjusts parameters so that the noisy image approaches the correct image.

2. An image processing device according to claim 1, wherein the calculation unit generates a plurality of two-dimensional images parallel to planes perpendicular to each direction by decomposing the target image in three directions, generates a second noise image which is a two-dimensional noise image by adding noise to each of the plurality of two-dimensional images, and generates the noise remover by machine learning the plurality of two-dimensional images and the second noise image as learning data.

3. An image processing device according to claim 1, wherein the calculation unit generates a plurality of two-dimensional images parallel to planes perpendicular to each direction by decomposing the correct image and the noise image in three directions, and generates the noise remover by machine learning using the plurality of two-dimensional images generated from the correct image and the plurality of two-dimensional images generated from the noise image as learning data.

4. An image processing device according to claim 1, wherein the calculation unit generates the noise remover in accordance with the type and level of the noise.

5. An image processing device according to claim 1, wherein the three-dimensional image with extremely low noise is an image generated by a charged particle beam device irradiating a sample with a high dose of charged particle beam.

6. An image processing method executed by an image processing device having a calculation unit that removes noise from an image, the image processing method comprising: an acquisition step of acquiring a reference image, which is a three-dimensional image that does not contain noise or has an extremely small amount of noise, and a noise image, which is a three-dimensional image in which noise has been added to the reference image; and a generation step of performing machine learning using the reference image and the noise image as learning data, and generating a noise remover by adjusting parameters so that the noise image approaches the reference image.

7. An image processing device having a calculation unit that removes noise from an image, wherein the calculation unit removes noise from the image using a noise remover generated by machine learning using as learning data a correct image, which is a three-dimensional image that does not contain noise or has an extremely small amount of noise, and a noisy image, which is a three-dimensional image in which noise has been added to the correct image.

Images