Program, image repair apparatus and image repair method

The program efficiently restores images with shot noise using a Gaussian Markov random field model, achieving high precision and reduced processing time, especially for scanning transmission electron microscope images.

JP2024160960APending Publication Date: 2024-11-15THE UNIV OF TOKYO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024071984
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-02
Filing Date
2024-04-25
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing image restoration methods for images containing shot noise suffer from low performance and require significant processing time.

Method used

A program configured to perform an acquisition step, an analysis step using a Markov random field model, and a generation step to generate an estimated image, utilizing a Gaussian Markov random field model to restore images containing shot noise.

Benefits of technology

The method enables high-precision image restoration in a short time, particularly for images captured by scanning transmission electron microscopes, with improved performance when multiple images are analyzed simultaneously.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160960000001_ABST
    Figure 2024160960000001_ABST
Patent Text Reader

Abstract

To provide a program or the like which can repair an image including shot noise at high precision in a short time.SOLUTION: According to one aspect of the invention, a program is provided. This program is configured to cause a computer to execute an acquisition step, an analysis step and a creation step. In the acquisition step, an observation image as an image including shot noise is acquired. In the analysis step, the observation image is analyzed using a Markov random field model. In the creation step, an estimated image estimated as the original image of the observation image is created based on the observation image and the analyzed result.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a program, an image restoration device, and an image restoration method. [Background technology]

[0002] Patent Document 1 discloses a technique for restoring an image containing Gaussian noise. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Journal of the Physical Society of Japan,2016,Vol.85,No.6 Summary of the Invention [Problem to be solved by the invention]

[0004] A regularization method is known as a technique for restoring an image containing shot noise. However, the regularization method has problems in that the image restoration performance is not very high and it takes a long time to restore the image.

[0005] In view of the above circumstances, the present invention provides a program etc. capable of restoring an image containing shot noise with high accuracy and in a short time. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a program. The program is configured to cause a computer to execute an acquisition step, an analysis step, and a generation step. In the acquisition step, an observed image that is an image including shot noise is acquired. In the analysis step, the observed image is analyzed using a Markov random field model. In the generation step, an estimated image that is estimated as an original image of the observed image is generated based on the observed image and the analysis result.

[0007] According to this aspect, an image containing shot noise can be restored with high precision and in a short time. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a hardware configuration of an image restoration device 100. FIG. [Diagram 2] FIG. 2 is a block diagram showing functions realized by the image restoration device 100 (control unit 110). [Diagram 3] FIG. 2 is an activity diagram showing the flow of information processing executed by the image restoration device 100. [Figure 4] FIG. 13 is a diagram showing the distribution of Bayes free energy in an observed image. [Diagram 5] FIG. 13 is a diagram showing a simulation image of a square. [Figure 6] 6A to 6C are diagrams showing the results of processing the observation images in FIG. 5, respectively. [Figure 7] FIG. 13 is a diagram showing a simulated image of a rectangle. [Figure 8] 8A to 8C are diagrams showing the results of processing the observation images in FIG. 7, respectively. [Figure 9] An image of an object actually captured by a scanning transmission electron microscope (hereinafter referred to as "actual data") [Figure 10] 10A to 10C are diagrams showing the results of processing the actual data in FIG. 9, respectively. [Figure 11] FIG. 13 is a diagram showing the results of image restoration processing using multiple simulation images simultaneously. [Figure 12] FIG. 13 is a diagram showing the results of image restoration processing using an average image obtained by averaging a number of simulation images. [Figure 13] 13 is a graph showing a change in RMSE when the number of simulation images used is changed. [Figure 14] FIG. 13 is a diagram showing the results of image restoration processing using multiple pieces of actual data simultaneously. [Figure 15] FIG. 13 is a diagram showing the results of image restoration processing using an average image obtained by averaging multiple pieces of actual data. [Figure 16] 1A to 1C are diagrams showing the results of imaging an object to be observed using a scanning transmission electron microscope equipped with the program of this embodiment. [Figure 17] FIG. 13 shows the results of inpainting a simulated image using a Poisson-Markov Random Field model or a Gaussian-Markov Random Field model. [Figure 18] 13A and 13B are diagrams showing the results of restoring an average image of multiple observed images, or restoring multiple observed images by simultaneously analyzing them, using a Poisson-Markov random field model or a Gaussian-Markov random field model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiments can be combined with each other.

[0010] Incidentally, a program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server, or may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] In addition, in various information processing according to an embodiment, an input and an output according to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of information referenced in such information processing (hereinafter referred to as reference information) is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a judgment formula such as a regression formula constructed by a statistical method), or may be a trained model that has previously trained the correlation between the input and the output, or may be a large-scale language model that can output a desired result by inputting a prompt.

[0012] In one embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In one embodiment, various information is handled, and this information is represented by, for example, physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit collection consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be performed on the circuit in the broad sense.

[0013] Furthermore, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, and a memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] 1. Hardware Configuration In the first section, the hardware configuration of the image restoration device 100 will be described.

[0015] 1 is a block diagram showing a hardware configuration of an image restoration device 100. The image restoration device 100 has a control unit 110, a storage unit 120, a display unit 130, an input unit 140, and a communication unit 150, and these components are electrically connected via a communication bus 160 inside the image restoration device 100. The image restoration device 100 is connected to a scanning transmission electron microscope (not shown) via a network. Each component will be further described.

[0016] The control unit 110 performs processing and control of the overall operation related to the image restoration device 100. The control unit 110 is, for example, a central processing unit (CPU) not shown. The control unit 110 realizes various functions related to the image restoration device 100 by reading out a predetermined program stored in the storage unit 120. That is, information processing by software stored in the storage unit 120 can be specifically realized by the control unit 110, which is an example of hardware, and executed as each functional unit included in the control unit 110. These will be further described in Section 2. Note that the control unit 110 is not limited to being single, and may be implemented to have multiple control units 110 for each function. Also, a combination of these may be used.

[0017] The storage unit 120 stores various information necessary for information processing of the image restoration device 100. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs related to the image restoration device 100 executed by the control unit 110, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the calculation of the program. Also, it may be a combination of these.

[0018] The display unit 130 may be included in the housing of the image restoration device 100 or may be externally attached. The display unit 130 displays a screen of a graphical user interface (GUI) that can be operated by a user. This is preferably implemented by using display devices such as a CRT display, a liquid crystal display, an organic EL display, and a plasma display depending on the type of the image restoration device 100. In the following description, the display unit 130 is assumed to be included in the housing of the image restoration device 100.

[0019] The input unit 140 may be included in the housing of the image restoration device 100, or may be externally attached. For example, the input unit 140 may be implemented as a touch panel integrated with the display unit 130. If it is a touch panel, the user can input a tap operation, a swipe operation, or the like. Of course, a switch button, a mouse, a QWERT keyboard, or the like may be adopted instead of the touch panel. That is, the input unit 140 accepts an operation input made by the user. The input is transferred to the control unit 110 as a command signal via the communication bus 160. Then, the control unit 110 can execute a predetermined control or calculation as necessary.

[0020] The communication unit 150 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as 5G / LTE / 3G, Bluetooth (registered trademark) communication, etc. as necessary. In other words, it is more preferable to implement it as a collection of multiple communication means. In other words, the image restoration device 100 communicates various information (e.g., electron microscope images) with the scanning transmission electron microscope via the communication unit 150 and the network.

[0021] 2. Functional configuration The functional configuration of this embodiment will be described in Section 2. As described above, information processing by software stored in the storage unit 120 is specifically realized by the control unit 110, which is an example of hardware, and can be executed as each functional unit included in the control unit 110.

[0022] FIG. 2 is a block diagram showing functions realized by the image restoration device 100 (control unit 110). As described above, the image restoration device 100 includes the control unit 110. Specifically, the image restoration device 100 (control unit 110) is configured to be able to execute each step of the program of this embodiment. The image restoration device 100 (control unit 110) includes an acquisition unit 111, an analysis unit 112, and a generation unit 113 corresponding to each step of the program of this embodiment. Here, the program of this embodiment is configured to cause a computer such as the image restoration device 100 to execute the acquisition step, analysis step, and generation step.

[0023] A scanning transmission electron microscope obtains an observation image by irradiating an observation object with an electron beam. The observation object is damaged by the irradiation of the electron beam. Therefore, it is necessary to reduce the amount of irradiation of the electron beam to a certain extent, but in this case, an observation image containing a lot of shot noise is obtained. Therefore, in this embodiment, an estimated image is generated that is estimated as an original image that does not contain shot noise.

[0024] The acquiring unit 111 is configured to acquire various pieces of information. The acquiring unit 111 is configured to execute an acquiring step. For example, the acquiring unit 111 acquires an observed image, which is an image including shot noise, from a scanning transmission electron microscope.

[0025] The analysis unit 112 is configured to analyze various information. The analysis unit 112 is configured to perform an analysis step. For example, the analysis unit 112 analyzes the acquired observation image using a Markov random field model.

[0026] The generating unit 113 is configured to generate various information. The generating unit 113 is configured to execute a generating step. For example, the generating unit 113 generates an estimated image that is estimated as an original image of the observed image based on the acquired observed image and the analyzed result.

[0027] 3. Overview of Image Restoration Methods (1) In Section 3, an overview of the image restoration method performed by the image restoration device 100 described above will be given.

[0028] The image restoration device 100 executes image processing based on the Bayesian inference framework. In this image processing, the observed image is analyzed using a Gaussian-Markov random field model. This Gaussian-Markov random field model assumes that there is an interaction between adjacent pixel values ​​of the image, and that the pixel values ​​are spatially continuous. It is also assumed that the observed image contains Gaussian noise.

[0029] Below, we explain based on the Gaussian-Markov random field model. Here, we formulate the image restoration method from one 2D observed image. The original image is assumed to follow the probability distribution of equation (1).

[0030]

number

[0031] Here, u is the original image, a is a hyperparameter that represents the smoothness of the original image, and N 2 indicates the number of pixels in the original image. The observed image follows the probability distribution of equation (2).

[0032]

number

[0033] Here, v is the observed image, and b is a hyperparameter that represents the noise intensity of the observed image. The posterior probability of the original image is given by equation (3) using Bayes' theorem.

[0034]

number

[0035] The estimated image that maximizes the posterior probability of the original image is given by equation (4).

[0036]

number

[0037] Next, we will explain how to estimate the hyperparameters a and b. Both hyperparameters a and b are considered to be high-order random variables, and the hyperparameters a and b are estimated by maximizing the posterior probability. The posterior probability of the hyperparameters a and b is given by equation (5).

[0038]

number

[0039] Here, by applying the discrete Fourier transform to equation (5), the multiple integrals are converted into the product of each Fourier integral. This allows the integral calculation to be performed precisely using Gaussian integrals. Transforming equation (5) gives equation (6).

[0040]

number

[0041] Now, by using logarithms in equation (6), we get equation (7).

[0042]

number

[0043] Here, F is the Bayes free energy. The Bayes free energy of the hyperparameters a and b can be calculated using equation (7). In addition, equation (7) can be used to determine not only the ratio of a to b, but also a and b individually.

[0044] According to this aspect, a model capable of restoring an image containing Gaussian noise can be applied to the purpose of restoring an image containing shot noise.

[0045] 4. Image restoration method flow In the fourth section, the flow of the image restoration method of the image restoration device 100 described above will be described. This image restoration method includes each step of the program of this embodiment. This image restoration method includes an acquisition step, an analysis step, and a generation step. In the acquisition step, an observed image that is an image containing shot noise is acquired. In the analysis step, the observed image is analyzed using a Markov random field model. In the generation step, an estimated image that is estimated as an original image of the observed image is generated based on the acquired observed image and the analysis result.

[0046] 3 is an activity diagram showing the flow of information processing executed by the image restoration device 100. Below, an explanation will be given along with each activity of this activity diagram. Here, the observed image is assumed to be a scanning transmission electron microscope image captured by a scanning transmission electron microscope. Therefore, the observed image is assumed to be an image including shot noise.

[0047] First, the scanning transmission electron microscope captures an image of an object to be observed (activity A110). Here, the imaging principle of the scanning transmission electron microscope will be explained. The scanning transmission electron microscope radiates a plane wave from a light source at a distant point. The radiated plane wave is incident on the object to be observed at a convergence angle α1 by the objective lens, and the object to be observed is scanned in a raster pattern by the scan coil. Electrons that are transmitted and scattered by the object to be observed form a convergent beam electron diffraction pattern. After that, the intensity projected at a certain point by the projection lens is measured by a detector at an angle α2. The measurement is then displayed on a monitor in synchronization with the scan.

[0048] Next, the scanning transmission electron microscope stores an observation image of the observation target (activity A120). That is, the control unit in the scanning transmission electron microscope stores the observation image in the storage unit in the scanning transmission electron microscope.

[0049] Next, the scanning transmission electron microscope transmits the observed image to the image restoration device 100 (activity A130). The control unit of the scanning transmission electron microscope reads out the observed image stored in the memory unit of the scanning transmission electron microscope. Next, the control unit of the scanning transmission electron microscope transmits the observed image to the image restoration device 100 via the communication unit of the scanning transmission electron microscope.

[0050] Next, the control unit 110 in the image restoration device 100 acquires an observed image from the scanning transmission electron microscope (activity A140). That is, in the acquisition step, an observed image is acquired.

[0051] In activity A140, for example, the following two-stage information processing is executed: (1) The communication unit 150 receives the observation image transmitted from the scanning transmission electron microscope. (2) The control unit 110 causes the memory unit 120 to store the received observation image.

[0052] Next, the control unit 110 in the image restoration device 100 calculates the Bayes free energy of the hyperparameters a and b from the observed image (activity A150). The hyperparameters are parameters used in analyzing the observed image. That is, in the analysis step, the Bayes free energy of the hyperparameters a and b is calculated from the observed image.

[0053] In activity A150, for example, the following three-stage information processing is executed. (1) The control unit 110 reads out the observed image stored in the storage unit 120. (2) The control unit 110 executes a calculation process to calculate the Bayes free energy from the observed image. (3) The control unit 110 stores data related to the calculated Bayes free energy in the storage unit 120.

[0054] Next, the control unit 110 in the image restoration device 100 searches for the minimum value of the Bayes free energy of the hyperparameters a and b (activity A160). The control unit 110 estimates the hyperparameters a and b based on the minimum value of the Bayes free energy searched for. That is, in the analysis step, the hyperparameters a and b are estimated by minimizing the Bayes free energy.

[0055] In activity A160, for example, the following four-stage information processing is executed. (1) The control unit 110 reads out the Bayes free energy data stored in the memory unit 120. (2) The control unit 110 executes a search process to search for the minimum value of the Bayes free energy. (3) The control unit 110 estimates the hyperparameters a and b based on the searched minimum value of the Bayes free energy. (4) The control unit 110 stores the estimated hyperparameters a and b in the memory unit 120.

[0056] Activities A150 to A160 can simplify the number of digits handled when estimating hyperparameters.

[0057] Next, the control unit 110 in the image restoration device 100 generates an estimated image that is estimated as an original image of the observed image (activity A170). That is, in the generating step, an estimated image is generated based on the observed image and the analyzed result.

[0058] In activity A170, for example, the following three-stage information processing is executed: (1) The control unit 110 reads out the observed image stored in the storage unit 120. (2) The control unit 110 reads out the estimated hyperparameters a and b stored in the storage unit 120. (3) The control unit 110 executes a generation process to generate an estimated image.

[0059] In activities A150 and A160, the analysis step may analyze all the observed images substantially simultaneously. The analysis process in this analysis step will be described in Sections 5 and 6.

[0060] According to such an aspect of the analysis step, an image including shot noise can be restored with high accuracy. Thus, according to the aspect of the present embodiment, it is particularly suitable for use with images captured by a scanning transmission electron microscope.

[0061] 5. Overview of Image Restoration Methods (2) In Section 5, we will explain the outline of the image restoration method executed in the activities A150 and A160 explained in Section 4. That is, we will formulate an image restoration method that uses multiple 2D observation images substantially simultaneously.

[0062] The probability distribution that the original image follows is the same as in the case of one image, and follows equation (8).

[0063]

number

[0064] Here, u is the original image, a is a hyperparameter that represents the smoothness of the original image, and N 2indicates the number of pixels in the original image. Here, it is assumed that T observed images are acquired. The probability distribution of an observed image obtained by using T observed images approximately simultaneously is assumed to follow equation (9).

[0065]

number

[0066] Here, v is the observed image, and b is a hyperparameter that represents the noise intensity of the observed image. The posterior probability of the original image is given by equation (10) using Bayes' theorem.

[0067]

number

[0068] The estimated image that maximizes the posterior probability of the original image is given by equation (11).

[0069]

number

[0070] Next, we will explain the estimation of hyperparameters a and b. Both hyperparameters a and b are considered to be high-order random variables, and the hyperparameters a and b are estimated by maximizing the posterior probability. The posterior probability of the hyperparameters a and b is given by equation (12).

[0071]

number

[0072] Now, by using logarithms in equation (12), we get equation (13).

[0073]

number

[0074] Here, F is the Bayes free energy. The Bayes free energy of the hyperparameters a and b can be calculated using equation (13). In addition, equation (13) can be used to determine not only the ratio of a to b, but also a and b individually.

[0075] 6. Overview of Image Restoration Methods (3) In Section 6, we will explain the outline of the image restoration method executed in activities A150 and A160 explained in Section 4. That is, we will formulate an image restoration method in which multiple two-dimensional observation images are averaged and then analyzed.

[0076] The probability distribution that the original image follows is the same as in the case of one image, and follows equation (14).

[0077]

number

[0078] Here, u is the original image, a is a hyperparameter that represents the smoothness of the original image, and N 2 indicates the number of pixels in the original image. Here, it is assumed that T observed images are acquired. The probability distribution of the average image obtained by averaging the observed images is assumed to follow equation (15).

[0079]

number

[0080] Here, m is the average image, and b is a hyperparameter that represents the noise intensity of the observed image. The posterior probability of the original image is given by Equation (16) using Bayes’ theorem.

[0081]

number

[0082] The estimated image that maximizes the posterior probability of the original image is given by equation (17).

[0083]

number

[0084] Next, we will explain the estimation of hyperparameters a and b. Both hyperparameters a and b are considered to be high-order random variables, and the hyperparameters a and b are estimated by maximizing the posterior probability. The posterior probability of the hyperparameters a and b is given by equation (18).

[0085]

number

[0086] Now, by using logarithms in equation (18), we obtain equation (19).

[0087]

number

[0088] Here, F is the Bayes free energy. The Bayes free energy of the hyperparameters a and b can be calculated using equation (19). In addition, equation (19) can be used to determine not only the ratio of a to b, but also a and b individually.

[0089] 7. Example 1 In Section 7, an embodiment in which the image restoration methods in Sections 4 to 6 are executed will be described.

[0090] Figure 4 shows the distribution of Bayes free energy in observed images. Figure 4(A) shows the distribution when hyperparameters a and b are estimated by averaging 20 observed images. Figure 4(B) shows the distribution when hyperparameters a and b are estimated by analyzing 20 observed images approximately simultaneously. In Figure 4, b is set to 10, and it is found that analyzing 20 observed images approximately simultaneously results in higher performance.

[0091] Fig. 5 shows a square simulation image. Fig. 5(A) shows an original image with 512 x 512 pixels. Fig. 5(B) shows one of 20 observed images obtained when an object to be observed is imaged using a scanning transmission electron microscope. Here, the noise intensity (hyper parameter) b of the observed image is set to 10. Fig. 5(C) shows an average image obtained by averaging the 20 observed images.

[0092] FIG. 6 shows the results of processing the observed images in FIG. 5. FIG. 6(A) shows an average image obtained by averaging 20 observed images (the same image as FIG. 5(C)). FIG. 6(B) shows a restored image (hereinafter also referred to as an "estimated image") from the average image in FIG. 6(A). FIG. 6(C) shows a restored image obtained by analyzing 20 observed images almost simultaneously. In FIG. 6(A), the root mean squared error (RMSE) was 0.70675, and in FIG. 6(B), the RMSE was 0.30912, the estimated value of a was 0.7063, the estimated value of b was 6.5000, and the restoration time was 112 milliseconds. In FIG. 6(C), the RMSE was 0.22465, the estimated value of a was 0.5188, the estimated value of b was 10.000, and the restoration time was 203 milliseconds.

[0093] From Fig. 6, we found the following: (1) An image cannot be restored simply by averaging 20 observed images, and (2) the restoration performance is higher when 20 observed images are analyzed almost simultaneously and restored than when the image is restored from the average image of the 20 observed images.

[0094] FIG. 7 shows a rectangular simulation image. FIG. 7(A) shows an original image with 264×240 pixels. FIG. 7(B) shows one of 20 observed images obtained when an object to be observed is imaged using a scanning transmission electron microscope. Here, the noise intensity (hyper parameter) b of the observed image is set to 10. FIG. 7(C) shows an average image obtained by averaging the 20 observed images.

[0095] FIG. 8 shows the results of processing the observed images in FIG. 7. FIG. 8(A) shows an average image obtained by averaging 20 observed images (the same image as FIG. 7(C)). FIG. 8(B) shows a restored image from the average image in FIG. 8(A). FIG. 8(C) shows a restored image obtained by analyzing 20 observed images almost simultaneously. In FIG. 8(A), the RMSE was 0.70507, and in FIG. 8(B), the RMSE was 0.34350, the estimated value of a was 0.8313, the estimated value of b was 6.0, and the restoration time was 46 milliseconds. In FIG. 8(C), the RMSE was 0.24574, the estimated value of a was 0.6125, the estimated value of b was 10.0, and the restoration time was 65 milliseconds.

[0096] According to FIG. 8, it is understood that the image restoration device 100 can be applied to a rectangular simulation image as well.

[0097] Fig. 9 shows images (hereinafter also referred to as "actual data") of an observation target actually captured by a scanning transmission electron microscope. In Fig. 9, 20 observation images were obtained. Fig. 9(A) shows one of the 20 observation images. Fig. 9(B) shows an average image obtained by averaging the 20 observation images.

[0098] Fig. 10 shows the results of processing the real data in Fig. 9. Fig. 10(A) shows a restored image from the average image in Fig. 9(B). Fig. 10(B) shows a restored image obtained by analyzing 20 observed images of the real data almost simultaneously. Fig. 10(C) shows an image obtained by restoring the observed image of the real data using a regularization method.

[0099] 10, the following was found: (1) the image restoration device 100 can be applied to real data; (2) the restoration performance is higher when the image restoration device 100 is used than when a regularization method is used; and (3) even when the real data is used, the restoration performance is higher when 20 observed images are analyzed and restored at approximately the same time than when the image is restored from an average image of the 20 observed images.

[0100] FIG. 11 shows the results of image restoration processing using multiple simulation images simultaneously. FIG. 11(A) shows the results using one simulation image. FIG. 11(B) shows the results using two simulation images simultaneously. FIG. 11(C) shows the results using four simulation images simultaneously. FIG. 11(D) shows the results using eight simulation images simultaneously. FIG. 11(E) shows the results using 16 simulation images simultaneously.

[0101] FIG. 12 shows the results of image restoration using an average image obtained by averaging multiple simulation images. FIG. 12(A) shows the results using one simulation image. FIG. 12(B) shows the results using an average image of two simulation images. FIG. 12(C) shows the results using an average image of four simulation images. FIG. 12(D) shows the results using an average image of eight simulation images. FIG. 12(E) shows the results using an average image of 16 simulation images.

[0102] Fig. 13 is a graph showing the transition of RMSE when the number of simulation images used is changed. Fig. 13(A) shows the transition of RMSE when image restoration processing is performed using multiple simulation images simultaneously. Fig. 13(B) shows the transition of RMSE when image restoration processing is performed using an average image of multiple simulation images.

[0103] 11 to 13, the following was found: (1) The more simulation images used, the higher the restoration performance. (2) As in Fig. 6, the restoration performance is higher when multiple simulation images are analyzed and restored almost simultaneously than when the average image of multiple simulation images is used.

[0104] Fig. 14 shows the results of image restoration processing using multiple pieces of real data simultaneously. Fig. 14(A) shows the results when one piece of real data is used. Fig. 14(B) shows the results when two pieces of real data are used simultaneously. Fig. 14(C) shows the results when four pieces of real data are used simultaneously. Fig. 14(D) shows the results when eight pieces of real data are used simultaneously. Fig. 14(E) shows the results when 16 pieces of real data are used simultaneously.

[0105] FIG. 15 shows the results of image restoration processing using an average image obtained by averaging multiple pieces of real data. FIG. 15(A) shows the results when one piece of real data is used. FIG. 15(B) shows the results when two pieces of real data are used as an average image. FIG. 15(C) shows the results when four pieces of real data are used as an average image. FIG. 15(D) shows the results when eight pieces of real data are used as an average image. FIG. 15(E) shows the results when 16 pieces of real data are used as an average image.

[0106] 14 and 15, the following was found: (1) As in FIG. 10, the image restoration device 100 can be applied to real data. (2) As in FIG. 10, as in the case of real data, restoration performance is higher when a plurality of real data are analyzed and restored at approximately the same time than when the plurality of real data are restored from an average image.

[0107] Fig. 16 is a diagram showing the result of imaging an object using a scanning transmission electron microscope equipped with the program of this embodiment. That is, this scanning transmission electron microscope corresponds to the image restoration device 100. Here, an image actually observed using this scanning transmission electron microscope is taken as an observed image, and an image obtained by restoring the observed image using the program of this embodiment is taken as a restored image. Fig. 16 shows an observed image and a restored image at an arbitrary time during observation, an observed image and a restored image 5 seconds after the arbitrary time, and an observed image and a restored image 10 seconds after the arbitrary time. The observed image and the restored image in Fig. 16 are images displayed on the same screen at the same time.

[0108] It can be seen that the observation image and the restored image at any given time point show the same part of the captured observation object. Similarly, it can be seen that the observation image and the restored image 5 seconds after the given time point, and the observation image and the restored image 10 seconds after the given time point show the same part of the captured observation object.

[0109] FIG. 16 shows that the scanning transmission electron microscope equipped with the program of this embodiment is capable of restoring an observation image of an observation target in a short time (almost in real time).

[0110] 8. Example 2 In Section 8, an embodiment will be described in which the image restoration methods in Sections 4 to 6 are executed using a Poisson-Markov Random Field model or a Gaussian-Markov Random Field model.

[0111] An image captured by a scanning transmission electron microscope (hereinafter also referred to as "STEM image") is degraded by shot noise, which is a type of Poisson noise. The method of restoring an image using a Gaussian-Markov random field model is applied on the assumption that the image is degraded by Gaussian noise. In other words, the method of restoring a STEM image using a Gaussian-Markov random field model approximates the degradation process of the image. On the other hand, the method of restoring an image using a Poisson-Markov random field model is applied on the assumption that the image is degraded by Poisson noise. In other words, the method of restoring a STEM image using a Poisson-Markov random field model incorporates the degradation process of the image. Therefore, it can be said that the method of restoring a STEM image using a Poisson-Markov random field model is theoretically more ideal than the method of restoring a STEM image using a Gaussian-Markov random field model. Therefore, in Example 2, the method of restoring a STEM image using a Poisson-Markov random field model and the method of restoring a STEM image using a Gaussian-Markov random field model were implemented, and their performances were compared.

[0112] Here, we show the probability distribution that the original image and the observed image follow when using the Poisson-Markov random field model. In this case, the original image follows the probability distribution of equation (20).

[0113]

number

[0114] Here, u is the original image, a is a hyperparameter that represents the smoothness of the original image, and N 2 indicates the number of pixels in the original image. The observed image is assumed to be degraded by Poisson noise and follows the probability distribution of Eq. (21).

[0115]

number

[0116] Here, v denotes the observed image. If we formulate a Poisson-Markov random field model by applying the probability distributions in equations (20) and (21), it can be said that the method of restoring a STEM image using the Poisson-Markov random field model is theoretically more ideal than the method of restoring a STEM image using the Gaussian-Markov random field model.

[0117] FIG. 17 is a diagram showing the results of restoring a simulation image using a Poisson-Markov random field model or a Gaussian-Markov random field model. Here, single-layer graphene was used as the observation target, and noise intensity was changed in three ways for comparison. FIG. 17(A) shows an original image with 128×128 pixels. FIG. 17(B) shows the results of restoring each input image using a Poisson-Markov random field model or a Gaussian-Markov random field model, with a simulation image in which weak noise was added to the original image, a simulation image in which moderate noise was added to the original image, and a simulation image in which strong noise was added to the original image prepared as input images.

[0118] As a result, the RMSE was 0.07106 when the Poisson-Markov random field model was used for input images with weak noise, and 0.06468 when the Gaussian-Markov random field model was used. The RMSE was 0.11788 when the Poisson-Markov random field model was used for input images in noise, and 0.11547 when the Gaussian-Markov random field model was used. The RMSE was 0.21736 when the Poisson-Markov random field model was used for input images with strong noise, and 0.20925 when the Gaussian-Markov random field model was used. In addition, the processing time when the Poisson-Markov random field model was used was more than 10 hours using a supercomputer, whereas the processing time when the Gaussian-Markov random field model was used was less than 1 second using a general laptop.

[0119] According to Fig. 17, we found the following: (1) The performance of the method for STEM image restoration using the Gaussian-Markov random field model is slightly better than that of the method for STEM image restoration using the Poisson-Markov random field model, and (2) the processing time of the method for STEM image restoration using the Gaussian-Markov random field model is much shorter than that of the method for STEM image restoration using the Poisson-Markov random field model.

[0120] Figure 18 shows the results of restoring an average image of multiple observed images, or restoring multiple observed images by simultaneously analyzing them, using a Poisson-Markov random field model or a Gaussian-Markov random field model. Here, strontium titanate (SrTiO3) was used as the observation target, and the number of observed images was changed to three different values ​​for comparison. Figure 18(A) shows an original image with 128 x 128 pixels. Figure 18(B) shows one of the observed images (actual data) of the observation target actually captured by a scanning transmission electron microscope. FIG. 18(C) shows the results when 20 observed images (actual data), 50 observed images (actual data), and 100 observed images (actual data) are prepared, and the average image of the observed images is used for restoration using a Gaussian-Markov random field model (hereinafter also referred to as "average image restoration"), and when all the observed images are used simultaneously for restoration using a Poisson-Markov random field model or a Gaussian-Markov random field model (hereinafter also referred to as "repair using the Poisson-Markov random field model" and "repair using the Gaussian-Markov random field model", respectively).

[0121] First, we show the results when 20 observed images were used. The RMSE for the average image restoration was 0.05731, the RMSE for the Poisson-Markov Random Field model restoration was 0.01404, and the RMSE for the Gaussian-Markov Random Field model restoration was 0.00955. Next, we show the results when 50 observed images were used. The RMSE for the average image restoration was 0.03482, the RMSE for the Poisson-Markov Random Field model restoration was 0.01199, and the RMSE for the Gaussian-Markov Random Field model restoration was 0.00819. Next, we show the results when 100 observed images were used. The RMSE for the average image restoration was 0.02475, the RMSE for the Poisson-Markov Random Field model restoration was 0.00988, and the RMSE for the Gaussian-Markov Random Field model restoration was 0.00725. In addition, the processing time when using the Poisson-Markov random field model was more than 10 hours using a supercomputer, whereas the processing time when using the Gaussian-Markov random field model was less than one second using an ordinary laptop.

[0122] According to Fig. 20, the following was found: (1) The restoration performance is higher when multiple observed images (actual data) are analyzed almost simultaneously and restored than when an average image of multiple observed images (actual data) is used for restoration. (2) The performance of the method for restoring STEM images using a Gaussian-Markov random field model is slightly better than that of the method for restoring STEM images using a Poisson-Markov random field model. (3) The processing time of the method for restoring STEM images using a Gaussian-Markov random field model is much shorter than that of the method for restoring STEM images using a Poisson-Markov random field model.

[0123] 9.Summary Section 9 provides a summary of Sections 1 through 8.

[0124] Normally, a regularization method is used to restore an image containing shot noise. This regularization method does not have a very high image restoration performance, and it takes a long time to restore the image. More specifically, when using a 1.8GHz CPU (product name: Intel Core i5), the regularization method takes 31 seconds to find one hyperparameter, so if 10 hyperparameters are to be found, it takes about 5 minutes.

[0125] The Gaussian-Markov Random Field model is used to restore an image containing Gaussian noise. The image restoration device 100 restores an image containing shot noise using the Gaussian-Markov Random Field model. The inventors of the present application have discovered that the regularization method is mathematically equivalent to the Gaussian-Markov Random Field model by examining the properties of the Markov Random Field model. That is, they have come to the conclusion that the Gaussian-Markov Random Field model works well for restoring an image containing Gaussian noise, even though the noise model is shot noise, which is a type of Poisson noise.

[0126] According to the aspects of the present embodiment, an image containing shot noise can be restored with high accuracy and in a short time. In addition, due to the simple configuration, the saved resources can be used for other core functions.

[0127] Although the embodiment of the present invention has been described above, the present invention is not limited to this, and can be modified as appropriate without departing from the technical concept of the invention.

[0128] 10. Variations In Section 10, a modification of this embodiment will be described.

[0129] 10-1.First modified example The control unit 110 performs write (storage) and read processes for various data and information in the memory unit 120, but this is not limited to this, and for example, information processing for each activity may be performed using a register or cache memory within the control unit 110.

[0130] 10-2. Second modified example In the analysis step, the observed image may be analyzed using various Markov random field models, without being limited to the Gaussian Markov random field model.

[0131] 10-3.Third modified example The observed image is not limited to a scanning transmission electron microscope image, but may be various electron microscope images (e.g., a scanning electron microscope image, a transmission electron microscope image, etc.). According to this aspect, images captured at a low dose in various electron microscopes can be restored with high accuracy.

[0132] 10-4.Fourth modified example In this embodiment, a two-dimensional image restoration method has been described, but the present invention is not limited to this, and a one-dimensional image restoration method may be used. The following description is based on a one-dimensional Gaussian-Markov random field model. The original image is assumed to follow the probability distribution of equation (22).

[0133]

number

[0134] Here, u is the original image, a is a hyperparameter that represents the smoothness of the original image, and N is the number of pixels in the original image. The observed image follows the probability distribution of equation (23).

[0135]

number

[0136] Here, v is the observed image, and b is a hyperparameter that represents the noise intensity of the observed image. The posterior probability of the original image is given by Equation (24) using Bayes’ theorem.

[0137]

number

[0138] The estimated image that maximizes the posterior probability of the original image is given by equation (25).

[0139]

number

[0140] Next, we will explain the estimation of hyperparameters a and b. Both hyperparameters a and b are considered to be high-order random variables, and the hyperparameters a and b are estimated by maximizing the posterior probability. The posterior probability of the hyperparameters a and b is given by equation (26).

[0141]

number

[0142] Here, by applying the discrete Fourier transform to equation (26), the multiple integrals are converted into the product of each Fourier integral. This allows the integral calculation to be performed precisely using Gaussian integrals. Transforming equation (26) gives equation (27).

[0143]

number

[0144] Now, by using logarithms in equation (27), we get equation (28).

[0145]

number

[0146] Here, F is the Bayes free energy. The Bayes free energy of the hyperparameters a and b can be calculated using equation (28). In addition, equation (28) can be used to determine not only the ratio of a to b, but also a and b individually.

[0147] According to this aspect, a model capable of restoring an image containing Gaussian noise can be applied to the purpose of restoring an image containing shot noise.

[0148] 11.Other It may be provided in any of the following ways:

[0149] (1) A program configured to cause a computer to execute an acquisition step, an analysis step, and a generation step, wherein in the acquisition step, an observed image that is an image including shot noise is acquired, in the analysis step, the observed image is analyzed using a Markov random field model, and in the generation step, an estimated image that is estimated as an original image of the observed image based on the observed image and the analysis results.

[0150] According to this aspect, an image including shot noise can be restored with high accuracy in a short time. In addition, because of the simple configuration, the saved resources can be used for other core functions.

[0151] (2) A program in accordance with the above (1), wherein in the analysis step, a Bayes free energy of a hyperparameter is calculated from the observed image, the hyperparameter being a parameter used in analyzing the observed image, and in the analysis step, the hyperparameter is estimated by minimizing the Bayes free energy.

[0152] According to this embodiment, it is possible to simplify the number of digits to be handled when estimating hyperparameters.

[0153] (3) The program according to (1) or (2) above, wherein the observed image is an electron microscope image.

[0154] According to this aspect, an image captured with a low dose of radiation in an electron microscope can be restored with high precision and in a short time.

[0155] (4) The program according to any one of (1) to (3) above, wherein the observed image is a scanning transmission electron microscope image.

[0156] According to this aspect, the present invention can be suitably used for images captured by a scanning transmission electron microscope.

[0157] (5) The program according to any one of (1) to (4) above, wherein the analysis step analyzes the observed image using a Gaussian-Markov random field model.

[0158] According to this aspect, a model capable of restoring an image containing Gaussian noise can be applied to the purpose of restoring an image containing shot noise.

[0159] (6) The program according to any one of (1) to (5) above, wherein in the analyzing step, all of the observation images are analyzed substantially simultaneously.

[0160] According to this embodiment, an image including shot noise can be restored with higher accuracy.

[0161] (7) An image restoration device comprising a control unit, the control unit being configured to be able to execute each step of the program recited in any one of (1) to (6) above.

[0162] According to this aspect, an image including shot noise can be restored with high accuracy in a short time. In addition, because of the simple configuration, the saved resources can be used for other core functions.

[0163] (8) An image restoration method comprising the steps of the program according to any one of (1) to (6) above.

[0164] According to this aspect, an image including shot noise can be restored with high accuracy in a short time. In addition, because of the simple configuration, the saved resources can be used for other core functions. Of course, this is not the case. [Explanation of symbols]

[0165] 100: Image restoration device 110: Control unit 111: Acquisition Department 112:Analysis Department 113 :Generation part 120: Storage section 130: Display section 140: Input section 150: Communications Department 160: Communication bus

Claims

1. A program, A method for generating a plurality of images, the method comprising: In the acquiring step, an observed image is acquired, which is an image including shot noise; In the analyzing step, the observed image is analyzed using a Markov random field model; In the generating step, an estimated image is generated based on the observed image and the result of the analysis, the estimated image being an original image of the observed image. program.

2. The program according to claim 1, In the analyzing step, a Bayes free energy of a hyperparameter is calculated from the observed image; The hyperparameters are parameters used in analyzing the observed image, In the analysis step, the hyperparameters are estimated by minimizing the Bayes free energy. program.

3. The program according to claim 1, The observation image is an electron microscope image. program.

4. The program according to claim 1, The observation image is a scanning transmission electron microscope image. program.

5. The program according to claim 1, In the analyzing step, the observed image is analyzed using a Gaussian-Markov random field model. program.

6. The program according to claim 1, In the analyzing step, all of the observation images are analyzed substantially simultaneously. program.

7. 1. An image restoration apparatus, comprising: A control unit is provided, The control unit is configured to be able to execute each step of the program according to any one of claims 1 to 6. Image restoration device.

8. 1. A method for image restoration, comprising: The program comprises the steps of any one of claims 1 to 6. Image restoration methods.