Diffraction Pattern Video Conversion System and Method for Mutually Converting Virtual TEM SADP Video and Actual TEM SADP Video Using Deep Learning

Abstract

An object of the present invention is to provide a diffraction pattern image conversion system and method for converting between a virtual TEM SADP image and an actual TEM SADP image using deep learning. [Means to Solve the Problem] The diffraction pattern image conversion system of the present invention is characterized by including an actual diffraction pattern image refinement unit that removes unnecessary information from the actual diffraction pattern image, a virtual diffraction pattern generation unit that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed, and an algorithm learning unit that learns an algorithm that can generate a diffraction pattern similar to an actual one from the virtual diffraction pattern using at least one of the actual diffraction pattern image from which the unnecessary information has been removed and the virtual diffraction pattern image.

Description

Technical Field

[0001] The present invention relates to a TEM SADP image generation system and method having high resolution by adaptively corresponding to input parameters. Furthermore, the present invention relates to a diffraction pattern image conversion system and method for mutually converting virtual TEM SADP images and actual TEM SADP images using deep learning.

Background Art

[0002] In an actual TEM SADP image, an electron beam is most transmitted and a bright point appears at the center of the image. Since the point has a relatively high brightness compared to the surrounding diffraction points, it becomes difficult to see the surrounding diffraction points. Therefore, when generally taking a TEM SADP image, as shown in FIG. 15, a beam stopper is used to block the bright point at the center of the image. However, in conventional simulation programs, the influence of such a beam stopper is not considered. In addition, in an actual TEM SADP, there are various forms of errors such as errors generated because the direction of the electron beam does not exactly match the zone axis, errors that can occur in the optical system, and errors that can occur in the process of obtaining a diffraction pattern through an image sensor such as a CCD / CMOS. However, conventional simulation programs do not consider such influences. Furthermore, depending on the TEM manufacturing company, there may be a difference in the quality of the obtained SADP image or a different mode of error, but such an influence is not considered in the simulation program.

Summary of the Invention

Problems to be Solved by the Invention

[0003] An object of the present invention is to provide a TEM SADP image generation system and method having high resolution by adaptively corresponding to input parameters. The present invention also provides a virtual diffraction pattern video generation system and method that can be used in a TEM. Furthermore, the present invention provides a technique capable of preventing a phenomenon in which a ringing effect, HOLZ (higher-order Laue zone), or blurred diffraction points are included in a diffraction pattern video. Furthermore, the present invention provides a technique capable of mathematically interpreting parameters input by a user to generate a diffraction pattern video. The present invention also provides a computing device that can be processed at high speed by utilizing CPU parallel processing or GPGPU. Furthermore, the present invention provides a technique that utilizes video processing techniques such as gamma correction. Furthermore, the present invention provides a technique that utilizes an SADP video generated by adaptively corresponding to input parameters. The object of the present invention is to provide a technique capable of preventing a phenomenon in which a material is destroyed by a large number of scanning beam outputs. Furthermore, the present invention provides a diffraction pattern video conversion system and method that mutually converts a virtual TEM SADP video and an actual TEM SADP video by using deep learning.

Means for Solving the Problems

[0004] In order to achieve the above-described object, the diffraction pattern video conversion system of the present invention includes an actual diffraction pattern video refinement unit that removes unnecessary information in an actual diffraction pattern video, a virtual diffraction pattern generation unit that obtains a virtual diffraction pattern video corresponding to the actual diffraction pattern video from which the unnecessary information has been removed, and at least one of the actual diffraction pattern video from which the unnecessary information has been removed and the virtual diffraction pattern video is used to generate a video belonging to the actual diffraction pattern domain from a video belonging to the virtual diffraction pattern domain, or to generate a video belonging to the virtual diffraction pattern domain from a video belonging to the actual diffraction pattern domain. It is characterized by including an actual-virtual mutual conversion algorithm learning unit.

[0005] The diffraction pattern video conversion system of the present invention includes an actual diffraction pattern video refinement unit that removes unnecessary information in an actual diffraction pattern video, a virtual diffraction pattern generation unit that acquires a virtual diffraction pattern video corresponding to the actual diffraction pattern video from which the unnecessary information has been removed, and an algorithm learning unit that uses a deep learning algorithm learned using at least one of the actual diffraction pattern video and the virtual diffraction pattern video to generate a diffraction pattern video belonging to the actual diffraction pattern domain from a diffraction pattern video belonging to the virtual diffraction pattern domain.

[0006] The diffraction pattern video conversion system of the present invention includes an actual diffraction pattern video refinement unit that removes unnecessary information in an actual diffraction pattern video, a virtual diffraction pattern generation unit that acquires a virtual diffraction pattern video corresponding to the actual diffraction pattern video from which the unnecessary information has been removed, and an algorithm learning unit that uses a deep learning algorithm learned using at least one of the actual diffraction pattern video and the virtual diffraction pattern video to generate a diffraction pattern video belonging to the virtual diffraction pattern domain from a diffraction pattern video belonging to the actual diffraction pattern domain.

[0007] In a computer-readable recording medium recording the program code of the present invention, the program code is used to perform a method including a step of removing unnecessary information in an actual diffraction pattern video, a step of generating a virtual diffraction pattern video corresponding to the actual diffraction pattern video from which the unnecessary information has been removed, and a step of using at least one of the actual diffraction pattern video from which the unnecessary information has been removed and the virtual diffraction pattern video to generate a video belonging to the actual diffraction pattern domain from a video belonging to the virtual diffraction pattern domain, or to generate a video belonging to the virtual diffraction pattern domain from a video belonging to the actual diffraction pattern domain.

Advantages of the Invention

[0008] The TEM SADP image generation system and method according to the present invention can prevent phenomena such as a HOLZ (High Order Laue Zone) being included in the diffraction pattern or a diffraction pattern with blurred diffraction points being generated by adaptively corresponding to input parameters, and can prevent the ringing effect of the diffraction pattern that may occur from the discontinuous points of the light source. Furthermore, the diffraction pattern image conversion system of the present invention can generate a TEM SADP image similar to the actual one corresponding to the virtual TEM SADP image, and thus can simulate various errors that may occur in an actual TEM experiment.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0010] As used herein, singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "composed of" or "including" should not necessarily be construed as including all of the plurality of components or steps described in the specification. Some of the components or steps may not be included, or additional components or steps may be further included. Also, terms such as "... part" and "module" described in the specification mean a unit that processes at least one function or operation, which can be implemented in hardware or software, or in a combination of hardware and software.

[0011] The present invention relates to a TEM (Transmission Electron Microscope) SADP (Selected Area Diffraction Pattern) image generation system and method that adaptively respond to input parameters and have high resolution, and can generate a high-quality TEM SADP image with resolution without the occurrence of the ringing effect of the diffraction pattern that can occur from the discontinuous points of the light source, the phenomenon that the HOLZ (High Order Laue Zone) is included in the diffraction pattern, and the phenomenon that the blurred diffraction points are included in the diffraction pattern.

[0012] To grasp the characteristics of the material, an electron beam is scanned through the TEM to obtain an SADP image, but when the number of output times of the electron beam increases, the material may be damaged. Therefore, the present invention can provide an SADP image by program without actually scanning the electron beam so as to prevent such damage to the material. Therefore, the SADP image generated in this way can be utilized in various fields.

[0013] Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 6 is a block diagram schematically showing the configuration of a TEM SADP image generation system according to an embodiment of the present invention, and FIG. 7 is a drawing showing a SADP image with the ringing effect eliminated generated using the TEM SADP image generation method of the present invention. As shown in FIG. 6, the TEM SADP image generation system of the present embodiment can generate a SADP image in which phenomena such as the ringing effect of the diffraction pattern, the phenomenon that HOLZ (higher-order Laue zone) is included in the diffraction pattern, and the phenomenon that blurred diffraction points are included do not occur by adaptively corresponding to input parameters.

[0014] Such a TEM SADP image generation system can include a parameter setting unit 600, a sample generation unit 602, a HKL vector generation unit 604, a light source generation unit 606, a diffraction pattern generation unit 608, and a control unit (not shown) that generally controls the operations of these units. Here, the TEM SADP image generation system can be a single device, for example, a server, or can be generally referred to by the name of a computing device.

[0015] The parameter setting unit 600 can set parameters for SADP image generation. For example, the parameter setting unit 600 can receive user input and set parameters. According to an embodiment, the parameter setting unit 600 can set parameters such as lattice constant, relative positions of atoms within the unit cell, zone axis, wavelength and intensity of the electron beam, camera distance, size of the diffraction pattern image, etc. These parameters can all be input by the user, or other parameters can be automatically generated when the user inputs some of them.

[0016] The sample generation unit 602 can generate a slab-shaped sample using the relative positions of atoms within the unit cell and the zone axis parameters. Here, the slab form can mean a thin plate shape. Of course, the generated sample is not limited to the slab form.

[0017] The HKL vector generation unit 604 can generate reciprocal lattice vectors that intersect with a virtual Ewald sphere. Here, the reciprocal lattice can be a parameter that is automatically generated by using a specific program according to the unit lattice set by the parameter setting unit 600.

[0018] The light source generation unit 606 can calculate the relative brightness of the electron beam reaching each atom in the sample.

[0019] The diffraction pattern generation unit 608 can accumulate the diffraction generated in the interaction between all atoms and electrons contained in the sample to generate a virtual SADP image. Parameters, reciprocal lattice vectors, the relative brightness of the electron beam reaching the atoms, etc. set in this process can be used.

[0020] In summary, the TEM SADP image generation system of this embodiment can adaptively generate a virtual SADP image corresponding to various input parameters, and the SADP image can be free from phenomena such as the ringing effect, the phenomenon that the HOLZ (higher-order Laue zone) pattern is included in the diffraction pattern, and the phenomenon that blurred diffraction points are included in the diffraction pattern. Also, the TEM SADP image generation system can utilize parallel processing or GPGPU to generate the SADP image at high speed. That is, the TEM SADP image generation system can generate a large number of virtual SADP images at high speed, and the generated SADP image can be almost the same as the actual SADP image.

[0021] On the other hand, although the parameters input by the user were specifically mentioned above, the parameters are not limited as long as the sample is generated using the parameters input by the user. That is, the TEM SADP image generation system can include a sample generation unit that generates a slab-shaped sample using parameters input by a user, a light source generation unit that determines the brightness of an electron beam reaching atoms in the sample using the input light source form and light source intensity, and a diffraction pattern generation unit that generates a virtual diffraction pattern image using the positions of the atoms in the sample and the determined brightness of the electron beam.

[0022] On the other hand, the reciprocal lattice vector, the brightness of the electron beam, and the diffraction pattern can be generated by mathematically applying the parameters input by the user. A detailed description thereof will be given later. Also, although the user inputs parameters to generate a TEM SADP image, it is also possible to extract parameters from an actual SADP image and generate a virtual TEM SADP image using the extracted parameters. That is, the TEM SADP image generation system can generate a number of virtual TEM SADP images based on the actual SADP image. Also, the number of slab layers, the reciprocal lattice vector, the brightness of the electron beam, etc. are not fixed and can change adaptively according to the parameters input by the user or the parameters extracted from the actual SADP image. A detailed description thereof will be given later.

[0023] Hereinafter, the TEM SADP image generation process will be specifically described with reference to the accompanying drawings. FIG. 8 is a drawing illustrating an example of the lattice constant of a material belonging to the cubic system. FIG. 9 is a drawing illustrating an example of the lattice constant of a material belonging to the hexagonal system, FIG. 10 is a drawing illustrating the relationship between an electron beam, an Ewald sphere, a reciprocal lattice, and a diffraction pattern, and FIG. 11 is a drawing illustrating before and after aligning unit cells. FIG. 12 is a drawing illustrating the result of fabricating a slab-shaped sample using unit cells aligned with a zone axis according to an embodiment of the present invention, and FIG. 13 is a drawing illustrating an example of a diffraction pattern generated by the TEM SADP image generation system of the present invention.

[0024] The parameter setting unit 600 can set parameters such as lattice constant, relative positions of atoms within the unit cell, zone axis, wavelength and intensity of the electron beam, camera distance, and size of the diffraction pattern image. These parameters can be input by the user or extracted from the actual SADP image. At this time, the lattice constant and the relative positions of atoms within the unit cell can be input in the form of files such as CIF (Crystallography Information File), FHI-aims, and XYZ.

[0025] The sample generation unit 602 can generate a slab-shaped sample using the input parameters for the lattice constant, relative positions of atoms within the unit cell, and zone axis. Specifically, the lattice constant can be composed of six variables: the magnitudes a, b, c of the lattice vectors and the angles α, β, γ between the lattice vectors. If a = b = c and α = β = γ = 90° as shown in FIG. 8, the corresponding substance belongs to the cubic crystal system. If a = b ≠ c, α = β = 90°, and γ = 120° as shown in FIG. 9, the corresponding substance belongs to the hexagonal crystal system. The relative positions of atoms within the unit cell can be expressed as shown in Table 1 below when representing the three-dimensional space within the unit cell between 0 and 1.

[0026]

Table 1

[0027] As shown in FIG. 11, the sample generation unit 602 can align unit cells so that the lattice plane corresponding to the direction of the electron beam and the zone axis is perpendicular, and generate a slab-shaped sample as shown in FIG. 12. Here, the slab-shaped sample can mean a structure in which unit cells are arranged in a plate shape. Although a cubic material was generated in FIG. 12, a hexagonal sample can also be generated in the same manner. At this time, the Rodrigues formula can be used to align the direction of the electron beam, which can be represented by a three-dimensional vector, and the zone axis.

[0028] According to one embodiment, the number of layers of the slab can be adaptively determined according to the input lattice constant and zone axis parameter, so as to prevent phenomena such as the high-order Laue zone (HOLZ) being included in the diffraction pattern and the diffraction pattern with blurred diffraction points being generated. That is, the sample generation unit 602 can adaptively determine the number of layers of the slab according to the input lattice constant and zone axis parameter so as to prevent the phenomenon that the high-order Laue zone (HOLZ) is included in the diffraction pattern and the diffraction pattern with blurred diffraction points is generated. The unit cells can be aligned in the slab layers determined in this way. As a result, even for the same material, the number of layers of the slab can be different according to the parameters input by the user.

[0029] As shown in FIG. 10, the HKL vector generation unit 604 can generate an inverse lattice vector that intersects the virtual Ewald sphere. Diffraction can occur in the inverse lattice that intersects the Ewald sphere, and thus the inverse lattice where diffraction occurs can be detected to obtain the diffraction pattern. Here, the inverse lattice can be a parameter that is automatically generated by using a specific program or mathematical formula according to the unit cell set by the parameter setting unit 600. Specifically, the HKL vector generation unit 604 can calculate the reciprocal lattice vectors h(x, y), k(x, y), and l(x, y) using the image coordinates (x, y) separated by a preset distance (d) from the origin where the electron beam is located and the wavelength (λ) of the electron beam by the following mathematical formulas 1 and 2.

[0030]

Number

[0031]

Number

[0032] As shown in Mathematical formula 1, if the image coordinates (x, y), the wavelength (λ), and the distance (d) from the origin where the electron beam is located are known, χ can be obtained, and by using the obtained χ, the reciprocal lattice vectors [h(x, y), k(x, y), l(x, y)] can be automatically obtained.

[0033] The light source generation unit 606 receives the form of the light source and the intensity of the light source as inputs and can obtain the brightness of the electron beam reaching each atom in the sample. At this time, the form of the light source can be flat or have a 2D Gaussian form with respect to the lattice plane perpendicular to the direction of the electron beam.

[0034] When the three-dimensional positions of the atoms in the sample are (x j , y j , z j ), assuming a flat light source form, the brightness of the electron beam reaching each atom is as shown in the following Mathematical formula 3.

[0035]

Number

[0036] Assuming the form of a 2D Gaussian light source, the brightness of the electron beam reaching each atom is as shown in Equation 4 below.

[0037]

Equation

[0038] Assuming the form of a 3D Gaussian light source, the brightness of the electron beam reaching each atom is as shown in Equation 5 below.

[0039]

Equation

[0040] In the light source generation unit 606, a 3D Gaussian can be utilized to create a continuous light source form. At this time, 3σ of the Gaussian can be set to be smaller than the width, length, and height of the sample, so as to ensure that no discontinuity points occur at the edges of the sample.

[0041] According to other embodiments, the light source generation unit 606 can also use a method of simultaneously utilizing a 2D Gaussian and an exponential decay function to remove possible discontinuity points in the width and length directions of the sample with the 2D Gaussian, and remove possible discontinuity points in the height direction of the sample with the exponential decay function.

[0042] On the other hand, by applying exponential decay to the direction reference of the electron beam, the brightness decrease of the electron beam as it passes through the sample can be simulated. The light source to which exponential decay is applied can be defined as shown in Equation 6 below.

[0043]

Number

[0044] The size and shape of the light source generated by the light source generation unit 606 can be adaptively varied according to the size of the input slab, the size of the diffraction pattern image, etc. As a result, the ringing effect of the diffraction pattern that can occur from the discontinuity of the light source can be prevented, which is illustrated in FIG. 7. That is, the light source generation unit 606 can adaptively vary and use the size and shape of the light source according to the size of the input slab and the size of the diffraction pattern image in order to prevent the ringing effect of the diffraction pattern.

[0045] The diffraction pattern generation unit 608 uses the reciprocal lattice vector [h(x, y), k(x, y), l(x, y)] obtained by the HKL vector generation unit 604, the positions of atoms in the sample, and the brightness (I(x j , y j , z j )) of the electron beam obtained in the light source generation unit 606 to calculate the accumulated diffraction pattern (F(h, k, l)) as shown in the following mathematical formula 7.

[0046]

Number

[0047] Subsequently, the diffraction pattern generation unit 608 can calculate the maximum value of the accumulated diffraction pattern and linearly normalize the accumulated diffraction pattern based on the calculated maximum value to generate a diffraction pattern image. According to other embodiments, the diffraction pattern generation unit 608 can also generate a diffraction pattern image by non-linearly normalizing using video processing techniques such as gamma correction. At this time, since the diffraction generated from the interaction between each atom and electron contained in the sample in the diffraction pattern generation unit 608 can be calculated independently, it can be calculated at high speed by utilizing CPU parallel processing or GPGPU (General Purpose computing on Graphics Processing Unit). The SADP image thus generated is illustrated in FIG. 13. As illustrated in FIG. 13, a ringing effect, a phenomenon in which HOLZ (higher-order Laue zone) is included in the diffraction pattern, and a diffraction pattern with blurred diffraction points do not occur in the SADP image.

[0048] On the other hand, the diffraction pattern generation unit 608 can apply various functions to produce the accumulated diffraction values into an SADP image. For example, the diffraction pattern generation unit 608 can also generate a diffraction pattern by applying a linear function, or can generate a diffraction pattern using a function used in gamma correction such as Vo = AVi γ As well. In summary, the SADP image generation system of this embodiment can generate an SADP image that utilizes the reciprocal lattice vector, the positions of atoms in the sample, and the brightness of the electron beam to prevent the generation of a ringing effect, a phenomenon in which HOLZ (higher-order Laue zone) is included in the diffraction pattern, and a diffraction pattern with blurred diffraction points.

[0049] Hereinafter, a system and method for converting an actual TEM SADP image and a virtual TEM SADP image using deep learning will be described with reference to the accompanying drawings. FIG. 14 is a drawing illustrating a system for mutually converting an actual TEM SADP image and a virtual TEM SADP image according to an embodiment of the present invention. FIG. 15 is a drawing illustrating an example of blocking a bright point at the center of an image using a beam stopper. FIG. 16 is a drawing illustrating an example of an actual TEM SADP image with scale information displayed. FIG. 17 is a drawing illustrating an example of removing scale information from an actual TEM SADP image. FIG. 18 is a drawing illustrating an example of a virtual TEM SADP image generated via a JEMS program. FIG. 19 is a drawing illustrating an example of a TEM SADP image generated in a form similar to the actual one from a virtual TEM SADP image. FIG. 20 is a drawing illustrating an example of an actual-virtual mutual conversion algorithm according to an embodiment of the present invention.

[0050] The system for mutually converting the actual TEM SADP image and the virtual TEM SADP image of this embodiment includes an actual diffraction pattern image refinement unit 1400, a virtual diffraction pattern generation unit 1402, an actual-virtual mutual conversion algorithm learning unit 1404, and a control unit that generally controls the operations of these units. The actual diffraction pattern image refinement unit 1400 can remove unnecessary portions in an actual TEM SADP image, for example, an actual TEM SADP image collected through an experiment or an actual TEM SADP image collected via the web. For example, as illustrated in FIG. 15, the actual diffraction pattern image refinement unit 1400 can remove annotation information, scale information, lattice plane index information, etc., added for additional information entry within the actual TEM SADP image.

[0051] According to one embodiment, the actual diffraction pattern image refinement unit 1400 can remove annotation information from the actual TEM SADP image and synthesize a background on the removed part using a commercial image processing program such as Photoshop or a hole-filling algorithm for filling in the lost parts of the image. The SADP image obtained through such a process is illustrated in FIG. 17. Of course, there is no limitation on the program as long as the removed part is filled. As another example, the actual diffraction pattern image refinement unit 1400 can remove scale information from the actual TEM SADP image with scale information displayed as shown in FIG. 16, and synthesize a background on the removed part via a hole-filling algorithm or the like to generate an SADP image as shown in FIG. 17.

[0052] The virtual diffraction pattern generation unit 1402 can generate a virtual TEM SADP image corresponding to the actual TEM SADP image using a TEM SADP simulation program such as JEMS, QSTEM, abTEM, Ladyne Software Suite, SingleCrystal, or Condor based on the input lattice constant and unit cell. Here, the lattice constant and unit cell can be input by the user or can also be extracted and obtained from the actual TEM SADP image.

[0053] When information regarding the lattice constant and unit cell is given in a file format such as CIF (Crystallography Information File), FHI-aims, or XYZ, a virtual TEM SADP image generated using the JEMS program is illustrated in FIG. 18. At this time, since the generated virtual SADP image is subsequently used for actual-virtual conversion algorithm learning, using one program to generate the SADP image rather than using various programs to generate the SADP image can reduce the confusion caused by data diversity in the learning process. For example, a virtual TEM SADP image corresponding to an actual TEM SADP image from which unnecessary parts have been removed using the same JEMS program can be generated, and other programs can also be not used to generate the virtual TEM SADP image.

[0054] According to another embodiment, the virtual diffraction pattern generation unit 1402 can also select a virtual SADP image corresponding to the actual TEM SADP image from among the virtual TEM SADP images generated in FIGS. 6 to 13.

[0055] From another perspective, the virtual diffraction pattern generation unit 1402 can also generate a virtual SADP image corresponding to the actual TEM SADP image through the method described in FIGS. 6 to 13. In this case, the virtual diffraction pattern generation unit 1402 may include a sample generation unit that generates a sample using at least one of a lattice constant, a relative position of atoms within a unit cell, and a zone axis parameter, a vector generation unit that generates a reciprocal lattice vector corresponding to the unit cell, a light source generation unit that obtains the brightness of an electron beam reaching atoms within the generated sample, a diffraction pattern generation unit that generates a virtual diffraction pattern image (TEM SADP image) using the generated reciprocal lattice vector, the positions of atoms within the sample, and the obtained brightness of the electron beam, and a selection unit that selects a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images.

[0056] The actual-virtual conversion algorithm learning unit 1404 can learn an actual-virtual conversion algorithm that generates an image belonging to the actual diffraction pattern domain from an image belonging to the virtual diffraction pattern domain or generates an image belonging to the virtual diffraction pattern domain from an image belonging to the actual diffraction pattern domain by utilizing the actual SADP image and the virtual SADP image.

[0057] Subsequently, the actual-virtual conversion algorithm learning unit 1404 can utilize a deep learning model, that is, use deep learning technology to generate an SADP image belonging to the actual diffraction pattern domain from an SADP image belonging to the virtual diffraction pattern domain. For example, as shown in FIG. 19, an SADP image generated in a form similar to the actual one can be generated from the virtual TEM SADP image displayed on the left side and displayed on the right side. At this time, if a virtual diffraction pattern having the same diffraction point position as the actual diffraction pattern is prepared, an algorithm that requires a pair of images belonging to both domains can be used. On the other hand, when only one of the actual diffraction pattern and the virtual diffraction pattern is prepared, an algorithm that does not require information about the pair of images belonging to both domains can be used.

[0058] Through such a series of processes, the actual-virtual conversion algorithm can generate a TEM SADP image similar to the actual one by comprehensively considering the influence of various errors included in the actual TEM SADP image and the influence of the beam stopper.

[0059] According to another embodiment, the actual-virtual conversion algorithm learning unit 1404 can not only generate an SADP image belonging to the actual diffraction pattern domain from an SADP image belonging to the virtual diffraction pattern domain, but also generate an SADP image belonging to the virtual diffraction pattern domain from an SADP image belonging to the actual diffraction pattern domain. If a deep learning application is created based on the TEM SADP image, the input can be the actual TEM SADP image. At this time, if the input SADP image belonging to the actual diffraction pattern domain is converted into an SADP image belonging to the virtual diffraction pattern domain, a deep learning model learned with the SADP image belonging to the virtual diffraction pattern domain can be utilized.

[0060] As shown in FIG. 20, to specifically explain the actual-virtual mutual conversion algorithm learning unit 1404, the actual-virtual mutual conversion algorithm learning unit 1404 can include an actual diffraction pattern classification unit, a virtual diffraction pattern classification unit, a Real2Sim conversion unit, and a Sim2Real conversion unit.

[0061] The actual diffraction pattern classification unit learns a deep learning model that classifies a virtual diffraction pattern converted into a form similar to the actual diffraction pattern via the Sim2Real conversion unit and a diffraction pattern captured via an actual TEM. At this time, the virtual diffraction pattern converted into a form similar to the actual diffraction pattern and the diffraction pattern captured via the actual TEM are all videos belonging to the actual diffraction pattern domain.

[0062] The virtual diffraction pattern classification unit learns a deep learning model that classifies an actual diffraction pattern converted into a form similar to the virtual diffraction pattern via the Real2Sim conversion unit and a virtual diffraction pattern generated through simulation. At this time, the actual diffraction pattern converted into a form similar to the virtual diffraction pattern and the virtual diffraction pattern generated through simulation are all videos belonging to the virtual diffraction pattern domain.

[0063] The Real2Sim conversion unit learns a deep learning model that converts a video belonging to the actual diffraction pattern domain into a video belonging to the virtual diffraction pattern domain. At this time, the goal of the Sim2Real conversion unit is to generate a video so similar that the virtual diffraction pattern classification unit cannot distinguish the video it generates from the video generated through simulation.

[0064] The Sim2Real conversion unit learns a deep learning model that converts a video belonging to the virtual diffraction pattern domain into a video belonging to the actual diffraction pattern domain. At this time, the goal of the Sim2Real conversion unit is to generate a video so similar that the actual diffraction pattern classification unit cannot distinguish the video it generates from the video captured via the actual TEM.

[0065] The four components belonging to the actual-virtual conversion algorithm learning unit 1404 influence each other and learn a deep learning model that can successfully achieve the goals given to each of them. Ultimately, it will be possible to smoothly convert the video belonging to the actual diffraction pattern domain and the video belonging to the virtual diffraction pattern domain with each other.

[0066] Since the process of acquiring actual TEM SADP videos through experiments requires a lot of time, if a deep learning model learned with SADP videos belonging to the virtual diffraction pattern domain can be used in actual applications, a large amount of learning datasets can be easily obtained, and an application with high performance can be constructed. To summarize, the system (video conversion system) for converting between the actual TEM SADP video and the virtual TEM SADP video of this embodiment removes unnecessary information from the collected actual TEM SADP video, generates a virtual TEM SADP video corresponding to the actual TEM SADP video from which the unnecessary information has been removed, and utilizes the generated virtual TEM SADP video and the actual TEM SADP video to learn the actual-virtual conversion algorithm. At this time, the actual-virtual conversion algorithm can also generate an SADP video belonging to the actual diffraction pattern domain from an SADP video belonging to the virtual diffraction pattern domain, and can also generate an SADP video belonging to the virtual diffraction pattern domain from an SADP video belonging to the actual diffraction pattern domain.

[0067] By generating a TEM SADP video similar to the actual one from the virtual TEM SADP, various errors that may occur in actual TEM experiments can be simulated. In addition, when creating a deep learning application that utilizes TEM SADP images, by reducing the gap between the actual TEM SADP and the virtual TEM SADP through an actual-virtual conversion algorithm, the virtual data generated in the simulation can be effectively used for deep learning.

[0068] On the other hand, the components of the above-described embodiments can be easily understood from a process perspective. That is, each component can be understood as its own process. Also, the processes of the above-described embodiments can be easily understood from the perspective of the components of the device. In addition, the above-described technical content can be embodied in the form of program instructions that can be executed via various computer means and recorded on a computer-readable medium. The computer-readable medium can include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium can be those specially designed and configured for the embodiments or those known to and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes such as those created by compilers but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be configured to operate as one or more software modules for performing the operations of the embodiments, and vice versa.

[0069] The above-described embodiments of the present invention are disclosed for illustrative purposes, and those of ordinary skill in the art who have knowledge of the present invention can make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the following claims.

Claims

1. An actual diffraction pattern image refinement unit that removes unnecessary information in an actual diffraction pattern image, A virtual diffraction pattern generation unit that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed, An actual-virtual mutual conversion algorithm learning unit that generates an image belonging to the actual diffraction pattern domain from an image belonging to the virtual diffraction pattern domain or generates an image belonging to the virtual diffraction pattern domain from an image belonging to the actual diffraction pattern domain by using at least one of the actual diffraction pattern image from which the unnecessary information has been removed and the virtual diffraction pattern image. A diffraction pattern image conversion system characterized by including.

2. The diffraction pattern image conversion system according to claim 1, wherein the diffraction pattern image is a TEM (Transmission Electron Microscope) SADP (Selected Area Diffraction Pattern) image.

3. The diffraction pattern image conversion system according to claim 2, wherein the unnecessary information is information regarding annotations, scales, or indexes.

4. The actual diffraction pattern image refinement unit detects the unnecessary information in the actual diffraction pattern image and fills the area where the unnecessary information was located by using the peripheral information of the detected unnecessary information by using a hole-filling algorithm. The diffraction pattern image conversion system according to claim 3, characterized by the above.

5. The virtual diffraction pattern generation unit uses a TEM SADP simulation program, The TEM SADP simulation program generates a virtual TEM SADP image corresponding to the actual TEM SADP image based on the input lattice constant and unit cell. The diffraction pattern image conversion system according to claim 2, characterized by the above.

6. The information regarding the lattice constant and the unit cell can be provided in a file form such as CIF (Crystallography Information File), FHI-aims, or XYZ. The diffraction pattern image conversion system according to claim 5, characterized by the above.

7. The virtual diffraction pattern generation unit is A sample generation unit that generates a sample using at least one of a lattice constant, the relative positions of atoms within a unit cell, and zone axis parameters; A vector generation unit that generates reciprocal lattice vectors corresponding to the unit cell; A light source generation unit that determines the brightness of an electron beam reaching atoms within the generated sample; A diffraction pattern generation unit that generates a virtual diffraction pattern image using the generated reciprocal lattice vectors, the relative positions of atoms within the sample, and the determined brightness of the electron beam; A selection unit that selects a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images, characterized in that the diffraction pattern image conversion system according to claim 2 includes the selection unit.

8. The sample generation unit adaptively determines the number of slab layers according to the lattice constant and the zone axis parameters among the input parameters so as to prevent a phenomenon in which a HOLZ (Higher Order Laue Zone) (higher-order Laue zone) is included in the diffraction pattern or a diffraction pattern with blurred diffraction points is generated. The light source generation unit adaptively varies the form and intensity of the light source according to the size of the slab or the size of the diffraction pattern image input so as to prevent the ringing effect of the diffraction pattern that can occur from the discontinuous points of the light source, characterized in that the diffraction pattern image conversion system according to claim 7 includes the light source generation unit.

9. The actual-virtual mutual conversion algorithm learning unit generates a diffraction pattern image belonging to the actual diffraction pattern domain from a diffraction pattern image belonging to the virtual diffraction pattern domain by utilizing a deep learning model, characterized in that the diffraction pattern image conversion system according to claim 2 includes the actual-virtual mutual conversion algorithm learning unit.

10. When the actual diffraction pattern image and the virtual diffraction pattern image are prepared, the actual-virtual mutual conversion algorithm learning unit generates a diffraction pattern image belonging to the actual diffraction pattern domain from a diffraction pattern image belonging to the virtual diffraction pattern domain by using a specific algorithm, characterized in that the diffraction pattern image conversion system according to claim 9 includes the actual-virtual mutual conversion algorithm learning unit.

11. The actual-virtual mutual conversion algorithm learning unit A Real2Sim conversion unit; A Sim2Real conversion unit; An actual diffraction pattern classification unit that trains a deep learning model to distinguish between a virtual diffraction pattern converted into a form similar to an actual diffraction pattern via the Sim2Real conversion unit and a diffraction pattern captured via an actual TEM, A virtual diffraction pattern classification unit that trains a deep learning model to distinguish between an actual diffraction pattern converted into a form similar to a virtual diffraction pattern via the Real2Sim conversion unit and a virtual diffraction pattern generated through simulation, The Real2Sim conversion unit trains a deep learning model that converts a video belonging to the actual diffraction pattern domain into a video belonging to the virtual diffraction pattern domain, The diffraction pattern video conversion system according to claim 2, wherein the Sim2Real conversion unit trains a deep learning model that converts a video belonging to the virtual diffraction pattern domain into a video belonging to the actual diffraction pattern domain.

12. An actual diffraction pattern video refinement unit that removes unnecessary information from an actual diffraction pattern video, A virtual diffraction pattern generation unit that obtains a virtual diffraction pattern video corresponding to the actual diffraction pattern video from which the unnecessary information has been removed, An algorithm learning unit that uses a deep learning algorithm learned using at least one of the actual diffraction pattern video and the virtual diffraction pattern video to generate a diffraction pattern video belonging to the actual diffraction pattern domain from a diffraction pattern video belonging to the virtual diffraction pattern domain. The diffraction pattern video conversion system is characterized by including.

13. The algorithm learning unit of the diffraction pattern video conversion system according to claim 12, wherein when the actual diffraction pattern video and the virtual diffraction pattern video are prepared, a specific algorithm is used to generate a diffraction pattern video belonging to the actual diffraction pattern domain from a diffraction pattern video belonging to the virtual diffraction pattern domain.

14. An actual diffraction pattern video refinement unit that removes unnecessary information from an actual diffraction pattern video, A virtual diffraction pattern generation unit that obtains a virtual diffraction pattern video corresponding to the actual diffraction pattern video from which the unnecessary information has been removed, A diffraction pattern image conversion system, comprising an algorithm learning unit that generates a diffraction pattern image belonging to a virtual diffraction pattern domain from a diffraction pattern image belonging to an actual diffraction pattern domain by using a deep learning algorithm learned by using at least one of the actual diffraction pattern image and the virtual diffraction pattern image.

15. In a computer-readable recording medium recording program code, the program code is used to perform steps of removing unnecessary information in an actual diffraction pattern image, generating a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed, and generating an image belonging to an actual diffraction pattern domain from an image belonging to a virtual diffraction pattern domain or generating an image belonging to the virtual diffraction pattern domain from an image belonging to the actual diffraction pattern domain by using at least one of the actual diffraction pattern image from which the unnecessary information has been removed and the virtual diffraction pattern image.

Images