Diffraction pattern image conversion system and method for converting between virtual TEM SADP image and actual TEM SADP image using deep learning
The system addresses TEM SADP image generation challenges by converting virtual and actual images using deep learning, preventing image distortion and simulating real TEM experiments, achieving high-quality SADP images.
Patent Information
- Application Number
- JP2024523278
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-03
- Filing Date
- 2022-10-12
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-10-12
AI Technical Summary
Conventional TEM SADP image generation systems fail to account for beam stoppers, alignment errors, and manufacturer-specific errors, leading to issues like bright spots obscuring diffraction points, HOLZ inclusion, and blurred patterns, and are inefficient in processing and simulating real TEM experiments.
A system utilizing deep learning to convert virtual and actual TEM SADP images, incorporating an actual diffraction pattern refiner, virtual diffraction pattern generator, and a mutual conversion algorithm to adaptively generate high-quality SADP images, preventing ringing effects and HOLZ inclusion, and utilizing CPU/GPGPU for high-speed processing.
The system effectively generates high-quality SADP images that mimic real TEM experiments, preventing image distortion and material destruction by simulating various errors, enabling efficient and accurate TEM SADP image generation.
Smart Images

Figure 2024540899000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a system and method for generating TEM SADP images that adaptively responds to input parameters and has high discrimination power. The present invention also relates to a diffraction pattern image conversion system and method for converting between a virtual TEM SADP image and an actual TEM SADP image using deep learning. [Background technology]
[0002] In an actual TEM SADP image, the electron beam is most transmitted, resulting in a bright spot appearing in the center of the image. This spot has a relatively high brightness compared to the surrounding diffraction spots, making the surrounding diffraction spots difficult to see. Therefore, when taking a TEM SADP image, a beam stopper is generally used to block the bright spot in the center of the image, as shown in Figure 15. However, conventional simulation programs do not take into account the effect of such a beam stopper. In addition, various types of errors exist in an actual TEM SADP, such as errors that occur when the direction of the electron beam does not exactly match the zone axis, errors that can occur in the optical system, errors that can occur in the process of acquiring a diffraction pattern through an image sensor such as a CCD / CMOS, etc. However, conventional simulation programs do not take these effects into account. Furthermore, the quality of the acquired SADP image may differ depending on the TEM manufacturer, and the type of error may differ. However, the simulation program does not take such effects into account. Summary of the Invention [Problem to be solved by the invention]
[0003] It is an object of the present invention to provide a TEM SADP image generating system and method that adaptively responds to input parameters and has high discrimination power. The present invention also provides a system and method for generating a virtual diffraction pattern image that can be used in a TEM. Furthermore, the present invention provides a technique for preventing the phenomenon of ringing effects, HOLZ (higher order Laue zones), or blurred diffraction points in a diffraction pattern image. Furthermore, the present invention provides a technique that can generate a diffraction pattern image by mathematically interpreting parameters input by a user. The present invention also provides a computing device capable of high-speed processing by utilizing CPU parallel processing or GPGPU. Additionally, the present invention provides techniques that take advantage of image processing techniques such as gamma correction. Furthermore, the present invention provides a technique for utilizing SADP images that are generated adaptively in response to input parameters. An object of the present invention is to provide a technique capable of preventing the phenomenon in which a material is destroyed by a large amount of scanning beam output. Furthermore, the present invention provides a diffraction pattern image conversion system and method for converting between a virtual TEM SADP image and an actual TEM SADP image using deep learning. [Means for solving the problem]
[0004] In order to achieve the above-mentioned object, the diffraction pattern image conversion system of the present invention is characterized by including an actual diffraction pattern image refinement unit that removes unnecessary information in an actual diffraction pattern image, a virtual diffraction pattern generation unit that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed, and a real-virtual mutual conversion algorithm learning unit that uses at least one of the actual diffraction pattern image from which the unnecessary information has been removed and the virtual diffraction pattern image to generate an image belonging to the actual diffraction pattern domain from an image belonging to the virtual diffraction pattern domain, or generate an image belonging to the virtual diffraction pattern domain from the image belonging to the actual diffraction pattern domain.
[0005] The diffraction pattern image conversion system of the present invention includes an actual diffraction pattern image refinement unit that removes unnecessary information in an actual diffraction pattern image, a virtual diffraction pattern generation unit that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed, and an algorithm learning unit that generates a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to a virtual diffraction pattern domain by using a deep learning algorithm trained using at least one of the actual diffraction pattern image and the virtual diffraction pattern image.
[0006] The diffraction pattern image conversion system of the present invention includes an actual diffraction pattern image refinement unit that removes unnecessary information in an actual diffraction pattern image, a virtual diffraction pattern generation unit that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed, and an algorithm learning unit that generates a diffraction pattern image belonging to a virtual diffraction pattern domain from a diffraction pattern image belonging to a real diffraction pattern domain by using a deep learning algorithm trained using at least one of the actual diffraction pattern image and the virtual diffraction pattern image.
[0007] In a computer-readable recording medium having program code recorded thereon, the program code is characterized in that it is used for performing a method including the steps of: removing unnecessary information in a real diffraction pattern image; generating a virtual diffraction pattern image corresponding to the real diffraction pattern image from which the unnecessary information has been removed; and generating an image belonging to a real diffraction pattern domain from an image belonging to a virtual diffraction pattern domain using at least one of the real diffraction pattern image from which the unnecessary information has been removed and the virtual diffraction pattern image, or generating an image belonging to the virtual diffraction pattern domain from the image belonging to the real diffraction pattern domain. Effect of the Invention
[0008] The TEM SADP image generating system and method according to the present invention adaptively responds to input parameters to prevent phenomena such as the inclusion of a High Order Laue Zone (HOLZ) in the diffraction pattern or the generation of a diffraction pattern with blurred diffraction points, and can prevent ringing effects in the diffraction pattern that can occur due to discontinuities in the light source. Furthermore, the diffraction pattern image conversion system of the present invention generates a TEM SADP image similar to the actual one corresponding to the virtual TEM SADP image, and therefore can simulate various errors that may occur in an actual TEM experiment. [Brief description of the drawings]
[0009] [Figure 1] 1 is a diagram illustrating an example of an actual TEM SADP image. [Diagram 2] 1 is a diagram illustrating a virtual SADP image generated by the JEMS program. [Diagram 3] 1 is a diagram illustrating a SADP image including a HOLZ (higher order Laue zone) pattern. [Figure 4] 1 is a diagram illustrating an SADP image including a blurred diffraction spot pattern. [Diagram 5] 1 is a diagram illustrating an SADP image including a ringing effect. [Figure 6] FIG. 1 is a block diagram illustrating a schematic configuration of a TEM SADP image generating system according to the present invention. [Figure 7] 1 is a diagram showing a SADP image in which a ringing effect is eliminated, which is generated using a TEM SADP image generating method according to the present invention; [Figure 8] 1 is a diagram illustrating an example of the lattice constant of a material belonging to a cubic system. [Figure 9] 1 is a diagram illustrating an example of the lattice constant of a material belonging to a hexagonal system. [Figure 10]1 is a diagram illustrating the relationship between an electron beam, an Ewald sphere, a reciprocal lattice, and a diffraction pattern. [Figure 11] 1 is a diagram illustrating a unit cell before and after alignment. [Figure 12] 1 is a diagram showing the results of fabricating a slab-shaped sample using a unit cell aligned with a zone axis according to the present invention. [Figure 13] 1 is a diagram illustrating an example of a diffraction pattern generated by the TEM SADP image generating system of the present invention. [Figure 14] 1 is a diagram illustrating a system for converting between an actual TEM SADP image and a virtual TEM SADP image according to the present invention. [Figure 15] 13 is a diagram illustrating an example in which a bright spot in the center of an image is blocked using a beam stopper. [Figure 16] 1 is a diagram illustrating an example of an actual TEM SADP image in which scale information is displayed. [Figure 17] 1 is a diagram illustrating an example in which scale information is removed from an actual TEM SADP image. [Figure 18] 1 illustrates an example of a virtual TEM SADP image generated via the JEMS program. [Figure 19] 1 is a diagram illustrating an example of a TEM SADP image generated in a form similar to an actual TEM SADP image from a virtual TEM SADP image. [Figure 20] 1 is a diagram illustrating an example of a real-virtual interconversion algorithm according to the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] As used herein, singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "include" should not be interpreted as including all of the components or steps described in the specification, and should be interpreted as including some components or steps not included, or including additional components or steps. In addition, terms such as "... unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be embodied in hardware or software, or a combination of hardware and software.
[0011] The present invention relates to a system and method for generating a TEM (Transmission Electron Microscope) SADP (Selected Area Diffraction Pattern) image having high discrimination power by adaptively responding to input parameters, and can generate a high-quality TEM SADP image with high discrimination power without the ringing effect of the diffraction pattern, which may occur due to discontinuities in the light source, the phenomenon in which a HOLZ (High Order Laue Zone) is included in the diffraction pattern, and the phenomenon in which a blurred diffraction point is included in the diffraction pattern.
[0012] In order to understand the characteristics of a material, an SADP image is obtained by scanning an electron beam on the material through a TEM, but if the number of times the electron beam is output is large, the material may be damaged. Therefore, in order to prevent such damage to the material, the present invention can provide an SADP image by a program without scanning an actual electron beam. Therefore, the SADP image generated in this way can be used in various fields.
[0013] Various embodiments of the present invention will now be described in detail with reference to the accompanying drawings. FIG. 6 is a block diagram illustrating a schematic configuration of a TEM SADP image generating system according to an embodiment of the present invention, and FIG. 7 is a diagram illustrating a SADP image in which ringing effect is eliminated, generated using the TEM SADP image generating method of the present invention. As shown in FIG. 6, the TEM SADP image generation system of this embodiment can adaptively respond to input parameters and generate SADP images that are free of the ringing effect in the diffraction pattern, the phenomenon in which the HOLZ (higher-order Laue zone) is included in the diffraction pattern, and the phenomenon in which blurred diffraction points are included.
[0014] The TEM SADP image generating system may include a parameter setting unit 600, a sample generating unit 602, an HKL vector generating unit 604, a light source generating unit 606, a diffraction pattern generating unit 608, and a control unit (not shown) for controlling the overall operation of these units. Here, the TEM SADP image generating system may be a single device, for example, a server, or may be commonly referred to as a computing device.
[0015] The parameter setting unit 600 may set parameters for generating an SADP image. For example, the parameter setting unit 600 may set parameters in response to a user input. According to an embodiment, the parameter setting unit 600 can set parameters such as a lattice constant, a relative position of atoms in a unit lattice, a zone axis, a wavelength and intensity of an electron beam, a camera distance, a size of a diffraction pattern image, etc. All of these parameters can be input by a user, or other parameters can be automatically generated when a user inputs some of them.
[0016] The sample preparation unit 602 may prepare a slab-shaped sample using the relative positions of atoms in a unit cell and zone axis parameters. Here, the slab shape may mean a thin plate shape. Of course, the sample to be prepared is not limited to the slab shape.
[0017] The HKL vector generating unit 604 can generate a reciprocal lattice vector that intersects with a virtual Ewald sphere. Here, the reciprocal lattice can be a parameter that is automatically generated by using a specific program according to the unit lattice set by the parameter setting unit 600.
[0018] The source generator 606 can calculate the relative brightness of the electron beam reaching each atom in the sample.
[0019] The diffraction pattern generator 608 may generate a virtual SADP image by accumulating diffractions generated in the interaction between all atoms and electrons contained in the sample. In this process, set parameters, reciprocal lattice vectors, and relative brightness of the electron beam reaching the atoms may be used.
[0020] In summary, the TEM SADP image generation system of this embodiment adaptively generates a virtual SADP image in response to various input parameters, and the SADP image is free from the ringing effect, the phenomenon in which a HOLZ (higher-order Laue zone) pattern is included in the diffraction pattern, and the phenomenon in which blurred diffraction points are included in the diffraction pattern. In addition, the TEM SADP image generation system can generate SADP images at high speed by utilizing parallel processing or GPGPU. That is, the TEM SADP image generating system can generate a large number of virtual SADP images at high speed, and the generated SADP images can be almost identical to the actual SADP images.
[0021] Meanwhile, although the above specifically refers to parameters input by a user, the parameters are not limited as long as the sample is generated using the parameters input by the user. That is, the TEM SADP image generating system may include a sample generating unit that generates a slab-shaped sample using parameters input by a user, a light source generating unit that determines the brightness of an electron beam that reaches atoms in the sample using an input light source shape and light source intensity, and a diffraction pattern generating unit that generates a virtual diffraction pattern image using the positions of atoms in the sample and the determined brightness of the electron beam.
[0022] Meanwhile, the reciprocal lattice vector, the brightness of the electron beam, and the diffraction pattern can be generated by mathematically applying parameters input by a user, which will be described in detail later. In addition, although the user inputs parameters to generate a TEM SADP image, the parameters can be extracted from an actual SADP image and a virtual TEM SADP image can be generated using the extracted parameters. That is, the TEM SADP image generating system can generate a number of virtual TEM SADP images based on an actual SADP image. In addition, the number of slab layers, reciprocal lattice vectors, and brightness of the electron beam are not fixed, but can be adaptively changed according to parameters input by the user or parameters extracted from an actual SADP image, which will be described in detail later.
[0023] Hereinafter, the process of generating a TEM SADP image will be described in detail with reference to the accompanying drawings. Figure 8 is a diagram showing an example of the lattice constant of a material belonging to a cubic system. Figure 9 is a diagram showing an example of the lattice constant of a material belonging to a hexagonal system. Figure 10 is a diagram showing the relationship between an electron beam, an Ewald sphere, a reciprocal lattice, and a diffraction pattern. Figure 11 is a diagram showing before and after aligning a unit lattice. Figure 12 is a diagram showing the result of fabricating a slab-shaped sample using a unit lattice aligned with a zone axis according to an embodiment of the present invention. Figure 13 is a diagram showing an example of a diffraction pattern generated by the TEM SADP image generating system of the present invention.
[0024] The parameter setting unit 600 can set parameters such as lattice constants, relative positions of atoms in a unit cell, zone axes, wavelength and intensity of an electron beam, camera distance, size of a diffraction pattern image, etc. These parameters can be input by a user or extracted from an actual SADP image. In this case, the lattice constants and the relative positions of atoms in the unit lattice can be input in a file format such as CIF (Crystallography Information File), FHI-aims, or XYZ.
[0025] The sample preparation unit 602 can prepare a slab-type sample using the input lattice constant, the relative positions of atoms in the unit lattice, and parameters for the zone axes. Specifically, the lattice constant can be composed of six variables: the magnitudes of the lattice vectors a, b, and c, and the angles between the lattice vectors α, β, and γ. If a=b=c, α=β=γ=90° as shown in Figure 8, the material belongs to the cubic crystal system, and if a=b≠c, α=β=90°, γ=120° as shown in Figure 9, the material belongs to the hexagonal crystal system. The relative positions of atoms within a unit cell can be expressed as shown in Table 1 below, when the three-dimensional space within the unit cell is expressed between 0 and 1.
[0026] [Table 1] (Relative positions of atoms within the unit cell of a LiAl material belonging to the cubic crystal system)
[0027] The sample preparation unit 602 may align the unit lattice so that the lattice plane corresponding to the direction of the electron beam and the zone axis is perpendicular to each other as shown in FIG 11, and may generate a slab-shaped sample as shown in FIG 12. Here, the slab-shaped sample may refer to a structure in which the unit lattices are arranged in a plate shape. Although a cubic crystal material is generated in FIG 12, a hexagonal crystal sample may also be generated in the same manner. In this case, the Rodrigues formula may be used to align the direction of the electron beam, which can be expressed as a three-dimensional vector, with the zone axis.
[0028] According to one embodiment, the number of slab layers is adaptively determined according to the input lattice constant and zone axis parameters, thereby preventing the phenomenon in which a HOLZ (higher-order Laue zone) is included in a diffraction pattern and the phenomenon in which a diffraction pattern with blurred diffraction points is generated. That is, the sample preparation unit 602 can adaptively determine the number of slab layers according to the input lattice constant and zone axis parameters so as to prevent the phenomenon that the HOLZ (higher order Laue zone) is included in the diffraction pattern or the diffraction pattern with blurred diffraction points from being generated. The unit lattice can be aligned in the slab layers determined in this way. As a result, even for the same material, the number of slab layers can be different depending on the parameters input by the user.
[0029] The HKL vector generating unit 604 can generate a reciprocal lattice vector intersecting with a virtual Ewald sphere as shown in Fig. 10. Diffraction can occur at the reciprocal lattice intersecting with the Ewald sphere, and therefore the reciprocal lattice at which diffraction occurs can be detected to obtain a diffraction pattern. Here, the reciprocal lattice can be a parameter that is automatically generated by utilizing a specific program or formula according to the unit lattice set by the parameter setting unit 600. Specifically, the HKL vector generator 604 can calculate reciprocal lattice vectors h(x,y), k(x,y), and l(x,y) using the image coordinates (x,y) spaced a preset distance (d) from the origin where the electron beam is located and the wavelength (λ) of the electron beam, using the following Equations 1 and 2.
[0030]
number
[0031]
number
[0032] As shown in Equation 1, if the image coordinates (x, y), wavelength (λ), and distance (d) from the origin where the electron beam is located are known, χ can be calculated, and the reciprocal lattice vectors [h(x, y), k(x, y), l(x, y)] can be automatically calculated using the calculated χ.
[0033] The light source generating unit 606 receives the shape and intensity of the light source as input, and can calculate the brightness of the electron beam that reaches each atom in the sample. In this case, the shape of the light source can be flat or have a 2D Gaussian shape based on a lattice plane perpendicular to the direction of the electron beam.
[0034] The three-dimensional position of an atom in a sample is (x j ,y j ,z j ), assuming a flat light source, the brightness of the electron beam reaching each atom is given by the following equation 3:
[0035]
number
[0036] Assuming a 2D Gaussian source configuration, the brightness of the electron beam reaching each atom is given by Equation 4 below.
[0037]
number
[0038] Assuming a 3D Gaussian source configuration, the brightness of the electron beam reaching each atom is given by Equation 5 below.
[0039]
number
[0040] The light source generating unit 606 can utilize a 3D Gaussian to create a continuous light source shape. At this time, the 3σ of the Gaussian can be set smaller than the width, length, and height of the sample so that no discontinuity occurs at the edge of the sample.
[0041] According to another embodiment, the light source generating unit 606 may use a method in which a two-dimensional Gaussian and an exponential decay function are simultaneously utilized to remove discontinuities that may occur in the horizontal and vertical directions of the sample using the two-dimensional Gaussian, and to remove discontinuities that may occur in the height direction of the sample using the exponential decay function.
[0042] Meanwhile, the brightness of the electron beam that decreases as the electron beam passes through the sample can be simulated by applying exponential decay to the direction of the electron beam. The light source to which exponential decay is applied can be defined as Equation 6 below.
[0043]
number
[0044] The size and shape of the light source generated by the light source generating unit 606 can be adaptively changed according to the size of the input slab and the size of the diffraction pattern image, etc., and as a result, the ringing effect of the diffraction pattern that can occur from discontinuity of the light source can be prevented, as shown in Fig. 7. That is, the light source generating unit 606 can adaptively change the size and shape of the light source according to the size of the input slab and the size of the diffraction pattern image to prevent the ringing effect of the diffraction pattern.
[0045] The diffraction pattern generating unit 608 generates a diffraction pattern based on the reciprocal lattice vectors [h(x,y), k(x,y), l(x,y)] obtained by the HKL vector generating unit 604, the positions of atoms in the sample, and the brightness (I(x j ,y j ,z j )) can be used to calculate the accumulated diffraction pattern (F(h,k,l)) as shown in Equation 7 below.
[0046]
number
[0047] Next, the diffraction pattern generating unit 608 may calculate a maximum value of the accumulated diffraction pattern, and linearly normalize the accumulated diffraction pattern based on the calculated maximum value to generate a diffraction pattern image. According to another embodiment, the diffraction pattern generating unit 608 may generate a diffraction pattern image by nonlinearly normalizing the diffraction pattern using an image processing technique such as gamma correction. In this case, the diffraction pattern generating unit 608 may independently calculate the diffraction generated by the interaction between each atom and electron contained in the sample, and may perform high-speed calculations using CPU parallel processing or GPGPU (General Purpose computing on Graphics Processing Unit). The SADP image generated in this manner is shown in FIG. 13. As shown in FIG. 13, the SADP image does not have a ringing effect, a phenomenon in which the HOLZ (Higher Order Laue Zone) is included in the diffraction pattern, or a diffraction pattern with blurred diffraction points.
[0048] Meanwhile, the diffraction pattern generator 608 can apply various functions to the accumulated diffraction values to generate a SADP image. For example, the diffraction pattern generator 608 can apply a linear function to generate a diffraction pattern, and Vo=AVi γ The diffraction pattern can also be generated using a function used in gamma correction, such as: In summary, the SADP image generation system of this embodiment can generate an SADP image that prevents the ringing effect, the phenomenon in which HOLZ (higher order Laue zone) is included in the diffraction pattern, and the phenomenon in which a diffraction pattern with blurred diffraction points is generated, by using the reciprocal lattice vector, the position of atoms in the sample, and the brightness of the electron beam.
[0049] Hereinafter, a system and method for converting a real TEM SADP image and a virtual TEM SADP image using deep learning will be described with reference to the accompanying drawings. FIG 14 is a diagram illustrating a system for mutually converting between a real TEM SADP image and a virtual TEM SADP image according to an embodiment of the present invention, FIG 15 is a diagram illustrating an example in which a bright point in the center of an image is blocked using a beam stopper, and FIG 16 is a diagram illustrating an example of a real TEM SADP image with scale information displayed. FIG 17 is a diagram illustrating an example in which scale information is removed from a real TEM SADP image, FIG 18 is a diagram illustrating an example of a virtual TEM SADP image generated through a JEMS program, and FIG 19 is a diagram illustrating an example of a TEM SADP image generated in a form similar to the real one from a virtual TEM SADP image. FIG 20 is a diagram illustrating an example of a real-virtual mutual conversion algorithm according to an embodiment of the present invention.
[0050] The system for converting between real and virtual TEM SADP images in this embodiment includes a real diffraction pattern image refinement unit 1400, a virtual diffraction pattern generation unit 1402, a real-virtual conversion algorithm learning unit 1404, and a control unit that controls the overall operation of these units. The actual diffraction pattern image refiner 1400 can remove unnecessary parts in an actual TEM SADP image, for example, an actual TEM SADP image collected through an experiment or an actual TEM SADP image collected through the web. For example, the actual diffraction pattern image refinement unit 1400 can remove annotation information, scale information, lattice plane index information, etc. that were added to fill in additional information in the actual TEM SADP image, as shown in FIG. 15.
[0051] According to an embodiment, the actual diffraction pattern image refiner 1400 can remove annotation information from an actual TEM SADP image and synthesize the background in the removed portion using a commercial image processing program such as Photoshop or a hole-filling algorithm that fills in missing parts of the image. The SADP image obtained through this process is shown in FIG. 17. Of course, there is no program restriction as long as the removed parts are filled. As another example, the actual diffraction pattern image refinement unit 1400 can remove scale information from an actual TEM SADP image in which scale information is displayed as shown in FIG. 16, and generate an SADP image as shown in FIG. 17 by synthesizing the background with the removed part through a hole-filling algorithm, etc.
[0052] The virtual diffraction pattern generator 1402 may generate a virtual TEM SADP image corresponding to an actual TEM SADP image based on the input lattice constant and unit lattice using a TEM SADP simulation program such as JEMS, QSTEM, abTEM, Ladyne Software Suite, SingleCrystal, Condor, etc. Here, the lattice constant and unit lattice may be input by a user or may be extracted and obtained from an actual TEM SADP image.
[0053] When information on lattice constants and unit lattices is given in file formats such as CIF (Crystallography Information File), FHI-aims, and XYZ, a virtual TEM SADP image generated using the JEMS program is shown in Figure 18. Since the virtual SADP image generated at this time is subsequently used for real-virtual mutual conversion algorithm learning, generating SADP images using one program rather than using various programs can reduce confusion caused by data diversity during the learning process. For example, the same JEMS program can be used to generate a virtual TEM SADP image that corresponds to an actual TEM SADP image with unwanted portions removed, and no other program can be used to generate the virtual TEM SADP image.
[0054] According to another embodiment, the virtual diffraction pattern generating unit 1402 may select a virtual SADP image corresponding to an actual TEM SADP image from among the virtual TEM SADP images generated in FIGS.
[0055] In another aspect, the virtual diffraction pattern generating unit 1402 may generate a virtual SADP image corresponding to an actual TEM SADP image through the method described with reference to FIGS. In this case, the virtual diffraction pattern generating unit 1402 may include a sample generating unit that generates a sample using at least one of the lattice constant, the relative positions of atoms in the unit lattice, and the zone axis parameters, a vector generating unit that generates a reciprocal lattice vector corresponding to the unit lattice, a light source generating unit that calculates the brightness of the electron beam reaching the atoms in the generated sample, a diffraction pattern generating unit that generates a virtual diffraction pattern image (TEM SADP image) using the generated reciprocal lattice vector, the positions of atoms in the sample, and the calculated brightness of the electron beam, and a selection unit that selects a virtual diffraction pattern image corresponding to an actual diffraction pattern image from the generated virtual diffraction pattern images.
[0056] The real-virtual mutual conversion algorithm learning unit 1404 can learn a real-virtual mutual conversion algorithm that uses a real SADP image and a virtual SADP image to generate an image belonging to a real diffraction pattern domain from an image belonging to a virtual diffraction pattern domain, or generate an image belonging to a virtual diffraction pattern domain from an image belonging to a real diffraction pattern domain.
[0057] Then, the real-virtual mutual conversion algorithm learning unit 1404 may generate an SADP image belonging to a real diffraction pattern domain from an SADP image belonging to a virtual diffraction pattern domain by using a deep learning model, i.e., a deep learning technique. For example, as shown in FIG 19, an SADP image generated in a form similar to the real one displayed on the right side may be generated from a virtual TEM SADP image displayed on the left side. In this case, if a virtual diffraction pattern having the same diffraction point positions as the real diffraction pattern is prepared, an algorithm that requires image pairs belonging to both domains can be used, whereas if only one of the real diffraction pattern and the virtual diffraction pattern is prepared, an algorithm that does not require information about image pairs belonging to both domains can be used.
[0058] Through this series of processes, the real-virtual mutual conversion algorithm can generate a TEM SADP image similar to the real one by comprehensively considering the influence of various errors contained in the real TEM SADP image and the influence of the beam stopper.
[0059] According to another embodiment, the real-virtual mutual conversion algorithm learning unit 1404 can generate a SADP image belonging to a real diffraction pattern domain from a SADP image belonging to a virtual diffraction pattern domain, and can also generate a SADP image belonging to a virtual diffraction pattern domain from a SADP image belonging to a real diffraction pattern domain. If a deep learning application is created based on a TEM SADP image, the input may be a real TEM SADP image. In this case, by converting the input SADP image belonging to the actual diffraction pattern domain into a SADP image belonging to a virtual diffraction pattern domain, it is possible to utilize the deep learning model trained on the SADP image belonging to the virtual diffraction pattern domain.
[0060] As shown in FIG. 20, to specifically describe the real-virtual mutual conversion algorithm learning unit 1404, the real-virtual mutual conversion algorithm learning unit 1404 can include a real diffraction pattern classification unit, a virtual diffraction pattern classification unit, a Real2Sim conversion unit, and a Sim2Real conversion unit.
[0061] The real diffraction pattern discrimination unit learns a deep learning model that distinguishes between a virtual diffraction pattern converted to a form similar to a real diffraction pattern through the Sim2Real conversion unit and a diffraction pattern photographed through a real TEM. At this time, the virtual diffraction pattern converted to a form similar to a real diffraction pattern and the diffraction pattern photographed through a real TEM are all images belonging to the real diffraction pattern domain.
[0062] The virtual diffraction pattern classification unit learns a deep learning model that distinguishes between an actual diffraction pattern converted into a form similar to the virtual diffraction pattern through the Real2Sim conversion unit and a virtual diffraction pattern generated through simulation. At this time, the actual diffraction pattern converted into a form similar to the virtual diffraction pattern and the virtual diffraction pattern generated through simulation are all images belonging to the virtual diffraction pattern domain.
[0063] The Real2Sim conversion unit trains a deep learning model that converts images belonging to the real diffraction pattern domain into images belonging to the virtual diffraction pattern domain. The goal of the Sim2Real conversion unit is to generate images that are so similar that the virtual diffraction pattern classification unit cannot distinguish between images generated by itself and images generated through simulation.
[0064] The Sim2Real conversion unit trains a deep learning model that converts images belonging to the virtual diffraction pattern domain into images belonging to the real diffraction pattern domain. The goal of the Sim2Real conversion unit is to generate images that are so similar that the real diffraction pattern classification unit cannot distinguish between the images it generates and images taken through a real TEM.
[0065] The four components belonging to the real-virtual mutual conversion algorithm learning unit 1404 influence each other to learn a deep learning model that can successfully achieve the goals assigned to each component, and ultimately become able to smoothly convert between images belonging to the real diffraction pattern domain and images belonging to the virtual diffraction pattern domain.
[0066] Since the process of acquiring actual TEM SADP images through experiments requires a lot of time, if a deep learning model trained on SADP images belonging to the virtual diffraction pattern domain can be used in real applications, it will be possible to easily obtain large amounts of training data sets and build high-performance applications. In summary, the system (image conversion system) for mutually converting between real TEM SADP images and virtual TEM SADP images according to the present embodiment can remove unnecessary information from collected real TEM SADP images, generate virtual TEM SADP images corresponding to the real TEM SADP images from which unnecessary information has been removed, and learn a real-virtual mutual conversion algorithm using the generated virtual TEM SADP images and real TEM SADP images. In this case, the real-virtual mutual conversion algorithm can generate a SADP image belonging to a real diffraction pattern domain from a SADP image belonging to a virtual diffraction pattern domain, and can also generate a SADP image belonging to a virtual diffraction pattern domain from a SADP image belonging to a real diffraction pattern domain.
[0067] By generating a TEM SADP image similar to the actual one from the virtual TEM SADP, various errors that may occur in an actual TEM experiment can be simulated. In addition, when creating a deep learning application using TEM SADP images, the gap between the real TEM SADP and the virtual TEM SADP is reduced through the real-virtual mutual conversion algorithm, so that the virtual data generated by the simulation can be effectively used for deep learning training.
[0068] Meanwhile, the components of the above-described embodiments can be easily understood from a process perspective, i.e., each component can be understood as a respective process, and the processes of the above-described embodiments can be easily understood from the perspective of the components of the apparatus. In addition, the technical contents described above may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments, or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine language codes, such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform operations of the embodiments, and vice versa.
[0069] The above-described embodiments of the present invention have been disclosed for illustrative purposes, and a person skilled in the art having ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the following claims.
Claims
1. an actual diffraction pattern image refinement unit for removing unnecessary information in the actual diffraction pattern image; a virtual diffraction pattern generating unit for acquiring a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and a real-to-virtual mutual conversion algorithm learning unit that uses at least one of the actual diffraction pattern image from which unnecessary information has been removed and the virtual diffraction pattern image to generate an image belonging to the real diffraction pattern domain from an image belonging to a virtual diffraction pattern domain, or generates an image belonging to the virtual diffraction pattern domain from the image belonging to the real diffraction pattern domain.
2. 2. The diffraction pattern image conversion system of claim 1, wherein the diffraction pattern image is a Transmission Electron Microscope (TEM) Selected Area Diffraction Pattern (SADP) image.
3. The diffraction pattern image conversion system of claim 2 , wherein the unnecessary information is information regarding annotation, scale, or index.
4. 4. The diffraction pattern image conversion system of claim 3, wherein the actual diffraction pattern image refinement unit detects the unnecessary information in the actual diffraction pattern image, and fills in the area where the unnecessary information was present using peripheral information of the detected unnecessary information using a hole-filling algorithm.
5. The virtual diffraction pattern generator utilizes a TEM SADP simulation program; 3. The diffraction pattern image conversion system of claim 2, wherein the TEM SADP simulation program generates a virtual TEM SADP image corresponding to the actual TEM SADP image based on an input lattice constant and a unit cell.
6. 6. The diffraction pattern image conversion system of claim 5, wherein the lattice constant and the information on the unit lattice can be provided in a file format such as Crystallography Information File (CIF), FHI-aims, or XYZ.
7. The virtual diffraction pattern generating unit includes: a sample preparation unit for preparing a sample using at least one of a lattice constant, a relative position of an atom in a unit lattice, and a zone axis parameter; A vector generating unit that generates a reciprocal lattice vector corresponding to the unit lattice; a light source generating unit for determining the brightness of the generated electron beam reaching atoms in the sample; a diffraction pattern generator that generates a virtual diffraction pattern image using the generated reciprocal lattice vectors, the relative positions of atoms in the sample, and the brightness of the electron beam that is obtained; 3. The diffraction pattern image conversion system of claim 2, further comprising: a selection unit for selecting a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images.
8. The sample preparation unit adaptively determines the number of layers of the slab according to the lattice constant and the zone axis parameter among the input parameters to prevent a phenomenon in which a HOLZ (Higher Order Laue Zone) is included in the diffraction pattern or a diffraction pattern with a blurred diffraction point is generated, 8. The diffraction pattern image conversion system of claim 7, wherein the light source generating unit adaptively changes the shape and intensity of the light source according to the size of an input slab or the size of a diffraction pattern image so as to prevent a ringing effect of the diffraction pattern that may occur due to discontinuities of the light source.
9. 3. The diffraction pattern image conversion system of claim 2, wherein the real-virtual mutual conversion algorithm learning unit generates a diffraction pattern image belonging to the real diffraction pattern domain from a diffraction pattern image belonging to the virtual diffraction pattern domain by utilizing a deep learning model.
10. 10. The diffraction pattern image conversion system of claim 9, wherein the real-virtual mutual conversion algorithm learning unit generates a diffraction pattern image belonging to the real diffraction pattern domain from the diffraction pattern image belonging to the virtual diffraction pattern domain using a specific algorithm when the real diffraction pattern image and the virtual diffraction pattern image are prepared.
11. The real-virtual mutual conversion algorithm learning unit includes: A Real2Sim conversion unit; A Sim2Real conversion unit; A real diffraction pattern discrimination unit that learns a deep learning model to discriminate between a virtual diffraction pattern converted into a form similar to a real diffraction pattern through the Sim2Real conversion unit and a real diffraction pattern photographed through a TEM; and a virtual diffraction pattern discrimination unit for learning a deep learning model for discriminating between an actual diffraction pattern converted into a form similar to the virtual diffraction pattern through the Real2Sim conversion unit and a virtual diffraction pattern generated through simulation, The Real2Sim conversion unit trains a deep learning model that converts an image belonging to the real diffraction pattern domain into an image belonging to the virtual diffraction pattern domain; The diffraction pattern image conversion system of claim 2 , wherein the Sim2Real conversion unit trains a deep learning model that converts an image belonging to the virtual diffraction pattern domain into an image belonging to the real diffraction pattern domain.
12. an actual diffraction pattern image refinement unit for removing unnecessary information in the actual diffraction pattern image; a virtual diffraction pattern generator for acquiring a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and an algorithm learning unit that generates a diffraction pattern image belonging to a real diffraction pattern domain from a diffraction pattern image belonging to a virtual diffraction pattern domain by using a deep learning algorithm trained using at least one of the actual diffraction pattern image and the virtual diffraction pattern image.
13. The diffraction pattern image conversion system of claim 12, wherein when the actual diffraction pattern image and the virtual diffraction pattern image are prepared, the algorithm learning unit generates a diffraction pattern image belonging to a pre-actual diffraction pattern domain from the diffraction pattern image belonging to the virtual diffraction pattern domain using a specific algorithm.
14. an actual diffraction pattern image refinement unit for removing unnecessary information in the actual diffraction pattern image; a virtual diffraction pattern generator for acquiring a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and an algorithm learning unit that generates a diffraction pattern image belonging to a virtual diffraction pattern domain from a diffraction pattern image belonging to a real diffraction pattern domain by using a deep learning algorithm trained using at least one of the actual diffraction pattern image and the virtual diffraction pattern image.
15. A computer-readable recording medium having program code recorded thereon, The program code comprises: removing unnecessary information in the actual diffraction pattern image; generating a virtual diffraction pattern image corresponding to the actual diffraction pattern image with the unwanted information removed; a step of generating an image belonging to a real diffraction pattern domain from an image belonging to a virtual diffraction pattern domain, or generating an image belonging to the virtual diffraction pattern domain from the image belonging to the real diffraction pattern domain, using at least one of the actual diffraction pattern image from which the unnecessary information has been removed and the virtual diffraction pattern image.
Citation Information
Patent Citations
Semiconductor image processing device
JP2021022056A
A Reconstruction Method of Lens-free Diffraction Image and Portable Analyzing Device of Water Particles Using this
KR102111124B1
Mitigating reality gap through training a simulation-to-real model using a vision-based robot task model
WO2021096561A1
