Transmission electron microscopic image processing method and device and transmission electron microscopic image reconstruction method and device

Abstract

The invention discloses a transmission electron microscope image processing and image reconstruction method and device, and relates to the technical field of transmission electron microscopes. For an image set acquired in a STEM mode of a TEM, the method comprises the following steps: setting one or more theoretical acquisition times in an acquisition time period corresponding to the image set, the image set comprising a plurality of images acquired at different times; searching one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points in the image; and generating an image corresponding to the theoretical acquisition time by using the searched pixel points on the one or two target images. According to the scheme provided by the embodiment of the invention, the distortion of the image of the automatic rotating sample shot by the TEM in the STEM mode can be adjusted, so that the reconstructed three-dimensional structure can truly reflect the microstructure of the material.

Description

Technical Field

[0001] The present invention relates to the technical field of transmission electron microscopy, and in particular to a transmission electron microscopy image processing and image reconstruction method and device. Background Art

[0002] In the scanning transmission electron microscopy (STEM) mode of a transmission electron microscope (TEM), the electron microscope and the camera are linked by an automatic control program. During the continuous rotation of the sample stage, the camera can collect a series of transmission electron microscopic two-dimensional images (also called TEM images) of the sample on the sample stage through video recording mode, and then reconstruct the three-dimensional structure of the sample (such as electronic devices, materials, etc.) by combining the series of TEM images.

[0003] Since the imaging principle of the STEM mode is that the convergent beam converges on the sample surface to perform point-by-point grid scanning and imaging, for example, if an image with 12×12 pixels needs to be collected, the convergent beam needs to sequentially scan the position of each pixel point in the 144 pixels, and the convergent beam needs to stay at the position of a pixel point for a certain period of time. During the STEM mode scanning and imaging process, the sample stage rotates continuously, so that there are certain differences in the sample stage rotation angles corresponding to each pixel point on the same image, that is, each pixel point on the same image is not collected at the same sample stage rotation angle, resulting in different degrees of distortion in the collected TEM image and the subsequently reconstructed three-dimensional image. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a transmission electron microscopy image processing and image reconstruction method and device. The solution provided by the embodiment of the present invention can adjust the distortion of the image of the automatically rotating sample taken by the TEM in the STEM mode, so that the reconstructed three-dimensional structure can more realistically reflect the microstructure of the material.

[0005] To achieve the above object, according to a first aspect of an embodiment of the present invention, a transmission electron microscopy image processing method is provided, which is for an image set collected in a STEM mode of a TEM, comprising:

[0006] Within a collection period corresponding to the image set, one or more theoretical collection times are set, wherein the image set includes a plurality of images collected at different times;

[0007] According to the acquisition time corresponding to the pixel points in the image, searching for one or two target images adjacent to the theoretical acquisition time;

[0008] An image corresponding to the theoretical acquisition time is generated by using the found pixel points on the one or two target images.

[0009] Optionally, the searching for one or two target images adjacent to the theoretical acquisition time includes:

[0010] For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations:

[0011] Determining a plurality of target pixel points corresponding to the positions of the theoretical pixel points from a plurality of images included in the image set;

[0012] The acquisition times corresponding to the plurality of target pixels are used to determine a first target pixel that coincides with the theoretical acquisition time or two second target pixels that are adjacent to the theoretical acquisition time.

[0013] Optionally, the generating an image corresponding to the theoretical acquisition time by using the found pixel points on one or two target images includes:

[0014] For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations:

[0015] In the case where a first target pixel point that coincides with the theoretical acquisition time is determined, the pixel value of the first target pixel point is used as the pixel value of the theoretical pixel point;

[0016] In the case where two second target pixels adjacent to the theoretical acquisition time are determined, the pixel value of the theoretical pixel is calculated using the pixel values ​​of the two second target pixels, the theoretical acquisition time, the acquisition time of any second target pixel, and the acquisition duration of a single image;

[0017] The calculated pixel values ​​of the theoretical pixel points are combined to obtain an image corresponding to the theoretical acquisition time.

[0018] Optionally, searching one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points of the image and using the pixel points on the found one or two target images to generate an image corresponding to the theoretical acquisition time includes:

[0019] Determine the position of the theoretical acquisition time in the image set according to the acquisition time corresponding to the pixel point of the image;

[0020] In the case where the theoretical acquisition time is located on the first target image, an image corresponding to the theoretical acquisition time is generated by using pixel points of the first target image and pixel points of a second target image adjacent to the first target image;

[0021] In view of the situation that the theoretical acquisition time is between two adjacent third target images, the image corresponding to the theoretical acquisition time is generated by using the pixel points of the two third target images.

[0022] Optionally, determining the position of the theoretical acquisition time in the image set includes:

[0023] The position of the theoretical acquisition time in the image set is determined according to the acquisition time of a single image, the theoretical acquisition time, and the acquisition sequence of the plurality of images.

[0024] Optionally, the transmission electron microscopy image processing method may further include:

[0025] Correcting the exposure time of the pixels set by the TEM;

[0026] The actual time consumed for capturing a single image is calculated according to the corrected exposure time and the number of pixels of the image, and the actual time consumed is determined as the acquisition time length of the single image.

[0027] Optionally, the using pixel points of the two third target images to generate an image corresponding to the theoretical acquisition time includes:

[0028] For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations:

[0029] Searching for a third target pixel point and a fourth target pixel point corresponding to the position of the theoretical pixel point in the two third target images respectively, and determining a second acquisition time of the third target pixel point;

[0030] Calculate the pixel value of a theoretical pixel by using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition time of a single image;

[0031] The calculated pixel values ​​of the theoretical pixel points are combined to obtain an image corresponding to the theoretical acquisition time.

[0032] Optionally, the generating an image corresponding to the theoretical acquisition time by using pixel points of the first target image and pixel points of a second target image adjacent to the first target image includes:

[0033] For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations:

[0034] Searching for a fifth target pixel point and a sixth target pixel point corresponding to the position of the theoretical pixel point in the first target image and the second target image respectively, and determining a fifth acquisition time corresponding to the fifth target pixel point;

[0035] Calculate the pixel value of a theoretical pixel by using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition time of a single image;

[0036] The calculated pixel values ​​of the theoretical pixel points are combined to obtain an image corresponding to the theoretical acquisition time.

[0037] Optionally, after determining the fifth acquisition time corresponding to the fifth target pixel point, the method further includes:

[0038] When the fifth acquisition time is consistent with the theoretical acquisition time, directly determining the pixel value of the fifth target pixel point to be the pixel value of the theoretical pixel point;

[0039] When the fifth acquisition time is inconsistent with the theoretical acquisition time, the step of calculating the pixel value of the theoretical pixel by using the fifth target pixel, the sixth target pixel, the fifth acquisition time and the acquisition duration of a single image is performed.

[0040] Optionally, the transmission electron microscopy image processing method further comprises:

[0041] In the case where the theoretical acquisition time is located on the first target image, the following operations are performed for each theoretical pixel point:

[0042] In a case where the fifth acquisition time of the fifth target pixel point is before the theoretical acquisition time, determining, according to the acquisition order of the plurality of images in the image set, that the next image of the first target image is the second target image;

[0043] In the case that the fifth acquisition time of the fifth target pixel point is after the theoretical acquisition time, according to the acquisition order of the multiple images in the image set, the previous image of the first target image is determined to be the second target image.

[0044] Optionally, the transmission electron microscopy image processing method further comprises:

[0045] receiving one or more sample stage rotation angles inputted from an external source;

[0046] Calculate the rotation time corresponding to the rotation angle of the sample stage using the sample rod rotation rate set by the TEM;

[0047] The theoretical collection time corresponding to the rotation angle of the sample stage is set by using the start time of the sample rod and the rotation time corresponding to the rotation angle of the sample stage.

[0048] Furthermore, according to a second aspect of an embodiment of the present invention, there is provided an image reconstruction method, comprising: the transmission electron microscopy image processing method provided by the embodiment of the first aspect mentioned above.

[0049] Furthermore, in a third aspect, an embodiment of the present invention provides a transmission electron microscopy image processing device, comprising: a setting unit, a position search unit and an image generation unit, wherein:

[0050] The setting unit is used to set one or more theoretical acquisition times within an acquisition period corresponding to the image set, wherein the image set includes a plurality of images acquired at different times;

[0051] The search unit is used to search for one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points of the image;

[0052] The image generation unit is used to generate an image corresponding to the theoretical acquisition time by using the found pixel points on one or two target images.

[0053] Furthermore, in a fourth aspect, an embodiment of the present invention provides an electronic device, including:

[0054] one or more processors;

[0055] a storage device for storing one or more programs,

[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect embodiment or the second aspect embodiment mentioned above.

[0057] Furthermore, in a fifth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method provided in the first aspect embodiment or the second aspect embodiment described above is implemented.

[0058] An embodiment of the above invention has the following advantages or beneficial effects: by setting one or more theoretical acquisition times, determining the position of the theoretical acquisition time in the image set, and then using the pixel points on the two images closest to the theoretical acquisition time to generate an image corresponding to the theoretical acquisition time, integrating the pixel points on the two images closest to the theoretical acquisition time, and based on the theoretical acquisition time, an image in the same plane can be generated, thereby improving image distortion, so that the generated image can more realistically reflect the microscopic two-dimensional structure of the material, and can adjust the distortion of the image of the automatically rotating sample taken by the TEM in the STEM mode, so that the reconstructed three-dimensional structure can more realistically reflect the microscopic structure of the material.

[0059] In addition, by generating images through the above scheme and adding them to the image set, the image distortion caused by the continuous rotation of the sample stage can be reduced, so that the image set can more realistically reflect the sample information, and effectively improve the authenticity and accuracy of the subsequent three-dimensional structure reconstructed based on the image set.

[0060] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0062] Figure 1 It is an exemplary two-dimensional image with 12×12 pixels collected by continuous tilting in STEM mode;

[0063] Figure 2 It is an exemplary three-dimensional image of 5×5 pixels and 8 images in total collected in STEM mode, sorted by acquisition time;

[0064] Figure 3 is a schematic diagram of the main process of the transmission electron microscopy image processing method according to an embodiment of the present invention;

[0065] Figure 4 It is a schematic diagram of the main process of searching for one or two target images adjacent to the theoretical acquisition time in the pixel dimension according to an embodiment of the present invention;

[0066] Figure 5 is a schematic diagram of a main process of generating an image corresponding to a theoretical acquisition time according to an embodiment of the present invention;

[0067] Figure 6 is a schematic diagram of a main process of generating an image corresponding to a theoretical acquisition time according to another embodiment of the present invention;

[0068] Figure 7This is a 1024×1024 two-dimensional image of the three remaining layers of a 28nm smartphone processor chip after thinning, collected by TEM in STEM mode;

[0069] Figure 8 Yes Figure 7 The two-dimensional image shown is an image processed by the technical solution provided by the embodiment of the present invention;

[0070] Fig. 9 is a schematic diagram of main units of a transmission electron microscopy image processing device according to an embodiment of the present invention;

[0071] Fig.10 is a schematic diagram of a system structure according to an embodiment of the present invention;

[0072] Fig.11 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The transmission electron microscope (TEM) with ultra-high spatial resolution can clearly characterize the microstructure of materials and has been widely used in chemistry, materials science, biological science and other fields. The imaging principle of the transmission electron microscope is: using an electron beam with an extremely short wavelength as a light source, the accelerated and focused electron beam is projected onto a very thin sample. The collision of electrons with atoms in the sample will change direction, resulting in three-dimensional angle scattering. The size of the scattering angle is related to factors such as the density and thickness of the sample, which will form images of different brightness. After being magnified and focused, the image is displayed on an imaging device (fluorescent screen, photosensitive coupling device).

[0074] At present, the two most commonly used imaging modes of transmission electron microscopes are transmission electron microscope (TEM) mode and scanning transmission electron microscope (STEM) mode. The technical solution provided in the embodiment of the present invention is mainly for the STEM mode. The STEM mode uses a converged electron beam to scan the sample surface point by point in a grid, and uses a ring detector at different angles below the sample to detect and record the imaging. In layman's terms, the convergent beam in the STEM mode is scanned point by point and then collected, just like the light of the laser scans point by point on the wall and then images. The STEM mode imaging obtains a two-dimensional image, and the subsequent three-dimensional reconstruction of the microscopic three-dimensional morphology or three-dimensional structure of the material through multiple two-dimensional images is required. Among them, electron microscopic tomographic reconstruction imaging technology is the most commonly used technology for observing the three-dimensional complete morphology of nanocrystalline samples. Its advantages are not only extremely high spatial resolution, but more importantly, a variety of signals can be used to perform three-dimensional imaging of the chemical composition, crystal orientation, dislocation, defects, etc. of the sample. Electron microscopic tomographic reconstruction imaging is combined with a dedicated tomographic scanning bracket, usually with a specified step size, generally 1-3°, to rotate the sample and image it at each rotation angle. After obtaining data exceeding a certain rotation angle (generally >120°), a series of images (tilt sequence) are aligned relative to a common origin and tilt axis, and then the three-dimensional reconstruction shape of the crystal is obtained using a related reconstruction algorithm. Generally speaking, the data collection time for tilt series images is about 45-60 minutes, which is relatively time-consuming and only applicable to nanomaterial samples that are stable under electron beam flow.

[0075] In order to overcome the problems existing in the existing tomography, a technical means for collecting three-dimensional reconstruction data by continuous tilt imaging is provided in the preliminary research of this patent (patent application number: 202410703691.5), which is a new electron microscopy tomography reconstruction imaging technology. This technology uses an automatic control program to link the electron microscope and the camera to complete continuous rotation while using high frame rate video to record the scanned image data. Continuous tilt imaging to collect data can realize the collection of complete image data in a short time, generally only five to six minutes. For example, for the method of collecting data by continuous tilt imaging, collecting a conventional 1024×1024 STEM image with a dwell time of 2μs for each pixel, it takes 4s to scan the image from beginning to end, which greatly reduces the consumption of electron dose and has a major breakthrough in the field of three-dimensional reconstruction of electron beam sensitive materials.

[0076] However, there are some limitations to collecting continuous tilt imaging data in STEM mode. First, the rotation of the sample rod is a continuous process and does not stop halfway. Second, for STEM mode, its imaging principle is to scan the sample surface point by point in a grid-like manner with a converging beam. Figure 1 is an image collected in STEM mode with 12×12 pixels and a dwell time of 500ns. Assume that the coordinate value of the first pixel is (1,1), that is, the collection process of this single image is that the convergent beam sequentially scans and images each pixel from the coordinate point (1,1)→(1,2)→…→(1,12)→(2,1)→…→(2,12)→…→(12,12) (that is, the convergent beam scans and images each pixel from left to right and from top to bottom on the sample), and the dwell time of each pixel is 500ns. Figure 1 As can be seen from the images shown, there are still some defects in collecting image data through continuous tilt imaging. The image quality of a single two-dimensional image is relatively fuzzy, and it is difficult to identify the shape of the sample photographed in a two-dimensional image. This is because, in the process of continuous tilt scanning of each pixel point, the angle corresponding to each pixel point has changed, that is, the acquisition position of each pixel point has changed, resulting in the distortion of the two-dimensional image composed of the collected pixels. For example, for a 1024×1024 STEM image with a dwell time of 2μs collected by continuous tilt, the sample rod is in a uniform and continuous rotation process. Assuming that the sample starts to rotate from 0°, when the beam is at the first pixel point (1,1), the sample is 0°; when the beam is at the last pixel point (1024,1024), 4s have passed, and the calculated rotation angle of the sample rod is: 0°+0.3°×4=1.2°. In other words, for an image collected using continuous tilt, the first pixel point and the last pixel point differ by 1.2°. Similarly, each pixel point on an image corresponds to a time, and each time corresponds to an angle. Therefore, for an image collected using continuous tilt, each pixel corresponds to a different angle. Such image data has angle distortion, and no angle can fully represent the image. Therefore, we need to correct the angle of the image to ensure that each image represents the same time and the same angle. The resulting image cannot be used for 3D reconstruction.

[0077] In addition, when using the STEM mode to collect continuously tilted image data, the rotation of the sample rod is a continuous process and will not stop in the middle. Due to the angular distortion, each angle represents each time, that is, there is also distortion in time, plus the drift caused by rotation, the image will be distorted. Therefore, in order to obtain a clear two-dimensional image or three-dimensional structure of the sample, a suitable algorithm must be used to correct these distortions and improve the distortion of the image.

[0078] The terms "first", "second", "third", "fourth", "fifth", etc. included in the embodiments of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific number or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is merely a way of distinguishing objects with the same attributes when describing the embodiments of the present invention.

[0079] The image set collected in the STEM mode involved in the embodiment of the present invention refers to an image set consisting of multiple two-dimensional images collected by continuously tilting the sample rod in the STEM mode of the TEM.

[0080] The theoretical acquisition time involved in the embodiment of the present invention refers to any value within the time interval consumed by obtaining the image set. For example, the time interval consumed by the image set is [t1, t n ], where t1 is the time of collecting the first pixel of the first image; t n is the time to collect the last pixel of the last image; the theoretical collection time is generally (t1, t n ), preferably, the theoretical acquisition time is [t1+m×T, t n -(m-1)×T], where m represents the number of pixels on a two-dimensional image, and T represents the time taken to collect one pixel; then, t1+m×T is the time corresponding to the last pixel collected in the first image, t n -(m-1)×T is the time corresponding to the first pixel in the last image.

[0081] It can be understood that after sorting the multiple two-dimensional images in the image set according to the order of acquisition time, in the sorting direction of the multiple two-dimensional images, any theoretical acquisition time actually corresponds to a two-dimensional area at a fixed height, and accordingly, the generated image corresponding to the theoretical acquisition time is the image within the two-dimensional area at the fixed height. Figure 2Taking the data collected in STEM mode with 5×5 pixels and 8 images in total as an example, a three-dimensional xyz coordinate system is established. The coordinates of the first pixel of the first image are (1,1,1), that is, the coordinates of the pixel of the i-th row and j-th column of the n-th image are (i,j,n). The scanning process of the light beam is (1,1,1)→(1,2,1)→…→(1,5,1)→(2,1,1)→…→(5,5,1)→…→(1,1,2)→(1,2,2)→…→(5,5,8), and each pixel corresponds to a time. At the same time, since the sample rod is in a state of uniform and continuous rotation, each time corresponds to an angle, so each pixel of the continuous tilt imaging data collected in STEM mode corresponds to an angle and time. Since the sample rod rotates continuously during the image data collection process, there are slight differences in the height of each pixel in the same two-dimensional image on the z-axis (it is precisely because of this slight difference that each pixel in the constructed two-dimensional image cannot actually represent the image data collected by the sample rod at a certain angle, ignoring the pixel effect caused by this slight difference, causing the image data to be distorted). In the xyz coordinate system, the two-dimensional images are stacked in the z-axis direction in chronological order, and a time point corresponds to a height of the z-axis, such as Figure 2 The time t1 shown corresponds to the height z1, the time t2 corresponds to the height z2, ..., the time t n Corresponding height z n Etc., then time t1 is used as the theoretical acquisition time, and the solution provided by the embodiment of the present invention can obtain the set of pixel points at height z1 corresponding to time t1, time t2 is used as the theoretical acquisition time, and the solution provided by the embodiment of the present invention can obtain the set of pixel points at height z2 corresponding to time t2, etc. Each pixel point in the two-dimensional image corresponding to each theoretical acquisition time is obtained based on the pixel points of each two-dimensional image in the image set. The technical solution provided by the embodiment of the present invention will be described in detail below.

[0082] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0083] The transmission electron microscopy image processing method provided in the embodiment of the present invention is for an image set collected in the STEM mode of a TEM. Preferably, the two-dimensional image in the image set is collected by continuously rotating the sample rod. It can be understood that the two-dimensional image in the image set can also be obtained by the existing method of rotating the sample rod once to collect a two-dimensional image. Specifically, Figure 3 As shown, the transmission electron microscopy image processing method may include the following steps:

[0084] Step S301: setting one or more theoretical acquisition times within an acquisition period corresponding to an image set, wherein the image set includes multiple images acquired at different times;

[0085] Step S302: searching for one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points in the image;

[0086] Step S303: Generate an image corresponding to the theoretical acquisition time using the pixel points on the one or two target images found.

[0087] As described above, the image set is mainly obtained by combining two-dimensional images obtained by collecting image data of electron microscopy tomography using continuous tilt imaging. The images processed by this technical solution can be subsequently reconstructed in three dimensions, or the images obtained by this technical solution can be reconstructed in three dimensions together with the existing images in the image set.

[0088] Among them, the program for implementing the above-mentioned transmission electron microscopy image processing can be embedded in the operating software carried by the TEM. After starting the operating software carried by the TEM, the program for implementing the transmission electron microscopy image processing can be synchronously started to perform the above-mentioned steps S301 to S303. In addition, the transmission electron microscopy image processing program can also be used as an independent software for terminal equipment or servers to perform image processing on the image set collected by the TEM in the STEM mode to provide a data basis for subsequent three-dimensional reconstruction. Further, the transmission electron microscopy image processing program can also be used as a part of the three-dimensional reconstruction program or embedded in the three-dimensional reconstruction program, so as to perform the image processing of the above-mentioned steps S301 to S303 on the image set collected by the TEM in the STEM mode through the three-dimensional reconstruction software, and perform three-dimensional reconstruction on the image processing result or the image processing result combined with the original image set to obtain the three-dimensional microstructure or three-dimensional micromorphology of the sample.

[0089] Specifically, the transmission electron microscopy image processing can be achieved by operating a setting button or by clicking a virtual operation control provided by the interface.

[0090] It can be understood that the multiple images acquired at different times in the image set provided by the embodiment of the present invention are multiple two-dimensional images acquired for the same sample within an uninterrupted acquisition time, and the pixels contained in the multiple two-dimensional images correspond to each other.

[0091] For the case where each 2D image in the image set is acquired under the continuous tilting of the sample rod, such as Figure 2As shown, after the multiple images acquired at different times in the image set are sorted in the order of acquisition time, the multiple images acquired at different times correspond to different heights on the z-axis. Based on this, the one or two target images adjacent to the theoretical acquisition time generally refer to one or two images adjacent to the z value corresponding to the theoretical acquisition time on the z-axis. Among them, if the acquisition time of a pixel point is exactly the same as the theoretical acquisition time, the image where the pixel point is located is the target image adjacent to the theoretical acquisition time.

[0092] The technical solution provided by the embodiment of the present invention sets one or more theoretical acquisition times, determines the position of the theoretical acquisition time in the image set, and then uses the pixel points on the two images closest to the theoretical acquisition time to generate an image corresponding to the theoretical acquisition time. The pixel points on the two images closest to the theoretical acquisition time are integrated, and images on the same plane can be generated based on the theoretical acquisition time, thereby improving image distortion, so that the generated image can more realistically reflect the microscopic two-dimensional structure of the material, and can adjust the distortion of the image of the automatically rotating sample taken by the TEM in the STEM mode, so that the reconstructed three-dimensional structure can more realistically reflect the microscopic structure of the material.

[0093] In addition, by generating images through the above scheme and adding them to the image set, the image distortion caused by the continuous rotation of the sample stage can be reduced, so that the image set can more realistically reflect the sample information, and effectively improve the authenticity and accuracy of the subsequent three-dimensional structure reconstructed based on the image set.

[0094] Furthermore, the above-mentioned search for one or two target images adjacent to the theoretical acquisition time and the use of pixels on the one or two target images found to generate an image corresponding to the theoretical acquisition time provided by the embodiment of the present invention involve pixel acquisition time or image acquisition time. Through multiple experimental processes, it is found that the time consumed by the light beam on an image includes not only the dwell time at each pixel, but also the time consumed when the light beam moves between each pixel. Therefore, in order to further improve the accuracy of searching for adjacent target pixels or target images for the theoretical acquisition time, the exposure time of the camera in the embodiment of the present invention is corrected, and the correction can be achieved by existing technical means. For example, for the continuous acquisition of 1024×1024 two-dimensional images by uninterrupted uniform rotation of the sample rod, the theoretical time consumption, actual time consumption and corrected pixel exposure time Dwell Time* of a single image obtained multiple times are shown in Table 1 below.

[0095] Table 1

[0096] 1024,500ns 0.52s 0.78s 740ns 1024,1μs 1.05s 2.04s 1.95μs 1024,2μs 2.10s 4.04s 3.85μs 1024,5μs 5.24s 6.87s 6.55μs

[0097] The actual time taken for a single two-dimensional image is ScanOneFrameTime*=DwellTime*×x×y; based on this, the acquisition time corresponding to the pixel point on the i-th row and j-th column of the n-th two-dimensional image in the image set can also be calculated. Specifically, the following calculation formula (1) can be used to calculate:

[0099] t (i,j,n) =(n-1)×ScanOneFrameTime * +[y(i-1)+(j-1)]×DwellTime * (1)

[0100] Where n represents the nth two-dimensional image in the image set, t (i,j,n) Indicates the acquisition time corresponding to the pixel point on the i-th row and j-th column of the n-th two-dimensional image; ScanOneFrameTime * It indicates the time taken to take a single two-dimensional image after correcting the exposure time; DwellTime* indicates the exposure time after correction; y indicates the number of pixels contained in each row of a single two-dimensional image.

[0101] That is, the technical solution provided by the embodiment of the present invention also includes: correcting the exposure time of the pixel points set by the TEM; calculating the actual time consumption of a single image according to the corrected exposure time and the number of pixel points of the image, and determining the actual time consumption as the acquisition time of the single image. Through this process, the accuracy of determining the target image or target pixel point for the theoretical acquisition time is further improved. It should be noted that the following embodiments involve the acquisition time of a single two-dimensional image and the acquisition time of each pixel point, etc., and are all explained by taking the exposure time Dwell Time* of the corrected pixel point as an example. The subsequent acquisition time of a single two-dimensional image and the acquisition time of each pixel point, etc. can also directly use the theoretical exposure time provided by the camera.

[0102] Furthermore, the specific implementation of the above-mentioned searching for one or two target images adjacent to the theoretical acquisition time can be completed from the dimension of pixels or from the dimension of images.

[0103] Specifically, a specific implementation method for searching for one or two target images adjacent to the theoretical acquisition time from the dimension of pixel points may include:

[0104] For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations: determine multiple target pixel points corresponding to the position of the theoretical pixel point from the multiple images included in the image set; use the acquisition times corresponding to the multiple target pixel points to determine the first target pixel point that coincides with the theoretical acquisition time or two second target pixel points that are adjacent to the theoretical acquisition time.

[0105] Among them, since the position of each pixel point in a two-dimensional image scanned by the laser beam is relatively fixed, that is, the x value and y value of the corresponding pixel point on each two-dimensional image are the same, the coordinates (x L ,y L ,t L ), x L and L That is, the x value and y value of each pixel in a two-dimensional image in the image set, t L is the theoretical acquisition time. For example, Figure 2 As shown, the coordinates of each pixel point on the first two-dimensional image are (1,1), (1,2), (1,3), (1,4),

[0106] (1,5), (2,1), (2,2), (2,3), (2,4), (2,5), (3,1), (3,2),

[0107] (3,3), (3,4), (3,5), (4,1), (4,2), (4,3), (4,4), (4,5),

[0108] (5,1), (5,2), (5,3), (5,4), (5,5). Accordingly, Figure 2 In the image set shown, the coordinates of each pixel point on the second to eighth two-dimensional images are also (1,1), (1,2), (1,3), (1,4), (1,5), (2,1), (2,2), (2,3), (2,4),

[0109] (2,5), (3,1), (3,2), (3,3), (3,4), (3,5), (4,1), (4,2),

[0110] (4,3), (4,4), (4,5), (5,1), (5,2), (5,3), (5,4), (5,5). The acquisition time corresponding to each pixel on each two-dimensional image and each pixel on the same two-dimensional image is different. Based on this, the theoretical pixel in the image corresponding to the theoretical acquisition time is: (1,1,z L ), (1,2,z L ), (1,3,z L ), (1,4,z L ), (1,5,z L ), (2,1,z L ), (2,2,z L ), (2,3,z L ), (2,4,zL ), (2,5,z L ),

[0111] (3,1,z L ), (3,2,z L ), (3,3,z L ), (3,4,z L ), (3,5,z L ),(4,1,z L ), (4,2,z L ), (4,3,z L ), (4,4,z L ),(4,5,z L ),(5,1,z L ),

[0112] (5,2,z L ), (5,3,z L ), (5,4,z L ),(5,5,z L ).

[0113] Among them, the multiple target pixel points corresponding to the position of the theoretical pixel point refer to the pixel points with the same x value and y value as the theoretical pixel point. For example, for Figure 2 In the 8 two-dimensional images shown, the theoretical pixel point (2,2,t L ) are the pixel points at the (2,2) position on each two-dimensional image.

[0114] It is worth noting that during the acquisition process of each pixel point, the acquisition time corresponding to the pixel point will be recorded, so the acquisition time corresponding to multiple target pixel points can be directly obtained.

[0115] The two second target pixels adjacent to the theoretical acquisition time refer to the pixels with the same x and y values ​​as the theoretical pixel, but the acquisition time is adjacent to the theoretical acquisition time. The acquisition time is adjacent to the theoretical acquisition time only for the pixels with the same x and y values ​​as the theoretical pixel. For example, for the theoretical pixel position (2,2) and the theoretical acquisition time t L , the pixel points with x=2, y=2 on each two-dimensional image are (2,2, t5), (2,2, t6), (2,2, t7), (2,2, t8), where t L If it is between t5 and t6, the second target pixel points adjacent to the theoretical acquisition time of the theoretical pixel point are determined to be (2, 2, t5) and (2, 2, t6).

[0116] Through the above process, it is achieved to search for the first target pixel point that coincides with the theoretical acquisition time or the two second target pixel points adjacent to the theoretical acquisition time for each theoretical pixel point. Since the first target pixel point or the two second target pixel points adjacent to the theoretical acquisition time actually contain the most information about the theoretical pixel point, by searching the first target pixel point or the two second target pixel points adjacent to the theoretical acquisition time, the subsequent calculation of the pixel value of the theoretical pixel point is more accurate.

[0117] Specifically, for the above-mentioned search for one or two target images adjacent to the theoretical acquisition time in the pixel dimension, the specific implementation method of the above-mentioned step S303 can be: for each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following Figure 4 The operation steps S401 to S403 are shown as follows:

[0118] Step S401: in the case where a first target pixel point that coincides with the theoretical acquisition time is determined, the pixel value of the first target pixel point is used as the pixel value of the theoretical pixel point;

[0119] Step S402: for the case where two second target pixels adjacent to the theoretical acquisition time are determined, the pixel value of the theoretical pixel is calculated using the pixel values ​​of the two second target pixels, the theoretical acquisition time, the acquisition time of any second target pixel, and the acquisition duration of a single image;

[0120] This step can be calculated using the following formula (2):

[0121]

[0122] Among them, P (i,j,tL) Indicates the calculated value corresponding to the theoretical acquisition time t L The pixel value of the pixel at the position corresponding to the i-th row and j-th column of (i,j,n) Represents the same as (i,j,t L ) a pixel value of a second target pixel point adjacent to it; t (i,j,n) Represents the same as (i,j,t L ) The acquisition time of a second target pixel adjacent to the target pixel; ScanOneFrameTime * Indicates the time taken for a single two-dimensional image; P (i,j,n+1) Represents the same as (i,j,t L ) is the pixel value of another second target pixel point adjacent to it; t (i,j,n+1) Represents the same as (i,j,t L ) is the acquisition time of another adjacent second target pixel point.

[0123] Step S403: combining the calculated pixel values ​​of each theoretical pixel point to obtain an image corresponding to the theoretical acquisition time.

[0124] The pixel values ​​of each theoretical pixel point calculated in combination refer to the pixel values ​​corresponding to the theoretical pixel points configured on the image corresponding to the x value and the y value according to the two-dimensional coordinates of each theoretical pixel point on the image, namely the x value and the y value.

[0125] A new image is generated through the above process, and the new image can correct the distortion of the pixels in the image set.

[0126] Furthermore, with respect to the image dimension, a specific implementation method of searching for one or two target images adjacent to the theoretical acquisition time and using pixel points on the one or two target images found to generate an image corresponding to the theoretical acquisition time may include: determining the position of the theoretical acquisition time in the image set according to the acquisition time corresponding to the pixel points of the image; for the case where the theoretical acquisition time is on the first target image, using pixel points of the first target image and pixel points of the second target image adjacent to the first target image to generate an image corresponding to the theoretical acquisition time; for the case where the theoretical acquisition time is between two adjacent third target images, using pixel points of the two third target images to generate an image corresponding to the theoretical acquisition time.

[0127] Since the pixels in each two-dimensional image in the image set have corresponding acquisition time, the two-dimensional images are sorted according to the acquisition time to obtain the following Figure 2In the order shown, the corresponding positions of each two-dimensional image on the z axis are different, and each value on the z axis corresponds to a time point. The position in the theoretical acquisition time image set can be determined by the z value corresponding to the theoretical acquisition time, or the position in the theoretical acquisition time image set can be determined directly by the theoretical acquisition time and the acquisition time of each two-dimensional image in the image set. For example, the acquisition time of each two-dimensional image is (t1, t2), (t3, t4), (t5, t6), etc., wherein the theoretical acquisition time can be between the acquisition time of two two-dimensional images, or can be within the acquisition time of one two-dimensional image, for example, the theoretical acquisition time is within (t3, t4) or the theoretical acquisition time is between (t3, t4) and (t5, t6). In the case where the theoretical acquisition time is within (t3, t4), the two-dimensional image corresponding to (t3, t4) is the first target image, and the two-dimensional image corresponding to (t1, t2) and the two-dimensional image corresponding to (t5, t6) are the second target image. For the theoretical acquisition time between (t3, t4) and (t5, t6), the two-dimensional image corresponding to (t3, t4) and the two-dimensional image corresponding to (t5, t6) are the third target image. Therefore, the first target image, the second target image, and the third target image involved in the embodiment of the present invention are to distinguish different determination methods of the target image, and they are not a limitation on the number of target images or the order of the target images.

[0128] Through the above process, it is possible to find an image adjacent to the theoretical acquisition time from the image dimension, and generate an image corresponding to the theoretical acquisition time using the pixel points on the adjacent image.

[0129] It is worth noting that, for the case where the theoretical acquisition time is located on the first target image, that is, there is just a pixel on the first target image whose acquisition time is equal to the theoretical acquisition time. The above-mentioned second target image adjacent to the first target image can be, after the image set is sorted according to the acquisition time, the second target image can be a two-dimensional image before the first target image and adjacent to the first target image, or a two-dimensional image after the first target image and adjacent to the first target image. Preferably, the coordinate position of the pixel point on the first target image corresponding to the theoretical acquisition time is determined, and for the pixel point collected before the pixel point on the first target image corresponding to the theoretical acquisition time, a two-dimensional image before the first target image and adjacent to the first target image is determined as the second target image; for the pixel point collected after the pixel point on the first target image corresponding to the theoretical acquisition time, a two-dimensional image after the first target image and adjacent to the first target image is determined as the second target image. For example, for a 5×5 two-dimensional image, the theoretical acquisition time corresponds to the position of the pixel point on the first target image is (2,2), then the pixels collected before this pixel point are (1,1), (1,2), (1,3), (1,4), (1,5), (2,1), and for these positions, the second target image is the two-dimensional image that is before and adjacent to the first target image; the theoretical acquisition time corresponds to the position of the pixel point on the first target image is (2,2), then the pixels collected after this pixel point are (2,3), (2,4), (2,5), (3,1), ..., (5,5), and for these positions, the second target image is the two-dimensional image that is after and adjacent to the first target image.

[0130] In a preferred embodiment, for determining adjacent target images in the image dimension, the specific implementation method of determining the position of the theoretical acquisition time in the image set may include: determining the position of the theoretical acquisition time in the image set according to the acquisition time of a single image, the theoretical acquisition time, and the acquisition order of multiple images. For example, if the acquisition time of an image is T1, then the result obtained by dividing the theoretical acquisition time by T1 and rounding up is the acquisition time of the image or between which two images the theoretical acquisition time is.

[0131] For the case where the theoretical acquisition time is between two adjacent third target images, the specific implementation of using the pixels of the two third target images to generate an image corresponding to the theoretical acquisition time may include:

[0132] like Figure 5 As shown, for each theoretical pixel point in the image corresponding to the theoretical acquisition time, the following operations from step S501 to step S503 are performed:

[0133] Step S501: searching for a third target pixel and a fourth target pixel corresponding to the position of a theoretical pixel in two third target images respectively, and determining a second acquisition time of the third target pixel;

[0134] For example, taking the theoretical pixel position (1,1) as an example, the corresponding third target pixel and fourth target pixel are the pixel corresponding to (1,1) in one third target image and the pixel corresponding to (1,1) in another third target image. For another example, taking the theoretical pixel position (2,3) as an example, the corresponding third target pixel and fourth target pixel are the pixel corresponding to (2,3) in one third target image and the pixel corresponding to (2,3) in another third target image.

[0135] The second acquisition time of the third target pixel point may be before the theoretical acquisition time or after the theoretical acquisition time, which is not limited here.

[0136] Step S502: Calculate the pixel value of the theoretical pixel using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition time of the single image;

[0137] This step can also be calculated using the following calculation formula (3).

[0138]

[0139] Among them, P (i,j,tL) Indicates the calculated value corresponding to the theoretical acquisition time t L The pixel value of the pixel at the position corresponding to the i-th row and j-th column of (i,j,m) Represents a third target image with (i,j,t L ) the pixel value of the third target pixel point adjacent to it; t (i,j,m) Represents the same as (i,j,t L ) The acquisition time of the third adjacent target pixel point; ScanOneFrameTime * Indicates the time taken for a single two-dimensional image; P (i,j,m+1) Indicates that the image with (i,j,t L )The pixel value of the fourth target pixel point adjacent to it.

[0140] In addition, this step can also be calculated using the above calculation formula (2), and it is only necessary to replace the pixel values ​​of the two second target pixel points in the above second calculation formula with the pixel values ​​of the third target pixel point and the pixel values ​​of the fourth target pixel point, and replace the acquisition time of the two second target pixel points in the above second calculation formula with the acquisition time corresponding to the third target pixel point and the acquisition time corresponding to the fourth target pixel point.

[0141] Step S503: combining the calculated pixel values ​​of the theoretical pixel points to obtain an image corresponding to the theoretical acquisition time.

[0142] Furthermore, for the case where the theoretical acquisition time is located on the first target image, a specific implementation method of generating an image corresponding to the theoretical acquisition time by using pixel points of the first target image and pixel points of a second target image adjacent to the first target image may be:

[0143] like Figure 6 As shown, for each theoretical pixel point in the image corresponding to the theoretical acquisition time, the following operation steps S601 to S604 are performed:

[0144] Step S601: searching for a fifth target pixel and a sixth target pixel corresponding to the position of the theoretical pixel in the first target image and the second target image respectively, and determining a fifth acquisition time corresponding to the fifth target pixel; if the fifth acquisition time is consistent with the theoretical acquisition time, executing step S602; if the fifth acquisition time is inconsistent with the theoretical acquisition time, executing step S603;

[0145] As described above, it can be understood that the corresponding positions of two pixels means that the x values ​​and y values ​​of the two pixels are equal, that is, the positions of the two pixels on their respective two-dimensional images are the same, such as (1,1) or (1,2).

[0146] Among them, for the case where the theoretical acquisition time is on the first target image, the following operations are performed for each theoretical pixel: when the fifth acquisition time of the fifth target pixel is before the theoretical acquisition time, the next image of the first target image is determined to be the second target image according to the acquisition order of multiple images in the image set; when the fifth acquisition time of the fifth target pixel is after the theoretical acquisition time, the previous image of the first target image is determined to be the second target image according to the acquisition order of multiple images in the image set.

[0147] Step S602: directly determine the pixel value of the fifth target pixel point as the pixel value of the theoretical pixel point, and execute step S604;

[0148] Step S603: Calculate the pixel value of the theoretical pixel using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition time of the single image;

[0149] The calculation process of this step can be completed by using the above calculation formula (2) or calculation formula (3). In the process of using the above calculation formula (2), it is only necessary to replace the pixel values ​​of the two second target pixels in the above calculation formula (2) with the pixel values ​​of the fifth target pixel and the pixel values ​​of the sixth target pixel, and replace the acquisition time of the two second target pixels in the above calculation formula (2) with the acquisition time corresponding to the fifth target pixel and the acquisition time corresponding to the sixth target pixel. In the process of using the above calculation formula (3), it is only necessary to replace the pixel value of the third target pixel and the pixel value of the fourth target pixel in the above calculation formula (3) with the pixel value of the fifth target pixel and the pixel value of the sixth target pixel, and replace the acquisition time of the third target pixel in the above calculation formula (3) with the acquisition time corresponding to the fifth target pixel.

[0150] Step S604: combining the calculated pixel values ​​of each theoretical pixel point to obtain an image corresponding to the theoretical acquisition time.

[0151] Furthermore, in order to improve the availability of the theoretical acquisition time, the transmission electron microscopy image processing method may further include: receiving one or more sample stage rotation angles inputted from the outside; calculating the rotation time corresponding to the sample stage rotation angle using the sample rod rotation rate set by the TEM; and setting the theoretical acquisition time corresponding to the sample stage rotation angle using the start time of the sample rod and the rotation time corresponding to the sample stage rotation angle. The theoretical acquisition time is made to be within the time range of the TEM scanning image, so as to further improve the availability and authenticity of the images generated by the above-mentioned various embodiments, and is conducive to correcting image distortion.

[0152] Furthermore, an embodiment of the present invention also provides an image reconstruction method, which may include: the transmission electron microscopy image processing method provided by any one of the above embodiments.

[0153] The following is a specific example to illustrate the results of image processing by the technical solution provided by the embodiment of the present invention. Specifically, for the sample (a 28nm process smartphone processing chip with three layers of about 300nm thickness remaining after thinning), the sample rod is continuously tilted in the STEM mode of the TEM, and the exposure time of each pixel is 2μs. A 1024×1024 two-dimensional image is collected as shown in the figure. Figure 7 As shown, from Figure 7 It can be clearly seen that the image has obvious angle distortion and image distortion. Figure 7 After the image shown in Figure 8 The two-dimensional image shown is from Figure 8It can be seen that, through the technical solution provided by the embodiment of the present invention, not only the angular distortion of the pixels of each picture is eliminated, but also the distortion and deformation of the image at a macroscopic level is corrected to a correct shape.

[0154] Further, Fig. 9 FIG. 1 is a schematic diagram of the structure of a transmission electron microscopy image processing device provided by an embodiment of the present invention. Fig. 9 As shown, the transmission electron microscopy image processing device 900 may include: a setting unit 901, a search unit 902 and an image generation unit 903, wherein:

[0155] A setting unit 901 is used to set one or more theoretical acquisition times within an acquisition period corresponding to an image set, wherein the image set includes multiple images acquired at different times;

[0156] A search unit 902 is used to search for one or two target images adjacent to a theoretical acquisition time according to the acquisition time corresponding to the pixel points of the image;

[0157] The image generation unit 903 is used to generate an image corresponding to the theoretical acquisition time by using the pixel points on the one or two target images found.

[0158] Furthermore, the search unit 902 is further used to perform the following operations for each theoretical pixel point in the image corresponding to the theoretical acquisition time: determine a plurality of target pixel points corresponding to the position of the theoretical pixel point from the plurality of images included in the image set; and determine a first target pixel point that coincides with the theoretical acquisition time or two second target pixel points that are adjacent to the theoretical acquisition time using the acquisition times corresponding to the plurality of target pixel points.

[0159] Furthermore, the image generation unit 903 is further used to perform the following operations for each theoretical pixel point in the image corresponding to the theoretical acquisition time: in the case of determining a first target pixel point that coincides with the theoretical acquisition time, using the pixel value of the first target pixel point as the pixel value of the theoretical pixel point; in the case of determining two second target pixel points adjacent to the theoretical acquisition time, using the pixel values ​​of the two second target pixel points, the theoretical acquisition time, the acquisition time of any second target pixel point and the acquisition time of a single image to calculate the pixel value of the theoretical pixel point; combining the calculated pixel values ​​of each theoretical pixel point to obtain an image corresponding to the theoretical acquisition time.

[0160] Furthermore, the search unit 902 is further used to determine the position of the theoretical acquisition time in the image set according to the acquisition time corresponding to the pixel points of the image; for the case where the theoretical acquisition time is located on the first target image, determine the second target image adjacent to the first target image; for the case where the theoretical acquisition time is located between two adjacent third target images, determine two third target images adjacent to the first target image.

[0161] Furthermore, the image generation unit 903 is further used to generate an image corresponding to the theoretical acquisition time by using pixel points of the first target image and pixel points of the second target image adjacent to the first target image when the theoretical acquisition time is on the first target image; and to generate an image corresponding to the theoretical acquisition time by using pixel points of the two third target images when the theoretical acquisition time is between two adjacent third target images.

[0162] Furthermore, the searching unit 902 is further configured to determine the position of the theoretical acquisition time in the image set according to the acquisition duration of a single image, the theoretical acquisition time, and the acquisition sequence of multiple images.

[0163] Furthermore, the image generation unit 903 is further used to correct the exposure time of the pixel points set by the TEM; according to the corrected exposure time and the number of pixels of the image, the actual time consumption of a single image is calculated, and the actual time consumption is determined as the acquisition time of a single image.

[0164] Furthermore, the image generation unit 903 is further used to perform the following operations for each theoretical pixel point in the image corresponding to the theoretical acquisition time: searching for the third target pixel point and the fourth target pixel point corresponding to the position of the theoretical pixel point in the two third target images respectively, and determining the second acquisition time of the third target pixel point; calculating the pixel value of the theoretical pixel point using the third target pixel point, the fourth target pixel point, the third acquisition time and the acquisition time of a single image; and combining the calculated pixel values ​​of each theoretical pixel point to obtain an image corresponding to the theoretical acquisition time.

[0165] Furthermore, the image generation unit 903 is further used to perform the following operations for each theoretical pixel point in the image corresponding to the theoretical acquisition time: searching for the fifth target pixel point and the sixth target pixel point corresponding to the position of the theoretical pixel point in the first target image and the second target image, respectively, and determining the fifth acquisition time corresponding to the fifth target pixel point; calculating the pixel value of the theoretical pixel point using the fifth target pixel point, the sixth target pixel point, the fifth acquisition time and the acquisition time of a single image; and combining the calculated pixel values ​​of each theoretical pixel point to obtain an image corresponding to the theoretical acquisition time.

[0166] Furthermore, the image generation unit 903 is further used to directly determine the pixel value of the fifth target pixel point as the pixel value of the theoretical pixel point when the fifth acquisition time is consistent with the theoretical acquisition time; and when the fifth acquisition time is inconsistent with the theoretical acquisition time, execute the step of calculating the pixel value of the theoretical pixel point using the fifth target pixel point, the sixth target pixel point, the fifth acquisition time and the acquisition time of a single image.

[0167] Furthermore, the search unit 902 is further used to perform the following operations for each theoretical pixel point in the case where the theoretical acquisition time is on the first target image: when the fifth acquisition time of the fifth target pixel point is before the theoretical acquisition time, determine the next image of the first target image as the second target image according to the acquisition order of multiple images in the image set; when the fifth acquisition time of the fifth target pixel point is after the theoretical acquisition time, determine the previous image of the first target image as the second target image according to the acquisition order of multiple images in the image set.

[0168] Furthermore, the setting unit 901 is further used to receive one or more sample stage rotation angles input from the outside; calculate the rotation time corresponding to the sample stage rotation angle using the sample rod rotation rate set by the TEM; and set a theoretical acquisition time corresponding to the sample stage rotation angle using the start time of the sample rod and the rotation time corresponding to the sample stage rotation angle.

[0169] Furthermore, an embodiment of the present invention also provides an image reconstruction device, which includes the above-mentioned transmission electron microscopy image processing method or transmission electron microscopy image processing device.

[0170] It is worth noting that the transmission electron microscopy image processing device and the image reconstruction device can be installed in the existing TEM software in the form of plug-ins, or can be installed in a terminal device or a server as an application.

[0171] Fig.10 An exemplary system architecture 1000 is shown in which a transmission electron microscopy image processing method or transmission electron microscopy image processing device according to an embodiment of the present invention can be applied.

[0172] like Fig.10As shown, the system architecture 1000 may include a first terminal device 1001 directly connected to the TEM device, a TEM device 1002, a network 1003, and a second terminal device 1004 or a server 1005 for transmission electron microscopy image processing. The network 1003 is used to provide a medium for a communication link between the first terminal device 1001 and the TEM device 1002, between the TEM device 1002 and the second terminal device 1003, between the first terminal device 1001 and the second terminal device 1004, or between the first terminal device 1001 and the server 1005. The network 1003 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0173] The user can use the first terminal device 1001 to interact with the TEM device 1002 through the network 1003 to obtain scanning information of the TEM device 1002 or send control information of the TEM device 1002, such as instructions for controlling the sample rod to rotate at a uniform speed, controlling the camera to enter a video shooting mode, etc. The first terminal device 1001 may be installed with software for controlling the TEM device or for displaying TEM scanning results, etc. (for example only).

[0174] The second terminal device 1004 or the server 1005 is used to obtain video data or a series of image data in the video data from the first terminal device 1001 and the TEM device 1002 respectively, and perform image processing or reconstruction on the system image data, and store and provide the obtained results to the user.

[0175] The first terminal device 1001 and the second terminal device 1004 may be various electronic devices having a display screen and supporting web browsing, including but not limited to a laptop computer and a desktop computer, and the like.

[0176] It should be noted that the transmission electron microscopic image processing method provided in the embodiment of the present invention is generally executed by the second terminal device 1004 or the server 1005, and accordingly, the transmission electron microscopic image processing device is generally disposed in the second terminal device 1004 or the server 1005. In addition, the transmission electron microscopic image processing method can also be performed by the first terminal device 1001.

[0177] In addition, the above-mentioned first terminal device 1001 and the second terminal device 1004 may also belong to the same terminal device. Accordingly, the transmission electron microscopy image processing device is installed in the first terminal device 1001 in the form of a plug-in to realize the processing of a series of transmission electron microscopy image data by the transmission electron microscopy image processing method provided by the embodiment of the present invention.

[0178] It should be understood that Fig.10The number of the second terminal devices or servers in the embodiment is only for illustration. Any number of second terminal devices or servers may be provided according to implementation requirements.

[0179] Reference below Fig.11 , which shows a schematic diagram of the structure of a computer system 1100 of a terminal device or a server suitable for implementing an embodiment of the present invention. Fig.11 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0180] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the system 1100 are also stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0181] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. The drive 910 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0182] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, the above-mentioned functions defined in the system of the present invention are executed.

[0183] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0184] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0185] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a setting unit, a search unit, and an image generation unit. The names of these modules or units do not constitute a limitation on the modules themselves in some cases, for example, the image generation unit may also be described as "a unit or module that generates an image corresponding to a theoretical acquisition time".

[0186] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: setting one or more theoretical acquisition times within the acquisition period corresponding to the image set, wherein the image set includes multiple images acquired at different times; searching for one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points in the image; and generating an image corresponding to the theoretical acquisition time using the pixel points on the one or two target images found.

[0187] According to the technical solution of the embodiment of the present invention, by setting one or more theoretical acquisition times, the position of the theoretical acquisition time in the image set is determined, and then the pixel points on the two images closest to the theoretical acquisition time are used to generate an image corresponding to the theoretical acquisition time, and the pixel points on the two images closest to the theoretical acquisition time are integrated, and based on the theoretical acquisition time, an image in the same plane can be generated, and image distortion is improved, so that the generated image can more realistically reflect the microscopic two-dimensional structure of the material, and the distortion of the image of the automatically rotating sample taken by the TEM in the STEM mode can be adjusted, so that the reconstructed three-dimensional structure can more realistically reflect the microscopic structure of the material. In addition, by generating an image through the above scheme and adding it to the image set, the image distortion caused by the continuous rotation of the sample stage can be reduced, so that the image set can more realistically reflect the sample information, and effectively improve the authenticity and accuracy of the subsequent three-dimensional structure reconstructed based on the image set.

[0188] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A transmission electron microscopy image processing method, characterized in that: A collection of images acquired in STEM mode for TEM, including: Within the acquisition period corresponding to the image set, one or more theoretical acquisition times are set, wherein the image set includes a plurality of images acquired at different times; According to the acquisition time corresponding to the pixel points in the image, searching for one or two target images adjacent to the theoretical acquisition time; An image corresponding to the theoretical acquisition time is generated by using the found pixel points on the one or two target images.

2. The transmission electron microscopy image processing method according to claim 1, characterized in that: The step of searching for one or two target images adjacent to the theoretical acquisition time includes: For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations: Determining a plurality of target pixel points corresponding to the positions of the theoretical pixel points from a plurality of images included in the image set; The acquisition times corresponding to the plurality of target pixels are used to determine a first target pixel that coincides with the theoretical acquisition time or two second target pixels that are adjacent to the theoretical acquisition time.

3. The transmission electron microscopy image processing method according to claim 2, characterized in that: The step of using the found pixel points on one or two target images to generate an image corresponding to the theoretical acquisition time includes: For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations: In the case where a first target pixel point that coincides with the theoretical acquisition time is determined, the pixel value of the first target pixel point is used as the pixel value of the theoretical pixel point; In the case where two second target pixels adjacent to the theoretical acquisition time are determined, the pixel value of the theoretical pixel is calculated using the pixel values ​​of the two second target pixels, the theoretical acquisition time, the acquisition time of any second target pixel, and the acquisition duration of a single image; The calculated pixel values ​​of the theoretical pixel points are combined to obtain an image corresponding to the theoretical acquisition time.

4. The transmission electron microscopy image processing method according to claim 1, characterized in that: The step of searching for one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points of the image and using the pixel points on the one or two target images found to generate an image corresponding to the theoretical acquisition time includes: Determine the position of the theoretical acquisition time in the image set according to the acquisition time corresponding to the pixel point of the image; In the case where the theoretical acquisition time is located on the first target image, an image corresponding to the theoretical acquisition time is generated by using pixel points of the first target image and pixel points of a second target image adjacent to the first target image; In view of the situation that the theoretical acquisition time is between two adjacent third target images, the image corresponding to the theoretical acquisition time is generated by using the pixel points of the two third target images.

5. The transmission electron microscopy image processing method according to claim 4, characterized in that: Determining the position of the theoretical acquisition time in the image set includes: The position of the theoretical acquisition time in the image set is determined according to the acquisition duration of a single image, the theoretical acquisition time, and the acquisition sequence of the plurality of images.

6. The transmission electron microscopy image processing method according to claim 3 or 5, characterized in that: Also includes: Correcting the exposure time of the pixels set by the TEM; The actual time consumed for capturing a single image is calculated according to the corrected exposure time and the number of pixels of the image, and the actual time consumed is determined as the acquisition time length of the single image.

7. The transmission electron microscopy image processing method according to claim 4, characterized in that: The step of using the pixels of the two third target images to generate an image corresponding to the theoretical acquisition time includes: For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations: Searching for a third target pixel point and a fourth target pixel point corresponding to the position of the theoretical pixel point in the two third target images respectively, and determining a second acquisition time of the third target pixel point; Calculate the pixel value of a theoretical pixel by using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition time of a single image; The calculated pixel values ​​of the theoretical pixel points are combined to obtain an image corresponding to the theoretical acquisition time.

8. The transmission electron microscopy image processing method according to claim 4, characterized in that: The step of using pixel points of the first target image and pixel points of a second target image adjacent to the first target image to generate an image corresponding to the theoretical acquisition time includes: For each theoretical pixel point in the image corresponding to the theoretical acquisition time, perform the following operations: Searching for a fifth target pixel point and a sixth target pixel point corresponding to the position of the theoretical pixel point in the first target image and the second target image respectively, and determining a fifth acquisition time corresponding to the fifth target pixel point; Calculate the pixel value of a theoretical pixel by using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition time of a single image; The calculated pixel values ​​of the theoretical pixel points are combined to obtain an image corresponding to the theoretical acquisition time.

9. The transmission electron microscopy image processing method according to claim 8, characterized in that: After determining the fifth acquisition time corresponding to the fifth target pixel point, the method further includes: When the fifth acquisition time is consistent with the theoretical acquisition time, directly determining the pixel value of the fifth target pixel point to be the pixel value of the theoretical pixel point; When the fifth acquisition time is inconsistent with the theoretical acquisition time, the step of calculating the pixel value of the theoretical pixel by using the fifth target pixel, the sixth target pixel, the fifth acquisition time and the acquisition duration of a single image is performed.

10. The transmission electron microscopy image processing method according to claim 8 or 9, characterized in that: Also includes: In the case where the theoretical acquisition time is located on the first target image, the following operations are performed for each theoretical pixel point: In a case where the fifth acquisition time of the fifth target pixel point is before the theoretical acquisition time, determining, according to the acquisition order of the plurality of images in the image set, that the next image of the first target image is the second target image; In the case that the fifth acquisition time of the fifth target pixel point is after the theoretical acquisition time, according to the acquisition order of the multiple images in the image set, the previous image of the first target image is determined to be the second target image.

11. The transmission electron microscopy image processing method according to any one of claims 1 to 5 and 7 to 9, characterized in that: Also includes: receiving one or more sample stage rotation angles inputted from an external source; Calculate the rotation time corresponding to the rotation angle of the sample stage using the sample rod rotation rate set by the TEM; The theoretical collection time corresponding to the rotation angle of the sample stage is set by using the start time of the sample rod and the rotation time corresponding to the rotation angle of the sample stage.

12. An image reconstruction method, characterized in that: include: The transmission electron microscopy image processing method according to any one of claims 1 to 11.

13. A transmission electron microscopy image processing device, characterized in that: include: A setting unit, a searching unit and an image generating unit, wherein: The setting unit is used to set one or more theoretical acquisition times within an acquisition period corresponding to the image set, wherein the image set includes a plurality of images acquired at different times; The search unit is used to search for one or two target images adjacent to the theoretical acquisition time according to the acquisition time corresponding to the pixel points of the image; The image generation unit is used to generate an image corresponding to the theoretical acquisition time by using the found pixel points on one or two target images.

14. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 12.

15. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images