A method and apparatus for transmission electron microscopy image processing and image reconstruction

Abstract

The application discloses a kind of transmission electron microscopy image processing, image reconstruction method and device, it is related to transmission electron microscope technical field.The method is aimed at the image set collected under the STEM mode of TEM, it can include: in the acquisition period corresponding to image set, set one or more theoretical acquisition time, wherein, image set includes multiple different time acquisition images;According to the acquisition time corresponding to the pixel point in image, find one or two target images adjacent to theoretical acquisition time;Using the pixel point on the one or two target images found, generate the image corresponding to theoretical acquisition time.The scheme provided in the embodiment of the application can adjust the distortion of the image of the automatically rotating sample photographed by TEM in the STEM mode, so that the reconstructed three-dimensional structure can more truly reflect the microstructure of the material.

Description

Technical Field

[0001] This invention relates to the field of transmission electron microscopy, and more particularly to a method and apparatus for transmission electron microscopy image processing and reconstruction. Background Technology

[0002] In the scanning transmission electron microscopy (STEM) mode of a transmission electron microscope (TEM), the electron microscope and camera are linked by an automatic control program. During the continuous rotation of the sample stage, the camera can acquire a series of two-dimensional transmission electron microscopic images (also known as TEM images) of the sample on the sample stage in video recording mode. Then, the three-dimensional structure of the sample (such as electronic devices, materials, etc.) can be reconstructed by combining the series of TEM images.

[0003] Because the imaging principle of STEM mode is to converge the beam to scan the sample surface point by point in a grid pattern, for example, if a 12×12 pixel image needs to be collected, the converged beam needs to sequentially scan the position of each of the 144 pixels. Moreover, the converged beam needs to stay at the position of a pixel for a certain period of time. During the STEM mode scanning imaging process, the sample stage rotates continuously, so that the rotation angle of the sample stage corresponding to each pixel in the same image is different. That is, the pixels in the same image are not collected under the same rotation angle of the sample stage, resulting in different degrees of distortion in the collected TEM image and the subsequently reconstructed 3D image. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for transmission electron microscopy image processing and image reconstruction. The solution provided by the embodiments of the present invention can adjust the distortion of automatically rotated sample images captured by TEM in STEM mode, so that the reconstructed three-dimensional structure can more realistically reflect the microstructure of the material.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a transmission electron microscopy (TEM) image processing method is provided, for an image set acquired in STEM mode of TEM, comprising:

[0006] Within the acquisition period corresponding to the image set, one or more theoretical acquisition times are set, wherein the image set includes multiple images acquired at different times;

[0007] Based on the acquisition time corresponding to the pixels in the image, find one or two target images adjacent to the theoretical acquisition time;

[0008] Using the pixels from one or two of the target images found, an image corresponding to the theoretical acquisition time is generated.

[0009] Optionally, finding one or two target images adjacent to the theoretical acquisition time includes:

[0010] For each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following operation:

[0011] From the multiple images included in the image set, determine multiple target pixels corresponding to the positions of the theoretical pixels;

[0012] By utilizing the acquisition times corresponding to multiple target pixels, a first target pixel that coincides with the theoretical acquisition time or two second target pixels that are adjacent to the theoretical acquisition time are determined.

[0013] Optionally, generating an image corresponding to the theoretical acquisition time using pixels from one or two of the found target images includes:

[0014] For each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following operation:

[0015] In the case where a first target pixel is identified that coincides with the theoretical acquisition time, the pixel value of the first target pixel is used as the pixel value of the theoretical pixel.

[0016] In the case where two second target pixels are identified that are adjacent to the theoretical acquisition time, the pixel value of the theoretical pixel is calculated using the pixel values ​​of the two second target pixels, the theoretical acquisition time, the acquisition time of any second target pixel, and the acquisition duration of a single image.

[0017] The pixel values ​​of each theoretical pixel point are combined and calculated to obtain an image corresponding to the theoretical acquisition time.

[0018] Optionally, the step of finding one or two target images adjacent to the theoretical acquisition time based on the acquisition time corresponding to the pixels of the image, and generating an image corresponding to the theoretical acquisition time using the pixels on the found one or two target images, includes:

[0019] Based on the acquisition time corresponding to each pixel in the image, determine the position of the theoretical acquisition time in the image set;

[0020] In the case where the theoretical acquisition time is located on the first target image, an image corresponding to the theoretical acquisition time is generated using the pixels of the first target image and the pixels of the second target image adjacent to the first target image.

[0021] In cases where the theoretical acquisition time falls between two adjacent third target images, an image corresponding to the theoretical acquisition time is generated using the pixels of the two third target images.

[0022] Optionally, determining the position of the theoretical acquisition time within the image set includes:

[0023] The position of the theoretical acquisition time in the image set is determined based on the acquisition duration of a single image, the theoretical acquisition time, and the acquisition order of multiple images.

[0024] Optionally, the above transmission electron microscopy image processing method may further include:

[0025] Correct the exposure time of the pixels set in the TEM;

[0026] Based on the corrected exposure time and the number of pixels in the image, the actual time taken to capture a single image is calculated, and the actual time taken is determined as the acquisition time of a single image.

[0027] Optionally, generating an image corresponding to the theoretical acquisition time using the pixels of the two third target images includes:

[0028] For each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following operation:

[0029] In the two third target images, respectively, find the third target pixel and the fourth target pixel that correspond to the position of the theoretical pixel, and determine the second acquisition time of the third target pixel;

[0030] The pixel value of the theoretical pixel is calculated using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition duration of a single image.

[0031] The pixel values ​​of each theoretical pixel point are combined and calculated to obtain an image corresponding to the theoretical acquisition time.

[0032] Optionally, generating an image corresponding to the theoretical acquisition time using pixels from the first target image and pixels from a second target image adjacent to the first target image includes:

[0033] For each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following operation:

[0034] In the first target image and the second target image, respectively, find the fifth target pixel and the sixth target pixel corresponding to the position of the theoretical pixel, and determine the fifth acquisition time corresponding to the fifth target pixel;

[0035] The pixel value of the theoretical pixel is calculated using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image.

[0036] The pixel values ​​of each theoretical pixel point are combined and calculated to obtain an image corresponding to the theoretical acquisition time.

[0037] Optionally, after determining the fifth acquisition time corresponding to the fifth target pixel, the method further includes:

[0038] If the fifth acquisition time is consistent with the theoretical acquisition time, the pixel value of the fifth target pixel is directly determined to be the pixel value of the theoretical pixel.

[0039] If the fifth acquisition time is inconsistent with the theoretical acquisition time, the step of calculating the pixel value of the theoretical pixel using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image is performed.

[0040] Optionally, the above transmission electron microscopy image processing method further includes:

[0041] For the case where the theoretical acquisition time is located on the first target image, the following operation is performed for each theoretical pixel:

[0042] If the fifth acquisition time of the fifth target pixel is before the theoretical acquisition time, the next image of the first target image is determined as the second target image according to the acquisition order of multiple images in the image set;

[0043] If the fifth acquisition time of the fifth target pixel is after the theoretical acquisition time, the image preceding the first target image is determined as the second target image according to the acquisition order of multiple images in the image set.

[0044] Optionally, the above transmission electron microscopy image processing method further includes:

[0045] Receives one or more sample stage rotation angles from external input;

[0046] Using the sample rod rotation rate set by the TEM, the rotation time corresponding to the rotation angle of the sample stage is calculated;

[0047] By utilizing the start-up time of the sample rod and the rotation duration corresponding to the rotation angle of the sample stage, a theoretical acquisition time corresponding to the rotation angle of the sample stage is set.

[0048] Furthermore, according to a second aspect of the present invention, an image reconstruction method is provided, comprising: the transmission electron microscopy image processing method provided in the first aspect embodiment above.

[0049] Furthermore, in a third aspect, embodiments of the present invention provide a transmission electron microscope image processing apparatus, comprising: a setting unit, a position finding unit, and an image generation unit, wherein,

[0050] The setting unit is used to set one or more theoretical acquisition times within the acquisition time period corresponding to the image set, wherein the image set includes multiple images acquired at different times;

[0051] The search unit is used to search for one or two target images that are adjacent to the theoretical acquisition time based on the acquisition time corresponding to the pixels of the image.

[0052] The image generation unit is used to generate an image corresponding to the theoretical acquisition time using the pixels on one or two of the target images found.

[0053] Furthermore, in a fourth aspect, embodiments of the present invention provide an electronic device, comprising:

[0054] One or more processors;

[0055] Storage device for storing one or more programs.

[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the first or second aspect embodiments described above.

[0057] Furthermore, in a fifth aspect, embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first or second aspect embodiments described above.

[0058] One embodiment of the above invention has the following advantages or beneficial effects: by setting one or more theoretical acquisition times, the position of the theoretical acquisition time in the image set is determined, and then the pixels on the two images closest to the theoretical acquisition time are used to generate an image corresponding to the theoretical acquisition time. By combining the pixels on the two images closest to the theoretical acquisition time, and based on the theoretical acquisition time, an image on the same plane can be generated, improving image distortion, so that the generated image can more realistically reflect the microscopic two-dimensional structure of the material, and can adjust the distortion of the image of the automatically rotated sample captured by TEM in STEM mode, so that the reconstructed three-dimensional structure can more realistically reflect the microscopic structure of the material.

[0059] In addition, by adding the images generated by the above method to the image set, the image distortion caused by the continuous rotation of the sample stage can be reduced, so that the image set can more realistically reflect the sample information and effectively improve the realism and accuracy of the three-dimensional structure reconstructed based on the image set.

[0060] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0061] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0062] Figure 1 This is an example of a 12×12 pixel two-dimensional image collected by continuous tilting in STEM mode;

[0063] Figure 2 This is an example of a 3D image collected in STEM mode with 5×5 pixels, consisting of a total of 8 images sorted by acquisition time.

[0064] Figure 3 This is a schematic diagram of the main flow of the transmission electron microscopy image processing method according to an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the main process of finding one or two target images adjacent to the theoretical acquisition time in the pixel dimension according to an embodiment of the present invention.

[0066] Figure 5 This is a schematic diagram of the main process for generating an image corresponding to the theoretical acquisition time according to an embodiment of the present invention;

[0067] Figure 6 This is a schematic diagram of the main process for generating an image corresponding to the theoretical acquisition time according to another embodiment of the present invention;

[0068] Figure 7This is a 1024×1024 two-dimensional image of the three remaining layers of a 28nm process smartphone processing chip after thinning, collected through the STEM mode of TEM.

[0069] Figure 8 Yes Figure 7 The two-dimensional image shown is the image processed by the technical solution provided in the embodiments of the present invention;

[0070] Figure 9 This is a schematic diagram of the main units of a transmission electron microscope image processing apparatus according to an embodiment of the present invention;

[0071] Figure 10 This is a schematic diagram of the system structure upon which the embodiments of the present invention depend;

[0072] Figure 11 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0073] Transmission electron microscopy (TEM), with its ultra-high spatial resolution, can clearly characterize the microstructure of materials and has been widely used in chemistry, materials science, and biological sciences. The imaging principle of TEM is as follows: using an electron beam with an extremely short wavelength as the light source, the accelerated and focused electron beam is projected onto a very thin sample. Electrons collide with atoms in the sample, changing their direction and producing solid-angle scattering. The size of the scattering angle is related to factors such as the sample's density and thickness, resulting in images of varying brightness. After magnification and focusing, the image is displayed on an imaging device (fluorescent screen, photocoupler).

[0074] Currently, the two most commonly used imaging modes in transmission electron microscopy are transmission electron microscopy (TEM) and scanning transmission electron microscopy (STEM). The technical solution provided in this invention mainly focuses on the STEM mode. The STEM mode uses a converged electron beam to scan the sample surface point by point in a grid pattern, and uses ring detectors at different angles below the sample to detect and record the images. In simple terms, the converging beam in STEM mode scans point by point and then collects the images, much like a laser beam scanning a wall point by point to create an image. STEM imaging yields two-dimensional images, which are subsequently used to reconstruct the microscopic three-dimensional morphology or structure of the material from multiple two-dimensional images. Electron microtomography (EMT) is the most commonly used technique for observing the complete three-dimensional morphology of nanocrystalline samples. Its advantages lie not only in its extremely high spatial resolution, but more importantly, in the diverse signals that allow for three-dimensional imaging of the sample's chemical composition, crystal orientation, dislocations, defects, etc. EMT, combined with a dedicated tomographic scanning scaffold, typically rotates the sample at a specified step size, generally 1-3°, and images are created at each rotation angle. After obtaining data exceeding a certain rotation angle (generally >120°), a series of images (tilted sequences) are aligned relative to a common origin and tilt axis, and then a relevant reconstruction algorithm is used to obtain the three-dimensional reconstructed shape of the crystal. Generally speaking, the data collection time for tilted sequence images is about 45-60 minutes, which is relatively time-consuming, and it is only applicable to nanomaterial samples that are stable under electron beam.

[0075] To overcome the problems of existing tomography, a technique for collecting 3D reconstruction data using continuous tilting imaging was provided in the early research of this patent (patent application number: 202410703691.5), which is a novel electron microscopy tomography reconstruction imaging technique. This technique utilizes an automatic control program to link the electron microscope and camera, completing continuous rotation while simultaneously recording scanned image data at a high frame rate. Continuous tilting imaging data collection enables the acquisition of complete image data in a short time, typically requiring only five to six minutes. For example, for collecting a conventional 1024×1024 STEM image with a 2μs dwell time per pixel using continuous tilting imaging, the entire image scan takes only 4 seconds, significantly reducing electron dose consumption and representing a major breakthrough in the field of 3D reconstruction of electron beam-sensitive materials.

[0076] However, collecting continuous tilt-based imaging data in STEM mode has certain limitations. First, the rotation of the sample rod is a continuous process that cannot be stopped midway. Second, the imaging principle of STEM mode is to converge the beam and scan the sample surface point by point in a grid pattern. Figure 1 This is an image with 12×12 pixels collected in STEM mode, with a dwell time of 500 ns. Assuming the coordinates of the first pixel are (1,1), the image collection process involves a convergent beam scanning image sequentially from coordinates (1,1)→(1,2)→…→(1,12)→(2,1)→…→(2,12)→…→(12,12) pixel by pixel (i.e., the convergent beam scans the sample pixel by pixel from left to right and top to bottom), with a dwell time of 500 ns for each pixel. Figure 1 The images shown demonstrate that continuous tilting imaging still has certain limitations. The image quality of a single 2D image is relatively blurry, making it difficult to identify the shape of the sample in the 2D image. This is because the angle corresponding to each pixel changes during continuous tilting scanning, meaning the acquisition position of each pixel changes, leading to distortion in the 2D image composed of the acquired pixels. For example, for a 1024×1024 STEM image acquired through continuous tilting with a dwell time of 2μs, the sample rod rotates at a uniform speed continuously. Assuming the sample starts rotating from 0°, when the beam is at the first pixel (1,1), the sample is at 0°; when the beam is at the last pixel (1024,1024), 4 seconds have passed. The rotation angle of the sample rod is calculated as: 0° + 0.3° × 4 = 1.2°. In other words, the first and last pixels in an image acquired using continuous tilting differ by 1.2°. Similarly, each pixel in an image corresponds to a time period, and each time period corresponds to an angle. Therefore, for an image collected using continuous tilting, each pixel corresponds to a different angle. This results in angular distortion in the image data, and no single angle can fully represent the image. Therefore, we need to perform angular correction on the images to ensure that each image represents the same time and angle. Images obtained in this way cannot be used for 3D reconstruction.

[0077] Furthermore, when collecting continuously tilted image data using STEM mode, the rotation of the sample rod is a continuous process that does not stop midway. Due to angular distortion, each angle represents a time interval, meaning there is also temporal distortion. Combined with the drift caused by rotation, the image will be distorted. Therefore, to obtain a clear two-dimensional image or three-dimensional structure of the sample, appropriate algorithms must be used to correct these distortions and improve the image distortion problem.

[0078] The terms "first," "second," "third," "fourth," and "fifth," etc., used in the terminology of this invention are used to distinguish similar objects and are not necessarily used to describe a specific number or order. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this invention.

[0079] The image set acquired in the STEM mode in the embodiments of the present invention refers to a set of multiple two-dimensional images acquired by continuously tilting the sample rod in the STEM mode of TEM.

[0080] The theoretical acquisition time in this embodiment of the invention refers to any value within the time interval consumed to obtain the image set. For example, the time interval consumed by the image set is [t1, t2]. n ], where t1 is the time to acquire the first pixel of the first image; t n The time taken to capture the last pixel of the last image is given; therefore, the theoretical acquisition time is generally (t1, t2). n Any one of the values ​​in ), preferably, the theoretical acquisition time is [t1+m×T, t n -(m-1)×T], where m represents the number of pixels in a two-dimensional image, and T represents the time spent acquiring one pixel; then, t1+m×T is the time taken to acquire the last pixel of the first image, t n -(m-1)×T is the time corresponding to the first pixel in the last image.

[0081] Understandably, by sorting multiple two-dimensional images in the image set according to their acquisition time, each theoretical acquisition time corresponds to a two-dimensional region at a fixed height along the sorting direction. Accordingly, the generated image corresponding to the theoretical acquisition time is an image within that two-dimensional region at that fixed height. Figure 2Taking a dataset of 8 images (5×5 pixels) collected in STEM mode as an example, a three-dimensional xyz coordinate system is established. The coordinates of the first pixel in the first image are (1,1,1), meaning the coordinates of the pixel in the i-th row and j-th column of the n-th image are (i,j,n). The scanning process of the beam is (1,1,1)→(1,2,1)→…→(1,5,1)→(2,1,1)→…→(5,5,1)→…→(1,1,2)→(1,2,2)→…→(5,5,8), with each pixel corresponding to a time interval. Simultaneously, since the sample rod rotates at a uniform speed continuously, each time interval corresponds to an angle. Therefore, each pixel in the continuous tilting imaging data collected in STEM mode corresponds to an angle and a time interval. Because the sample rod rotates continuously during image data acquisition, there are slight differences in the height of each pixel in the same 2D image along the z-axis. (It is precisely because of these slight differences that the individual pixels in the constructed 2D image do not actually represent the image data acquired by the sample rod at a single angle. Ignoring the pixel influence caused by this slight difference leads to image data distortion.) In the xyz coordinate system, 2D images are stacked sequentially along the z-axis in chronological order. Therefore, one time point corresponds to one height on the z-axis, such as... Figure 2 The time t1 corresponds to height z1, time t2 corresponds to height z2, ..., time t n Corresponding height z n If time t1 is taken as the theoretical acquisition time, the set of pixels at height z1 corresponding to time t1 can be obtained through the scheme provided in the embodiments of the present invention. If time t2 is taken as the theoretical acquisition time, the set of pixels at height z2 corresponding to time t2 can be obtained through the scheme provided in the embodiments of the present invention, and so on. Each pixel in the two-dimensional image corresponding to each theoretical acquisition time is obtained based on the pixels of each two-dimensional image in the image set. The technical solution provided in the embodiments of the present invention will be described in detail below.

[0082] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0083] The transmission electron microscopy (TEM) image processing method provided in this embodiment of the invention is for image sets acquired in STEM mode of TEM. Preferably, the two-dimensional images in the image set are acquired by continuously rotating the sample rod. It can be understood that the two-dimensional images in this image set can also be obtained using existing methods that acquire one two-dimensional image per rotation of the sample rod. Specifically, as... Figure 3 As shown, the transmission electron microscopy image processing method may include the following steps:

[0084] Step S301: Within the acquisition time period corresponding to the image set, set one or more theoretical acquisition times, wherein the image set includes multiple images acquired at different times;

[0085] Step S302: Based on the acquisition time corresponding to the pixels in the image, find one or two target images adjacent to the theoretical acquisition time;

[0086] Step S303: Using the pixels on one or two target images found, generate an image corresponding to the theoretical acquisition time.

[0087] As mentioned above, the image set is mainly a combination of two-dimensional images obtained by acquiring image data from electron microtomography using continuous tilt imaging. Subsequently, three-dimensional reconstruction can be performed on the images obtained by this technical solution, or the images obtained by this technical solution can be combined with existing images in the image set for three-dimensional reconstruction.

[0088] The program for performing the aforementioned transmission electron microscopy (TEM) image processing can be embedded in the operating software of the TEM. Therefore, upon starting the operating software of the TEM, the TEM image processing program can be started simultaneously to execute steps S301 to S303. Alternatively, the TEM image processing program can also be used as a standalone software application on a terminal device or server to process images acquired by the TEM in STEM mode, providing a data foundation for subsequent 3D reconstruction. Furthermore, the TEM image processing program can also be part of or embedded within a 3D reconstruction program, allowing the 3D reconstruction software to perform the image processing steps S301 to S303 on the images acquired by the TEM in STEM mode, and then perform 3D reconstruction using the processed image result or by combining the processed result with the original image set to obtain the 3D microstructure or 3D morphology of the sample.

[0089] Specifically, the above-mentioned transmission electron microscopy image processing can be achieved by operating the setting buttons or by clicking the virtual operation controls provided on the interface.

[0090] Understandably, the multiple images acquired at different times in the image set provided in this embodiment of the invention are multiple two-dimensional images acquired for the same sample within a continuous acquisition time. The pixels contained in these multiple two-dimensional images correspond to each other.

[0091] For cases where each two-dimensional image in the image set was acquired while the sample rod was continuously tilted, such as... Figure 2As shown, after sorting the multiple images acquired at different times in this image set according to their acquisition time, these images acquired at different times correspond to different heights on the z-axis. Based on this, one or two target images adjacent to the theoretical acquisition time generally refer to one or two images adjacent to the z-value corresponding to that theoretical acquisition time on the z-axis. Specifically, if the acquisition time of a pixel is exactly the same as the theoretical acquisition time, then the image containing that pixel is the target image adjacent to that theoretical acquisition time.

[0092] The technical solution provided by this invention sets one or more theoretical acquisition times, determines the position of the theoretical acquisition time in the image set, and then uses the pixels on the two images closest to the theoretical acquisition time to generate an image corresponding to the theoretical acquisition time. By combining the pixels on the two images closest to the theoretical acquisition time, and based on the theoretical acquisition time, an image on the same plane can be generated, improving image distortion and enabling the generated image to more realistically reflect the microscopic two-dimensional structure of the material. It can adjust the distortion of images of automatically rotated samples captured by TEM in STEM mode, so that the reconstructed three-dimensional structure can more realistically reflect the microscopic structure of the material.

[0093] In addition, by adding the images generated by the above method to the image set, the image distortion caused by the continuous rotation of the sample stage can be reduced, so that the image set can more realistically reflect the sample information and effectively improve the realism and accuracy of the three-dimensional structure reconstructed based on the image set.

[0094] Furthermore, the above-mentioned method of finding one or two target images adjacent to the theoretical acquisition time and generating an image corresponding to the theoretical acquisition time using the pixels on the found one or two target images involves pixel acquisition time or image acquisition time. Through multiple experiments, it was found that the time consumed by the light beam on an image includes not only the dwell time at each pixel but also the time consumed by the light beam moving between pixels. Therefore, to further improve the accuracy of finding adjacent target pixels or target images for the theoretical acquisition time, the exposure time of the camera in this embodiment of the invention is corrected. This correction can be achieved using existing technical means. For example, for continuously acquiring 1024×1024 two-dimensional images by continuously rotating the sample rod at a constant speed, the theoretical time, actual time, and corrected pixel exposure time (Dwell Time*) of each single image obtained through multiple implementations are shown in Table 1 below.

[0095] Table 1

[0096] 1024,500ns 0.52s 0.78s 740ns 1024,1μs 1.05s 2.04s 1.95μs 1024,2μs 2.10s 4.04s 3.85μs 1024.5μs 5.24s 6.87s 6.55μs

[0097] The actual time taken to capture a single 2D image is ScanOneFrameTime*=DwellTime*×x×y; based on this, the acquisition time corresponding to the pixel in the i-th row and j-th column of the n-th 2D image in the image set can also be calculated. Specifically, it can be calculated using the following formula (1):

[0098] t (i,j,n) = (n-1)×ScanOneFrameTime * +[y(i-1)+(j-1)]×DwellTime * (1)

[0099] Where n represents the nth two-dimensional image in the image set, t (i,j,n) ScanOneFrameTime represents the acquisition time corresponding to the pixel in the i-th row and j-th column of the n-th 2D image. * This represents the time taken to capture a single 2D image after exposure time correction; DwellTime* represents the corrected exposure time; y represents the number of pixels in each row of a single 2D image.

[0100] The technical solution provided in this embodiment of the invention further includes: correcting the exposure time of pixels set in the TEM; calculating the actual time consumed for a single image based on the corrected exposure time and the number of pixels in the image, and determining the actual time consumed as the acquisition duration of a single image. This process further improves the accuracy of determining the target image or target pixel based on the theoretical acquisition time. It should be noted that the following embodiments involving the acquisition time of a single two-dimensional image and the acquisition time of each pixel are all described using the corrected pixel exposure time (Dwell Time*) as an example. The subsequent acquisition time of a single two-dimensional image and the acquisition time of each pixel can also directly use the theoretical exposure time provided by the camera.

[0101] Furthermore, the specific implementation of finding one or two target images adjacent to the theoretical acquisition time can be done from the dimension of pixels or from the dimension of the image.

[0102] Specifically, a specific implementation method for finding one or two target images adjacent to the theoretical acquisition time from the pixel dimension may include:

[0103] For each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following operations: determine multiple target pixels corresponding to the position of the theoretical pixel from the multiple images included in the image set; use the acquisition time corresponding to the multiple target pixels to determine the first target pixel that coincides with the theoretical acquisition time or two second target pixels that are adjacent to the theoretical acquisition time.

[0104] Since the positions of each pixel in a two-dimensional image scanned by a laser beam are relatively fixed, that is, the x and y values ​​of corresponding pixels in each two-dimensional image are the same, the coordinates (x, y) of each theoretical pixel in the image obtained at the theoretical acquisition time provided in this embodiment of the invention are... L ,y L ,t L In ), x L and y L That is, the x and y values ​​of each pixel in a two-dimensional image within a set of images, t L This refers to the theoretical data collection time. For example, such as... Figure 2 As shown, the coordinates of each pixel on the first 2D image are (1,1), (1,2), (1,3), (1,4).

[0105] (1,5), (2,1), (2,2), (2,3), (2,4), (2,5), (3,1), (3,2),

[0106] (3,3), (3,4), (3,5), (4,1), (4,2), (4,3), (4,4), (4,5),

[0107] (5,1), (5,2), (5,3), (5,4), (5,5). Accordingly, Figure 2 In the image set shown, the coordinates of each pixel in the second to eighth two-dimensional images are also (1,1), (1,2), (1,3), (1,4), (1,5), (2,1), (2,2), (2,3), (2,4).

[0108] (2,5), (3,1), (3,2), (3,3), (3,4), (3,5), (4,1), (4,2),

[0109] (4,3), (4,4), (4,5), (5,1), (5,2), (5,3), (5,4), (5,5). The acquisition time for each pixel in each 2D image and for each pixel in the same 2D image is different. Based on this, the theoretical pixels in the image corresponding to the theoretical acquisition time are: (1,1,z... L ), (1,2,z L ), (1,3,z L ), (1,4,z L ), (1,5,z L ), (2,1,z L ), (2,2,z L ), (2,3,z L ), (2,4,z L), (2,5,z L ),

[0110] (3,1,z L ), (3,2,z L ), (3,3,z L ), (3,4,z L ), (3,5,z L ), (4,1,z L ), (4,2,z L ), (4,3,z L ), (4,4,z L ), (4,5,z L ), (5,1,z L ),

[0111] (5,2,z L ), (5,3,z L ), (5,4,z L ), (5,5,z L ).

[0112] Among them, multiple target pixels corresponding to the position of the theoretical pixel refer to pixels with the same x and y values ​​as the theoretical pixel. For example, for... Figure 2 In the eight two-dimensional images shown, the theoretical pixel point (2,2,t) is... L The multiple target pixels corresponding to ) are the pixels at position (2,2) on each two-dimensional image.

[0113] It is worth noting that during the acquisition of each pixel, the acquisition time corresponding to that pixel is recorded. Therefore, the acquisition time corresponding to multiple target pixels can be obtained directly.

[0114] In this context, two second target pixels adjacent to the theoretical acquisition time refer to pixels with the same x and y values ​​as the theoretical pixel, but whose acquisition time is adjacent to the theoretical acquisition time. "Adjacent to the theoretical acquisition time" only applies to pixels with the same x and y values ​​as the theoretical pixel. For example, for a theoretical pixel at position (2,2) with a theoretical acquisition time of t... L The pixels with x=2 and y=2 on each of the two-dimensional images are (2,2,t5), (2,2,t6), (2,2,t7), and (2,2,t8), respectively, where t L If the pixel is located between t5 and t6, then the second target pixel adjacent to the theoretical acquisition time of the theoretical pixel is determined to be (2,2,t5) and (2,2,t6).

[0115] Through the above process, a first target pixel that coincides with the theoretical acquisition time or two second target pixels adjacent to the theoretical acquisition time are found for each theoretical pixel. Since the first target pixel or the two second target pixels adjacent to the theoretical acquisition time actually contain the most information of the theoretical pixel, finding the first target pixel or the two second target pixels adjacent to the theoretical acquisition time makes the subsequent calculation of the pixel value of the theoretical pixel more accurate.

[0116] Specifically, regarding the above-mentioned method of finding one or two target images adjacent to the theoretical acquisition time at the pixel level, the specific implementation of step S303 can be as follows: for each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following... Figure 4 The operation steps S401 to S403 are shown below:

[0117] Step S401: For the case where the first target pixel is determined to coincide with the theoretical acquisition time, the pixel value of the first target pixel is taken as the pixel value of the theoretical pixel.

[0118] Step S402: For the case where two second target pixels adjacent to the theoretical acquisition time are determined, the pixel value of the theoretical pixel is calculated using the pixel values ​​of the two second target pixels, the theoretical acquisition time, the acquisition time of any second target pixel, and the acquisition duration of a single image.

[0119] This step can be calculated using the following formula (2):

[0120]

[0121] Among them, P (i,j,tL) This indicates that the calculated value corresponds to the theoretical acquisition time t. L The pixel value of the pixel at the position corresponding to the i-th row and j-th column; P (i,j,n) Represents the relationship between (i,j,t) L The pixel value of the adjacent second target pixel; t (i,j,n) Represents the relationship between (i,j,t) L The acquisition time of an adjacent second target pixel; ScanOneFrameTime * P represents the time taken to process a single 2D image; (i,j,n+1) Represents the relationship between (i,j,t) L The pixel value of the adjacent second target pixel; t (i,j,n+1) Represents the relationship between (i,j,t) L The acquisition time of the adjacent second target pixel.

[0122] Step S403: Combine the pixel values ​​of each theoretical pixel point to obtain the image corresponding to the theoretical acquisition time.

[0123] The pixel value of each theoretical pixel calculated by combination refers to configuring the pixel value corresponding to the theoretical pixel on the image according to the two-dimensional coordinates of each theoretical pixel on the image, namely the x value and y value.

[0124] The above process generates a new image that corrects the distortion of pixels in the image set.

[0125] Furthermore, regarding the image dimension, the specific implementation of finding one or two target images adjacent to the theoretical acquisition time and generating an image corresponding to the theoretical acquisition time using the pixels on the found one or two target images may include: determining the position of the theoretical acquisition time in the image set based on the acquisition time corresponding to the pixels of the image; if the theoretical acquisition time is located on the first target image, generating an image corresponding to the theoretical acquisition time using the pixels of the first target image and the pixels of the second target image adjacent to the first target image; if the theoretical acquisition time is located between two adjacent third target images, generating an image corresponding to the theoretical acquisition time using the pixels of the two third target images.

[0126] Since each pixel in the two-dimensional image set has a corresponding acquisition time, sorting the two-dimensional images according to their acquisition time yields the following results: Figure 2As shown in the diagram, each 2D image has a different position on the z-axis, and each value on the z-axis corresponds to a time point. Therefore, the position within the image set at the theoretical acquisition time can be determined using the z-value corresponding to the theoretical acquisition time, or directly using the theoretical acquisition time and the acquisition times of each 2D image in the image set. For example, the acquisition times of each 2D image are (t1, t2), (t3, t4), (t5, t6), etc. The theoretical acquisition time can lie between the acquisition times of two 2D images or within the acquisition time of a single 2D image; for instance, the theoretical acquisition time may be within (t3, t4) or between (t3, t4) and (t5, t6). In the case where the theoretical acquisition time is within (t3, t4), the 2D image corresponding to (t3, t4) is the first target image, and the 2D images corresponding to (t1, t2) and (t5, t6) are the second target images. For theoretical acquisition times between (t3, t4) and (t5, t6), the two-dimensional images corresponding to (t3, t4) and (t5, t6) are the third target images. Therefore, the first, second, and third target images mentioned in the embodiments of the present invention are used to distinguish different methods of determining target images, and are not a limitation on the number or order of target images.

[0127] The above process enables the search for images adjacent to the theoretical acquisition time from the image dimension, and the generation of an image corresponding to the theoretical acquisition time using the pixels on the adjacent images.

[0128] It is worth noting that, for the case where the theoretical acquisition time is located on the first target image, that is, there exists a pixel on the first target image whose acquisition time is equal to the theoretical acquisition time, the second target image adjacent to the first target image can be, after sorting the images in the image set according to the acquisition time, a two-dimensional image that precedes and is adjacent to the first target image, or a two-dimensional image that follows and is adjacent to the first target image. Preferably, when determining the coordinate position of the pixel on the first target image corresponding to the theoretical acquisition time, for pixels acquired before the pixels on the first target image corresponding to the theoretical acquisition time, a two-dimensional image that precedes and is adjacent to the first target image is determined as the second target image; for pixels acquired after the pixels on the first target image corresponding to the theoretical acquisition time, a two-dimensional image that follows and is adjacent to the first target image is determined as the second target image. For example, for a 5×5 two-dimensional image, the theoretical acquisition time corresponds to the position of the pixel (2,2) on the first target image. Then, the pixels acquired before this pixel are (1,1), (1,2), (1,3), (1,4), (1,5), and (2,1). For these positions, the second target image is the two-dimensional image that is before and adjacent to the first target image. The theoretical acquisition time corresponds to the position of the pixel (2,2) on the first target image. Then, the pixels acquired after this pixel are (2,3), (2,4), (2,5), (3,1), ..., (5,5). For these positions, the second target image is the two-dimensional image that is after and adjacent to the first target image.

[0129] In a preferred embodiment, for determining adjacent target images in the image dimension, the specific implementation of determining the position of the theoretical acquisition time in the image set may include: determining the position of the theoretical acquisition time in the image set based on the acquisition duration of a single image, the theoretical acquisition time, and the acquisition order of multiple images. For example, if the acquisition duration of an image is T1, then the theoretical acquisition time divided by T1 and rounded up indicates which image's acquisition time or between which two images the theoretical acquisition time falls within.

[0130] For cases where the theoretical acquisition time lies between two adjacent third target images, the specific implementation method for generating an image corresponding to the theoretical acquisition time using the pixels of the two third target images may include:

[0131] like Figure 5 As shown, for each theoretical pixel in the image corresponding to the theoretical acquisition time, the following steps S501 to S503 are performed:

[0132] Step S501: In the two third target images, find the third target pixel and the fourth target pixel that correspond to the position of the theoretical pixel, and determine the second acquisition time of the third target pixel;

[0133] For example, taking the theoretical pixel position (1,1) as an example, its corresponding third and fourth target pixels are the pixels corresponding to (1,1) in one third target image and the pixels corresponding to (1,1) in another third target image, respectively. As another example, taking the theoretical pixel position (2,3) as an example, its corresponding third and fourth target pixels are the pixels corresponding to (2,3) in one third target image and the pixels corresponding to (2,3) in another third target image, respectively.

[0134] The second acquisition time of the third target pixel can be before or after the theoretical acquisition time, and is not limited here.

[0135] Step S502: Calculate the pixel value of the theoretical pixel using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition duration of a single image;

[0136] This step can also be calculated using the following formula (3).

[0137]

[0138] Among them, P (i,j,tL) This indicates that the calculated value corresponds to the theoretical acquisition time t. L The pixel value of the pixel at the position corresponding to the i-th row and j-th column; P (i,j,m) In a third target image, the elements in the image (i,j,t) are considered to be in the same position as (i,j,t). L The pixel value of the third adjacent target pixel; t (i,j,m) Represents the relationship between (i,j,t) L The acquisition time of the adjacent third target pixel; ScanOneFrameTime * P represents the time taken to process a single 2D image; (i,j,m+1) This indicates that the image contains (i,j,t) and another third target image. L The pixel value of the fourth adjacent target pixel.

[0139] Alternatively, this step can also be calculated using the above calculation formula (2). It is only necessary to replace the pixel values ​​of the two second target pixels in the above second calculation formula with the pixel values ​​of the third target pixel and the fourth target pixel, and replace the acquisition time of the two second target pixels in the above second calculation formula with the acquisition time corresponding to the third target pixel and the acquisition time corresponding to the fourth target pixel.

[0140] Step S503: Combine the pixel values ​​of each theoretical pixel point to obtain an image corresponding to the theoretical acquisition time.

[0141] Furthermore, for the case where the theoretical acquisition time is located on the first target image, the specific implementation method for generating an image corresponding to the theoretical acquisition time using the pixels of the first target image and the pixels of the second target image adjacent to the first target image can be as follows:

[0142] like Figure 6 As shown, for each theoretical pixel in the image corresponding to the theoretical acquisition time, the following operation steps S601 to S604 are performed:

[0143] Step S601: In the first target image and the second target image respectively, find the fifth target pixel and the sixth target pixel corresponding to the position of the theoretical pixel, and determine the fifth acquisition time corresponding to the fifth target pixel; if the fifth acquisition time is consistent with the theoretical acquisition time, proceed to step S602; if the fifth acquisition time is inconsistent with the theoretical acquisition time, proceed to step S603.

[0144] As mentioned above, it can be understood that two corresponding pixel positions mean that the x-values ​​and y-values ​​of the two pixels are equal. That is, the two pixels are in the same position on their respective two-dimensional images, such as (1,1) or (1,2), etc.

[0145] Specifically, for the case where the theoretical acquisition time is on the first target image, the following operations are performed for each theoretical pixel: if the fifth acquisition time of the fifth target pixel is before the theoretical acquisition time, the next image of the first target image is determined as the second target image according to the acquisition order of multiple images in the image set; if the fifth acquisition time of the fifth target pixel is after the theoretical acquisition time, the previous image of the first target image is determined as the second target image according to the acquisition order of multiple images in the image set.

[0146] Step S602: Directly determine the pixel value of the fifth target pixel to be the pixel value of the theoretical pixel, and then proceed to step S604;

[0147] Step S603: Calculate the pixel value of the theoretical pixel using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image;

[0148] This calculation process can be completed using either the above calculation formula (2) or the above calculation formula (3). When using the above calculation formula (2), simply replace the pixel values ​​of the two second target pixels in the above calculation formula (2) with the pixel values ​​of the fifth and sixth target pixels, and replace the acquisition times of the two second target pixels in the above calculation formula (2) with the acquisition times corresponding to the fifth and sixth target pixels. When using the above calculation formula (3), simply replace the pixel values ​​of the third and fourth target pixels in the above calculation formula (3) with the pixel values ​​of the fifth and sixth target pixels, and replace the acquisition time of the third target pixel in the above calculation formula (3) with the acquisition time corresponding to the fifth target pixel.

[0149] Step S604: Combine the pixel values ​​of each theoretical pixel point to obtain the image corresponding to the theoretical acquisition time.

[0150] Furthermore, to improve the usability of the theoretical acquisition time, the above-described transmission electron microscopy image processing method may further include: receiving one or more sample stage rotation angles input from an external source; calculating the rotation duration corresponding to the sample stage rotation angle using the sample rod rotation rate set by the TEM; and setting the theoretical acquisition time corresponding to the sample stage rotation angle using the sample rod start-up time and the rotation duration corresponding to the sample stage rotation angle. This ensures that the theoretical acquisition time is within the time range of the TEM scan image, thereby further improving the usability and realism of the images generated in the above embodiments and facilitating the correction of image distortion.

[0151] Furthermore, embodiments of the present invention also provide an image reconstruction method. This image reconstruction method may include: the transmission electron microscopy image processing method provided in any of the above embodiments.

[0152] The following example illustrates the results of image processing using the technical solution provided in this embodiment of the invention. Specifically, for a sample (a 28nm process smartphone processing chip with approximately 300nm thickness remaining after thinning), a 1024×1024 two-dimensional image was collected using the STEM mode of TEM, with the sample holder continuously tilted and each pixel exposed for 2μs. Figure 7 As shown, from Figure 7 It is evident that the image suffers from significant angular distortion and image warping. The transmission electron microscopy image processing method described above in this embodiment of the invention is used to process this image. Figure 7 After the image shown, the following is obtained: Figure 8 The two-dimensional image shown is from Figure 8As can be seen, the technical solution provided by the embodiments of the present invention not only eliminates the angular distortion of pixels in each image, but also corrects the distortion of the image to the correct shape on a macroscopic level.

[0153] Furthermore, Figure 9 This is a schematic diagram of the structure of a transmission electron microscope image processing device provided in an embodiment of the present invention. Figure 9 As shown, the transmission electron microscope image processing device 900 may include: a setting unit 901, a searching unit 902, and an image generation unit 903, wherein,

[0154] Setting unit 901 is used to set one or more theoretical acquisition times within the acquisition time period corresponding to the image set, wherein the image set includes multiple images acquired at different times;

[0155] The search unit 902 is used to search for one or two target images that are adjacent to the theoretical acquisition time based on the acquisition time corresponding to the pixels of the image.

[0156] The image generation unit 903 is used to generate an image corresponding to the theoretical acquisition time using the pixels on one or two target images found.

[0157] Furthermore, the search unit 902 is further configured to perform the following operations for each theoretical pixel in the image corresponding to the theoretical acquisition time: determine multiple target pixels corresponding to the position of the theoretical pixel from multiple images included in the image set; and determine a first target pixel that coincides with the theoretical acquisition time or two second target pixels that are adjacent to the theoretical acquisition time using the acquisition time corresponding to the multiple target pixels.

[0158] Furthermore, the image generation unit 903 is further configured to perform the following operations for each theoretical pixel in the image corresponding to the theoretical acquisition time: for the case where a first target pixel is determined to coincide with the theoretical acquisition time, the pixel value of the first target pixel is used as the pixel value of the theoretical pixel; for the case where two second target pixels are determined to be adjacent to the theoretical acquisition time, the pixel value of the theoretical pixel is calculated using the pixel values ​​of the two second target pixels, the theoretical acquisition time, the acquisition time of any second target pixel, and the acquisition duration of a single image; and the calculated pixel values ​​of each theoretical pixel are combined to obtain the image corresponding to the theoretical acquisition time.

[0159] Furthermore, the search unit 902 is further configured to determine the position of the theoretical acquisition time in the image set based on the acquisition time corresponding to the pixel of the image; for the case where the theoretical acquisition time is located on the first target image, determine the second target image adjacent to the first target image; for the case where the theoretical acquisition time is located between two adjacent third target images, determine the two third target images adjacent to the first target image.

[0160] Furthermore, the image generation unit 903 is further configured to, for the case where the theoretical acquisition time is located on the first target image, generate an image corresponding to the theoretical acquisition time using the pixels of the first target image and the pixels of the second target image adjacent to the first target image; and for the case where the theoretical acquisition time is located between two adjacent third target images, generate an image corresponding to the theoretical acquisition time using the pixels of the two third target images.

[0161] Furthermore, the search unit 902 is further used to determine the position of the theoretical acquisition time in the image set based on the acquisition duration of a single image, the theoretical acquisition time, and the acquisition order of multiple images.

[0162] Furthermore, the image generation unit 903 is further used to correct the exposure time of the pixels set in the TEM; based on the corrected exposure time and the number of pixels in the image, it calculates the actual time consumed for a single image and determines the actual time consumed as the acquisition time of a single image.

[0163] Furthermore, the image generation unit 903 is further configured to perform the following operations for each theoretical pixel in the image corresponding to the theoretical acquisition time: in the two third target images, respectively, find the third target pixel and the fourth target pixel corresponding to the position of the theoretical pixel, and determine the second acquisition time of the third target pixel; calculate the pixel value of the theoretical pixel using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition duration of a single image; and combine the calculated pixel values ​​of each theoretical pixel to obtain the image corresponding to the theoretical acquisition time.

[0164] Furthermore, the image generation unit 903 is further configured to perform the following operations for each theoretical pixel in the image corresponding to the theoretical acquisition time: find the fifth target pixel and the sixth target pixel corresponding to the position of the theoretical pixel in the first target image and the second target image respectively, and determine the fifth acquisition time corresponding to the fifth target pixel; calculate the pixel value of the theoretical pixel using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image; and combine the calculated pixel values ​​of each theoretical pixel to obtain the image corresponding to the theoretical acquisition time.

[0165] Furthermore, the image generation unit 903 is further configured to directly determine the pixel value of the fifth target pixel as the pixel value of the theoretical pixel when the fifth acquisition time is consistent with the theoretical acquisition time; and to perform the step of calculating the pixel value of the theoretical pixel using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image when the fifth acquisition time is inconsistent with the theoretical acquisition time.

[0166] Furthermore, the lookup unit 902 is further configured to perform the following operations for each theoretical pixel when the theoretical acquisition time is on the first target image: when the fifth acquisition time of the fifth target pixel is before the theoretical acquisition time, determine the next image of the first target image as the second target image according to the acquisition order of multiple images in the image set; when the fifth acquisition time of the fifth target pixel is after the theoretical acquisition time, determine the previous image of the first target image as the second target image according to the acquisition order of multiple images in the image set.

[0167] Furthermore, the setting unit 901 is further used to receive one or more sample stage rotation angles input from the outside; calculate the rotation duration corresponding to the sample stage rotation angle using the sample rod rotation rate set by the TEM; and set the theoretical acquisition time corresponding to the sample stage rotation angle using the sample rod start time and the rotation duration corresponding to the sample stage rotation angle.

[0168] Furthermore, embodiments of the present invention also provide an image reconstruction apparatus. This image reconstruction apparatus includes the above-described transmission electron microscopy image processing method or transmission electron microscopy image processing apparatus.

[0169] It is worth noting that the aforementioned transmission electron microscopy image processing device and the aforementioned image reconstruction device can be installed as plug-ins in existing TEM software, or they can be installed as applications on terminal devices or servers.

[0170] Figure 10 An exemplary system architecture 1000 for a transmission electron microscope image processing method or apparatus to which embodiments of the present invention can be applied is shown.

[0171] like Figure 10As shown, the system architecture 1000 may include a first terminal device 1001 directly connected to the TEM device, a TEM device 1002, a network 1003, and a second terminal device 1004 or a server 1005 for transmission electron microscopy image processing. The network 1003 serves as a medium for providing communication links between the first terminal device 1001 and the TEM device 1002, between the TEM device 1002 and the second terminal device 1003, between the first terminal device 1001 and the second terminal device 1004, or between the first terminal device 1001 and the server 1005. The network 1003 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0172] Users can use the first terminal device 1001 to interact with the TEM device 1002 via the network 1003 to obtain scanning information from the TEM device 1002 or send control information to the TEM device 1002, such as commands to control the sample rod to rotate at a constant speed or to control the camera to enter video recording mode. The first terminal device 1001 may be equipped with software for controlling the TEM device or for displaying TEM scanning results (example only).

[0173] The second terminal device 1004 or server 1005 is used to acquire video data or a series of image data from the video data from the first terminal device 1001 and TEM device 1002 respectively, and to perform image processing or reconstruction on the system image data, and to store and provide the results to the user.

[0174] The first terminal device 1001 and the second terminal device 1004 can be various electronic devices with displays and support web browsing, including but not limited to laptops and desktop computers.

[0175] It should be noted that the transmission electron microscopy image processing method provided in this embodiment of the invention is generally executed by the second terminal device 1004 or the server 1005, and correspondingly, the transmission electron microscopy image processing device is generally disposed in the second terminal device 1004 or the server 1005. Alternatively, the transmission electron microscopy image processing method can also be performed by the first terminal device 1001.

[0176] In addition, the first terminal device 1001 and the second terminal device 1004 mentioned above may also belong to the same terminal device. Accordingly, the transmission electron microscopy image processing device is installed in the first terminal device 1001 in the form of a plug-in, so as to realize the processing of a series of transmission electron microscopy image data by the transmission electron microscopy image processing method provided in the embodiments of the present invention.

[0177] It should be understood that Figure 10The number of second terminal devices or servers shown is merely illustrative. Depending on implementation needs, there can be any number of second terminal devices or servers.

[0178] The following is for reference. Figure 11 It shows a schematic diagram of the structure of a computer system 1100 suitable for implementing terminal devices or servers of the present invention. Figure 11 The terminal device or server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0179] like Figure 11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1102 or programs loaded from storage section 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of the system 1100. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0180] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. Drive 910 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0181] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs the functions defined above in the system of this invention.

[0182] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0184] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a setting unit, a searching unit, and an image generation unit. The names of these modules or units do not necessarily limit the module itself; for example, the image generation unit may also be described as "a unit or module that generates images corresponding to the theoretical acquisition time."

[0185] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to include: setting one or more theoretical acquisition times within an acquisition period corresponding to an image set, wherein the image set includes multiple images acquired at different times; finding one or two target images adjacent to the theoretical acquisition time based on the acquisition time corresponding to pixels in the images; and generating an image corresponding to the theoretical acquisition time using the pixels on the found one or two target images.

[0186] According to the technical solution of this invention, by setting one or more theoretical acquisition times, the position of the theoretical acquisition time in the image set is determined. Then, using the pixels on the two images closest to the theoretical acquisition time, an image corresponding to the theoretical acquisition time is generated. By combining the pixels on the two images closest to the theoretical acquisition time, and based on this theoretical acquisition time, images on the same plane can be generated, improving image distortion and enabling the generated image to more realistically reflect the microscopic two-dimensional structure of the material. This can adjust the distortion of images of automatically rotating samples captured by TEM in STEM mode, allowing the reconstructed three-dimensional structure to more realistically reflect the microscopic structure of the material. Furthermore, by adding the image generated by the above method to the image set, the image distortion caused by continuous rotation of the sample stage can be reduced, enabling the image set to more realistically reflect sample information and effectively improving the realism and accuracy of the subsequent three-dimensional structure reconstructed based on this image set.

[0187] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for processing transmission electron microscopy images, characterized in that, The image set acquired in STEM mode for TEM includes: Within the acquisition period corresponding to the image set, one or more theoretical acquisition times are set, wherein the image set includes multiple images acquired at different times; Based on the acquisition time corresponding to the pixels in the image, find one or two target images adjacent to the theoretical acquisition time; Using the pixels on one or two of the target images found, an image corresponding to the theoretical acquisition time is generated; Based on the acquisition time corresponding to the pixels in the image, the process includes finding one or two target images adjacent to the theoretical acquisition time, utilizing the pixels from the found one or two target images, and generating an image corresponding to the theoretical acquisition time, including: Determining the position of the theoretical acquisition time in the image set based on the acquisition time corresponding to the pixels of the image specifically includes: determining the position of the theoretical acquisition time in the image set based on the acquisition duration of a single image, the theoretical acquisition time, and the acquisition order of multiple images; In the case where the theoretical acquisition time is located on the first target image, an image corresponding to the theoretical acquisition time is generated using the pixels of the first target image and the pixels of the second target image adjacent to the first target image. For cases where the theoretical acquisition time falls between two adjacent third target images, an image corresponding to the theoretical acquisition time is generated using the pixels of the two third target images. Specifically, this involves: for each theoretical pixel in the image corresponding to the theoretical acquisition time, performing the following operations: finding the third target pixel and the fourth target pixel corresponding to the position of the theoretical pixel in each of the two third target images, and determining the third acquisition time of the third target pixel; calculating the pixel value of the theoretical pixel using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition duration of a single image; and combining the calculated pixel values ​​of each theoretical pixel to obtain the image corresponding to the theoretical acquisition time. The transmission electron microscopy image processing method further includes: correcting the exposure time of the pixels set in the TEM; calculating the actual time consumed for a single image based on the corrected exposure time and the number of pixels in the image, and determining the actual time consumed as the acquisition time of a single image; The transmission electron microscopy image processing method further includes: receiving one or more sample stage rotation angles input from the outside; calculating the rotation duration corresponding to the sample stage rotation angle using the sample rod rotation rate set by the TEM; and setting the theoretical acquisition time corresponding to the sample stage rotation angle using the start-up time of the sample rod and the rotation duration corresponding to the sample stage rotation angle.

2. The transmission electron microscopy image processing method according to claim 1, characterized in that, The step of generating an image corresponding to the theoretical acquisition time using pixels from the first target image and pixels from a second target image adjacent to the first target image includes: For each theoretical pixel in the image corresponding to the theoretical acquisition time, perform the following operation: In the first target image and the second target image, respectively, find the fifth target pixel and the sixth target pixel corresponding to the position of the theoretical pixel, and determine the fifth acquisition time corresponding to the fifth target pixel; The pixel value of the theoretical pixel is calculated using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image. The pixel values ​​of each theoretical pixel point are combined and calculated to obtain an image corresponding to the theoretical acquisition time.

3. The transmission electron microscopy image processing method according to claim 2, characterized in that, After determining the fifth acquisition time corresponding to the fifth target pixel, the process also includes: If the fifth acquisition time is consistent with the theoretical acquisition time, the pixel value of the fifth target pixel is directly determined to be the pixel value of the theoretical pixel. If the fifth acquisition time is inconsistent with the theoretical acquisition time, the step of calculating the pixel value of the theoretical pixel using the fifth target pixel, the sixth target pixel, the fifth acquisition time, and the acquisition duration of a single image is performed.

4. The transmission electron microscopy image processing method according to claim 2 or 3, characterized in that, Also includes: For the case where the theoretical acquisition time is located on the first target image, the following operation is performed for each theoretical pixel: If the fifth acquisition time of the fifth target pixel is before the theoretical acquisition time, the next image of the first target image is determined as the second target image according to the acquisition order of multiple images in the image set; If the fifth acquisition time of the fifth target pixel is after the theoretical acquisition time, the image preceding the first target image is determined as the second target image according to the acquisition order of multiple images in the image set.

5. An image reconstruction method, characterized in that, include: The transmission electron microscopy image processing method according to any one of claims 1 to 4.

6. A transmission electron microscope image processing device, characterized in that, include: The system comprises a setting unit, a search unit, and an image generation unit, among which... The setting unit is used to set one or more theoretical acquisition times within the acquisition time period corresponding to the image set, wherein the image set includes multiple images acquired at different times; The search unit is used to search for one or two target images that are adjacent to the theoretical acquisition time based on the acquisition time corresponding to the pixels of the image. The image generation unit is used to generate an image corresponding to the theoretical acquisition time using the pixels on one or two of the target images found. The search unit is further configured to determine the position of the theoretical acquisition time in the image set based on the acquisition time corresponding to the pixels of the image; for the case where the theoretical acquisition time is on the first target image, determine the second target image adjacent to the first target image; for the case where the theoretical acquisition time is between two adjacent third target images, determine the two third target images adjacent to the first target image. The image generation unit is further configured to, for the case where the theoretical acquisition time is located on the first target image, generate an image corresponding to the theoretical acquisition time using the pixels of the first target image and the pixels of the second target image adjacent to the first target image; and for the case where the theoretical acquisition time is located between two adjacent third target images, generate an image corresponding to the theoretical acquisition time using the pixels of the two third target images. The search unit is further used to determine the position of the theoretical acquisition time in the image set based on the acquisition duration of a single image, the theoretical acquisition time, and the acquisition order of multiple images; The image generation unit is further used to correct the exposure time of the pixels set by the TEM; based on the corrected exposure time and the number of pixels in the image, the actual time consumption of a single image is calculated, and the actual time consumption is determined as the acquisition time of a single image; The image generation unit is further configured to perform the following operations for each theoretical pixel in the image corresponding to the theoretical acquisition time: in the two third target images, respectively, find the third target pixel and the fourth target pixel corresponding to the position of the theoretical pixel, and determine the third acquisition time of the third target pixel; calculate the pixel value of the theoretical pixel using the third target pixel, the fourth target pixel, the third acquisition time, and the acquisition duration of a single image; and combine the calculated pixel values ​​of each theoretical pixel to obtain the image corresponding to the theoretical acquisition time. The setting unit is further configured to receive one or more sample stage rotation angles input from the outside; calculate the rotation duration corresponding to the sample stage rotation angle using the sample rod rotation rate set by the TEM; and set the theoretical acquisition time corresponding to the sample stage rotation angle using the start-up time of the sample rod and the rotation duration corresponding to the sample stage rotation angle.

7. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images