Atomic segmentation and localization method in high-resolution transmission electron microscopy based on UUnet

Abstract

The present invention relates to an atomic segmentation and positioning technology under a high-resolution transmission electron microscope based on UUnet. The process of the method includes: S1: using a high-resolution transmission electron microscope to generate an atomic image, and cutting the original image into N sub-images, and manually calibrating the center of mass of each atom in each sub-image, and then using a point spread function G to spread the calibrated atomic coordinates into a circle, and using the calibrated result image as a training set for atomic segmentation processing; S2: using a UUnet model-based and a Swin-Unet model-based UUnet adaptive fusion network to segment each atom in the sub-image; S3: using Hoffman circle detection to accurately locate the center of mass of all atoms in the atomic segmentation result image.

Description

Technical Field

[0001] The invention relates to an atomic segmentation and positioning method under a high-resolution transmission electron microscope based on UUnet, and belongs to the field of computer vision. Background Art

[0002] The change of atomic spacing in crystalline materials (strain, grain boundary, twin boundary) will affect the performance of crystalline materials. For example, nickel-based high-temperature alloys are key structural materials for manufacturing high-temperature and high-pressure components such as aircraft engines and gas turbines. Introducing more twin boundaries in engineering alloys can significantly increase the life of the alloy. Among them, the precise positioning of the atomic center of mass is the key to analyzing the change of atomic spacing, and the analysis of the change in atomic spacing is the key to analyzing the atomic structure.

[0003] However, locating the atomic center of mass is a very challenging task. First, high-resolution transmission electron microscope atomic images are subject to various noise interferences during the shooting process. Secondly, the current methods used to record high-quality atomic scale data are mostly carried out by manual calibration. The annotation of the atomic center of mass is limited by the professional knowledge required for the atomic structure itself, and even experts in this field will make annotation errors. In addition, the high time cost will affect the efficiency of using the atomic center of mass to analyze its structure, and these factors make manual annotation impractical. In recent years, people have made significant progress in reliable atomic scale imaging of materials, mainly due to the widespread application of scanning electron microscopes and high-resolution transmission electron microscope imaging technologies. In addition, image processing and recognition technology has developed by leaps and bounds, involving computer science, information science and other aspects, and is also widely used in target tracking, material analysis and other fields. In the field of material analysis, the segmentation algorithm using a deep learning framework can achieve accurate atomic segmentation effects through a small amount of training data sets, and then locate the atomic center of mass of the segmentation results, which can realize the function of automatically locating the atomic center of mass, which is efficient and robust. In addition, the color display of the spacing changes of the atomic center of mass automatically located can also facilitate professionals to quickly analyze the state of the atomic structure. Therefore, the present invention has important significance in the field of computer vision. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides an atomic segmentation and positioning method under a high-resolution transmission electron microscope based on UUnet, which solves the problem of low efficiency in locating the center of mass of atoms in high-resolution transmission electron microscope atomic images by relying on manual calibration, and has good practicality and robustness.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A method for atomic segmentation and positioning under a high-resolution transmission electron microscope based on UUnet, comprising the following steps:

[0007] Step S1: Generate an atomic image using a high-resolution transmission electron microscope, crop the original image into N sub-images, and manually calibrate the center of mass of each atom in each sub-image. Then, use the point spread function G to spread the calibrated atomic coordinates into a circle, and use the calibrated result image as a training set for atomic segmentation processing;

[0008] Step S2: using the Unet model-based and Swin-Unet model-based UUnet adaptive fusion network to segment each atom in the sub-image;

[0009] Step S3: Use Huffman network detection to accurately locate the mass centers of all atoms in the atomic segmentation result image.

[0010] Furthermore, in step S1, a point spread function G is designed to generate an atomic training data set, referring to formula (1);

[0011]

[0012] Among them G ij represents the intensity value of the pixel (i, j) in the true atomic image, (x u ,y r ) represents the coordinates of the atomic center of mass in the sub-image, where u = {u1, u2, ..., u n}, r = {r1, r2, ..., r n}.

[0013] Furthermore, in step S2, UUnet includes three modules:

[0014] M1 module: It uses the 8-layer convolutional block in the Unet network. First, it performs 4 downsampling operations on the cropped sub-image. Each downsampling operation will learn feature information such as texture and edge. Then it performs 4 upsampling operations. Each upsampling operation will learn the global information in the sub-image. After the Unet network executes the 8-layer convolutional block, the resolution of the atomic image is the same as the original image, that is, the feature information A of this layer.

[0015] M2 module: First, each cropped sub-image (M×M×R) is divided into multiple small blocks The size of each small block is (m×m×R), where (M>m), and the position information is added to each small block, and then small blocks are rearranged into one large block Then the large block is input into two consecutive transformers to learn feature information such as texture and edge. Finally, the 3-layer encoding block and 3-layer decoding block in the Swin-Unet network are used. First, the feature information such as texture and edge is learned through 3 encoding blocks, and then the global information is learned through 3 decoding blocks. After the Swin-Unet network executes the decoding block 3 times, a 3×3×64 convolution layer is constructed to adjust the number of feature channels of the image at this resolution, and the output result is used as the global information B.

[0016] M3 module: Design a layer of learnable weights W1 and W2 with the same size as the image resolution for the Unet network and Swin-Unet network respectively, and fuse the features of A and B in this layer to obtain C:

[0017] C=W1*A+W2*B (2)

[0018] Where C is the feature information fused from the two networks, including the texture edge and global information of atoms, and * represents the point-to-point multiplication operation between matrices.

[0019] The beneficial effects of the present invention are mainly manifested in: the present invention diffuses the calibrated atomic coordinates into a circle through the designed point spread function G, and uses the calibrated result image as a training set for atomic segmentation processing, adopts the designed UUnet adaptive fusion network to segment each atom in the sub-image, and uses the Hoffman circle to detect the center of mass of each atom in the segmented result image, thereby realizing an automated method for accurately locating the center of mass of the atom, which not only has low time cost consumption but also can meet higher accuracy requirements, provides convenience for atomic structure analysis, and is more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the process of atomic segmentation and positioning technology under high-resolution transmission electron microscopy based on UUnet;

[0021] Figure 2 The synthesis process of the atomic training data set in the present invention;

[0022] Figure 3 It is a flowchart of atomic segmentation based on UUnet network in the present invention;

[0023] Figure 4 This is a diagram showing the result of automatic positioning of the atomic mass center in the present invention. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0025] Reference Figure 1 to Figure 4, an atomic segmentation and positioning method under high-resolution transmission electron microscopy based on UUnet, comprising the following steps:

[0026] Step S1: Generate an atomic image using a high-resolution transmission electron microscope, crop the original image into N sub-images, and manually calibrate the center of mass of each atom in each sub-image. Then, use the point spread function G to spread the calibrated atomic coordinates into a circle, and use the calibrated result image as a training set for atomic segmentation processing;

[0027] Step S2: using the Unet model-based and Swin-Unet model-based UUnet adaptive fusion network to segment each atom in the sub-image;

[0028] Step S3: Use Huffman circle detection to accurately locate the mass centers of all atoms in the atomic segmentation result image.

[0029] Figure 1 This is a flow chart of the atomic image segmentation and positioning technology under high-resolution transmission electron microscope imaging. The method includes five modules in total, namely, cropping the atomic image of high-resolution transmission electron microscope imaging, manually calibrating the atomic center of mass, synthesizing a training data set, using the UUnet network to segment the atomic image, and accurately positioning the atomic center of mass.

[0030] Figure 2 This is the synthesis process of the atomic training data set in the present invention. First, the atomic images required for the experiment are generated by processing with high-resolution transmission electron microscopy imaging technology. Then, 786 sub-images with a resolution of 256×256 are cropped from 10 nanoporous gold atomic images with a resolution of 4096×4224, and the mass centers of all atoms in the sub-images are manually calibrated. Finally, the calibrated atomic coordinates are diffused into a circle through the designed point spread function G, and the calibrated result image is used as a training set for atomic segmentation processing, referring to formula (1):

[0031]

[0032] Among them G ij represents the intensity value of the pixel (i, j) in the true atomic image, (x u ,y r ) represents the coordinates of the atomic center of mass in the sub-image, where u = {u1, u2, ..., u n}, r = {r1, r2, ..., r n}.

[0033] like Figure 3 As shown in Figure 1, the main process of UUnet network segmentation of atomic images is divided into the following three parts.

[0034] Step S11 (corresponding to M1 module): The Unet network architecture uses a U-shaped CNN network. The left side of the U-shaped structure is the downsampling layer, and the right side is the upsampling layer. Each sampling includes two consecutive convolution blocks and operations to map various features. Each convolution block consists of a 3×3 convolution layer, a normalization layer, and a linear activation layer, followed by a 2×2 maximum pooling operation with a stride of 2. A total of 8 convolution blocks in the Unet network are used. First, the cropped sub-image is placed in the Unet network, and 4 downsampling operations are performed on the cropped sub-image, and then 4 upsampling operations are performed. Each downsampling operation will process the feature information such as texture and edge of each atom in the segmented sub-image. After the downsampling operation, the feature information of the atomic image is magnified by 2 times, and the spatial resolution of the image is reduced by 2 times. After each upsampling operation, the global information of the atomic image is extracted. After each upsampling operation, the feature information of the atomic image is reduced by 2 times, and the spatial resolution of the image is expanded by 2 times. After the Unet network executes 8 layers of convolution blocks, the resolution of the atomic image is the same as the original image, that is, the feature information of this layer is A.

[0035] Step S12 (corresponding to module M2): First, divide each cropped sub-image (M×M×R) into multiple small blocks Where (M>m), and add position information to each small block, and then small blocks are rearranged into a human block Then the block is input into two consecutive transformers to learn feature information such as texture and edge. Then the 3-layer encoding block and 3-layer decoding block in the Swin-Unet network are used. Swin-Unet also adopts a U-shaped structure, with the encoding block on the left side of the U-shaped structure, the decoding block on the right side of the U-shaped structure, and the skip connection in the middle. The encoding block consists of 3 layers with the same structure, each of which includes a linear embedding layer and two consecutive transformer modules. The rearranged large block passes through 3 encoding blocks and 3 decoding blocks in turn. After passing through each encoding block, the feature information of the input image will be magnified by 2 times, and the resolution of the input image will be halved. At the same time, the feature information such as texture and edge in the atomic image is learned. After passing through each decoding block, the resolution of the input image is enlarged by 2 times, the feature information is reduced by half, and the global information in the atomic image is learned. When the Swin-Unet network executes the decoding block 3 times, a 3×3×64 convolution layer is constructed to adjust the number of feature channels of the image at this resolution, and the output result is used as the global information B.

[0036] Step S13 (corresponding to M3 module): Design a layer of learnable weights W1 and W2 with the same size as the image resolution for the Unet network and Swin-Unet network respectively, and perform feature fusion C (Formula 2) on the recorded global information A and B in this layer:

[0037] C=W1*A+W2*B (2)

[0038] Where C is the feature information fused from the two networks, including the texture edge and global information of atoms, and * represents the point-to-point multiplication operation between matrices.

[0039] Figure 4 (a) is the result of using Hoffman circle detection technology to accurately locate the center of mass of atoms in the atomic segmentation result image. Figure 4 (b) is the positioning effect diagram of the obtained center of mass in the original image.

[0040] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for atomic segmentation and positioning under high-resolution transmission electron microscopy based on UUnet, characterized in that: The method comprises the following steps: Step S1: Generate an atomic image using a high-resolution transmission electron microscope, crop the original image into N sub-images, and manually calibrate the center of mass of each atom in each sub-image. Then, use the point spread function G to spread the calibrated atomic coordinates into a circle, and use the calibrated result image as a training set for atomic segmentation processing; Step S2: using the Unet model-based and Swin-Unet model-based UUnet adaptive fusion network to segment each atom in the sub-image; Step S3: using Huffman circle detection to accurately locate the mass centers of all atoms in the atomic segmentation result image; In step S1, a point spread function G is designed to generate an atomic training data set, referring to formula (1): Among them G ij represents the intensity value of the pixel (i, j) in the true atomic image, (x u ,y r ) represents the coordinates of the atomic center of mass in the sub-image, where u = {u1, u2, ..., u n }, r = {r1, r2, ..., r n }.

2. The atomic segmentation and positioning method under high-resolution transmission electron microscopy based on UUnet as claimed in claim 1, characterized in that: In step S2, UUnet includes three modules: M1 module: It uses the 8-layer convolutional block in the Unet network. First, it performs 4 downsampling operations on the cropped sub-image. Each downsampling operation will learn feature information such as texture and edge. Then it performs 4 upsampling operations. Each upsampling operation will learn the global information in the sub-image. After the Unet network executes the 8-layer convolutional block, the resolution of the atomic image is the same as the original image, that is, the feature information A of this layer. M2 module: First, each cropped sub-image (M×M×R) is divided into multiple small blocks The size of each small block is (m×m×R), where M>m, and the position information is added to each small block, and then Small blocks are rearranged into one large block Then the large block is input into two consecutive transformers to learn feature information such as texture and edge. Finally, the 3-layer encoding block and 3-layer decoding block in the Swin-Unet network are used. First, the feature information such as texture and edge is learned through 3 encoding blocks, and then the global information is learned through 3 decoding blocks. After the Swin-Unet network executes the decoding block 3 times, a 3×3×64 convolution layer is constructed to adjust the number of feature channels of the image at this resolution, and the output result is used as the global information B. M3 module: Design a layer of learnable weights W1 and W2 with the same size as the image resolution for the Unet network and Swin-Unet network respectively, and fuse the features of A and B in this layer to obtain C: C=W1*A+W2*B (2) Where C is the feature information fused from the two networks, including the texture edge and global information of atoms, and * represents the point-to-point multiplication operation between matrices.

Images