An improved method for phase segmentation of crystal TEM images based on deep learning

Abstract

The present invention discloses an improved method for phase division of crystal TEM images based on deep learning, comprising: a feature region division part and a phase identification part. The feature region division part generates a four-dimensional data by executing a fast Fourier transform algorithm through a sliding window, and then performs principal component analysis to estimate the number of important components in the four-dimensional data. Thereafter, the four-dimensional data is decomposed into a feature matrix and a corresponding coefficient matrix using non-negative matrix decomposition, wherein the feature matrix is ​​composed of a fast Fourier transform amplitude spectrum, and the coefficient matrix is ​​composed of a thermal map corresponding to the fast Fourier transform amplitude spectrum. The phase identification part uses a phase identification algorithm to obtain phase information in the fast Fourier transform amplitude spectrum, and the final phase region can be obtained by analyzing the thermal map corresponding to the fast Fourier transform amplitude spectrum. The present invention realizes the automatic division of the phase region of the crystal TEM image. Compared with the existing methods, the boundary of the final phase region of this method is more accurate.

Description

Technical Field

[0001] The present invention relates to a method for automatically processing transmission electron microscope (TEM) images. Specifically, the method comprises using a sliding window to traverse the image and calculate the fast Fourier transform amplitude spectrum within each sliding window to generate a four-dimensional data set. A machine learning method is used to decompose the four-dimensional data into a coefficient matrix representing characteristic regions and a feature matrix representing the fast Fourier transform amplitude spectrum of the characteristic regions. Finally, a phase recognition method based on deep learning is used to analyze the fast Fourier transform amplitude spectrum in the feature matrix, thereby completing the automatic phase region segmentation. Specifically, the method relates to an improved method for phase segmentation of crystal TEM images based on deep learning. Background Art

[0002] The analysis of material TEM images currently relies primarily on manual experience, which is cumbersome, time-consuming, and labor-intensive. To accelerate the efficiency of materials research, Chinese patent application number CN202211670815.1 discloses a method for phase region segmentation in crystal TEM images based on deep learning. This method uses deep learning and computer vision technology to automatically identify phases in TEM images and divides and merges phase regions using a sliding window. This method achieves automatic phase region segmentation and can help researchers find the approximate location of the phase of interest. However, the phase region boundaries obtained by this method are not precise enough, and the method is significantly affected by the accuracy of the neural network during fine-grained analysis, requiring the use of a more complex neural network to ensure accurate results. Another example is the paper "Automatic microscopic image analysis by moving window local Fourier transform and machine learning," which discloses a method for feature region segmentation in microscopic images based on machine learning. This method can segment multiple feature regions from electron microscope images to help researchers find the accurate boundaries between each feature region. However, this method still requires manual analysis of each feature region and does not achieve fully automated analysis. Summary of the Invention

[0003] In order to reduce the workload of materials researchers in manually analyzing TEM images and quickly help them find the exact location of the phase of interest, the present invention combines the advantages of existing methods and designs an improved machine learning-based crystal TEM image phase division method. This method uses a sliding window algorithm and a fast Fourier transform algorithm to generate a four-dimensional data set, and decomposes the four-dimensional data into a coefficient matrix representing the characteristic area heat map and a feature matrix representing the fast Fourier transform amplitude spectrum of the characteristic area through a machine learning method. The fast Fourier transform amplitude spectrum in the characteristic matrix is ​​analyzed by the phase identification method disclosed in the Chinese patent publication number CN202211670815.1. The threshold segmentation algorithm and the connected domain algorithm are used to obtain the boundary of the characteristic area of ​​the characteristic area heat map corresponding to the fast Fourier transform amplitude spectrum of the identified phase. The boundary of the characteristic area is the final phase area boundary, thereby realizing the automatic division of the phase of the crystal TEM image.

[0004] Specifically, the present invention provides an improved method for phase region segmentation of crystal TEM images based on machine learning, which mainly includes the following steps:

[0005] S1, use a square sliding window algorithm to traverse the TEM image and calculate the Fourier transform amplitude spectrum of each window;

[0006] S2, generating all Fourier transform amplitude spectra generated by the sliding window into a four-dimensional data set;

[0007] S3, using a principal component analysis method to analyze the number of important components in the four-dimensional data set, where the number of important components corresponds to the number of characteristic regions in the TEM image;

[0008] S4. Performing non-negative matrix decomposition on the four-dimensional data set according to the number of important components. This step will obtain a feature matrix and a coefficient matrix. The feature matrix is ​​composed of multiple fast Fourier transform amplitude spectra, and the coefficient matrix is ​​composed of multiple heat maps. The number of fast Fourier transform amplitude spectra in the feature matrix is ​​consistent with the number of heat maps in the coefficient matrix, and there is a one-to-one correspondence.

[0009] S5. Use the phase identification method disclosed in the Chinese patent publication number CN202211670815.1 to analyze the fast Fourier transform amplitude spectrum in the feature matrix. If the phase of interest is identified from a certain fast Fourier transform amplitude spectrum, use threshold segmentation and edge detection algorithm on the heat map corresponding to the fast Fourier transform amplitude spectrum to obtain the boundary of the phase area.

[0010] The four-dimensional data in step S2 includes two real space dimensions and two reciprocal space dimensions, the real space dimension information represents the position information of the sliding window in the TEM image, and the reciprocal space dimension information represents the Fourier transform amplitude spectrum within the sliding window;

[0011] When using principal component analysis to analyze the number of important components in step S3, a PCA Scree Plot needs to be generated. The horizontal axis represents the number of components, and the vertical axis represents the proportion of the variance of each component to the total variance. The higher the proportion, the more important the component. Usually, there is an obvious inflection point in the PCA Scree Plot, and the number of components before the inflection point is the number of important components.

[0012] The thermal map in the coefficient matrix described in step S4 needs to be enlarged to the same size as the original TEM image before performing the threshold segmentation and edge detection algorithm;

[0013] The phase identification method described in step S5 is the phase identification method disclosed in Chinese Patent Publication No. CN202211670815.1. However, the phase region division method disclosed in Chinese Patent Publication No. CN202211670815.1 proposes a specially optimized neural network for accurate spot segmentation. In contrast, the Fourier transform amplitude spectrum obtained by non-negative matrix decomposition in step S4 of the present invention only requires a simple U-Net network to achieve accurate segmentation.

[0014] The boundary of the physical phase region described in step S5 needs to be drawn in the original TEM image by colored lines to facilitate observation by researchers.

[0015] Beneficial effects: The present invention realizes automatic and objective division of phase regions in crystal TEM images. Using the present invention for TEM image analysis can help researchers quickly find the region where the phase of interest is located. Compared with the phase region division method disclosed in the Chinese patent publication number CN202211670815.1, the phase boundary obtained by the present invention is more accurate, and no complex neural network model is required in the phase identification process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the general flow of the automatic image processing system of the present invention;

[0017] Figure 2 is a description of the neural network architecture used by the deep learning method described in the present invention;

[0018] Figure 3 It is a schematic diagram showing the comparison results of the present invention, existing methods and manual analysis in dividing phase boundaries. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0020] like Figure 1 、 Figure 2 、 Figure 3 As shown, the present invention provides an improved method for phase segmentation of crystal TEM images based on machine learning, comprising the following steps:

[0021] S1, use a sliding window to traverse the entire TEM image and calculate the fast Fourier transform amplitude spectrum of each window;

[0022] S2. Generate a four-dimensional data set from all the fast Fourier transform amplitude spectra in step S1. When the side length of the sliding window is wSize, the sliding step is Stpe, the width of the TEM image is W, and the height is H, the size of the four-dimensional data is ;

[0023] S3. Generate a PCA Scree Plot of the four-dimensional data set using a principal component analysis method. There is usually a relatively obvious inflection point in the PCA Scree Plot. The number of components before the inflection point represents the number n of important components in the four-dimensional data, representing the number of protected n characteristic regions in the TEM image.

[0024] S4. Setting the output parameters of non-negative matrix decomposition according to the number n of important components, and decomposing the four-dimensional data into a feature matrix and a coefficient matrix using non-negative matrix decomposition, wherein the feature matrix is ​​composed of n fast Fourier transform amplitude spectra, each fast Fourier transform amplitude spectrum represents the physical phase information of a characteristic area, and the coefficient matrix is ​​composed of n heat maps, each heat map represents the distribution of its corresponding fast Fourier transform amplitude spectrum in the four-dimensional data described in step S2, and also represents the range of a characteristic area;

[0025] S5. Perform phase identification on the fast Fourier transform amplitude spectrum in the feature matrix generated in step S4 using the phase identification method disclosed in Chinese patent publication number CN202211670815.1, generate a spatial map using a threshold segmentation algorithm for the thermal map corresponding to the fast Fourier transform amplitude spectrum of the identified phase, obtain the boundary of the phase using an edge detection algorithm for the spatial map, and draw it in the TEM image using colored lines.

[0026] The window size and sliding step size of the sliding window algorithm described in step S1 determine the resolution of the final analysis result. Smaller window and sliding step sizes mean greater sensitivity to phase transformations in the TEM image and finer phase boundaries, but require more analysis time.

[0027] The first two dimensions of the four-dimensional data in step S2 represent the position information of the sliding window in real space, and the last two dimensions represent the information of the fast Fourier transform amplitude spectrum of the sliding window;

[0028] In step S3, finding the location of the inflection point in the PCA Scree Plot is achieved using the Python knee library;

[0029] The size of the fast Fourier transform amplitude spectrum in the feature matrix described in step S4 is consistent with the size of the sliding window. Compared with the result of directly performing fast Fourier transform on the image, the spot clarity in the fast Fourier transform amplitude spectrum obtained from the four-dimensional data described in step S2 by non-negative matrix decomposition is much higher. Therefore, when using the phase recognition method disclosed in the Chinese patent with publication number CN202211670815.1 for phase recognition in step S5, only a simpler U-Net algorithm is needed to complete the spot segmentation, without the need to use the LCA-UNet disclosed in the patent. The heatmaps in the coefficient matrix of non-negative matrix decomposition are stored in dictionary A. Each heatmap represents a feature region, for example, A={feature region a: heatmap a, feature region b: heatmap b, ...}. The Fourier transform amplitude spectrum in the feature matrix is ​​stored in dictionary B, for example, B={feature region a: Fourier transform amplitude spectrum a, feature region b: Fourier transform amplitude spectrum b, ...}. When performing phase identification, it is necessary to traverse the contents of dictionary B. The results of phase identification are stored in dictionary C, with the feature region number as the key and the phase information as the value, for example, C={feature region a: m-ZrO2, feature region b: t-ZrO2, ...}.

[0030] When obtaining the physical phase area in step S5, it is only necessary to traverse the information in C in the dictionary and index the thermal map corresponding to the physical phase area from the dictionary A according to the key value. Before performing threshold segmentation on the thermal map, it is necessary to first enlarge the thermal map to the same size as the original TEM image. The threshold is pre-set through manual experience or determined by the Otsu method. The edge detection algorithm described in step S5 is a connected domain algorithm.

[0031] In a specific embodiment of the present invention, HRTEM images of in-situ oxidation experiments of commercial Zircaloy-4 were selected as method test data. The automatic processing method of the present invention was used for analysis and compared with the manual analysis results. Figure 3As shown: The TEM image shown in the figure is a high-resolution TEM image of the embodiment. The area selected by the black dotted line in the image is the area belonging to ZrO2 analyzed manually, and the area selected by the white solid line is the area belonging to ZrO2 analyzed automatically by the present invention. The two are basically consistent. Compared with the phase region division method disclosed in the Chinese patent publication number CN202211670815.1, the results of the present invention are more accurate at the boundary, such as Figure 3 shown.

[0032] The technical means disclosed in the solutions of the present invention are not limited to those disclosed in the above-mentioned embodiments, but also include technical solutions composed of any combination of the above-mentioned technical features. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An improved method for phase segmentation of crystal TEM images based on deep learning, characterized by: The method comprises the following steps: 1) Use a square sliding window to traverse the TEM image and calculate the fast Fourier transform amplitude spectrum within each window; 2) constructing the fast Fourier transform amplitude spectrum generated in step 1) into four-dimensional (4D) data, wherein the four-dimensional data comprises two real space dimensions and two reciprocal space dimensions, wherein the reciprocal space dimensions correspond to the fast Fourier transform amplitude spectrum of each subregion, and the real space dimensions correspond to the position of the sliding window in the TEM image; 3) using a principal component analysis algorithm to estimate the number n of important components in the 4D data, and then using non-negative matrix factorization to decompose the 4D data into a feature matrix and a coefficient matrix based on the number of important components, wherein the feature matrix is ​​composed of n fast Fourier transform amplitude spectra, each fast Fourier transform amplitude spectrum representing a characteristic structure, and the coefficient matrix corresponds to the feature matrix and is composed of n heat maps representing the distribution of the corresponding characteristic structures in the 4D data, and the heat maps have a one-to-one correspondence with the fast Fourier transform amplitude spectra; 4) using a phase identification algorithm to perform phase identification on the fast Fourier transform amplitude spectrum in the characteristic matrix generated in step 3); 5) The thermal map corresponding to the fast Fourier transform amplitude spectrum identified in step 4) is enlarged to the same size as the original TEM image, and a threshold segmentation algorithm is used to segment the thermal map into a binary image consisting of background and physical phase regions. The connected domain algorithm is then used to obtain the boundaries of the feature regions, and the boundaries of the physical phase regions are depicted with colored lines to complete the division of the physical phase regions.

2. The improved method for crystal TEM image phase segmentation based on deep learning according to claim 1, characterized in that: The principal component analysis algorithm in step 3) needs to generate a PCA Scree Plot when estimating the number of important components in the 4D data. The number of important components is determined based on the positions of the inflection points in the PCA Scree Plot. The number of important components represents the number of characteristic regions in the TEM image.

Images