Atomic force microscope image restoration method based on long short-term memory network

Abstract

The present invention discloses an atomic force microscope image restoration method of a long short-term memory network. The method of the present invention first uses the dilation method in mathematical morphology to obtain a simulation sample, then normalizes the simulation sample, inputs it into an LSTM model for training, and uses an improved MAE loss function and an Adam optimization algorithm to improve the LSTM model, and then performs regularization and denormalization on the training sample. Finally, the actual AFM image is input into the trained model to obtain an AFM restored image, and a step-by-step interpolation adaptive algorithm is used for processing, so as to accurately obtain the position of the feature point in the sample. The present invention has stability and good robustness, and can effectively restore the AFM image, thereby improving the measurement accuracy.

Description

Technical Field

[0001] The invention relates to the setting of standard instruments such as grids and steps in nano-metrology, and to an atomic force microscope image restoration method based on a long short-term memory (LSTM) network. Background Art

[0002] Since the atomic force microscope (AFM) uses the interaction force between a tiny probe and the object to be measured for imaging, it is widely used in the field of nanoscience metrology. AFM obtains relevant geometric parameters by obtaining high-resolution imaging of nano-metrology standard instruments and calibrates them, thereby realizing the transfer of measurement values ​​from standard metrology instruments to working metrology instruments. However, during the scanning imaging process of AFM, due to the influence of the needle tip, the image obtained by scanning is the result of the combined action of the atomic force probe and the sample, rather than a true description of the sample morphology. This phenomenon is the "artifact effect" of AFM. This phenomenon reduces the accuracy of the measurement results of standard devices and affects the transfer and traceability of measurement values ​​in nanoscience metrology.

[0003] From the perspective of improving the accuracy of measuring instruments, probes with smaller curvature radii can be used to reduce the "artifact effect", but such probes often have poor durability, high costs, and cannot eliminate the "artifact effect" in principle. Since nanomaterials have high stiffness and hardness and are not prone to deformation and wear, AFM uses contact measurement when scanning samples. Based on this measurement method, the true surface of the sample can be obtained through a certain algorithm based on the imaging of the sample. However, this deconvolution process is non-deterministic and the image cannot be restored using an accurate mathematical expression, so the process has high requirements for algorithm selection.

[0004] At present, the main algorithms for restoring AFM images include restoration algorithms based on radial basis function neural networks and multi-layer perceptrons, probe blind reconstruction algorithms based on mathematical morphology, and back propagation neural network algorithms. Among them, the probe blind reconstruction algorithm often over-relies on the AFM measurement image, which makes the robustness of the reconstructed probe morphology poor, affecting the restoration of the AFM image. The radial basis function neural network and back propagation neural network algorithms are subject to the upper limit of the depth of the neural network during training, and are often accompanied by the problem of gradient decay or explosion leading to training failure. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a method for restoring AFM images based on a long short-term memory (LSTM) recurrent neural network.

[0006] The technical solution adopted by the present invention is:

[0007] An atomic force microscope image restoration method based on long short-term memory network comprises the following steps:

[0008] Step 1: Use the dilation method in mathematical morphology to obtain simulation samples;

[0009] Assume that the expression of the probe cross-sectional profile shape when the needle tip is located at the origin position in the coordinate system is s(x), and the expression of the sample cross-sectional profile shape is f(x). The needle tip's horizontal coordinate is used to define the position of the probe in the scanning line. Assuming that the needle tip's horizontal coordinate is x0, the needle tip's vertical coordinate h represents the height value measured at the sample scanning point at x0. At this time, assuming that the expression of the probe cross-sectional profile shape is t(x), the relationship between s(x) and t(x) satisfies the expression

[0010] t(x)=s(x)+h (1)

[0011] The relationship between f(x) and t(x) satisfies the expression

[0012]

[0013] AFM contact measurement is equivalent to the expansion operation in mathematical morphology. The mathematical expression of the expansion operation is:

[0014]

[0015] Therefore, according to formula (3), the expression of h can be written as

[0016]

[0017] Combining equation (1) and equation (2), equation (3) can be transformed into

[0018]

[0019] Since h and x satisfy a one-to-one mapping relationship, h can be regarded as a function of x, and h(x) is used to represent the change curve of the height result measured by the probe at a certain horizontal scanning line of the sample, that is, the result of each row of the simulation image. h(x) contains the relationship between the sample profile, the needle tip profile and the measured image.

[0020] The simulation process uses a nanogrid standard device as the sample to be tested and a silicon nitride needle tip as the scanning device. The simulation scanning process can obtain 2000 sets of sample images, of which 1600 sets of images are used as training sets and 400 sets of images are used as test sets. The sample selection of the training set needs to include all probe types, but not all nanogrid types.

[0021] Step 2: Normalize the simulated sample image obtained in step 1;

[0022] The AFM image is decomposed row by row to achieve dimensionality reduction, so that the row vectors become training samples of the network.

[0023] Since the AFM simulation image is a 140×140 matrix, the input data and output results are both 1×140 row vectors. In order to suppress the phenomenon of non-convergence and overfitting during the training process and thus improve the stability and performance of the neural network, the input data is normalized, that is,

[0024]

[0025] In formula (6), x is the value in the input matrix, x max and x min are the maximum and minimum values ​​in the input matrix, respectively, * is the result after the input data is normalized.

[0026] Step 3: Input the data obtained in step 2 into the LSTM model for training;

[0027] The LSTM model contains a forget gate F t , Input Gate I t , output gate O t and candidate memory cells Four parts. The gate input of LSTM is the current time step input X t With the previous time step hidden state H t-1 , which matches the gradient of each point in the AFM scan line determined by the height value of the adjacent point. The output is calculated by the fully connected layer of the sigmoid activation function, and its expression is

[0028]

[0029] In order to obtain candidate memory cells and have a larger gradient when the AFM scan line input is near 0, and the model converges faster, the tanh activation function is used, and its expression is

[0030]

[0031] LSTM can store high-dimensional AFM scan line vector information through memory cells, thereby obtaining the degree of correlation between each scan point. For example, the correlation of points in the flat area of ​​the scan line is weak, while the correlation of points in the area with large changes in the scan line height is strong. Suppose the memory cell C in the previous time step t-1 , the current time step memory cell C t , in obtaining C t When F t and I t The Hadamard product of the matrix controls the flow of information between points in the scan line:

[0032]

[0033] If F t The elements in are always close to 1 and I t The elements in are always approximately 0, and the past memory cells will be saved and passed to the current time step. This process can preserve the effective information in the AFM scan line for a long time, solve the gradient attenuation problem caused by the increase in the dimension of the scan line vector, and improve the reliability of training. At the same time, the dependency between points with large spacing in the scan line can be obtained, and image restoration in areas with severe "artifact effects" can be achieved.

[0034] In getting C t Afterwards, through O t To control from H t The flow of information to the hidden state at the current time step:

[0035] H t =O t ⊙tanh(C t ) (10)

[0036] For the nanometer grid scanning line, when O t When the elements in are approximately 0 or equal to 0, C t The information in the nanogrid is often only the information of the flat area of ​​the nanogrid, not the required characteristic information, and is generally reserved for one's own use; t When the elements in are far away from 0, then C t contains characteristic information such as nano-grid step points and step positions, which will be transmitted to H t For use by the output layer.

[0037] Step 4: Use the MSE loss function and Adam optimization algorithm to continuously improve the LSTM model;

[0038] The modified mean absolute error (MMAE) is used as the loss function to evaluate the restoration effect of AFM images, and the 1-norm is used in the summation process. The expression of the loss function is:

[0039]

[0040] In formula (11), n ​​represents the number of training samples, l represents the vector dimension of the network output, Represents the estimated value of each row of the AFM restored image, y i Represents the true value of each row of the AFM restored image.

[0041] The learning rate parameter adaptation formula of the Adam algorithm is:

[0042]

[0043] In formula (12), β 0 represents the initial given hyperparameter, β i represents the hyperparameters after iteration i.

[0044] Step 5: Regularize and denormalize the training samples;

[0045] For the output layer after the LSTM model, the Dropout regularization method is used to randomly discard some output features of this layer during the training process, thereby reducing the overfitting phenomenon and making the trained model suitable for more types of AFM scan sample images.

[0046] All sample data in the LSTM model training process are in the interval [0,1], and the final restored image needs to be close to the actual sample, so all data needs to be denormalized before obtaining the output result.

[0047] Step 6: Input the actual AFM image into the trained model to obtain the AFM restored image;

[0048] For the AFM restored image, an adaptive algorithm with step-by-step interpolation is used to obtain the position of the characteristic points in the sample more accurately.

[0049] Beneficial effects of the present invention: After simulating the scanning of samples to collect AFM simulation images, the present invention obtains the AFM image deconvolution process through LSTM model training, and finally uses the model for actual AFM image restoration. The present invention can solve the problems of cost consumption and image quality degradation caused by needle tip wear in the process of obtaining actual large-scale AFM images. Using the LSTM model for deconvolution operation helps to repair the locations with serious "artifact effects" in the AFM image, and at the same time, it also solves the problem of gradient disappearance or explosion during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the principle of obtaining the AFM simulation image in the present invention.

[0051] Figure 2 This is a three-dimensional image of a simulated needle tip using the method of the present invention.

[0052] Figure 3 This is a three-dimensional image of a nano-grating simulated by the method of the present invention.

[0053] Figure 4 It is a three-dimensional diagram of the result of simulating nano-grating scanning by the method of the present invention.

[0054] Figure 5 It is an algorithm flow chart of the method of the present invention.

[0055] Figure 6It is a structural diagram of the LSTM model of the method of the present invention.

[0056] Figure 7 It is a graph of loss value-iteration number in neural network training according to the method of the present invention.

[0057] Figure 8 It is a top view of the nano grating in the test results of the method of the present invention. DETAILED DESCRIPTION

[0058] The present invention simulates the process of AFM scanning the nano-grid of the nano-metering standard instrument through Matlab simulation to obtain neural network training samples. Afterwards, the scanning height curves of each row in the measurement image of the nano-grid by AFM are equivalent to time series samples, which are input into the constructed LSTM framework for training to obtain a prediction model, and then the test data is input into the model to obtain the prediction results, which are compared with the actual results, proving that the model can effectively restore the actual morphology of the sample, which has practical significance for processing AFM scanning data and improving measurement accuracy, and can achieve docking with related AFM imaging software.

[0059] The present invention comprises the following steps:

[0060] Step 1: Use the dilation method in mathematical morphology to obtain simulation samples;

[0061] The scanning process of AFM contact measurement does not need to consider the tiny deformation caused by the interaction force between the probe and the sample. Assume that when the needle tip is located at the origin of the coordinate system, the expression of the cross-sectional profile of the probe is s(x), and the expression of the cross-sectional profile of the sample is f(x). The horizontal coordinate of the needle tip is used to define the position of the probe in the scanning line. Assuming that the horizontal coordinate of the needle tip is x0, the vertical coordinate h of the needle tip represents the height value measured at the sample scanning point at x0. At this time, assuming that the expression of the cross-sectional profile of the probe is t(x), the relationship between s(x) and t(x) satisfies the expression

[0062] t(x)=s(x)+h (1)

[0063] The relationship between f(x) and t(x) satisfies the expression

[0064]

[0065] AFM contact measurement is equivalent to the expansion operation in mathematical morphology. The mathematical expression of the expansion operation is:

[0066]

[0067] Therefore, according to formula (3), the expression of h can be written as

[0068]

[0069] Combining equation (1) and equation (2), equation (3) can be transformed into

[0070]

[0071] Since h and x satisfy a one-to-one mapping relationship, h can be regarded as a function of x, and h(x) is used to represent the change curve of the height result measured by the probe at a certain horizontal scanning line of the sample, that is, the result of each row of the simulation image. h(x) contains the relationship between the sample profile, the needle tip profile and the measured image.

[0072] In the simulation process, nano-grid standard devices were selected as the tested samples and silicon nitride needle tips were used as scanning devices.

[0073] In order to better obtain the characteristic vectors of nanogrids and probes in training, the following scheme can be used to set the parameters of the two: the nanogrid line width W is 20-40nm, the spacing is 5nm, the height H is 10-30nm, the spacing is 5nm, a total of 25 nanogrid types are set, the corresponding matrix size is 140×140, and the resolution is 1nm / pixel; the silicon nitride needle tip curvature radius R is 10-30nm, the spacing is 5nm, the cone angle θ is 5°-80°, the spacing is 5nm, a total of 80 needle tip types are set, the corresponding matrix size is 30×30, and the resolution is 1nm / pixel. The simulation scanning process can obtain 2000 sets of sample images, of which 1600 sets of images are used as training sets and 400 sets of images are used as test sets. At the same time, since in the actual scanning process, it is often believed that the probe types are limited, and the nanogrid morphology is infinite, the sample selection of the training set needs to include all probe types, but not all nanogrid types.

[0074] Step 2: Normalize the simulated sample image obtained in step 1;

[0075] Since the AFM image obtained in step 1 is equivalent to a set of row vectors, the AFM image can be decomposed row by row to achieve dimensionality reduction, so that the row vectors become the training samples of the network. In this process, since the AFM simulation image is a 140×140 matrix, the input data and output results are both 1×140 row vectors. In order to suppress the phenomenon of non-convergence and overfitting during the training process, and thus improve the stability and performance of the neural network, the input data needs to be normalized, that is,

[0076]

[0077] In formula (6), x is the value in the input matrix, x max and x min are the maximum and minimum values ​​in the input matrix, respectively, *is the result after the input data is normalized.

[0078] Step 3: Input the data obtained in step 2 into the LSTM model for training;

[0079] Since the convolution process in the AFM scanning sample is caused by the gradient change of the sample morphology and has nothing to do with the height value of the sample itself, the focus of the AFM image deconvolution process is to study the changes in the sample scanning height curve. For the input matrix, if the position of each data in the matrix is ​​changed, the data in the output matrix itself will change, not just the spatial change. Therefore, the data in the input matrix is ​​ordered and can be regarded as a time series. At the same time, since the input matrix and the output matrix are of the same size, the LSTM model is suitable for training this process.

[0080] The LSTM model contains a forget gate F t , Input Gate I t , output gate O t and candidate memory cells Four parts. The gate input of LSTM is the current time step input X t With the previous time step hidden state H t-1 , which matches the gradient of each point in the AFM scan line determined by the height value of the adjacent point. The output is calculated by the fully connected layer of the sigmoid activation function, and its expression is

[0081]

[0082] In order to obtain candidate memory cells and have a larger gradient when the AFM scan line input is near 0, and the model converges faster, the tanh activation function is used, and its expression is

[0083]

[0084] LSTM can store high-dimensional AFM scan line vector information through memory cells, thereby obtaining the degree of correlation between each scan point. For example, the correlation of points in the flat area of ​​the scan line is weak, while the correlation of points in the area with large changes in the scan line height is strong. Suppose the memory cell C in the previous time step t-1 , the current time step memory cell C t , in obtaining C t When F t and I t The Hadamard product of the matrix controls the flow of information between points in the scan line:

[0085]

[0086] If F t The elements in are always close to 1 and It The elements in are always approximately 0, and the past memory cells will be saved and passed to the current time step. This process can preserve the effective information in the AFM scan line for a long time, solve the gradient attenuation problem caused by the increase in the dimension of the scan line vector, and improve the reliability of training. At the same time, the dependency between points with large spacing in the scan line can be obtained, and image restoration in areas with severe "artifact effects" can be achieved.

[0087] In getting C t After that, you need to pass O t To control from H t The flow of information to the hidden state at the current time step:

[0088] H t =O t ⊙tanh(C t ) (10)

[0089] For the nanometer grid scanning line, when O t When the elements in are approximately 0 or equal to 0, C t The information in the nanogrid is often only the information of the flat area of ​​the nanogrid, not the required characteristic information, and is generally reserved for one's own use; t When the elements in are far away from 0, then C t contains characteristic information such as nano-grid step points and step positions, which will be transmitted to H t For use by the output layer.

[0090] Step 4: Use the MSE loss function and Adam optimization algorithm to continuously improve the LSTM model;

[0091] In the process of adjusting the LSTM model to achieve gradient descent, since the output AFM scanning behavior is in the form of a vector, the modified mean absolute error (MMAE) is used as the loss function to evaluate the restoration effect of the AFM image, and the 1-norm is used in the summation process. The expression of the loss function is

[0092]

[0093] In formula (11), n ​​represents the number of training samples, l represents the vector dimension of the network output, Represents the estimated value of each row of the AFM restored image, y i Represents the true value of each row of the AFM restored image.

[0094] In the iterative process of gradient descent, the Adam optimization algorithm is used. The Adam algorithm performs an exponentially weighted moving average on the small batch stochastic gradient, corrects the deviation of some variables, and uses an adaptive learning rate to continuously optimize the network parameters. Compared with the stochastic gradient descent method, this algorithm can more effectively update the network weights and ultimately speed up the convergence speed. The learning rate parameter adaptive formula of the Adam algorithm is:

[0095]

[0096] In formula (12), β 0 represents the initial given hyperparameter, β i represents the hyperparameters after iteration i.

[0097] Step 5: Regularize and denormalize the training samples;

[0098] For the output layer after the LSTM model, the Dropout regularization method is used to randomly discard some output features of this layer during the training process, thereby reducing the overfitting phenomenon and making the trained model suitable for more types of AFM scan sample images.

[0099] All sample data in the LSTM model training process are in the interval [0,1], and the final restored image needs to be close to the actual sample, so all data needs to be denormalized before obtaining the output result.

[0100] Step 6: Input the actual AFM image into the trained model to obtain the AFM restored image;

[0101] For the AFM restored image, an adaptive algorithm with step-by-step interpolation is used to obtain the position of the characteristic points in the sample more accurately.

[0102] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0103] Example:

[0104] Step 1: Use the dilation method in mathematical morphology to obtain simulation samples;

[0105] according to Figure 1The AFM simulation image schematic shown in the figure uses the following scheme to set the parameters: the nano-grating line width W is 20-40nm, the spacing is 5nm, the height H is 10-30nm, the spacing is 5nm, the corresponding matrix size is 140×140, and the resolution is 1nm / pixel; the silicon nitride tip curvature radius R is 10-30nm, the cone angle θ is 5°-80°, the spacing is 5nm, the corresponding matrix size is 30×30, and the resolution is 1nm / pixel. Of the 2000 sets of sample images obtained in the simulation scanning process, 1600 sets of images are used as training sets and 400 sets of images are used as test sets. The sample selection of the training set must include all probe types. The simulated tip model, simulated nano-grating model, and simulated AFM image are shown as follows: Figure 2 , Figure 3 and Figure 4 shown.

[0106] After obtaining the AFM simulation image, the sample needs to be trained. The algorithm process used is as follows: Figure 5 shown.

[0107] Step 2: Normalize the simulated sample image obtained in step 1.

[0108] Step 3: Input the data obtained in step 2 into the LSTM model for training.

[0109] The LSTM model consists of four parts: forget gate, input gate, output gate, and candidate memory cells. The gate input of LSTM is the current time step input and the previous time step hidden state, and the output is obtained by the fully connected layer of the sigmoid function. In the memory layer, the current time step memory cell is obtained by the Hadamard product of the matrix, and the current time step hidden state is obtained by the tanh function, see Figure 6 .

[0110] Step 4: Use the improved MAE loss function and Adam optimization algorithm to continuously improve the LSTM model.

[0111] The loss value-iteration number curve during training is as follows Figure 7 shown.

[0112] Step 5: Regularize and denormalize the training samples;

[0113] For the output layer after the LSTM model, the Dropout regularization method is used to randomly discard the output features of this layer during the training process, thereby reducing the overfitting phenomenon and making the trained model suitable for more types of AFM images.

[0114] Step 6: Input the actual AFM image into the trained model to obtain the AFM restored image;

[0115] The output result obtained by inputting the actual AFM image into the trained model is processed using a step-by-step interpolation adaptive algorithm, so that the position of the feature points in the sample can be obtained more accurately.

[0116] The test results of the method of the present invention are as follows Figure 8 As shown in the figure, (a) is a top view of an actual nano-grating sample, (b) is a top view of an actual nano-grating sample after AFM scanning, and (c) is a top view of the AFM restored image obtained by the trained model of (b). At this point, the atomic force microscope image restoration is achieved.

[0117] The above is only a preferred specific implementation of the method of the present invention, but the protection scope of the method of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the method of the present invention should be included in the protection scope of the method of the present invention. Therefore, the protection scope of the method of the present invention should be based on the protection scope of the claims.

Claims

1. Atomic force microscope image restoration method based on long short-term memory network, It is characterized in that Follow the steps below to implement it: Step 1: Use the dilation method in mathematical morphology to obtain simulation samples; Step 2: Normalize the simulation samples; Step 3: Input the normalized simulation samples into the LSTM model for training; The LSTM model includes four parts: a forget gate, an input gate, an output gate, and candidate memory cells; the gate input of the LSTM is the current time step input and the previous time step hidden state, and the output is obtained by the fully connected layer of the sigmoid function; In the memory layer, the memory cell of the current time step is obtained by the Hadamard product of the matrix, thereby controlling the flow of information in the row vector of the AFM image, and the hidden state of the current time step is obtained by the tanh function; Step 4: Use the MSE loss function and Adam optimization algorithm to continuously improve the LSTM model; Step 5: Regularize and denormalize the training samples; Step 6: Input the actual AFM image into the trained model to obtain the AFM restored image; Wherein, the expression of the tanh function is: LSTM stores the AFM scan line vector information with a dimension of 1×140 through memory cells, thereby obtaining the degree of correlation between each scanning point; set the memory cell C of the previous time step t-1 , the current time step memory cell C t , in obtaining C t When F t and I t The Hadamard product of the matrix controls the flow of information between points in the scan line: If F t The elements in are always close to 1 and I t The elements in are always approximately 0, and the past memory cells will be saved through time and passed to the current time step; In getting C t Afterwards, through O t To control from H t The flow of information to the hidden state at the current time step: H t =O t ⊙tanh(C t ) For the nanometer grid scanning line, when O t When the elements in are approximately 0 or equal to 0, C t The information in is the information of the flat area of ​​the nano grid; when O t When the elements in are far away from 0, then C t contains the nano-grid step point and step position feature information, which will be transmitted to H t For use by the output layer; The process of obtaining the simulation sample in step 1 is as follows: The three-dimensional process of AFM contact measurement is converted into a two-dimensional process, and the binary function representing the process is converted into a univariate function; that is, s(x) is used to represent the cross-sectional profile of the probe when the needle tip is located at the origin in the coordinate system, and f(x) is the cross-sectional profile of the sample. Then, the sample height h(x) obtained by AFM scanning is related to the relationship between f(x) and s(x); Set the parameters of the simulated nanogrid and simulated silicon nitride needle tip; The expansion method in mathematical morphology is used for simulation to obtain multiple sets of training samples; In step 4, in order to satisfy the requirement that the output result is in vector form, the improved MAE loss function MMAE and Adam optimization algorithm are used to continuously improve the LSTM model; the expression of the MMAE loss function is: Where n represents the number of training samples, l represents the vector dimension of the network output, Represents the estimated value of each row of the AFM restored image, y i Represents the true value of each row of the AFM restored image; The learning rate parameter adaptation formula of the Adam algorithm is: where β 0 represents the initial given hyperparameter, β i represents the hyperparameters after iteration i.

2. The method for restoring an atomic force microscope image based on a long short-term memory network according to claim 1, It is characterized in that The normalization process in step 2 is as follows: The simulation samples obtained in step 1 are equivalent to a set of row vectors, and the AFM image is decomposed row by row to achieve dimensionality reduction, so that the row vectors become training samples of the network; in this process, the input data and output results are both 1×140 row vectors, and the input data is normalized.

3. The method for restoring an atomic force microscope image based on a long short-term memory network according to claim 1, It is characterized in that The regularization and denormalization processing of the training samples in step 5 is as follows: For the output layer after the LSTM model, the Dropout regularization method is used to randomly discard some output features of this layer during the training process, thereby reducing the overfitting phenomenon and making the trained model applicable to more types of scanned sample images; In order to make the AFM restored image close to the actual sample, all data are denormalized before obtaining the output results.

Images