A microscope sample drift correction method based on image processing algorithm
Through image processing-based methods, the feature points and velocity fields of the microscope image are extracted, and the image offset problem caused by sample drift in electron microscope video is solved, efficient drift correction is achieved, and the stability and accuracy of the microscope characterization results are improved. sex.
Patent Information
- Application Number
- CN202210329018.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-03-30
AI Technical Summary
When shooting videos, electron microscopes cause image shift due to sample drift, which affects the characterization results. Existing methods such as cross-correlation algorithms and the use of crystal structures to calculate the drift distance have disadvantages of large time expenditures and prior knowledge, which are difficult to effectively correct.
Using an image processing method, the feature points of the microscope image are extracted, the velocity field of the feature points is calculated and their position is estimated, the drift distance of each frame of the image is calculated, and the specific steps include semantic segmentation neural network training and SIFT feature points The use of detection and Lucas-Canade algorithm is combined with bilinear or bicubital interpolation algorithm for drift correction.
It realizes efficient correction of sample drift in microscope video, improves the stability and accuracy of characterization results, reduces calculation time, and requires no prior knowledge.
Smart Images

Figure CN114627028B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of microscope testing, and in particular to a microscope sample drift correction method based on an image processing algorithm at a nanometer scale. Background Art
[0002] Electron microscopes are widely used in research in the fields of materials, physics, chemistry, semiconductors, and life sciences for their atomic-level characterization capabilities. The imaging process of an electron microscope uses an electron beam as a light source to act on a sample, and by collecting transmitted electrons and scattered electrons at different scattering angles, an image with atomic structure information is displayed in the imaging system. Due to the high magnification of the microscope, the stability of the sample stage and sample fixation is extremely high to ensure that mechanical drift does not occur, but it is still impossible to completely avoid drift caused by factors such as thermal drift caused by electron beam irradiation, drift caused by ambient temperature, and improper operation of the operator when installing the sample. These tiny vibrations or offsets with a moving speed of several nanometers per minute will cause the characterization area to move away from the observation center, causing drift in the continuous images taken, thereby affecting the characterization results. Therefore, solving the drift problem of electron microscopes is very important for improving the characterization quality of electron microscopes. The commonly used methods for the drift problem of microscope images are: calculating the drift distance using the cross-correlation algorithm and inferring the drift distance using the crystal structure. Both methods have their own shortcomings. The cross-correlation algorithm requires pixel-by-pixel matching calculations, which is time-consuming; using the crystal structure to infer the drift distance requires prior knowledge of the crystal structure. Summary of the invention
[0003] The purpose of the present invention is to solve the problem of sample drift when operating a microscope to shoot video. A method for correcting microscope video drift based on an image using an image processing algorithm is proposed. The method extracts feature points and calculates the velocity field of the feature points for point tracking to calculate the drift distance, and has good correction capability.
[0004] A microscope sample drift correction method based on an image processing algorithm of the present invention comprises the following steps:
[0005] Step 1, extracting feature points of the first frame of the microscope image;
[0006] Step 2, calculate the velocity field of the feature point and estimate the position of the feature point in the next frame image;
[0007] Step 3, calculating the drift distance of each frame of the microscope image relative to the first frame;
[0008] Step 4: Correct each frame of the image according to the drift distance.
[0009] Furthermore, in step 1, feature points of the first frame of the microscope image are extracted. The specific steps are as follows: Step 1.1, collect microscope images, segment foreground and background, and create an image semantic segmentation dataset, which is used to train the subsequent semantic segmentation neural network;
[0010] Step 1.2, build a semantic segmentation neural network; build a neural network based on the type of data set, the size of the data set, and the equipment used to train the network.
[0011] Step 1.3, select appropriate optimization function, loss function, learning rate and number of iterations to train the neural network for semantic segmentation of microscope images according to the depth of the neural network model, the type and size of the data set;
[0012] Step 1.4: The microscope image to be corrected is input into the trained neural network model to perform foreground and background segmentation to obtain a semantically segmented image.
[0013] Step 1.5, using the feature point detection algorithm SIFT to extract the feature points of the first frame of the microscope image after semantic segmentation; this step can remove the influence of pixel changes and noise on feature point extraction and velocity field calculation.
[0014] Furthermore, the semantic segmentation neural network constructed in step 1.2 adopts any one of the structures of convolution and deconvolution, or convolution and upsampling.
[0015] The convolutional neural network block consists of two-dimensional convolution, global normalization or batch normalization, and ReLU activation function, and the deconvolutional neural network block consists of two-dimensional deconvolution, global normalization or batch normalization, and ReLU activation function;
[0016] The upsampling adopts any one of linear interpolation, bilinear interpolation, and bicubic interpolation.
[0017] Furthermore, the semantic segmentation neural network constructed in step 1.2; specifically: the first half of the semantic segmentation neural network architecture uses two layers of convolution blocks and one layer of pooling layer stacked four times and then one layer of convolution blocks; the second half of the semantic segmentation neural network uses one upsampling layer and two layers of convolution blocks stacked four times and then one layer of two-dimensional convolution layer;
[0018] The convolution block is constructed in sequence by a two-dimensional convolution layer, a global normalization layer, and a ReLU activation function layer; the pooling layer is any one of average pooling and maximum pooling.
[0019] Furthermore, the loss function of training in step 1.2 adopts any one of BCELoss or BCELosswithlogits; the optimization method of training adopts any one of Adam, SGD, GD, Momentum, RMSProp and Adagrad.
[0020] Furthermore, the feature point in step 1 is any one of Harris corner points, contour points, SIFT feature points, ORB feature points or Fast corner points.
[0021] Furthermore, in step 2, the method for calculating the velocity field of the feature point and estimating the position in the next frame adopts any one of the Lucas-Canade algorithm or the pyramid Lucas-Canade algorithm.
[0022] Starting from the first frame image, the two-dimensional velocity field (optical flow field) of the feature points is calculated for every two adjacent frames, and the position of the feature points in the next frame image is estimated by the calculated instantaneous velocity. The above steps are repeated on subsequent images, so that the feature points extracted from the first frame can be tracked in the entire microscope image.
[0023] Furthermore, the method for correcting each frame of the image according to the drift distance in step 4 adopts any one of direct pixel-level translation, sub-pixel bilinear interpolation algorithm, or sub-pixel bicubic interpolation algorithm.
[0024] Beneficial effects: The application scenario of the present invention is that a user operates a microscope to continuously take multiple microscope photos, and the image drift of the multiple microscope photos taken continuously can be corrected. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is the semantic segmentation result map.
[0026] Figure 2 This is the SIFT feature detection result diagram.
[0027] Figure 3 This is a graph of object drift distance in the TEM video.
[0028] Figure 4 This is a comparison picture before and after correction.
[0029] Figure 5 This is a diagram of the semantic segmentation neural network architecture of the present invention.
[0030] Figure 6 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0031] A microscope sample drift correction method based on an image processing algorithm of the present invention comprises the following steps:
[0032] Step 1: Extract feature points of the first frame of the microscope image, specifically including the following steps:
[0033] Step 1.1, collect microscope images and create an image semantic segmentation dataset.
[0034] The labels of the semantic segmentation dataset are in the form of foreground and background, with the foreground highlighted with a pixel value of 255 and the background represented by a black pixel value of 0. The size of the image and the label are both 480×480, and the number of image and label pairs is around 150.
[0035] Step 1.2: Build a semantic segmentation neural network. Build appropriate neural network layers based on the type and size of the dataset and the equipment used to train the network.
[0036] The neural network in the present invention can adopt any one of the structures of convolution and deconvolution and convolution and upsampling. In this embodiment, the convolution and upsampling structure is adopted.
[0037] The convolution block is constructed in sequence by a two-dimensional convolution layer with a convolution kernel size of 3×3, a global normalization layer, and a ReLU activation function layer; the pooling layer uses maximum pooling, retaining the maximum pixel among every four pixels; and the upsampling layer uses a linear upsampling layer. The first half of the neural network architecture uses two layers of convolution blocks and one layer of pooling layer stacked four times and then one layer of convolution blocks; the second half of the neural network uses one upsampling layer and two layers of convolution blocks stacked four times and then one layer of two-dimensional convolution layer.
[0038] The architecture of the semantic segmentation neural network is as follows Figure 5As shown in the figure, starting from the far left: (1) Perform two convolution operations on the 480*480 image with a channel number of 1. The image size after convolution is 480*480 and the number of channels is 64, that is, 64 feature maps are extracted; (2) Perform a pooling operation to reduce the image size to 240*240, and perform two convolution operations. The image size is 120*120 and the number of channels is increased to 128; (3) Perform a second pooling operation and perform two convolution operations. The image size becomes 120*120 and the number of channels is increased to 128. 256; (4) Perform the third pooling operation and perform two convolutions, the image size becomes 60*60 and the number of channels is 512; (5) Perform the fourth pooling operation and perform two convolutions, the image size becomes 30*30 and the number of channels remains 512; (6) Perform the above operation, and concatenate the result of step (5) with the result of the above operation to form a 60*60 image with a channel number of 1024, and perform two more convolutions. After convolution, the image size becomes 60*60 and the number of channels is 256. (7) The operation is the same as step (6), the image size is 120*120, and the number of channels is 128. (8) The operation is the same as step (6), the image size is 240*240, and the number of channels is 64. (9) Upsample the result of step (8) and concatenate it with the result of step 1 in the channel direction to form an image of size 480*480 and number of channels 128; perform two more convolutions to make the image size 480*480 and number of channels 64; finally perform one more convolution to restore the image to size 480*480 and number of channels 1.
[0039] After training the training set, the semantic segmentation network can pass a picture to be segmented through each network layer and output a segmented result.
[0040] Step 1.3, select appropriate optimization function, loss function, learning rate and number of iterations to train the neural network for semantic segmentation of microscope images according to the depth of the neural network model, the type and size of the data set;
[0041] Specifically, the semantic segmentation neural network in step 2 is trained using the data set in step 1. The loss function for training uses the cross entropy loss function for binary classification.
[0042] The optimization method for training adopts any one of Adam, SGD, GD, Momentum, RMSProp and Adagrad. The optimization algorithm for training in this embodiment adopts RMSProp, the number of iterations of training is 40 times, and the learning rate during training is 0.00001.
[0043] Step 1.4: Input the microscope image to be corrected into the trained neural network model to obtain the semantically segmented image. The result after semantic segmentation is as follows: Figure 1 shown.
[0044] Step 1.5, feature point detection SIFT algorithm is used to select feature points for the first frame of the microscope video, where the threshold of the Hessian matrix determinant response value is set to 100. The result of extracting SIFT feature points in the first frame of the TEM image after semantic segmentation is shown in the figure below: Figure 2 As shown:
[0045] Step 2: Starting from the first frame image of the microscope, the two-dimensional velocity field (optical flow field) of the feature point is calculated for every two adjacent frames, and the coordinates of the feature point in the next frame are estimated by the calculated instantaneous velocity. The calculation of the two-dimensional velocity field is based on the assumption that the pixels of the same object in two adjacent frames remain unchanged, that is, I(x+δx,y+δy,t+δt)=I(x,y,t). Based on this assumption, the calculation formula of the feature point velocity can be obtained: That is I x V x +I y V y =-I t Repeat the above steps on subsequent images, so that the feature points extracted from the first frame can be tracked throughout the entire microscope video.
[0046] Step 3: According to the coordinates of the feature points in each frame in step 2, the drift distance of each frame relative to the first frame can be calculated.
[0047] The average drift distance in the x direction is Where n is the number of feature points. Similarly, the average drift distance in the y direction is The drift distance of an object in a TEM image is Figure 3 As shown:
[0048] Step 4: According to the drift distance of each frame relative to the first frame calculated in step 3, drift correction can be performed on each frame. If the drift distance is an integer, pixel-level correction is used, and the pixel is directly shifted, and the removed part is filled with pixel value 0. For a decimal drift distance, a bilinear interpolation algorithm is used for pixel shifting. The two interpolation formulas in the x direction are: The linear interpolation formula in the y direction is: For the removed part, the processing method is the same as pixel-level translation. The result of drift correction is shown in Figure 4. Figure 4 There are 10 frames between two adjacent pictures. The first picture is the result before correction, and the second picture is the result after correction.
[0049] like Figure 6As shown, it is a flow chart of the present invention: first, the image in the TEM video is input for semantic segmentation, and the feature points of the first frame image after segmentation are extracted, and then it is determined whether there is a next frame image. If so, the velocity field of the feature points is calculated through the two adjacent frames of images and the position of the feature points in the next frame is estimated. Then, the drift distance of the object in the frame image relative to the first frame can be calculated and corrected. If there is no next frame, the algorithm ends.
Claims
1. A microscope sample drift correction method based on image processing algorithm, characterized in that: The steps include: Step 1, extracting feature points of the first frame of the microscope image; Step 2, calculate the velocity field of the feature point and estimate the position of the feature point in the next frame image; Step 3, calculating the drift distance of each frame of the microscope image relative to the first frame; Step 4: Correct each frame of the image according to the drift distance; In step 1, feature points of the first frame of the microscope image are extracted. The specific steps are as follows: Step 1.1, collect microscope images, segment foreground and background, and create an image semantic segmentation dataset; Step 1.2, build a semantic segmentation neural network; Step 1.3, select the optimization method and loss function to train the image semantic segmentation neural network; Step 1.4, the microscope image to be corrected is input into the trained neural network model to perform foreground and background segmentation to obtain a semantically segmented image; Step 1.5, extracting feature points of the first frame of the microscope image after semantic segmentation using a feature point detection algorithm; In step 4, the method for correcting each frame of the image according to the drift distance adopts any one of direct pixel-level translation, sub-pixel bilinear interpolation algorithm or sub-pixel bicubic interpolation algorithm.
2. A microscope sample drift correction method based on image processing algorithm according to claim 1, characterized in that: The semantic segmentation neural network built in step 1.2 adopts either a convolution and deconvolution structure or a convolution and upsampling structure.
3. A microscope sample drift correction method based on image processing algorithm according to claim 2, characterized in that: The convolutional neural network block consists of two-dimensional convolution, global normalization or batch normalization, and ReLU activation function, and the deconvolutional neural network block consists of two-dimensional deconvolution, global normalization or batch normalization, and ReLU activation function.
4. The microscope sample drift correction method based on image processing algorithm according to claim 2, characterized in that: The upsampling adopts any one of linear interpolation, bilinear interpolation, and bicubic interpolation.
5. The microscope sample drift correction method based on image processing algorithm according to claim 1, characterized in that: The semantic segmentation neural network built in step 1.2; Specifically, the first half of the semantic segmentation neural network architecture uses two layers of convolution blocks and one layer of pooling layer stacked four times and then one layer of convolution blocks; the second half of the semantic segmentation neural network uses one upsampling layer and two layers of convolution blocks stacked four times and then one layer of two-dimensional convolution layer; The convolution block is constructed in sequence by a two-dimensional convolution layer, a global normalization layer, and a ReLU activation function layer; the pooling layer is any one of average pooling and maximum pooling.
6. The microscope sample drift correction method based on image processing algorithm according to claim 1, characterized in that: The loss function for training in step 1.3 is either BCELoss or BCELosswithlogits. The optimization method for training is any one of Adam, SGD, GD, Momentum, RMSProp and Adagrad.
7. The microscope sample drift correction method based on image processing algorithm according to claim 1, characterized in that: The feature point is any one of Harris corner points, contour points, SIFT feature points, ORB feature points or Fast corner points.
8. The microscope sample drift correction method based on image processing algorithm according to claim 1, characterized in that: The method for calculating the velocity field of the feature point and estimating the position in the next frame in step 2 adopts either the Lucas-Canade algorithm or the pyramid Lucas-Canade algorithm.
Citation Information
Patent Citations
Image distortion correction method of scanning electron microscope
CN107590787A
Transmission electron microscope sample drift correction method based on artificial intelligence algorithm
CN109540946A
Rapid panoramic stitching method and system for microscopic images
CN111626936A
Systems, methods and computer programs for a microscope and microscope system
EP3879328A1
