Method for removing noise in FIBSEM noise image

Abstract

The invention provides a method for removing noise in an FIBSEM noise image, which takes a DnCNN network model as a framework, obtains an improved DnCNN network denoising model by building an initial improved DnCNN network denoising model and optimizing model parameters, and performs denoising on the FIBSEM noise image by using the improved DnCNN network denoising model, and comprises the following steps: S1, building the initial improved DnCNN network denoising model; s2, model parameter optimization: S2.1, data preprocessing; s2.2, carrying out residual learning training; s2.3, testing the model; s3, circulating; and S4, denoising. According to the method, the improved DnCNN network model is adopted, residual learning and batch normalization technologies are combined, so that the denoising process is more stable, the method can adapt to different noise levels, and noise and curtain effects caused by sample surface topography fluctuation and component difference are effectively reduced for noise removal of the FIBSEM image.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular to a method for removing noise from a FIBSEM noise image. Background Art

[0002] Focused ion beam scanning electron microscopy plays a vital role in nanoscale image acquisition. However, during the scanning process, the morphological fluctuations and composition differences of the sample surface will lead to inconsistent FIB etching rates at different locations, resulting in noise and curtain effects in FIBSEM images, which significantly affects the image quality.

[0003] At present, the cutting-edge research on FIBSEM image denoising is mainly focused on the following directions. Effective solutions usually include: using a lower beam current for cross-section refinement, depositing a protective layer on the sample surface to flatten the sample surface, and using swing cutting technology to achieve multi-angle processing of the ion beam. Among them, reducing the ion beam current can effectively reduce the curtain effect caused by FIB processing, but this method is time-consuming and cannot completely eliminate the effect. Depositing a protective layer on the sample surface can not only effectively protect the sample surface, but also overcome the curtain effect caused by surface unevenness to a certain extent (this method is basically used in the cross-section processing process), but it still cannot eliminate the curtain effect caused by differences in internal sample components. In addition, although the traditional image filtering operation can reduce noise to a certain extent, there are still large errors in the subsequent threshold segmentation and volume fraction calculation, which affects the accuracy of the component division results.

[0004] Therefore, it is urgent to develop more effective methods for denoising FIBSEM images to improve image quality and the accuracy of analysis results. Summary of the invention

[0005] The purpose of the present invention is to solve the above problems in the prior art and to propose a method for removing noise from FIBSEM noise images.

[0006] The object of the present invention can be achieved by the following technical scheme: a method for removing noise from a FIBSEM noise image, using a DnCNN network model as a framework, by building an initial improved DnCNN network denoising model and optimizing model parameters, an improved DnCNN network denoising model is obtained, and the FIBSEM noise image is denoised using the improved DnCNN network denoising model. The method for removing noise from a FIBSEM noise image comprises the following steps:

[0007] S1 builds the initial improved DnCNN network denoising model: The initial improved DnCNN network denoising model includes loading the preprocessed noise image unit, input layer, combined convolution layer and output layer. The combined convolution layer includes 1 head convolution layer, 15 stable enhanced convolution layers and 1 tail convolution layer;

[0008] The parameters of the head convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0009] The parameters of the tail convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0010] The stable enhanced convolution layer includes a convolution operation unit, a batch normalization unit, and a rectified linear unit. The ReLU activation function is used in the rectified linear unit. The parameters of the convolution operation unit are as follows: the convolution kernel size is 3×3, the step size is 1, and the number of channels is 64;

[0011] S2 Model parameter optimization: S2.1 Data preprocessing: Perform artificial denoising on the noisy images acquired by the FIBSEM system to obtain a noise-free image set, and then divide the noise-free image set into a noise-free training set and a noise-free test set using the train_test_split function;

[0012] S2.2 Residual learning training: After random cropping, geometric transformation and noise addition, the original images in the noise-free training set are subjected to an artificial noise image training set. The images in the artificial noise image training set and the original images in the noise-free training set are used as data pairs to train the initial improved DnCNN network denoising model for parameter optimization. During the training, the error is calculated using the L1 loss function, and the weight of the L1 loss function is set to 1.

[0013] S2.3 Model testing: Add the set noise to the original images in the noise-free test set to form an artificial test sample set, and use the trained initial improved DnCNN network denoising model to perform denoising operations to obtain the denoised test image set;

[0014] By comparing the images in the denoised test image set with the original images in the noise-free test set, the peak signal-to-noise ratio is used as the evaluation index to evaluate the denoising effect of the improved DnCNN network denoising model.

[0015] S3 loop: repeat steps S2.2 and S2.3 until the denoising effect reaches the expected effect, and the improved DnCNN network denoising model is obtained;

[0016] S4 denoising: The noisy target image acquired by the FIBSEM system is denoised using the improved DnCNN network denoising model to obtain the target image.

[0017] In the above-mentioned method for removing noise from FIBSEM noise images, in step S2.1, when using the train_test_split function for division, the parameter test_size=0.2 is set, and 80% of the noise-free image set data is designated as the noise-free training set data, and 20% of the noise-free image set data is designated as the noise-free sample test set data.

[0018] In the above method for removing noise from FIBSEM noise images, the loading preprocessed noise image unit uses DataLoader to load data from a custom dataset and preprocess the data, converting the image from HWC format to CHW format, converting it to a PyTorch tensor and adjusting the channel format;

[0019] The input layer parameters are as follows: the input shape is (1300, 2000, 1), and the number of input channels is set to 1.

[0020] In the above method for removing noise from FIBSEM noise images, in step S2.1, the geometric transformation is specifically flipping and rotating;

[0021] In the noise adding operation, noise at different noise levels is added according to the characteristics of vertical stripes in FIBSEM images, and the dataset size is expanded through the attention mechanism and image patch method to ensure that the model can fully learn the characteristic relationship between noise and image during training.

[0022] In the above-mentioned method for removing noise from FIBSEM noise images, in step S2.2, in a continuously running training loop, the initial improved DnCNN network denoising model repeatedly receives training data, performs forward propagation and back propagation, updates model parameters, and regularly records training information and saves the model; in order to ensure the reproducibility of the training process, a random seed is set so that the same random result can be obtained in each run.

[0023] In the above-mentioned method for removing noise from FIBSEM noise images, in step S2.1, the initial improved DnCNN network denoising model is automatically saved every 5000 training steps to save the current model parameters and the state of the optimizer.

[0024] In the above-mentioned method for removing noise from a FIBSEM noise image, in step S4, the specific process of obtaining a target image is as follows: the noisy target image obtained by using the FIBSEM system is processed by using a loading preprocessed noise image unit as input data; the input data is input into a combined convolution layer through an input layer, a head convolution layer and a stable enhanced convolution layer are used in the combined convolution layer to extract image features, and then the image features are reorganized in a tail convolution layer to generate a noisy residual image, the input image is subtracted from the residual image to obtain a denoised image, and finally output through an output layer to obtain a target image.

[0025] Compared with the prior art, the method for removing noise from FIBSEM noise images has the following beneficial effects:

[0026] The present invention utilizes residual learning and batch normalization technology to improve the training efficiency and denoising performance of the model, and significantly improves the image quality; by artificially adding noise, the diversity of training data is enhanced, so that the model can more comprehensively understand the relationship between noise and image during the learning process, improve the generalization ability of the model, enhance the model's learning ability for different noise features, and ensure the accuracy of the denoising result; in general, the present invention adopts an improved DnCNN network model, combined with residual learning and batch normalization technology, so that the denoising process is more stable, can adapt to different noise levels, and effectively reduces the noise and curtain effect caused by the surface morphology fluctuations and composition differences of the sample for noise removal of FIBSEM images, thereby improving the clarity and accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The flowchart of the development of the network model of the present invention.

[0028] Figure 2 This is a network model structure diagram of the improved DnCNN of the present invention.

[0029] Figure 3 It is a diagram showing the pixel value distribution of the noise image, clean image and residual image of the present invention.

[0030] Figure 4 It is the residual image restored by using the residual patch in the present invention.

[0031] Figure 5 (a) is the original image;

[0032] (b) is the denoised image of the DnCNN network;

[0033] (c) is the threshold segmentation image of the original image;

[0034] (d) is the threshold segmentation image after denoising by the DnCNN network. DETAILED DESCRIPTION

[0035] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0036] like Figure 1 As shown, the method for removing noise from FIBSEM noise images takes the DnCNN network model as a framework, obtains an improved DnCNN network denoising model by building an initial improved DnCNN network denoising model and optimizing model parameters, and uses the improved DnCNN network denoising model to denoise the FIBSEM noise image. The method for removing noise from FIBSEM noise images includes the following steps:

[0037] S1 builds the initial improved DnCNN network denoising model: The initial improved DnCNN network denoising model includes loading the preprocessed noise image unit, input layer, combined convolution layer and output layer. The combined convolution layer includes 1 head convolution layer, 15 stable enhanced convolution layers and 1 tail convolution layer;

[0038] The parameters of the head convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0039] The parameters of the tail convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0040] The stable enhanced convolution layer includes a convolution operation unit, a batch normalization unit, and a rectified linear unit. The ReLU activation function is used in the rectified linear unit. The parameters of the convolution operation unit are as follows: the convolution kernel size is 3×3, the step size is 1, and the number of channels is 64;

[0041] Batch Normalization: Batch normalization technology is used to stabilize the training process and prevent gradient explosion or disappearance, ensuring that the model has good stability during training.

[0042] S2 Model parameter optimization: S2.1 Data preprocessing: Perform artificial denoising on the noisy images acquired by the FIBSEM system to obtain a noise-free image set, and then divide the noise-free image set into a noise-free training set and a noise-free test set using the train_test_split function;

[0043] S2.2 Residual learning training: After random cropping, geometric transformation and noise addition, the original images in the noise-free training set are subjected to an artificial noise image training set. The images in the artificial noise image training set and the original images in the noise-free training set are used as data pairs to train the initial improved DnCNN network denoising model for parameter optimization. During the training, the error is calculated using the L1 loss function, and the weight of the L1 loss function is set to 1.

[0044] The core idea of ​​residual learning is to introduce skip connections to allow the network to directly learn the residual between input and output, rather than directly learning complex mapping relationships. In the DnCNN network, the network output is the noise residual, and the residual is subtracted from the input image to obtain the denoised image, which enhances the denoising effect and obtains a clean image.

[0045] After the original image is randomly cropped, geometrically transformed, and noise is added, an artificial noise image training set is obtained to improve the generalization ability of the model.

[0046] S2.3 Model testing: Add the set noise to the original images in the noise-free test set to form an artificial test sample set, and use the trained initial improved DnCNN network denoising model to perform denoising operations to obtain the denoised test image set;

[0047] By comparing the images in the denoised test image set with the original images in the noise-free test set, the peak signal-to-noise ratio is used as the evaluation index to evaluate the denoising effect of the improved DnCNN network denoising model.

[0048] S3 loop: repeat steps S2.2 and S2.3 until the denoising effect reaches the expected effect, and the improved DnCNN network denoising model is obtained.

[0049] S4 denoising: The noisy target image acquired by the FIBSEM system is denoised using the improved DnCNN network denoising model to obtain the target image.

[0050] In step S2.1, when using the train_test_split function for division, the parameter test_size=0.2 is set, and 80% of the noise-free image set data is designated as the noise-free training set data, and 20% of the noise-free image set data is designated as the noise-free sample test set data.

[0051] The parameter test_size=0.2 of this function specifies that 80% of the data is used for training and 20% of the data is used for testing. In order to fairly evaluate the model performance, the test data set is not involved in model training. This division ensures that the training data set accounts for the majority so that the model can learn more features, while the test set is used to evaluate the generalization ability of the model.

[0052] The Load Preprocessed Noise Image unit uses DataLoader to load data from a custom dataset and preprocess the data, converting the image from HWC format to CHW format, converting it to a PyTorch tensor and adjusting the channel format;

[0053] The input layer parameters are as follows: the input shape is (1300, 2000, 1), and the number of input channels is set to 1.

[0054] In step S2.1, the geometric transformation is specifically performed by flipping and rotating;

[0055] In the noise adding operation, noise at different noise levels is added according to the characteristics of vertical stripes in FIBSEM images, and the dataset size is expanded through the attention mechanism and image patch method to ensure that the model can fully learn the characteristic relationship between noise and image during training.

[0056] In step S2.2, in the continuously running training loop, the initial improved DnCNN network denoising model will repeatedly receive training data, perform forward propagation and back propagation, update model parameters, and regularly record training information and save the model; in order to ensure the reproducibility of the training process, a random seed is set so that the same random results can be obtained in each run.

[0057] In step S2.1, the initial improved DnCNN network denoising model is automatically saved every 5000 training steps to save the current model parameters and optimizer status.

[0058] In step S4, the specific process of obtaining the target image is as follows: the noisy target image obtained by the FIBSEM system is processed by the loading preprocessed noise image unit as input data; the input data is input into the combined convolution layer through the input layer, the head convolution layer and the stable enhanced convolution layer are used to extract image features in the combined convolution layer, and then the image features are reorganized in the tail convolution layer to generate a noisy residual image, the input image is subtracted from the residual image to obtain a denoised image, and finally output through the output layer to obtain the target image.

[0059] For example, the following operations:

[0060] This embodiment aims to describe the detailed process of FIBSEM image denoising using the improved DnCNN network. Figure 1 , which is the development process of the network model.

[0061] 1. Data acquisition and preprocessing: The original image was acquired under the FIBSEM imaging system. The image format was HWC (height, width, channel) format with a size of 2000 × 1300. When loading the data, the noisy image was converted into a PyTorch tensor and adjusted to the CHW (channel, height, width) format. After that, the image was divided into multiple 40 × 40 small image blocks and noise was added.

[0062] 2. Model workflow: Figure 2To improve the network model structure diagram of DnCNN. The noisy target image obtained by the FIBSEM system is processed by the loading preprocessed noise image unit as input data; the input data is input into the combined convolution layer through the input layer, and the head convolution layer and the stable enhanced convolution layer are used in the combined convolution layer to extract image features, and then the image features are reorganized in the tail convolution layer to generate a noisy residual image. The residual image is the noise feature learned by the network, representing the noise pattern recognized by the network. The input image is subtracted from the residual image to obtain the denoised image, and finally output through the output layer to obtain the target image.

[0063] In the denoising process of improving the DnCNN network, Figure 3 The pixel value distribution of the noisy image, clean image, and residual image is shown. Figure 4 The residual image restored by the residual patch helps us understand the idea of ​​residual learning and the process of generating residual images, so as to observe the noise distribution characteristics of network learning. The "noise image" minus the "residual image" gives the "denoised image", that is, the denoised image completed by residual learning.

[0064] 3. Result analysis and application: The "denoised image" is used for subsequent analysis and characterization of pore characteristics. First, the difference between the original image and the denoised image is analyzed from the two values ​​of PSNR and SSIM. After calculation, the PSNR is 32dB, indicating that the error between the denoised image and the original clean image is small and the image quality is high. The SSIM is 0.77, indicating that the denoised image has a good similarity with the original image in terms of structure, brightness and contrast, reflecting the effectiveness of the network in retaining image structure information.

[0065] Secondly, the image denoising effect is evaluated from the aspect of porosity. Figure 5 , (a) (b) are the original image and the improved DnCNN network denoising image, respectively, (c) (d) are the threshold segmentation images of the original image and the threshold segmentation images after denoising by the improved DnCNN network. The artificially calculated porosity is 6.1%. It is compared with the real data. The noise is removed through the denoising process. The porosity of the "denoised image" is 6.3%, while the porosity of the original image is 7.9%. In comparison, the porosity of the image denoised by the improved DnCNN network model is closer to the real data, showing the pore structure more clearly, which is convenient for further pore connectivity analysis and feature extraction.

[0066] The specific embodiments described herein are merely examples of the spirit of the present invention. A person skilled in the art of the present invention may make various modifications or supplements to the specific embodiments described or replace them in a similar manner, but will not deviate from the spirit of the present invention or exceed the defined scope. Although the present invention is described and described in detail in the drawings and the foregoing description, such descriptions and descriptions are considered to be illustrative or exemplary rather than restrictive. It should be understood that within the scope of the following claims, changes and modifications may be made by a person of ordinary skill. Specifically, the present invention encompasses additional embodiments having any combination of features from the above-mentioned different embodiments. With respect to the use of the expression "generally" or "substantially", this patent application should be understood to disclose that these features and values ​​are also fully satisfied, i.e., there is no aforementioned characterization as "generally" or "substantially".

Claims

1. A method for removing noise from a FIBSEM noisy image, characterized in that: Taking the DnCNN network model as the framework, an improved DnCNN network denoising model is obtained by building an initial improved DnCNN network denoising model and optimizing model parameters. The improved DnCNN network denoising model is used to denoise the FIBSEM noise image. The method for removing noise from the FIBSEM noise image includes the following steps: S1 builds the initial improved DnCNN network denoising model: The initial improved DnCNN network denoising model includes loading the preprocessed noise image unit, input layer, combined convolution layer and output layer. The combined convolution layer includes 1 head convolution layer, 15 stable enhanced convolution layers and 1 tail convolution layer; The parameters of the head convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64; The parameters of the tail convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64; The stable enhanced convolution layer includes a convolution operation unit, a batch normalization unit, and a rectified linear unit. The ReLU activation function is used in the rectified linear unit. The parameters of the convolution operation unit are as follows: the convolution kernel size is 3×3, the step size is 1, and the number of channels is 64; S2 Model parameter optimization: S2.1 Data preprocessing: Perform artificial denoising on the noisy images acquired by the FIBSEM system to obtain a noise-free image set, and then divide the noise-free image set into a noise-free training set and a noise-free test set using the train_test_split function; S2.2 Residual learning training: After random cropping, geometric transformation and noise addition, the original images in the noise-free training set are subjected to an artificial noise image training set. The images in the artificial noise image training set and the original images in the noise-free training set are used as data pairs to train the initial improved DnCNN network denoising model for parameter optimization. During the training, the error is calculated using the L1 loss function, and the weight of the L1 loss function is set to 1. S2.3 Model testing: Add the set noise to the original images in the noise-free test set to form an artificial test sample set, and use the trained initial improved DnCNN network denoising model to perform denoising operations to obtain the denoised test image set; By comparing the images in the denoised test image set with the original images in the noise-free test set, the peak signal-to-noise ratio is used as an evaluation index to evaluate the denoising effect of the improved DnCNN network denoising model. S3 loop: repeat steps S2.2 and S2.3 until the denoising effect reaches the expected effect, and the improved DnCNN network denoising model is obtained; S4 denoising: The noisy target image acquired by the FIBSEM system is denoised using the improved DnCNN network denoising model to obtain the target image.

2. The method for removing noise from a FIBSEM noise image according to claim 1, characterized in that: In step S2.1, when using the train_test_split function for division, the parameter test_size=0.2 is set, and 80% of the noise-free image set data is designated as the noise-free training set data, and 20% of the noise-free image set data is designated as the noise-free sample test set data.

3. The method for removing noise from a FIBSEM noise image according to claim 1, characterized in that: The Load Preprocessed Noise Image unit uses DataLoader to load data from a custom dataset and preprocess the data, converting the image from HWC format to CHW format, converting it to a PyTorch tensor and adjusting the channel format; The input layer parameters are as follows: the input shape is (1300, 2000, 1), and the number of input channels is set to 1.

4. The method for removing noise from a FIBSEM noise image according to claim 1, characterized in that: In step S2.1, the geometric transformation is specifically performed by flipping and rotating; In the noise adding operation, noise at different noise levels is added according to the characteristics of vertical stripes in FIBSEM images, and the dataset size is expanded through the attention mechanism and image patch method to ensure that the model can fully learn the characteristic relationship between noise and image during training.

5. The method for removing noise from a FIBSEM noise image according to claim 1, characterized in that: In step S2.2, in the continuously running training loop, the initial improved DnCNN network denoising model will repeatedly receive training data, perform forward propagation and back propagation, update model parameters, and regularly record training information and save the model; in order to ensure the reproducibility of the training process, a random seed is set so that the same random results can be obtained in each run.

6. The method for removing noise from a FIBSEM noise image according to claim 1, characterized in that: In step S2.1, the initial improved DnCNN network denoising model is automatically saved every 5000 training steps to save the current model parameters and optimizer status.

7. The method for removing noise from a FIBSEM noise image according to claim 1, characterized in that: In step S4, the specific process of obtaining the target image is as follows: the noisy target image obtained by the FIBSEM system is processed by the loading preprocessed noise image unit as input data; the input data is input into the combined convolution layer through the input layer, the head convolution layer and the stable enhanced convolution layer are used to extract image features in the combined convolution layer, and then the image features are reorganized in the tail convolution layer to generate a noisy residual image, the input image is subtracted from the residual image to obtain a denoised image, and finally output through the output layer to obtain the target image.

Images