A method for removing noise from FIBSEM noisy images

Abstract

The present invention provides a method for removing noise from noisy FIBSEM images. Using a DnCNN network model as a framework, the method constructs an initial improved DnCNN network denoising model and optimizes model parameters to obtain an improved DnCNN network denoising model. The improved DnCNN network denoising model is then used to denoise noisy FIBSEM images. The method comprises the following steps: S1, constructing the initial improved DnCNN network denoising model; S2, optimizing model parameters; S2.1, data preprocessing; S2.2, residual learning training; S2.3, model testing; S3, looping; and S4, denoising. The method utilizes the improved DnCNN network model, combined with residual learning and batch normalization techniques, to achieve a more stable denoising process that can adapt to varying noise levels. Specifically for denoising FIBSEM images, the method effectively reduces noise and the curtain effect caused by sample surface topography fluctuations and compositional differences.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular to a method for removing noise from a FIBSEM noise image. Background Art

[0002] Focused ion beam scanning electron microscopy (FIB) technology plays a crucial role in nanoscale image acquisition. However, during the scanning process, surface topography and compositional variations in the sample can lead to inconsistent FIB etching rates at different locations, resulting in noise and curtain effects in FIBSEM images, significantly affecting image quality.

[0003] At present, the cutting-edge research on FIBSEM image denoising is mainly concentrated in the following directions. Effective solutions usually include: using a lower beam current for cross-section refinement, depositing a protective layer on the sample surface to smooth the sample surface, and using swing cutting technology to achieve multi-angle processing of the ion beam. Among them, reducing the ion beam current can effectively reduce the curtain effect caused by FIB processing, but this method is time-consuming and cannot completely eliminate the effect. Depositing a protective layer on the sample surface can not only effectively protect the sample surface, but also overcome the curtain effect caused by surface unevenness to a certain extent (this method is basically used in the cross-section processing process), but it still cannot eliminate the curtain effect caused by differences in the internal composition of the sample. In addition, although traditional image filtering operations can reduce noise to a certain extent, there are still large errors in the subsequent threshold segmentation and volume fraction calculation, which affects the accuracy of the component division results.

[0004] Therefore, it is urgent to develop more effective methods to denoise FIBSEM images in order to improve image quality and the accuracy of analysis results. Summary of the Invention

[0005] The purpose of the present invention is to address the above problems in the existing technology and propose a method for removing noise from FIBSEM noise images.

[0006] The object of the present invention can be achieved by the following technical solutions: a method for removing noise from FIBSEM noisy images, using a DnCNN network model as a framework, building an initial improved DnCNN network denoising model and optimizing model parameters to obtain an improved DnCNN network denoising model, and using the improved DnCNN network denoising model to denoise FIBSEM noisy images. The method for removing noise from FIBSEM noisy images includes the following steps:

[0007] S1 builds the initial improved DnCNN network denoising model: The initial improved DnCNN network denoising model includes loading the preprocessed noise image unit, input layer, combined convolution layer and output layer. The combined convolution layer includes 1 head convolution layer, 15 stable enhanced convolution layers and 1 tail convolution layer;

[0008] The parameters of the head convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0009] The parameters of the tail convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0010] The stable enhanced convolution layer includes a convolution operation unit, a batch normalization unit, and a rectified linear unit. The ReLU activation function is used in the rectified linear unit. The parameters of the convolution operation unit are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0011] S2 Model parameter optimization: S2.1 Data preprocessing: Perform manual denoising on the noisy images acquired by the FIBSEM system to obtain a noise-free image set, which is then divided into a noise-free training set and a noise-free test set using the train_test_split function;

[0012] S2.2 Residual Learning Training: After random cropping, geometric transformation, and noise addition, the original images in the noise-free training set are subjected to an artificial noise image training set. The images in the artificial noise image training set and the original images in the noise-free training set are used as data pairs to train the initial improved DnCNN network denoising model for parameter optimization. During training, the error is calculated using the L1 loss function, and the weight of the L1 loss function is set to 1.

[0013] S2.3 Model Testing: Add the specified noise to the original images in the noise-free test set to form an artificial test sample set. Use the trained initial improved DnCNN network denoising model to perform denoising operations to obtain the denoised test image set.

[0014] By comparing the images in the denoised test set with the original images in the noise-free test set, the peak signal-to-noise ratio is used as the evaluation index to evaluate the denoising effect of the improved DnCNN network denoising model;

[0015] S3 loop: repeat steps S2.2 and S2.3 until the denoising effect reaches the expected effect, and the improved DnCNN network denoising model is obtained;

[0016] S4 denoising: The noisy target image acquired by the FIBSEM system is denoised using the improved DnCNN network denoising model to obtain the target image.

[0017] In the above-mentioned method for removing noise from FIBSEM noise images, in step S2.1, when using the train_test_split function for division, the parameter test_size is set to 0.2, and 80% of the noise-free image set data is specified as the noise-free training set data, and 20% of the noise-free image set data is specified as the noise-free sample test set data.

[0018] In the above method for removing noise from FIBSEM noise images, the Load Preprocessed Noise Image unit uses DataLoader to load data from a custom dataset and preprocess the data, converting the image from HWC format to CHW format, converting it to a PyTorch tensor and adjusting the channel format;

[0019] The input layer parameters are as follows: the input shape is (1300, 2000, 1), and the number of input channels is set to 1.

[0020] In the above-mentioned method for removing noise from FIBSEM noise images, in step S2.1, the geometric transformation is specifically performed by flipping and rotating;

[0021] In the noise addition operation, noise at different noise levels is added according to the characteristics of vertical stripes in FIBSEM images. The dataset size is expanded through the attention mechanism and image patch method to ensure that the model can fully learn the characteristic relationship between noise and image during training.

[0022] In the above-mentioned method for removing noise from FIBSEM noisy images, in step S2.2, in a continuously running training loop, the initial improved DnCNN network denoising model repeatedly receives training data, performs forward propagation and backward propagation, updates model parameters, and regularly records training information and saves the model; in order to ensure the reproducibility of the training process, a random seed is set so that the same random results can be obtained in each run.

[0023] In the above-mentioned method for removing noise from FIBSEM noisy images, in step S2.1, the initial improved DnCNN network denoising model is automatically saved every 5000 training steps to save the current model parameters and optimizer status.

[0024] In the above-mentioned method for removing noise from a FIBSEM noisy image, in step S4, the specific process of obtaining a target image is as follows: the noisy target image obtained by the FIBSEM system is processed using the loading preprocessed noise image unit and used as input data; the input data is input into the combined convolution layer through the input layer, the head convolution layer and the stable enhanced convolution layer are used in the combined convolution layer to extract image features, and then the image features are recombined in the tail convolution layer to generate a noisy residual image, the input image is subtracted from the residual image to obtain a denoised image, and finally the target image is output through the output layer.

[0025] Compared with the existing technology, this method for removing noise from FIBSEM noise images has the following beneficial effects:

[0026] The present invention utilizes residual learning and batch normalization techniques to improve the training efficiency and denoising performance of the model, significantly enhancing image quality. By artificially adding noise, the diversity of training data is enhanced, enabling the model to more comprehensively understand the relationship between noise and image during the learning process, thereby improving the generalization ability of the model and enhancing the model's ability to learn different noise characteristics, thereby ensuring the accuracy of the denoising results. Overall, the present invention utilizes an improved DnCNN network model in combination with residual learning and batch normalization techniques to make the denoising process more stable and adaptable to different noise levels. For FIBSEM image noise removal, it effectively reduces the noise and curtain effect caused by surface morphology fluctuations and composition differences of the sample, thereby improving the clarity and accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart for developing the network model of the present invention.

[0028] Figure 2 This is the network model structure diagram of the improved DnCNN of the present invention.

[0029] Figure 3 This is a diagram showing the pixel value distribution of the noise image, clean image, and residual image of the present invention.

[0030] Figure 4 It is the residual image restored by using the residual patch in the present invention.

[0031] Figure 5 (a) is the original image;

[0032] (b) is the denoised image of the DnCNN network;

[0033] (c) is the threshold segmentation image of the original image;

[0034] (d) is the threshold segmentation image after denoising by the DnCNN network. DETAILED DESCRIPTION

[0035] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0036] like Figure 1 As shown, the method for removing noise from FIBSEM noisy images uses the DnCNN network model as a framework. By building an initial improved DnCNN network denoising model and optimizing model parameters, an improved DnCNN network denoising model is obtained. The improved DnCNN network denoising model is used to denoise the FIBSEM noisy image. The method for removing noise from the FIBSEM noisy image includes the following steps:

[0037] S1 builds the initial improved DnCNN network denoising model: The initial improved DnCNN network denoising model includes loading the preprocessed noise image unit, input layer, combined convolution layer and output layer. The combined convolution layer includes 1 head convolution layer, 15 stable enhanced convolution layers and 1 tail convolution layer;

[0038] The parameters of the head convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0039] The parameters of the tail convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0040] The stable enhanced convolution layer includes a convolution operation unit, a batch normalization unit, and a rectified linear unit. The ReLU activation function is used in the rectified linear unit. The parameters of the convolution operation unit are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64;

[0041] Batch Normalization: Batch normalization technology is used to stabilize the training process and prevent gradient explosion or disappearance, ensuring that the model has good stability during training.

[0042] S2 Model parameter optimization: S2.1 Data preprocessing: Perform manual denoising on the noisy images acquired by the FIBSEM system to obtain a noise-free image set, which is then divided into a noise-free training set and a noise-free test set using the train_test_split function;

[0043] S2.2 Residual Learning Training: After random cropping, geometric transformation, and noise addition, the original images in the noise-free training set are subjected to an artificial noise image training set. The images in the artificial noise image training set and the original images in the noise-free training set are used as data pairs to train the initial improved DnCNN network denoising model for parameter optimization. During training, the error is calculated using the L1 loss function, and the weight of the L1 loss function is set to 1.

[0044] The core idea of ​​residual learning is to introduce skip connections, allowing the network to directly learn the residual between input and output, rather than directly learning complex mapping relationships. In the DnCNN network, the network output is the noise residual. Subtracting this residual from the input image yields a denoised image, enhancing the denoising effect and producing a clean image.

[0045] After the original image is randomly cropped, geometrically transformed, and noise is added, an artificial noise image training set is obtained to improve the generalization ability of the model.

[0046] S2.3 Model Testing: Add the specified noise to the original images in the noise-free test set to form an artificial test sample set. Use the trained initial improved DnCNN network denoising model to perform denoising operations to obtain the denoised test image set.

[0047] By comparing the images in the denoised test image set with the original images in the noise-free test set, the peak signal-to-noise ratio is used as the evaluation indicator to evaluate the denoising effect of the improved DnCNN network denoising model.

[0048] S3 loop: Repeat steps S2.2 and S2.3 until the denoising effect reaches the expected effect, and the improved DnCNN network denoising model is obtained.

[0049] S4 denoising: The noisy target image acquired by the FIBSEM system is denoised using the improved DnCNN network denoising model to obtain the target image.

[0050] In step S2.1, when using the train_test_split function for splitting, the parameter test_size is set to 0.2, and 80% of the noise-free image set data is designated as the noise-free training set data, and 20% of the noise-free image set data is designated as the noise-free sample test set data.

[0051] The parameter test_size = 0.2 in this function specifies that 80% of the data is used for training and 20% for testing. To fairly evaluate model performance, the test dataset is not used in model training. This partitioning ensures that the training dataset accounts for the majority, allowing the model to learn more features, while the test dataset is used to evaluate the model's generalization ability.

[0052] The Load Preprocessed Noise Image unit uses DataLoader to load data from a custom dataset and preprocess the data, converting the image from HWC format to CHW format, converting it to a PyTorch tensor, and adjusting the channel format;

[0053] The input layer parameters are as follows: the input shape is (1300, 2000, 1), and the number of input channels is set to 1.

[0054] In step S2.1, the geometric transformation is specifically performed by flipping and rotating;

[0055] In the noise addition operation, noise at different noise levels is added according to the characteristics of vertical stripes in FIBSEM images. The dataset size is expanded through the attention mechanism and image patch method to ensure that the model can fully learn the characteristic relationship between noise and image during training.

[0056] In step S2.2, in the continuously running training loop, the initial improved DnCNN network denoising model will repeatedly receive training data, perform forward propagation and backpropagation, update model parameters, and regularly record training information and save the model; in order to ensure the reproducibility of the training process, a random seed is set so that the same random results can be obtained in each run.

[0057] In step S2.1, the initial improved DnCNN network denoising model is automatically saved every 5000 training steps to save the current model parameters and optimizer status.

[0058] In step S4, the specific process of obtaining the target image is as follows: the noisy target image obtained by the FIBSEM system is processed by the pre-processed noise image unit and used as input data; the input data is input into the combined convolution layer through the input layer, the head convolution layer and the stable enhanced convolution layer are used in the combined convolution layer to extract image features, and then the image features are recombined in the tail convolution layer to generate a noisy residual image, the input image is subtracted from the residual image to obtain a denoised image, and finally output through the output layer to obtain the target image.

[0059] For example, the following operations:

[0060] This embodiment aims to describe the detailed process of FIBSEM image denoising using the improved DnCNN network. Figure 1 , which is the development process of the network model.

[0061] 1. Data Acquisition and Preprocessing: The original image was acquired using a FIBSEM imaging system in HWC (height, width, channels) format with dimensions of 2000 × 1300. During data loading, the noisy image was converted to a PyTorch tensor and resized to CHW (channels, height, width) format. The image was then partitioned into multiple 40 × 40 tiles and noise was added.

[0062] 2. Model workflow: Figure 2To improve the network model structure of DnCNN, a noisy target image acquired using a FIBSEM system is processed using the pre-processed noise image unit and used as input data. This input data is fed into the combined convolutional layer through the input layer. The head convolutional layer and the stabilized enhanced convolutional layer are used in the combined convolutional layer to extract image features. The image features are then recombined in the tail convolutional layer to generate a noisy residual image. This residual image represents the noise features learned by the network and the noise pattern recognized by the network. The denoised image is obtained by subtracting the residual image from the input image, and finally outputted through the output layer to obtain the target image.

[0063] In the denoising process of improving the DnCNN network, Figure 3 The pixel value distribution of the noisy image, clean image and residual image is shown. Figure 4 The residual image restored using the residual patch helps us understand the concept of residual learning and the process of generating residual images, so that we can observe the noise distribution characteristics learned by the network. The "noise image" minus the "residual image" yields the "denoised image," which is the denoised image achieved through residual learning.

[0064] 3. Results Analysis and Application: The denoised image is used for subsequent analysis and characterization of pore characteristics. We first analyzed the differences between the original and denoised images using PSNR and SSIM. The calculated PSNR was 32dB, indicating a low error between the denoised image and the original clean image, indicating high image quality. An SSIM of 0.77 indicates that the denoised image is highly similar to the original image in terms of structure, brightness, and contrast, demonstrating the network's effectiveness in preserving image structural information.

[0065] Secondly, the image denoising effect is evaluated from the perspective of porosity. Figure 5 (a) (b) show the original image and the denoised image using the improved DnCNN network, respectively. (c) (d) show the thresholded segmentation of the original image and the thresholded segmented image after denoising using the improved DnCNN network. The manually calculated porosity is 6.1%. When compared to real data, the denoising process removes the noise, resulting in a "denoised image" with a porosity of 6.3%, while the porosity of the original image is 7.9%. In comparison, the porosity of the denoised image using the improved DnCNN network model is closer to the real data, more clearly demonstrating the pore structure and facilitating further pore connectivity analysis and feature extraction.

[0066] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art of the invention may make various modifications or additions to the described specific embodiments or replace them in a similar manner, but will not deviate from the spirit of the invention or exceed the defined scope. Although the present invention is described and described in detail in the drawings and the foregoing description, such illustrations and descriptions are considered to be illustrative or exemplary rather than restrictive. It should be understood that within the scope of the following claims, changes and modifications may be made by those skilled in the art. Specifically, the present invention covers further embodiments having any combination of features from the different embodiments described above. With respect to the use of the expression "generally" or "substantially", this patent application should be understood to disclose that these features and values ​​are also fully satisfied, i.e., there is no aforementioned characterization as "generally" or "substantially".

Claims

1. A method for removing noise from a FIBSEM noise image, characterized in that: Using the DnCNN network model as a framework, an improved DnCNN network denoising model is obtained by building an initial improved DnCNN network denoising model and optimizing model parameters. The improved DnCNN network denoising model is used to denoise FIBSEM noisy images. The method for removing noise from FIBSEM noisy images includes the following steps: S1 builds the initial improved DnCNN network denoising model: The initial improved DnCNN network denoising model includes loading the preprocessed noise image unit, input layer, combined convolution layer and output layer. The combined convolution layer includes 1 head convolution layer, 15 stable enhanced convolution layers and 1 tail convolution layer; The parameters of the head convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64; The parameters of the tail convolution layer are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64; The stable enhanced convolution layer includes a convolution operation unit, a batch normalization unit, and a rectified linear unit. The ReLU activation function is used in the rectified linear unit. The parameters of the convolution operation unit are as follows: the convolution kernel size is 3×3, the stride is 1, and the number of channels is 64; S2 Model parameter optimization: S2.1 Data preprocessing: Perform manual denoising on the noisy images acquired by the FIBSEM system to obtain a noise-free image set, which is then divided into a noise-free training set and a noise-free test set using the train_test_split function; S2.2 Residual Learning Training: After random cropping, geometric transformation, and noise addition, the original images in the noise-free training set are subjected to an artificial noise image training set. The images in the artificial noise image training set and the original images in the noise-free training set are used as data pairs to train the initial improved DnCNN network denoising model for parameter optimization. During training, the error is calculated using the L1 loss function, and the weight of the L1 loss function is set to 1. S2.3 Model Testing: Add the specified noise to the original images in the noise-free test set to form an artificial test sample set. Use the trained initial improved DnCNN network denoising model to perform denoising operations to obtain the denoised test image set. By comparing the images in the denoised test set with the original images in the noise-free test set, the peak signal-to-noise ratio is used as the evaluation index to evaluate the denoising effect of the improved DnCNN network denoising model; S3 loop: repeat steps S2.2 and S2.3 until the denoising effect reaches the expected effect, and the improved DnCNN network denoising model is obtained; S4 denoising: The noisy target image acquired by the FIBSEM system is denoised using the improved DnCNN network denoising model to obtain the target image.

2. The method for removing noise from a FIBSEM noise image according to claim 1, wherein: In step S2.1, when using the train_test_split function for splitting, the parameter test_size is set to 0.2, and 80% of the noise-free image set data is designated as the noise-free training set data, and 20% of the noise-free image set data is designated as the noise-free sample test set data.

3. The method for removing noise from a FIBSEM noise image according to claim 1, wherein: The Load Preprocessed Noise Image unit uses DataLoader to load data from a custom dataset and preprocess the data, converting the image from HWC format to CHW format, converting it to a PyTorch tensor, and adjusting the channel format; The input layer parameters are as follows: the input shape is (1300, 2000, 1), and the number of input channels is set to 1.

4. The method for removing noise from a FIBSEM noise image according to claim 1, wherein: In step S2.1, the geometric transformation is specifically performed by flipping and rotating; In the noise addition operation, noise at different noise levels is added according to the characteristics of vertical stripes in FIBSEM images. The dataset size is expanded through the attention mechanism and image patch method to ensure that the model can fully learn the characteristic relationship between noise and image during training.

5. The method for removing noise from a FIBSEM noise image according to claim 1, wherein: In step S2.2, in the continuously running training loop, the initial improved DnCNN network denoising model will repeatedly receive training data, perform forward propagation and backpropagation, update model parameters, and regularly record training information and save the model; in order to ensure the reproducibility of the training process, a random seed is set so that the same random results can be obtained in each run.

6. The method for removing noise from a FIBSEM noise image according to claim 1, wherein: In step S2.1, the initial improved DnCNN network denoising model is automatically saved every 5000 training steps to save the current model parameters and optimizer status.

7. The method for removing noise from a FIBSEM noise image according to claim 1, wherein: In step S4, the specific process of obtaining the target image is as follows: the noisy target image obtained by the FIBSEM system is processed by the pre-processed noise image unit and used as input data; the input data is input into the combined convolution layer through the input layer, the head convolution layer and the stable enhanced convolution layer are used in the combined convolution layer to extract image features, and then the image features are recombined in the tail convolution layer to generate a noisy residual image, the input image is subtracted from the residual image to obtain a denoised image, and finally output through the output layer to obtain the target image.

Images