A multi-model collaborative deep learning image processing method applicable to high-voltage and low-voltage scanning electron microscopes

Abstract

The core challenge of traditional scanning electron microscope (SEM) images is their susceptibility to Gaussian and Poisson noise and blurring, resulting in insufficient image resolution. To address this, this invention proposes a multi-model collaborative deep learning image processing method suitable for both high-pressure and low-pressure SEMs. The method proceeds in a progressive sequence of "denoising – deblurring – super-resolution." First, an improved DnCNN model removes Poisson-Gaussian mixed noise from the noisy image. Then, a generator model using a generative adversarial network (GAN) architecture deblurs the denoised image, restoring image detail clarity. Finally, an ESRGAN super-resolution model magnifies the deblurred image by 4 times, improving its spatial resolution. This invention allows for flexible selection of a fixed-round training mode or an early-stop training mode, and enables quantitative evaluation of processing results. It compares different processing sequences to identify the optimal process, providing technical support for the accurate analysis of SEM images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a multi-model collaborative deep learning image processing method applicable to high-voltage and low-voltage scanning electron microscopes, which optimizes the quality and further improves the resolution of high-voltage and low-voltage scanning electron microscope images affected by noise and blur. Background Technology

[0002] Based on the different accelerating voltages, scanning electron microscopes (SEMs) can be divided into two main categories: high-voltage SEMs and low-voltage SEMs. Each has its own advantages in different application scenarios. High-voltage SEMs typically have an accelerating voltage of no less than 10 kV, and are characterized by high electron beam intensity, high imaging resolution, and strong sample selectivity. They can clearly present the fine microstructure of samples with good conductivity, such as metallic materials and semiconductor devices. However, their performance deteriorates when observing sensitive insulating samples such as biological samples and polymer materials. The reasons are: (1) High-energy electron beams can easily damage sensitive samples during imaging; (2) The samples have poor conductivity, and charge accumulation occurs during imaging, leading to image distortion and affecting imaging resolution. Low-voltage SEMs are generally below 5 kV. During imaging, there is little sample damage and no charge accumulation. They are suitable for observing sensitive samples such as biological samples and insulating materials. However, the electron beam intensity is insufficient, and the acquired images are easily affected by a mixture of Gaussian noise and Poisson noise. At the same time, the image quality will be further degraded due to factors such as electron beam scattering and external environment, resulting in increased blurring, loss of details, and decreased resolution.

[0003] For both high-pressure and low-pressure scanning electron microscopes (SEMs), image quality directly determines the accuracy of subsequent microscopic analysis. Existing SEM image processing methods suffer from the following shortcomings: First, noise removal often employs denoising algorithms with a single noise model, making it difficult to simultaneously adapt to the high-intensity noise of high-pressure SEMs and the mixed noise characteristics of low-pressure SEMs. Second, deblurring methods largely rely on traditional iterative algorithms, resulting in slow convergence and artifacts that compromise the realism of microstructures. Third, super-resolution techniques, lacking deep integration with denoising and deblurring, amplify both noise and blur in the image when implemented independently. Furthermore, existing processing methods generally lack flexible training control mechanisms, making it difficult to adjust training strategies according to the actual image quality requirements of different types of SEMs, and their effectiveness is limited to a single metric, failing to comprehensively reflect image fidelity, structural similarity, and perceptual quality. Finally, existing processing methods have not yet conducted comparative studies on different processing sequences, while the combination order of different processing steps significantly impacts image optimization.

[0004] To jointly address the noise, blurring, and low resolution issues in high-voltage and low-voltage scanning electron microscope (SEM) images and contribute to the independent controllability of domestically produced high-end scientific instruments, this invention proposes a multi-model collaborative deep learning image processing scheme for SEM that is flexible in training, comprehensive in evaluation, and highly applicable. This scheme systematically optimizes SEM images acquired by secondary electron detectors, providing technical support for subsequent image quality evaluation and microscopic analysis. Summary of the Invention

[0005] This invention discloses a multi-model collaborative deep learning image processing method applicable to high-pressure and low-pressure scanning electron microscopes. Through a progressive processing architecture of "denoising-deblurring-super-resolution", combined with a flexible training control mode and a comprehensive index evaluation system, it aims to simultaneously achieve effective denoising, accurate deblurring and super-resolution reconstruction of high-pressure and low-pressure scanning electron microscope images, thereby providing high-quality image data for subsequent analysis of the two types of scanning electron microscope images.

[0006] The core idea of ​​this invention is to process scanning electron microscope images in the order of "denoising first, then deblurring, and finally super-resolution," avoiding the amplification of noise and blurring in subsequent processing. Each stage is closely integrated, with the output of one stage serving as the input of the next, forming a complete processing chain. The overall processing system mainly includes a denoising module, a deblurring module, a super-resolution module, a training control module, and an evaluation module, as detailed below:

[0007] The first part, the denoising module, employs an improved DnCNN (Denoising Convolutional Neural Network) model. This network has 17 layers, with only one input and one output channel, making it suitable for grayscale image processing. The model optimizes the MSE loss function and separates noise from the image signal through residual learning. The training data uses grayscale processing results from the CIFAR10 dataset. Noisy samples are constructed by generating Poisson-Gaussian mixed noise: first, image pixel values ​​are scaled from the 0-1 range to the 0-255 range to generate Poisson noise, which is then normalized and compared with... Gaussian noise is superimposed, and finally cropped to the 0-1 range to ensure the image's validity.

[0008] The second part, the deblurring module, is built based on a GAN (Generative Adversarial Network) and consists of a generator and a discriminator. The generator employs an "initial convolution - two downsampling layers - six residual blocks - two upsampling layers - output convolution" structure, preserving image details through residual connections and outputting a deblurred image with the same size as the input. The discriminator uses a four-layer convolutional block structure to distinguish the deblurred image output by the generator from the real, sharp image. During training, MSE pixel loss and adversarial loss are jointly optimized to improve the fidelity and realism of the deblurred image.

[0009] The third part, the super-resolution processing module, employs the RealESRGAN_x4plus model based on the ESRGAN architecture, loading a pre-trained weight file. The deblurred image is used as input, and the model processes it to achieve a 4x resolution upscaling. Then, BICUBIC (bicubic interpolation) is used for fine-tuning to adjust the super-resolution image back to the original image size for easier subsequent comparative analysis. This module generates low-resolution (LR) images by calling external scripts, and then generates super-resolution (SR) images based on the LR images.

[0010] Part Four, Training Control Module: Supports two training modes: fixed epoch (training rounds) training mode and early-stop training mode. Fixed epoch mode specifies the number of training rounds for the denoising and deblurring modules by setting the `denoise_epoch` and `deblur_epoch` parameters respectively, disabling the early-stop mechanism to ensure the model is trained to the specified number of rounds. Early-stop training mode determines the maximum number of training rounds by setting the `max_epoch` parameter. When the loss value continuously exceeds a preset tolerance value and the change is less than 1e-5, training automatically stops to avoid overfitting. When the specified number of training rounds exceeds the existing model's training rounds, forced retraining is enabled by setting the `force_train` parameter. Specific parameters can be changed according to actual needs.

[0011] Part 5, Evaluation Module: Three quantitative metrics are used to comprehensively evaluate the processing effect: PSNR (Peak Signal-to-Noise Ratio) is used to evaluate image fidelity; a higher value indicates less distortion. When PSNR reaches approximately 30dB, the image reconstruction quality is generally considered good with minimal distortion. SSIM (Structural Similarity Score) is used to evaluate image structural similarity; a value closer to 1 indicates better structural consistency. LPIPS (Perceptual Similarity Points) is used to evaluate image perceptual quality; a lower value indicates better perceptual performance. Specific evaluation steps include: PSNR and SSIM evaluation between the original image and the denoised image; PSNR, SSIM, and LPIPS evaluation between the denoised image and the deblurred image; and PSNR, SSIM, and LPIPS evaluation between the deblurred image and the super-resolution image.

[0012] This invention also conducted comparative experiments on different processing sequences of denoising, deblurring, and super-resolution, including various combinations such as "deblurring-denoising-super-resolution", "super-resolution-denoising-deblurring", and "denoising-deblurring-super-resolution". The results show that the processing sequence of "denoising first, then deblurring, and finally super-resolution" can effectively avoid problems such as noise amplification and blurring during super-resolution and noise interference with detail restoration during deblurring. The PSNR, SSIM, and other quantitative indicators of the processed image are better than other processing sequences, making it the optimal processing sequence for adapting to the characteristics of scanning electron microscope images.

[0013] The present invention also provides a storage medium storing program instructions for processing scanning electron microscope images, which, when executed by a processor, implement any of the aforementioned methods.

[0014] The advantages and positive effects of this invention are as follows:

[0015] (1) The present invention adopts a progressive architecture of "denoising-deblurring-super-resolution", which avoids the problem of amplifying noise and blurring in the super-resolution process, significantly improving the overall quality and enhancing the practicality and efficiency of the system. At the same time, through a large number of comparative experiments with different processing sequences, it is verified that this processing sequence is the optimal sequence for adapting to the noise and blurring characteristics of scanning electron microscope images. Compared with other processing sequences, it can achieve higher image fidelity and structural similarity, providing a process reference for the processing of scanning electron microscope images.

[0016] (2) In view of the mixed noise characteristics of scanning electron microscope images, the present invention uses a Poisson-Gaussian mixed noise model to train the denoising module. Compared with a single noise model, the denoising effect is better. In actual processing, the average PSNR of the denoised image can reach more than 30dB and the average SSIM can reach more than 0.8.

[0017] (3) The present invention adopts a deblurring model based on GAN architecture. Through joint optimization of pixel loss and adversarial loss, it can not only ensure the fidelity of image structure, but also improve the clarity of details, avoiding the problem of artifacts easily generated by traditional algorithms, and achieving good perception effect.

[0018] (4) This invention supports two training modes, which can be flexibly selected according to actual needs. The fixed epoch mode is suitable for scenarios that require stable training results, while the early stop mode is suitable for scenarios that quickly obtain the optimal model.

[0019] (5) The present invention uses three quantitative indicators to comprehensively evaluate the processing effect, which can reflect the image processing level from three dimensions: fidelity, structural similarity and perceptual quality, and provide support for the objective evaluation of the processing effect.

[0020] (6) The present invention has strong adaptability, covering both high-voltage and low-voltage scanning electron microscopes, and does not require separate algorithm adjustments for different types of scanning electron microscopes, thus achieving one-stop image optimization.

[0021] (7) The present invention has a compact structure, clear logical steps, and automatic saving of process status and results, which facilitates subsequent analysis and retrieval and has broad application prospects. Attached Figure Description

[0022] Figure 1 A flowchart for multi-model collaborative deep learning image processing;

[0023] Figure 2 A schematic diagram of the DnCNN model structure for the denoising module;

[0024] Figure 3 Here are the logic flowcharts for the two training modes;

[0025] Figure 4 A schematic diagram of the generator model structure for the deblurring module;

[0026] Figure 5 A schematic diagram of the discriminator model structure for the deblurring module;

[0027] Figure 6 This is a flowchart of the super-resolution module's workflow. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0029] A multi-model collaborative deep learning image processing method suitable for high-pressure and low-pressure scanning electron microscopes, including dataset construction and module training and deployment.

[0030] After setting up the environment, the dataset was prepared. The training dataset for the denoising module used the CIFAR10 public dataset. The RGB images were converted to grayscale images using the transforms.Compose function to simulate the grayscale characteristics of SEM images, and 32×32 image patches were randomly cropped as training samples. The training dataset for the deblurring module used the GoPro dataset, organized according to the directory structure of GOPR sequence folder / blur (blurred images) and GOPR sequence folder / sharp (sharp images). The image size was adjusted to 256×256 to match the commonly used imaging resolution of SEMs. For the test dataset, 210 original SEM images with a resolution of 256×256 were collected and placed in the test folder as test data to verify the model's processing effect on the two types of SEM images.

[0031] Multi-model collaborative deep learning image processing workflow as follows Figure 1 As shown, it includes a noise reduction module, a deblurring module, a super-resolution processing module, and an index evaluation module.

[0032] This invention employs three quantitative indicators—PSNR, SSIM, and LPIPS—to comprehensively evaluate the processing effect, objectively reflecting the quality improvement of high-voltage and low-voltage scanning electron microscope images at each processing stage from three dimensions: fidelity, structural similarity, and perceived quality. PSNR is used to evaluate image fidelity; a higher value indicates less image distortion. Its calculation formula is as follows:

[0033]

[0034]

[0035] In equations (1) and (2) The size of the grayscale image is denoted as . The original image is denoted as . The processed image is denoted as , This represents the maximum value of a pixel in the image (255 for grayscale images). This represents the mean squared error between the processed image and the reference image. SSIM is used to evaluate image structural similarity; a value closer to 1 indicates better structural similarity. The calculation formula is:

[0036]

[0037] In formula (3) , respectively, are the pixel mean values ​​of the reference image x and the processed image y. Let x and y be the pixel variances, respectively. Let x and y be the pixel covariance. The constant is used to avoid a denominator of 0. LPIPS is used to evaluate the perceptual quality of images. It measures image similarity based on deep learning. The core idea is to use a pre-trained neural network to extract multi-level feature representations of images and calculate the Euclidean distance in the feature space. The specific calculation formula is defined internally by the LPIPS model. The core is to measure the difference between two images at the level of human visual perception. The lower the value, the better the perceptual effect.

[0038] The first step is to denoise the original scanning electron microscope image, such as... Figure 2 As shown. First, the DnCNN model is initialized. During initialization, the number of input and output channels is set to 1, the number of intermediate layer channels to 64, the convolutional kernel size to 3×3, and the padding to 1. Model parameters are randomly initialized. For training configuration, the batch size (batch_size) for a single training iteration is set to 64, the learning rate is 1e-3, and the mainstream deep learning optimizer, Adam, is selected. Adam combines the advantages of momentum and RMSprop, dynamically adjusting the learning rate for each parameter by calculating the first and second moments of the gradient. Its core formula includes:

[0039] First-order moment estimation:

[0040] Second-order moment estimation:

[0041] Parameter update:

[0042] in, This is the moment estimate after bias correction. The initial learning rate, To prevent the denominator from being zero, a minimum value is set. Furthermore, in this embodiment, the optimization objective at this stage is the MSE (mean squared error) loss function, which can effectively measure the pixel-level difference between noisy and clean images.

[0043] The aforementioned DnCNN model is a convolutional neural network for image denoising based on an end-to-end architecture. It is based on a residual learning architecture, and its core is to learn the noise residual rather than directly learning the clean image. That is, the input is a noisy image, the output is a noise residual, and the denoising is achieved by subtracting the predicted noise part from the noisy input. Figure 2 In this architecture, the network structure is a 17-layer modular design. The first layer uses a 3×3 convolutional kernel to achieve feature dimensionality upscaling (expanding the number of input channels c to 64) and activate non-linear features. The middle layers use BatchNorm (batch normalization) and ReLU (corrected linear unit) activation, with a 3×3 kernel size and padding of 1 to maintain the same size. The output layer uses a convolutional kernel of the same size to perform channel dimensionality reduction (restoring 64 to c). Residual learning reduces the training difficulty, and BatchNorm improves the convergence speed, making the denoising effect far superior to traditional denoising algorithms (such as Wiener filtering and median filtering). Traditional algorithms can only handle single types of noise and are prone to blurring details, while DnCNN can adapt to mixed noise while preserving image edges and textures.

[0044] Furthermore, the training process employed a Poisson-Gaussian mixture noise model to construct noisy samples to adapt to the different noise characteristics of high-pressure and low-pressure scanning electron microscopes. Specifically, this included: first, scaling the image pixel values ​​from the 0-1 range to the 0-255 range to generate Poisson noise (simulating the brightness dependence of low-pressure electron microscope noise); then, after normalization, and... The Gaussian noise (simulating high-intensity random noise in a high-voltage electron microscope) is superimposed; finally, it is cropped to the 0-1 interval to ensure the image's legality, so that the model can learn the removal rules of the two types of noise simultaneously during training.

[0045] The training mode can be selected from two modes according to actual needs: fixed epoch training and early stop training. Figure 3 As shown. In fixed epoch mode, by setting denoise_epoch=140 and force_train=True, the model is trained for 140 epochs and then saved to a fixed location. During training, the average loss of the training set is calculated and the training progress is monitored in each epoch. In early stop mode, by setting max_epoch=80 and patience value=25, training is automatically stopped and the current optimal model is saved when the change in loss value is less than 1e-5 for 25 consecutive epochs. After the model training is completed, the saved model weights are loaded, the original image input from a high-voltage or low-voltage scanning electron microscope is received, and noise and image signals are automatically separated through forward propagation. The denoised image is then output and saved.

[0046] In addition, the force_train parameter is used to control whether to force retraining. When the specified number of training epochs is greater than the number of training epochs of the existing model, this parameter needs to be enabled to ensure that the model is trained to the target number of epochs. If the number of training epochs of the existing model already meets the requirements, it is not necessary to enable this parameter and directly load the existing model for processing to improve efficiency.

[0047] The image evaluation after denoising is carried out by calculating PSNR and SSIM to assess the noise removal effect and verify whether the denoising module can preserve image details while suppressing noise. In this embodiment, the average PSNR of the denoised image can reach more than 30dB and the average SSIM can reach more than 0.8, indicating that the noise is effectively suppressed and the image structural details are well preserved. It can handle both the high-intensity noise of high-voltage scanning electron microscopes and the mixed noise of low-voltage scanning electron microscopes.

[0048] The second step is to deblur the image. The entire process is based on GAN and consists of two main parts: a generator and a discriminator, as shown in Figures 4 and 5, respectively.

[0049] Figure 4 In this model, the generator employs a symmetrical structure of "initial convolution - two downsampling operations - 6 residual blocks - two upsampling operations - output convolution." Residual connections preserve image detail and prevent feature loss during downsampling. Specifically: the input layer is a 1-channel denoised image; the initial convolution kernel size is 7×7, padding=3, and the number of channels is expanded to 64; the two downsampling convolution kernel sizes are 3×3, stride=2, padding=1, resulting in 128 and 256 channels respectively; the two upsampling transposed convolutions are 3×3, stride=2, padding=1, and output_padding=1, resulting in 128 and 64 channels respectively; finally, the output convolution is also 7×7, padding=3, restoring the number of channels to 1.

[0050] Figure 5 In this model, the discriminator employs a 4-layer convolutional block structure. The process includes: an input layer of a 1-channel image; followed by four convolutional blocks performing BatchNorm-LeakyReLU convolution with 64, 128, 256, and 512 channels respectively, with the first convolutional block having no BatchNorm; next, a 4×4 convolutional layer with stride=1 and padding=1; and finally, an adaptive average pooling layer and a 1-channel output layer. The discriminator distinguishes between the deblurred image output by the generator and the real, sharp image, outputting a discrimination probability value. During model initialization, the parameters of both the generator and the discriminator are randomly initialized.

[0051] In the deblurring module, the batch size is set to 8, the learning rate to 1e-4, the optimizer is Adam, and betas is set to (0.5, 0.999). Joint optimization using MSE pixel loss and adversarial loss is employed. MSE loss ensures pixel-level similarity between the generated image and the real image, while adversarial loss enhances the realism and detail reproduction of the generated image. The generator loss is the adversarial loss. Pixel loss is addressed by balancing the effects of the two types of loss through weight adjustments.

[0052] In addition, the training mode also supports both fixed epoch and early stopping. In fixed epoch mode, deblur_epoch=320 and force_train=True, after the model is trained for 320 epochs, the generator and discriminator weights are saved to the corresponding directories respectively; in early stopping mode, max_epoch=80 and patience value=30, when the generator loss changes by less than 1e-5 for 30 consecutive epochs, training is automatically stopped and the optimal model is saved.

[0053] Finally, the image evaluation of the deblurring module was performed by calculating PSNR, SSIM, and LPIPS to evaluate the denoised and deblurred images, verifying the module's ability to restore details and its perceptual quality. In this embodiment, the average PSNR of the deblurred image reached over 17dB, the average SSIM reached 0.7, and the LPIPS value decreased to 0.4. Although the PSNR decreased, this was due to the temporary increase in pixel-level differences caused by the restoration of details during the deblurring process. The perceptual quality of the image remained good, and the details in the blurred areas were effectively restored, laying the foundation for subsequent super-resolution processing.

[0054] The third part involves super-resolution processing of the images. The RealESRGAN_x4plus model based on the ESRGAN architecture effectively extracts deep features of the images, achieving a 4x resolution magnification. Through transfer learning, the model adapts to the characteristics of scanning electron microscope images. Before practical application, a small number of clear images from high-pressure and low-pressure scanning electron microscopes can be used for fine-tuning to improve the model's adaptability.

[0055] As an engineering improvement of ESRGAN, RealESRGAN_x4plus enables super-resolution optimization of low-resolution images. Specifically, it introduces a real image degradation model, replacing traditional residual blocks with Residual Dense Blocks (RRDB) to enhance feature extraction capabilities, and jointly optimizes through perceptual loss and GAN loss. The perceptual loss extracts image features through a pre-trained VGG network, calculating the distance between the generated image and the real image in the feature space, thus compensating for the limitation of MSE, which only focuses on pixels. During the pre-training stage, perceptual loss and GAN loss ensure visual quality, while the fine-tuning stage (for SEM images) uses MSE loss to ensure structural similarity. The core formula combines perceptual loss, GAN loss, and MSE loss. Furthermore, the model optimizes the network structure to improve robustness and supports 4x resolution magnification by default. This model does not require retraining from scratch, providing pre-trained weights (v00.pth), and can be directly ported to SEM image processing.

[0056] The flowchart of the super-resolution processing module is as follows: Figure 6 As shown: First, the pre-trained v00.pth weight file is loaded, and the external script run_generate_lr.py is called to take the deblurred image as high-resolution input and generate the corresponding low-resolution (LR) image, which is then saved. Next, the inference_realesrgan.py script is executed, the LR image directory is input, the magnification factor is set to 4, and the super-resolution (SR) image is output and saved through the RealESRGAN_x4plus model. Finally, BICUBIC interpolation is used to fine-tune the super-resolution image to the size of the original SEM image, which is convenient for subsequent comparative analysis and practical applications, and ensures that the processed image retains details while maintaining the same observation angle as the original image.

[0057] In the image evaluation of the super-resolution processing module, the deblurred image and the super-resolution image are evaluated by calculating PSNR, SSIM, and LPIPS to verify whether the super-resolution module maintains image fidelity and structural consistency while improving resolution. In this embodiment, the average PSNR of the super-resolution image can reach over 34dB, the average SSIM can reach over 0.8, and the LPIPS value can be reduced to 0.4, indicating that the image maintains good fidelity and perceptual quality while improving resolution.

[0058] Furthermore, this embodiment conducted a systematic comparative experiment on the processing order of the trained denoising, deblurring, and super-resolution modules to verify the optimal workflow adapted to the characteristics of scanning electron microscope (SEM) images. Three typical processing order combinations were designed: "deblurring-denoising-super-resolution," "super-resolution-denoising-deblurring," and "denoising-deblurring-super-resolution." To eliminate interference from irrelevant variables, all schemes used the same model parameter settings and basic processing flow, adjusting only the execution order of the three modules to ensure the comparability of the experiments. The experimental samples consisted of 210 high-voltage and low-voltage SEM images from the test dataset.

[0059] The results show that after using the proposed "denoising-deblurring-super-resolution" processing method, the image's PSNR, SSIM, LPIPS, and other quantitative indicators are significantly better than other processing sequences. Specifically, PSNR is improved by an average of 3 dB compared to other sequences, SSIM is improved by at least 0.04 on average, and LPIPS is reduced by an average of 0.06. The proposed "denoising-deblurring-super-resolution" processing method avoids problems such as amplifying noise and blurring during super-resolution, noise interfering with detail restoration during deblurring, and the denoising stage damaging details in the super-resolution image. It is suitable for intelligent evaluation of high-voltage and low-voltage scanning electron microscope images.

Claims

1. A multi-model collaborative deep learning image processing method suitable for high-pressure and low-pressure scanning electron microscopes, comprising a denoising module, a deblurring module, a super-resolution module, a training control module, and an index evaluation module, characterized in that: (1) Denoising module: The improved DnCNN model is used to denoise the original scanning electron microscope images containing Gaussian noise and Poisson noise. The model is optimized by generating training data with mixed Poisson-Gaussian noise and outputs denoised images. (2) Deblurring module: The denoised image obtained by the denoising module is input into the deblurring model based on generative adversarial network. The deblurred image is output through adversarial training optimization between the generator and the discriminator. (3) Super-resolution processing module: The deblurred image obtained by the deblurring processing module is input into the ESRGAN super-resolution model, and the image is magnified by 4 times to output a high-resolution scanning electron microscope image. (4) Training control module: Supports two modes: fixed epoch training and early stop training. In fixed epoch mode, the number of training epochs for the denoising module and the deblurring module are specified by setting the denoise_epoch and deblur_epoch parameters, respectively. In early stop training mode, the maximum number of training epochs is determined by setting the max_epoch parameter. When the loss value continuously exceeds the preset patience value and the change is less than 1e-5, training is automatically stopped to avoid overfitting. When the specified number of training epochs is greater than the number of training epochs of the existing model, forced retraining is enabled by setting the force_train parameter. (5) Evaluation module: The quantitative evaluation of each processing stage is achieved by calculating the PSNR and SSIM indices of the original image and the denoised image, the PSNR, SSIM and LPIPS indices of the denoised image and the deblurred image, and the PSNR, SSIM and LPIPS indices of the original image and the super-resolution image.

2. The method as described in claim 1, characterized in that: The improved DnCNN model in the denoising module contains a 17-layer network structure with 1 input and 1 output channel. The intermediate layers use 64-channel convolutional kernels, and noise separation is achieved through residual learning. The MSE loss function is used for optimization during training.

3. The method as described in claim 1, characterized in that: The noise generation process in the denoising module is as follows: first, the image pixel values ​​are scaled to the 0-255 range to generate Poisson noise, which is then normalized and superimposed with Gaussian noise, and then cropped to the 0-1 range to obtain a noisy image, where the Gaussian noise intensity σ=25 / 255.

0.

4. The method as described in claim 1, characterized in that: The generator of the deblurring model in the deblurring module includes an initial convolutional layer, two downsampling layers, six residual blocks, and two upsampling layers. It is jointly optimized using the MSE loss function and the adversarial loss function. The discriminator adopts a four-layer convolutional block structure and outputs a single-channel discrimination result.

5. The method as described in claim 1, characterized in that: The ESRGAN super-resolution model in the super-resolution processing module adopts the RealESRGAN_x4plus network structure, and the output image is finely adjusted to the original image size through BICUBIC interpolation.

6. The method as described in claim 1, characterized in that: Without departing from the principles of this invention, appropriate adjustments to the model structure, training parameters, and indicator thresholds of each module are all within the scope of protection of this invention. For example, regarding the model structure, the number of convolutional layers and channels in the DnCNN model of the denoising module can be adjusted according to actual needs; the number of residual blocks in the deblurring module can be increased or decreased according to actual needs; and the magnification factor of the super-resolution module can be adjusted to 2x or 8x to adapt to different resolution requirements. Regarding training parameters, batch_size, learning rate, patience value in early stopping mode, etc., can be optimized according to hardware conditions. For example, when the GPU memory is large, the batch_size can be increased; and the σ value of noise generation can be adjusted according to the noise intensity of the specific scanning electron microscope.

7. The method as described in claim 1, characterized in that: A multi-model collaborative deep learning image processing method applicable to high-pressure and low-pressure scanning electron microscopes is not only applicable to the field of high-pressure and low-pressure scanning electron microscopes, but can also be used to process focused ion beam secondary electron images, transmission electron microscope images, and optical microscope images. Similar quality optimization effects can be achieved simply by adjusting the noise model and training dataset, and should also be considered within the scope of protection of this invention.

Images