A lightweight scanning electron microscope image super-resolution method based on diffusion model

Abstract

The present invention discloses a lightweight scanning electron microscope image super-resolution method based on a diffusion model, which relates to the field of electron microscope characterization and analysis technology. The method comprises the following steps: using a scanning electron microscope to collect a low-quality, low-resolution image group and a high-quality, high-resolution image group; cropping and constructing them into training data sets; constructing an encoder, a decoder, and a lightweight inference model; training the model using the sampling time, a control input image in the data set, and a reference real image as input, optimizing and updating the weight parameters in the lightweight inference model to obtain a trained lightweight inference model; and applying the trained model to scanning electron microscope image super-resolution. The present invention provides a lightweight scanning electron microscope image super-resolution method based on a diffusion model to reduce the computational burden of the diffusion model training and inference processes, improve image quality, and provide more accurate and reliable data support for microstructural analysis in materials science.

Description

Technical Field

[0001] The present invention relates to the technical field of electron microscope characterization and analysis, and in particular to a lightweight scanning electron microscope image super-resolution method based on a diffusion model. Background Art

[0002] Microstructure not only determines a material's macroscopic properties but also holds the key to understanding phenomena in materials science. Scanning electron microscopy, with its exceptional nanoscale resolution and high depth of field, has become a common tool for observing microstructure in materials science research. Scanning electron microscopy scans the surface of a sample with an electron beam. The resulting signal is collected and amplified by a detector, ultimately generating a high-contrast two-dimensional image that reveals detailed features of the sample's surface, providing researchers with a window into the microscopic world of materials.

[0003] Although scanning electron microscopes have many advantages, their image acquisition process is inevitably affected by noise. The presence of noise not only reduces the quality of the image, but also affects the accuracy of image analysis and processing. In order to improve the signal-to-noise ratio of the image, the dwell time of the electron beam at each pixel can be increased, that is, a slow scan mode can be adopted. However, although this method can improve image quality, it also leads to a significant increase in image acquisition time. In addition, long-term focusing of high-energy electron beams can damage certain samples and is not suitable for applications such as in situ imaging that require fast and continuous imaging, such as dealing with chemical reaction scenarios, stress relaxation and other problems. Especially when pursuing high-resolution images, the acquisition time will increase significantly, which makes it more difficult to find a balance between image quality and acquisition efficiency.

[0004] The Denosing Diffusion Probabilistic Model (DDPM) is an advanced generative model framework that excels in various data generation and restoration tasks. DDPM learns the data distribution by simulating the inverse process of data generation—gradually removing noise from noisy data. It has been applied in a variety of fields, including image denoising, speech enhancement, text generation, and data restoration. A notable feature of DDPM is its powerful learning capability, which allows it to capture complex patterns in the data and gradually restore its true characteristics during the denoising process. For super-resolution tasks, images generated using this approach often exhibit superior visual quality and detail preservation. However, the training and inference processes of DDPM are cumbersome, computationally demanding, and often time-consuming. Therefore, there is an urgent need for a lightweight model approach that can accelerate the training and inference of DDPM-based SEM image denoising tasks, reduce the computational burden, and improve image quality to provide more accurate and reliable data support for microstructural analysis in materials science. Summary of the Invention

[0005] The present invention provides a lightweight SEM image super-resolution method based on a diffusion model to solve the problems of large computing resource requirements and long time consumption of existing DDPM-based image super-resolution methods, and achieves efficient denoising and super-resolution of low-quality SEM images to obtain high-quality SEM images.

[0006] The object of the present invention is achieved through the following technical solution: a lightweight scanning electron microscope image super-resolution method based on a diffusion model, the method comprising the following steps:

[0007] S1, using the fast scan mode and slow scan mode in the scanning electron microscope to collect image data pairs,

[0008] S2. Cropping the two sets of images separately to construct a training dataset, where the cropped fast-scan mode images are used as control input images, and the cropped slow-scan mode images are used as reference real images;

[0009] S3. Construct a lightweight inference model; the lightweight inference model includes an encoder, a decoder, and several cross-attention modules; input sampling time information, a control input image, and a reference real image through the decoder, upsample the image output by the decoder, and input it into the cross-attention module and the decoder to obtain an inferred noise distribution image, and subtract the upsampled control input image from the noise distribution image to obtain a super-resolution image;

[0010] S4. Use the training dataset to train the lightweight inference model;

[0011] S5. Input the fast scan mode and slow scan mode images into the trained lightweight inference model to generate super-resolution images.

[0012] Furthermore, the image data are collected in the fast scan mode and the slow scan mode specifically as follows: when collecting image data, the sample is pasted on the scanning electron microscope sample stage using conductive glue, and the sample stage is fixed in the scanning electron microscope chamber using screws; an area with rich features is selected for image collection, and during the shooting process, it is necessary to ensure that the shooting field of view in the fast scan and slow scan modes remains completely consistent.

[0013] Further, the encoder includes a time embedding block, a control input channel, a reference real channel and an upsampling block;

[0014] The time embedding block is used to add sampling time information to the model;

[0015] The control input channel and the reference real channel both include two groups of residual block-pixel decomposition block combinations, which are input into the control input image and the reference real image respectively;

[0016] The upsampling block adopts nearest neighbor interpolation, linear interpolation, bilinear interpolation or bicubic interpolation, and the upsampling multiple is the same as the super-resolution multiple.

[0017] Furthermore, the cross-attention module includes two residual blocks and one cross-attention block.

[0018] Furthermore, the decoder includes two groups of residual block-pixel reassembly block combinations.

[0019] Furthermore, the training uses mean square error as a loss function and an Adam optimizer as an optimization algorithm to train the model; adjusts parameters to train the model, and determines the optimal hyperparameters of the prediction model; after determining the optimal hyperparameters, uses the data set to train a lightweight scanning electron microscope image super-resolution model.

[0020] On the other hand, the present invention provides a lightweight scanning electron microscope image super-resolution device based on a diffusion model, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the lightweight scanning electron microscope image super-resolution method based on the diffusion model.

[0021] On the other hand, the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for lightweight scanning electron microscope image super-resolution based on a diffusion model is implemented.

[0022] Beneficial effects of the present invention:

[0023] To address the difficulty in balancing SEM image quality and acquisition efficiency, the present invention relies on real data, is based on a diffusion model, and lightweights the inference process. This reduces the computational burden of achieving super-resolution SEM images, accelerates training and inference speeds, and provides more accurate and reliable data support for microstructural analysis in materials science by improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the lightweight scanning electron microscope image super-resolution method based on the diffusion model provided by the present invention.

[0025] Figure 2 This is a model structure diagram of the lightweight scanning electron microscope image super-resolution method based on the diffusion model provided by the present invention.

[0026] Figure 3 Super-resolution results of low-quality and low-resolution images of IN718 alloy fracture samples using the traditional DDPM method and the lightweight diffusion model.

[0027] Figure 4Schematic diagram of the lightweight scanning electron microscope image super-resolution device based on the diffusion model provided by the present invention. DETAILED DESCRIPTION

[0028] The purpose of this invention is to provide a lightweight scanning electron microscope image super-resolution method based on a diffusion model to reduce the computational burden of the diffusion model training and inference processes, improve image quality, and provide more accurate and reliable data support for microstructural analysis in materials science.

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] In this embodiment, based on a Tescan Clara scanning electron microscope, SEM images of the microstructure of nickel-based polycrystalline fracture samples are collected to construct a data set, and a lightweight diffusion model is trained to achieve super-resolution of the scanning electron microscope images. At the same time, the problem of high computational burden of traditional diffusion models and long training and inference processes is solved.

[0031] The specific steps are as follows:

[0032] like Figure 1-3 As shown in FIG, the lightweight SEM image super-resolution method based on the diffusion model has the following steps:

[0033] S1. Image data pairs were collected using a Tescan Clara scanning electron microscope. The image data were from an IN718 fracture specimen after a high-temperature in-situ tensile test.

[0034] Before the in-situ experiment, the IN718 sample was dog-bone shaped, 50 mm long, and had an observation gauge section of 1.5 mm in length and 1.35 mm in width. The sample was ground and polished, and then the polished sample was corroded to focus on the surface microstructure morphology. The sample was subjected to an in-situ tensile test at 650°C until fracture. 60 pairs of images were collected in the same field of view using a Tescan Clara scanning electron microscope, including different microstructure morphologies. The electron beam dwell time of the low-quality, low-resolution image was 320 nanoseconds per pixel and the resolution was 320×240 pixels, and the electron beam dwell time of the high-quality, high-resolution image was 3.2 microseconds per pixel and the resolution was 1280×960 pixels. In this embodiment, the super-resolution magnification is 4 times, but is not limited to this.

[0035] S2. Cropping the two sets of images separately to construct a training dataset and a test dataset. In this example, the low-quality, low-resolution images are cropped to 64×64 pixels, and the high-quality, high-resolution images are cropped to 256×256 pixels, but this is not limited to this. Randomly select 80% of the image pairs as the training set, the cropped low-quality, low-resolution images as control input images, and the cropped high-quality, high-resolution images as reference real images; the remaining 20% ​​of the images are used as the test set to verify the trained model;

[0036] S3. Build a lightweight inference model, such as Figure 2 As shown, it includes an encoder, N cross attention modules and a decoder. In this embodiment, N is 2, but not limited to this;

[0037] The encoder includes a time embedding block, four groups of residual block-pixel de-reassembly block combinations and an upsampling block. In this embodiment, the downsampling factor of each pixel de-reassembly block is 2, but not limited to this. The upsampling block uses the nearest neighbor interpolation method, but not limited to this. The cross attention module includes two residual blocks and a cross attention block. The decoder includes two groups of residual block-pixel reassembly block combinations. The upsampling factor of each pixel reassembly block should be the same as the downsampling factor of the pixel de-reassembly block in the encoder. In this embodiment, the upsampling factor of each pixel reassembly block is 2, but not limited to this.

[0038] S4. The sampling time information t in each cycle is added to the first residual block of the encoder and the first residual block in each subsequent cross-attention module through the time embedding module. The initial noise image is combined with the control input image after the two groups of residual blocks-pixel decomposition blocks and the upsampling block and input into the subsequent cross-attention module, wherein the upsampling multiple is the same as the super-resolution multiple. In this embodiment, the upsampling multiple is 4 times, but not limited to this; the cross-attention module receives the output of the encoder and the sampling time information after passing through the time embedding module; the output after several cross-attention modules is combined with two groups of residual blocks-pixel recomposition blocks to obtain an inferred noise distribution image; in this embodiment, the number of cross-attention modules is 2, but not limited to this; the upsampled control input image is subtracted from the inferred noise distribution image to obtain an inferred super-resolution image;

[0039] The performance of the inference model is evaluated using peak signal-to-noise ratio, structural similarity index, and Fréchet distance. In this embodiment, mean square error is used as the loss function and Adam optimizer is used as the optimization algorithm, but not limited thereto. A lightweight scanning electron microscope image super-resolution model is trained using the entire dataset, and the optimal hyperparameters are adjusted and determined based on the experimental results. In this embodiment, the learning rate is 10-4 , the batch size is 4, and the number of optimization iterations is 2×10 6 , but not limited to this;

[0040] S5: Input the fast scan mode image into the trained lightweight inference model to generate a super-resolution image. In this embodiment, the low-resolution image in the test set is input into the trained model to complete the scanning electron microscope image super-resolution task. In this embodiment, the peak signal-to-noise ratio of the evaluation index is measured to be 24.26dB, the structural similarity index is 0.478, and the Fréchet distance is 116.247. The calculated model computational complexity evaluation index parameter amount is 11.583M, which is 80.931M lower than the traditional DDPM method, and the floating-point operations per second are 26.7G, which is 121.293G lower than the traditional DDPM method. In addition, Figure 3 The figure shows the comparison results of three groups of images, which respectively show low-quality low-resolution images, traditional DDPM method super-resolution images, lightweight diffusion model super-resolution images proposed in this paper, and high-quality high-resolution images.

[0041] The results show that the use of a lightweight diffusion model for super-resolution of scanning electron microscope images can obtain excellent quantitative data and visual effects, while significantly reducing the amount of model calculations.

[0042] It can be seen that the present invention provides a lightweight scanning electron microscope image super-resolution method based on a diffusion model to reduce the computational burden of the diffusion model training and inference processes, improve image quality, and provide more accurate and reliable data support for microstructural analysis in materials science.

[0043] Corresponding to the aforementioned embodiment of a lightweight scanning electron microscope image super-resolution method based on a diffusion model, the present invention also provides an embodiment of a lightweight scanning electron microscope image super-resolution device based on a diffusion model.

[0044] See also Figure 4 An embodiment of the present invention provides a lightweight scanning electron microscope image super-resolution device based on a diffusion model, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a lightweight scanning electron microscope image super-resolution method based on a diffusion model in the above embodiment.

[0045] The embodiment of a lightweight scanning electron microscope image super-resolution device based on a diffusion model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown in the figure, a hardware structure diagram of a lightweight scanning electron microscope image super-resolution device based on a diffusion model provided by the present invention is provided in any device with data processing capability, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0046] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0047] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0048] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for lightweight scanning electron microscope image super-resolution based on a diffusion model in the above embodiment is implemented.

[0049] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0050] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the lightweight scanning electron microscope image super-resolution method based on a diffusion model.

[0051] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0052] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A lightweight scanning electron microscope image super-resolution method based on a diffusion model, characterized in that: The steps of this method are as follows: S1, using the fast scan mode and slow scan mode in the scanning electron microscope to collect image data pairs, S2. Cropping the two sets of images separately to construct a training dataset, where the cropped fast-scan mode images are used as control input images, and the cropped slow-scan mode images are used as reference real images; S3. Build a lightweight inference model; the lightweight inference model includes an encoder, a decoder, and several cross-attention modules; The decoder inputs sampling time information, a control input image, and a reference real image, upsamples the decoder output image, and inputs it into a cross-attention module and a decoder to obtain an inferred noise distribution image, and subtracts the upsampled control input image from the noise distribution image to obtain a super-resolution image; the encoder includes a time embedding block, a control input channel, a reference real channel, and an upsampling block, and the control input channel and the reference real channel each include two groups of residual block-pixel de-reconstruction block combinations, which are respectively input into the control input image and the reference real image; the decoder includes two groups of residual block-pixel reconstruction block combinations; S4. Use the training dataset to train the lightweight inference model; S5. Input the quick scan mode image into the trained lightweight inference model to generate a super-resolution image.

2. The method for lightweight scanning electron microscope image super-resolution based on a diffusion model according to claim 1, characterized in that: The fast scan mode and slow scan mode are specifically used to collect image data as follows: when collecting image data, the sample is attached to the scanning electron microscope sample stage using conductive glue, and the sample stage is fixed in the scanning electron microscope chamber using screws; an area with rich features is selected for image collection, and during the shooting process, it is necessary to ensure that the shooting field of view remains completely consistent in both the fast scan mode and the slow scan mode.

3. The method for lightweight scanning electron microscope image super-resolution based on diffusion model according to claim 1, characterized in that: The time embedding block is used to add sampling time information to the model; The upsampling block adopts nearest neighbor interpolation, linear interpolation, bilinear interpolation or bicubic interpolation, and the upsampling multiple is the same as the super-resolution multiple.

4. The method for lightweight scanning electron microscope image super-resolution based on a diffusion model according to claim 1, characterized in that: The cross-attention module includes two residual blocks and a cross-attention block; the cross-attention module receives the output of the encoder and the sampling time information after passing through the time embedding module.

5. The lightweight SEM image super-resolution method based on a diffusion model according to claim 1, characterized in that: The training uses mean square error as a loss function and Adam optimizer as an optimization algorithm to train the model; a lightweight scanning electron microscope image super-resolution model is trained using the data set, and the optimal hyperparameters are adjusted and determined based on the experimental results.

6. A lightweight scanning electron microscope image super-resolution device based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the lightweight scanning electron microscope image super-resolution method based on a diffusion model according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a lightweight scanning electron microscope image super-resolution method based on a diffusion model according to any one of claims 1 to 5 is implemented.

Images