Light-weight scanning electron microscope image super-resolution method based on diffusion model

Abstract

The invention discloses a light-weight scanning electron microscope image super-resolution method based on a diffusion model, and relates to the technical field of electron microscope characterization analysis, and the method comprises the following steps: collecting a low-quality low-resolution image group and a high-quality high-resolution image group by using a scanning electron microscope; respectively clipping to construct a training data set; constructing an encoder, a decoder and a lightweight reasoning model; training the model by taking the sampling time, the control input image in the data set and the reference real image as input, and optimizing and updating weight parameters in the lightweight inference model to obtain a trained lightweight inference model; and applying the trained model to the scanning electron microscope image super-resolution. The invention provides a lightweight scanning electron microscope image super-resolution method based on a diffusion model, so as to reduce the calculation burden of the training process and the reasoning process of the diffusion model, improve the image quality and provide more accurate and reliable data support for microstructure analysis of material science.

Description

Technical Field

[0001] The invention relates to the technical field of electron microscope characterization and analysis, and in particular to a lightweight scanning electron microscope image super-resolution method based on a diffusion model. Background Art

[0002] Microstructure not only determines the macroscopic properties of materials, but is also the key to understanding material science phenomena. Scanning electron microscopy has become a common tool for observing microstructure in material science research due to its excellent nanoscale resolution and high depth of field. The scanning electron microscope scans the surface of the sample with an electron beam, and the excited signal is collected and amplified by the detector, ultimately generating a high-contrast two-dimensional image that reveals the detailed features of the sample surface, providing researchers with a window to observe and analyze the microscopic world of materials.

[0003] Although scanning electron microscopes have many advantages, their image acquisition process is inevitably affected by noise. The presence of noise not only reduces the quality of the image, but also affects the accuracy of image analysis and processing. In order to improve the signal-to-noise ratio of the image, the residence time of the electron beam at each pixel can be increased, that is, the slow scan mode can be adopted. However, although this method can improve the image quality, it will also lead to a significant increase in image acquisition time. In addition, the long-term focusing of the high-energy electron beam will damage certain samples and is not suitable for applications such as in situ imaging that require fast and continuous imaging, such as dealing with chemical reaction scenarios, stress relaxation and other problems. Especially when pursuing high-resolution images, the acquisition time will increase significantly, which makes it more difficult to find a balance between image quality and acquisition efficiency.

[0004] DDPM (Denosing Diffusion Probabilistic Model) is an advanced generative model framework that performs well in handling various data generation and recovery tasks. DDPM learns the distribution of data by simulating the inverse process of data generation, that is, gradually removing noise from noisy data. It has been applied in many fields such as image denoising, speech enhancement, text generation, and data recovery. A notable feature of DDPM is its powerful learning ability, which can capture complex patterns in data and gradually restore the true characteristics of data during the denoising process. For super-resolution tasks, images generated by this method usually perform better in visual performance and detail retention. However, the training and inference process of DDPM is relatively cumbersome, has high requirements for computing resources, and is usually time-consuming. Therefore, there is an urgent need for a model lightweight method that can speed up the training and inference speed of SEM image denoising tasks based on DDPM, reduce the computational burden, and provide more accurate and reliable data support for microstructure analysis in materials science by improving image quality. Summary of the invention

[0005] The present invention provides a lightweight SEM image super-resolution method based on a diffusion model to solve the problems of large computing resource requirements and long time consumption of existing DDPM-based image super-resolution methods, and achieves efficient denoising and super-resolution of low-quality SEM images to obtain high-quality SEM images.

[0006] The objective of the present invention is achieved through the following technical solution: a lightweight scanning electron microscope image super-resolution method based on a diffusion model, the method steps are as follows:

[0007] S1, using the fast scan mode and slow scan mode in the scanning electron microscope to collect image data pairs,

[0008] S2, cropping the two groups of images respectively to construct a training data set, wherein the cropped fast scan mode images are used as control input images, and the cropped slow scan mode images are used as reference real images;

[0009] S3, construct a lightweight inference model; the lightweight inference model includes an encoder, a decoder and several cross-attention modules; the decoder inputs sampling time information, a control input image and a reference real image, upsamples the image output by the decoder and inputs it into the cross-attention module and the decoder to obtain an inferred noise distribution image, and subtracts the upsampled control input image from the noise distribution image to obtain a super-resolution image;

[0010] S4. Use the training data set to train the lightweight inference model;

[0011] S5. Input the fast scan mode and slow scan mode images into the trained lightweight inference model to generate super-resolution images.

[0012] Furthermore, the fast scan mode and the slow scan mode are specifically used to collect image data: when collecting image data, the sample is adhered to the scanning electron microscope sample stage using conductive glue, and the sample stage is fixed in the scanning electron microscope chamber using screws; an area with rich features is selected for image collection, and during the shooting process, it is necessary to ensure that the shooting field of view in the fast scan and slow scan modes remains completely consistent.

[0013] Further, the encoder includes a time embedding block, a control input channel, a reference real channel and an upsampling block;

[0014] The time embedding block is used to add sampling time information to the model;

[0015] The control input channel and the reference real channel both include two groups of residual block-pixel anti-recombination block combinations, which are input into the control input image and the reference real image respectively;

[0016] The up-sampling block adopts nearest neighbor interpolation, linear interpolation, bilinear interpolation or bicubic interpolation, and the up-sampling multiple is the same as the super-resolution multiple.

[0017] Furthermore, the criss-cross attention module includes two residual blocks and a criss-cross attention block.

[0018] Furthermore, the decoder comprises two groups of residual block-pixel reassembly block combinations.

[0019] Furthermore, the training uses mean square error as a loss function and uses Adam optimizer as an optimization algorithm to train the model; adjusts parameters to train the model, and determines the optimal hyperparameters of the prediction model; after determining the optimal hyperparameters, uses the data set to train a lightweight scanning electron microscope image super-resolution model.

[0020] On the other hand, the present invention provides a lightweight scanning electron microscope image super-resolution device based on a diffusion model, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the lightweight scanning electron microscope image super-resolution method based on the diffusion model is implemented.

[0021] On the other hand, the present invention provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the diffusion model-based lightweight scanning electron microscope image super-resolution method is implemented.

[0022] Beneficial effects of the present invention:

[0023] To address the problem of difficulty in balancing the image quality and acquisition efficiency of scanning electron microscopes, the present invention relies on real data drive, is based on a diffusion model and lightweights the inference process, thereby reducing the computational burden of achieving super-resolution of electron microscope images, speeding up training and inference, and providing more accurate and reliable data support for microstructure analysis in materials science by improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the lightweight scanning electron microscope image super-resolution method based on the diffusion model provided by the present invention.

[0025] Figure 2 This is a model structure diagram of the lightweight scanning electron microscope image super-resolution method based on the diffusion model provided by the present invention.

[0026] Figure 3 Super-resolution results of low-quality and low-resolution images of IN718 alloy fracture samples using the traditional DDPM method and the lightweight diffusion model.

[0027] Figure 4Schematic diagram of a lightweight scanning electron microscope image super-resolution device based on a diffusion model provided by the present invention. DETAILED DESCRIPTION

[0028] The purpose of the present invention is to provide a lightweight scanning electron microscope image super-resolution method based on a diffusion model to reduce the computational burden of the diffusion model training process and the inference process, improve the image quality and provide more accurate and reliable data support for the microstructure analysis of materials science.

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] In this embodiment, based on the Tescan Clara scanning electron microscope, SEM images of the microstructure are collected in the nickel-based polycrystalline fracture sample to construct a data set, and a lightweight diffusion model is trained to achieve super-resolution of the scanning electron microscope image. At the same time, the problem that the traditional diffusion model has a high computational burden and the training and reasoning process is time-consuming is solved.

[0031] The specific steps are as follows:

[0032] like Figure 1-3 As shown in FIG. 1 , the lightweight SEM image super-resolution method based on the diffusion model has the following steps:

[0033] S1, using Tescan Clara scanning electron microscope to collect image data pairs, the image data source is IN718 fracture specimen after high temperature in-situ tensile test;

[0034] Before the in-situ experiment, the IN718 sample was dog-bone shaped, 50 mm long, and contained an observation gauge section with a length of 1.5 mm and a width of 1.35 mm; the sample was ground and polished, and then the polished sample was corroded to focus on the surface microstructure morphology; the sample was subjected to an in-situ tensile test at 650°C until it broke; 60 pairs of images were collected under the same field of view using a Tescan Clara scanning electron microscope, including different microstructure morphologies, wherein the electron beam residence time of the low-quality and low-resolution image was 320 nanoseconds / pixel, the resolution was 320×240 pixels, and the electron beam residence time of the high-quality and high-resolution image was 3.2 microseconds / pixel, and the resolution was 1280×960 pixels; in this embodiment, the super-resolution multiple is 4 times, but is not limited thereto;

[0035] S2. Crop the two groups of images separately to form a training data set and a test data set. In this example, the low-quality and low-resolution images are cropped to 64×64 pixels, and the high-quality and high-resolution images are cropped to 256×256 pixels, but not limited to this. Randomly select 80% of the image pairs as the training set, the cropped low-quality and low-resolution images as the control input images, and the cropped high-quality and high-resolution images as the reference real images; the remaining 20% ​​of the images are used as the test set to verify the trained model.

[0036] S3. Build a lightweight inference model, such as Figure 2 As shown, it includes an encoder, N cross attention modules and a decoder. In this embodiment, N is 2, but not limited to this;

[0037] The encoder includes a time embedding block, four groups of residual block-pixel de-reassembly block combinations and an upsampling block. In this embodiment, the downsampling multiple of each pixel de-reassembly block is 2, but not limited to this. The upsampling block adopts the nearest neighbor interpolation method, but not limited to this; the cross attention module includes two residual blocks and a cross attention block; the decoder includes two groups of residual block-pixel reassembly block combinations, and the upsampling multiple of each pixel reassembly block should be the same as the downsampling multiple of the pixel de-reassembly block in the encoder. In this embodiment, the upsampling multiple of each pixel reassembly block is 2, but not limited to this;

[0038] S4, the sampling time information t in each cycle process is added to the first residual block of the encoder and the first residual block in each subsequent cross-attention module through the time embedding module, the initial noise image is combined through two groups of residual blocks-pixel anti-recombination blocks, and the control input image after the combination of two groups of residual blocks-pixel anti-recombination blocks and the upsampling block is combined and input into the subsequent cross-attention module, wherein the upsampling multiple is the same as the super-resolution multiple. In this embodiment, the upsampling multiple is 4 times, but not limited to this; the cross-attention module receives the output of the encoder and the sampling time information after passing through the time embedding module; the output after several cross-attention modules is combined through two groups of residual blocks-pixel recombination blocks to obtain the inferred noise distribution image; in this embodiment, the number of cross-attention modules is 2, but not limited to this; the up-sampled control input image is subtracted from the inferred noise distribution image to obtain the inferred super-resolution image;

[0039] The performance of the inference model is evaluated using peak signal-to-noise ratio, structural similarity index, and Fréchet distance. In this embodiment, mean square error is used as the loss function and Adam optimizer is used as the optimization algorithm, but not limited thereto. A lightweight scanning electron microscope image super-resolution model is trained using all data sets, and the optimal hyperparameters are adjusted and determined based on the experimental results. In this embodiment, the learning rate is 10-4 , the batch size is 4, and the number of optimization iterations is 2×10 6 , but not limited to this;

[0040] S5: Input the fast scan mode image into the trained lightweight inference model to generate a super-resolution image. In this embodiment, the low-resolution image in the test set is input into the trained model to complete the SEM image super-resolution task. In this embodiment, the peak signal-to-noise ratio of the evaluation index is measured to be 24.26dB, the structural similarity index is 0.478, and the Fréchet distance is 116.247. The calculated model calculation complexity evaluation index parameter is 11.583M, which is 80.931M lower than the traditional DDPM method, and the number of floating-point operations per second is 26.7G, which is 121.293G lower than the traditional DDPM method. In addition, Figure 3 Three groups of image comparison results are shown, which respectively show low-quality low-resolution images, traditional DDPM method super-resolution images, lightweight diffusion model super-resolution images proposed in this invention, and high-quality high-resolution images.

[0041] The results show that the use of a lightweight diffusion model for super-resolution of SEM images can obtain excellent quantitative data and visual effects while significantly reducing the amount of model calculations.

[0042] It can be seen that the present invention provides a lightweight scanning electron microscope image super-resolution method based on a diffusion model to reduce the computational burden of the diffusion model training process and the inference process, improve the image quality, and provide more accurate and reliable data support for the microstructure analysis of materials science.

[0043] Corresponding to the aforementioned embodiment of a lightweight scanning electron microscope image super-resolution method based on a diffusion model, the present invention also provides an embodiment of a lightweight scanning electron microscope image super-resolution device based on a diffusion model.

[0044] See also Figure 4 A lightweight scanning electron microscope image super-resolution device based on a diffusion model provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a lightweight scanning electron microscope image super-resolution method based on a diffusion model in the above embodiment.

[0045] The embodiment of a lightweight scanning electron microscope image super-resolution device based on a diffusion model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 4 As shown, it is a hardware structure diagram of a lightweight scanning electron microscope image super-resolution device based on a diffusion model provided by the present invention, in which any device with data processing capability is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0046] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0047] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0048] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a lightweight scanning electron microscope image super-resolution method based on a diffusion model in the above embodiment is implemented.

[0049] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0050] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the diffusion model-based lightweight scanning electron microscope image super-resolution method is implemented.

[0051] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0052] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A lightweight scanning electron microscope image super-resolution method based on a diffusion model, characterized in that: The steps of this method are as follows: S1, using the fast scan mode and slow scan mode in the scanning electron microscope to collect image data pairs, S2, cropping the two groups of images respectively to construct a training data set, wherein the cropped fast scan mode images are used as control input images, and the cropped slow scan mode images are used as reference real images; S3. Construct a lightweight inference model; the lightweight inference model includes an encoder, a decoder, and several cross attention modules; The sampling time information, control input image and reference real image are input through the decoder, the image output by the decoder is upsampled and input into the cross attention module and the decoder to obtain the inferred noise distribution image, and the upsampled control input image is subtracted from the noise distribution image to obtain the super-resolution image; S4. Use the training data set to train the lightweight inference model; S5. Input the quick scan mode image into the trained lightweight inference model to generate a super-resolution image.

2. The lightweight SEM image super-resolution method based on diffusion model according to claim 1, characterized in that: The fast scan mode and slow scan mode are specifically used to collect image data: when collecting image data, the sample is adhered to the scanning electron microscope sample stage using conductive glue, and the sample stage is fixed in the scanning electron microscope chamber using screws; an area with rich features is selected for image collection, and during the shooting process, it is necessary to ensure that the shooting field of view in the fast scan mode and the slow scan mode remains completely consistent.

3. The lightweight SEM image super-resolution method based on diffusion model according to claim 1, characterized in that: The encoder includes a time embedding block, a control input channel, a reference real channel and an upsampling block; The time embedding block is used to add sampling time information to the model; The control input channel and the reference real channel both include two groups of residual block-pixel anti-recombination block combinations, which are input into the control input image and the reference real image respectively; The up-sampling block adopts nearest neighbor interpolation, linear interpolation, bilinear interpolation or bicubic interpolation, and the up-sampling multiple is the same as the super-resolution multiple.

4. The lightweight SEM image super-resolution method based on diffusion model according to claim 1, characterized in that: The cross-attention module includes two residual blocks and a cross-attention block; the cross-attention module receives the output of the encoder and the sampling time information after passing through the time embedding module.

5. The lightweight SEM image super-resolution method based on diffusion model according to claim 1, characterized in that: The decoder includes two sets of residual block-pixel reconstructed block combinations.

6. The lightweight SEM image super-resolution method based on diffusion model according to claim 1, characterized in that: The training uses mean square error as a loss function and Adam optimizer as an optimization algorithm to train the model; a lightweight scanning electron microscope image super-resolution model is trained using the data set, and the optimal hyperparameters are adjusted and determined based on experimental results.

7. A lightweight scanning electron microscope image super-resolution device based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a lightweight scanning electron microscope image super-resolution method based on a diffusion model as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a lightweight scanning electron microscope image super-resolution method based on a diffusion model as described in any one of claims 1 to 6 is implemented.

Images