A microscopic electron microscope image processing method and system based on multi-scale scaling

Abstract

The present invention provides a microscopic electron microscope image processing method and system based on multi-scale scaling. The method comprises acquiring an image of a target object using a microscopic electron microscope, storing the acquired microscopic electron microscope image in a file, performing bilinear interpolation downsampling on the image, and saving the bilinear interpolation downsampling image as a low-resolution image (LR); rotating, scaling, flipping, adjusting brightness and contrast, and adding random noise to the low-resolution image (LR) to generate additional variants; obtaining a microscopic image dataset; and constructing a multi-scale scaling microscopic electron microscope image super-resolution (MS-MESR) network model. The present invention enables flexible scaling control, achieving more selective scaling results, and significantly improving the resolution and quality of microscopic electron microscope images, as well as their applicability under different scaling factors.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution processing, and in particular to a microscopic electron microscope image processing method and system based on multi-scale scaling. Background Art

[0002] Scanning electron microscopes (SEMs) are powerful tools for observing the microstructure of materials and biological samples. These devices are capable of displaying sample details at high resolution and are widely used in fields such as materials science, biomedicine, and semiconductor manufacturing. For analyzing microscopic features, high-resolution images provide accurate and comprehensive information, making them crucial for researchers conducting image analysis. However, despite the high resolution offered by microscopic electron microscopes, image quality and detail accuracy are still affected by various factors, such as image noise, image blur, and resolution limitations.

[0003] Traditional microscopic electron microscope images require the following series of operations inside the device to obtain high-resolution images:

[0004] Image denoising: For example, filters (such as Gaussian filtering, mean filtering, etc.) are used to reduce noise in images. However, these methods often lead to the loss of image details, especially when processing high-noise images, which may erase important structural information.

[0005] Image sharpening: Improve the contrast and details of an image by using sharpening filters (such as Laplacian filters and edge enhancement filters). However, this method may amplify noise and artifacts in the image, making the actual structure of the image difficult to discern.

[0006] Multi-frame image synthesis: This method improves resolution by synthesizing multiple images. Although this method can improve image clarity to a certain extent, the synthesis process is complex and computationally intensive, the operation is tedious, and may introduce errors.

[0007] This series of operations results in a high computational complexity for generating high-quality SEM images, consuming significant computing resources. The process is complex and requires high operator input. This processing time is long, impacting the efficiency of real-time image analysis and applications. For example, a 3840 x 2400 pixel image takes twice as long to process as a 960 x 600 pixel image. This high cost and the high cost of SEM equipment lead to high unit labor costs, and the long processing time incurs significant costs. Summary of the Invention

[0008] In view of the shortcomings of existing traditional microscopic electron microscope image processing methods, the purpose of the present invention is to provide a microscopic electron microscope image processing method and system based on multi-scale scaling to solve the above-mentioned obvious defects in the prior art.

[0009] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0010] In a first aspect, the present invention provides a method for processing a microscopic electron microscope image based on multi-scale scaling, the method comprising the following steps:

[0011] Step 1: Acquire an image of the target object through a microscopic electron microscope, store the acquired microscopic electron microscope image in a file, perform bilinear difference downsampling on the image, and save the image after bilinear difference downsampling as a low-resolution image LR; rotate, scale, flip, adjust the brightness and contrast of the low-resolution image LR, and add random noise to generate additional variants; obtain a microscopic image dataset;

[0012] Step 2: Constructing a multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model

[0013] The multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model includes a decoding part consisting of a pre-trained autoencoder, a latent diffusion model LDM, an implicit neural decoder and a multi-layer perceptron MLP; the decoding part consisting of the implicit neural decoder and the multi-layer perceptron is used to reconstruct the latent feature vector z' in the last time step state and feed it back to the image space;

[0014] Input low-resolution image LR, encode it in pixel space through the autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector h, w, and c are the height, width, and number of channels of the input low-resolution image LR respectively;

[0015] In the latent diffusion model LDM, random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is spliced ​​with the random input noise and used as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is spliced ​​again with the latent feature vector z obtained by the pre-trained autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state The latent feature vector z is input into the implicit neural decoder. At the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted. The latent feature vector z' in the last time step state is input into the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image.

[0016] In the multi-layer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ;

[0017]

[0018] Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U;

[0019] The decoded output image is expressed as follows:

[0020]

[0021] Among them, I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system, It means that it conforms to the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * Perform Euclidean distance interpolation, and after splicing, the multi-layer perceptron MLP performs the final pixel space to image space decoding process, F θ Functional representation of the mathematical model of the multi-layer perceptron MLP;

[0022] Step 3: Use the microscopic image dataset to train the MS-MESR network model. The trained MS-MESR network model is used to process microscopic electron microscope images in real time, obtain super-resolution images corresponding to low-resolution images, and perform detail restoration and enhancement.

[0023] Furthermore, the pre-trained autoencoder consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the tail, and also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv.

[0024] Furthermore, the latent diffusion model (LDM) is a diffusion model trained in the latent space of a pre-trained autoencoder to achieve high-quality image synthesis. The entire LDM diffusion process consists of a forward process and a backward process. The forward process gradually adds random noise to the input data, while the backward process constructs the required data samples from the noise. The forward process is a fixed process, and the noise latent feature vector z at time step T is expressed as:

[0025]

[0026] Where I represents the identity matrix; z0 represents the eigenvector of the original input data in the latent space, represents 1-β from time step 1 to T T Multiplication; “:=” indicates the equivalent of the symbol, β is a hyperparameter that represents the noise intensity added at each time step during the diffusion process; It is a hyperparameter sequence that represents the noise intensity in each time step i(0~T), and ∈ refers to the random Gaussian noise that conforms to the standard normal distribution in the diffusion model; conversely, in the reverse process, it is the inversion of the forward process.

[0027] Furthermore, the multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model is a two-stage process. The two-stage alignment strategy defines the target loss function of the entire network as:

[0028] L align =λ1L dm +λ2L recon

[0029] Where λ1 and λ2 are custom parameters for calculation; L dm represents the denoising objective loss function of the latent diffusion model; L recon Represents the reconstruction loss L of the real image obtained using a micro electron microscope SEM recon ;

[0030] The reconstruction loss L of the real image obtained using a micro electron microscope SEM recon Defined as:

[0031]

[0032] Among them, I0 is the original input image, Represents the latent feature vector from the prediction The generated image;

[0033] The denoising objective loss function of the latent diffusion model is expressed as:

[0034]

[0035] Among them, z0 represents the potential feature vector in the initial state.

[0036] Furthermore, the implicit neural decoder includes a shallow 3×3 convolutional layer Conv and a main decoding part, which is composed of two sub-parts in series, each sub-part including a group normalization layer, a Sigmoid activation function layer and a 3×3 convolutional layer Conv connected in sequence; the potential feature vector z' is input into the shallow 3×3 convolutional layer Conv to obtain shallow feature information, and the shallow feature information is passed through the main decoding part to obtain deep feature information, and a residual mechanism is introduced to add the shallow feature information and the deep feature information, and the addition result is spliced ​​with the coordinate information corresponding to the image pixels in the latent space coordinate system as the input of the multi-layer perceptron.

[0037] Furthermore, during the training process, the quality of the super-resolution image is evaluated by calculating the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM).

[0038] In a second aspect, the present invention provides a microscopic electron microscope image processing system based on multi-scale scaling, which executes the method described.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The method of the present invention adopts a multi-scale zoom microscopic electron microscope image super-resolution MS-MESR (Multi-Scale Microscopic Electron microscopy image Super-Resolution) network model. In the decoding part including the multi-layer perceptron MLP, it can realize flexible zoom factor control. Compared with the traditional fixed zoom learning method, it realizes more selective results of the zoom factor, significantly improves the resolution and quality of the microscopic electron microscope image, and the applicability under different zoom factors, provides more accurate and clear image support for the application of microscopic electron microscopes, is a more efficient processing method, and can obtain higher quality microscopic electron microscope images.

[0041] The symmetric autoencoder-implicit neural decoder architecture of this invention maintains symmetry between the convolutional layers of the autoencoder and the implicit neural decoder. This design enhances information preservation and detail recovery in image super-resolution tasks. Through symmetric convolution and transposed convolution operations, this architecture gradually restores the image's high resolution while effectively preserving the feature information extracted during the encoding process. This architecture not only improves training stability, but also reduces overfitting and simplifies model debugging and optimization, resulting in clearer, more accurate, and high-resolution images.

[0042] The implicit neural decoder in this paper combines shallow and deep convolutional layers, effectively fusing multi-level feature information through group normalization, sigmoid activation, and 3×3 convolutional layers. This architecture utilizes a residual mechanism to combine shallow and deep features, enhancing the ability to recover image detail while significantly reducing detail loss and artifacts, improving the visual quality of the image. This design enables the model to more accurately recover image details and edges, improving the quality of the resulting image.

[0043] The two-stage alignment strategy in this invention improves the speed of learning and reasoning and reduces error propagation in model training by optimizing the alignment process in the two stages of feature extraction and image generation. The objective function of the overall network is defined by combining reconstruction loss and denoising target loss, thereby improving the accuracy and stability of the model. The final generated image has higher resolution and quality.

[0044] Furthermore, the present invention offers significant advantages in terms of resolution improvement. It can recover more details from low-resolution images, breaking through the limitations of traditional resolution, achieving clearer and more detailed images while reducing detail loss. It can effectively preserve and enhance image details, reducing detail loss and artifacts caused by traditional denoising and sharpening processes.

[0045] In terms of computational efficiency, although super-resolution processing requires certain computing resources, its processing efficiency is higher than that of traditional multi-frame synthesis technology, achieving the practical purpose of reducing costs and increasing efficiency. The present invention can complete super-resolution processing in a shorter time, significantly improve image quality, improve the visual effect of the image, and improve the accuracy of sample observation, thereby enhancing the reliability of scientific research and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the structure of the multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model in the present invention.

[0047] Figure 2 Schematic diagram of the connection structure between the implicit neural decoder and the multi-layer perceptron in the present invention.

[0048] Figure 3 Schematic diagram of the two-stage alignment strategy in the present invention. DETAILED DESCRIPTION

[0049] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the figures and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0050] Throughout this specification, references to "one embodiment," "some embodiments," and the like indicate that a particular feature, structure, material, or characteristic described in connection with that embodiment is included in at least one embodiment of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments.

[0051] The present invention is based on a multi-scale micro-electron microscope image processing method. Its overall steps are: preparing a micro-electron microscope image dataset, constructing a multi-scale micro-electron microscope image super-resolution MS-MESR network model, and training the multi-scale micro-electron microscope image super-resolution MS-MESR network model using the micro-electron microscope image dataset. The specific steps are as follows:

[0052] Step 1: Obtain a microscopic image dataset

[0053] An image of the target object is captured using a microscopic electron microscope (SEM). The image is then stored in a file and bilinearly downsampled. This image is saved as a low-resolution image (LR), which serves as the input image for network training. To further improve the generalization and robustness of the model, data augmentation techniques are applied. Images are rotated, scaled, flipped, brightness and contrast adjusted, and random noise added to generate additional variations. This helps expand the diversity of the training data, enhances the model's generalization, and ensures that the model can better handle a variety of real-world situations.

[0054] Depending on the actual situation, the target object can be photographed at different resolutions. After processing by the method of the present invention, a higher resolution image can be obtained relative to the input. The types of target objects in the microscopic image dataset of the present invention can be as diverse as possible, thereby improving the generalization ability of the model.

[0055] Step 2: Constructing a multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model

[0056] The multi-scale zoom microscopic electron microscope image super-resolution MS-MESR network model, its structure is as follows Figure 1 shown.

[0057] The multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model includes a decoding part consisting of a pre-trained autoencoder, a latent diffusion model LDM (Latent Diffusion Model), an implicit neural decoder and a multi-layer perceptron MLP.

[0058] Input low-resolution image LR, encode it in pixel space through the autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector During the training process, the potential feature vector z is continuously expanded and contracted in dimension;

[0059] After the feature map information of LR is obtained by autoencoding and extracted as a potential feature vector z, in the latent diffusion model LDM, the random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The potential feature vector z is spliced ​​with the random input noise and used as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is spliced ​​again with the potential feature vector z obtained by the autoencoder to obtain the potential feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state The latent feature vector z is input into the implicit neural decoder. At the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted. The latent feature vector z' in the last time step state is input into the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image.

[0060] In the multi-layer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ;

[0061]

[0062] Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U;

[0063] The decoded output image is expressed as follows:

[0064]

[0065] Among them, I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system, It means that it conforms to the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * Perform Euclidean distance interpolation, and after splicing, the multi-layer perceptron MLP performs the final pixel space to image space decoding process, F θ A function representation of the mathematical model of a multi-layer perceptron (MLP).

[0066] Step 3: Use the microscopic image dataset to train the MS-MESR network model. The trained MS-MESR network model is used to process microscopic electron microscope images in real time, obtain super-resolution images corresponding to low-resolution images, and perform detail restoration and enhancement.

[0067] The pre-trained autoencoder in the present invention follows the basic convolutional and transposed convolutional neural network structure. The pre-trained autoencoder (CNN structure) consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the end, which also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv. The main function of the autoencoder is to transform the input image into a Perform encoding operation to obtain the initial feature map information, and input image Extracted as latent feature vector The subsequent decoding part consisting of the implicit neural decoder and the multi-layer perceptron reconstructs the potential feature vector z' in the last time step state and feeds it back to the image space.

[0068] The convolution parts of the autoencoder and the implicit neural decoder in the network model of the present invention are symmetrical. The main advantage of using a symmetrical structure is that it can effectively enhance the information retention and detail restoration capabilities in image super-resolution tasks. Through convolution and transposed convolution operations that are symmetrical to the autoencoder, the high resolution of the image can be gradually restored, and the feature information extracted during the encoding process can be maintained and refined, thereby improving the reconstruction quality of image details and edges. This special structure not only improves the stability and generalization ability of the training process, but also simplifies the debugging and optimization process of the model, helps to reduce overfitting and improve the reconstruction effect of the final image. In addition, it supports more natural information flow and high-quality image restoration through structural consistency and regularity.

[0069] The structure of the implicit neural decoder is as follows Figure 2As shown in the figure, the input is the latent feature vector z' at the previous time step. It passes through a shallow 3×3 convolutional layer (Conv) to obtain shallow feature information. This then enters the main decoding section, which consists of two subsections: a group normalization layer, a sigmoid activation function layer, and a 3×3 convolutional layer (Conv). This section converts the shallow feature information into deep feature information. A residual mechanism is introduced to add the shallow and deep feature information. This addition result is then concatenated with the coordinate information corresponding to the image pixels in the latent space coordinate system and input into a multilayer perceptron for the final decoding operation.

[0070] Specifically, the autoencoder-implicit neural decoder structure increases the effective information by expanding the dimension of the potential feature vector. The feature vector passes through the multi-layer perceptron MLP, transfers from the pixel space to the image space, and then decodes it into an image. In the multi-layer perceptron, its learning process is realized by mapping the RGB pixel values ​​of the image to the output coordinates.

[0071] In addition, the key to the multi-scale zoom super-resolution technology of the present invention lies in the decoding part composed of an implicit neural decoder and a multi-layer perceptron. In the multi-layer perceptron, a zoom multiple sequence s∈(1,M] is set, where M represents the maximum zoom multiple. A zoom multiple N is randomly selected from s to form a subsequence s* of zoom multiples from 1 to N, and the zoom multiple s* is mapped to a vector set When the scaling factor is adjusted in the implicit space, the modeling process of the working parameters of the multilayer perceptron at different scaling factors can be expressed as:

[0072] U=Reshape(s * )

[0073]

[0074] in, express The regularization result of , setting the parameter δ = 1e-8 to avoid the denominator of the regularization process to zero; Reshape represents the mapping function; Represents an element in the vector set U;

[0075] The working parameters f of the multilayer perceptron with different scaling factors in the subsequence corresponding to the random scaling factor N in the implicit space can be obtained. θ Expressed as:

[0076]

[0077] The decoded output image is expressed as follows:

[0078]

[0079] Among them, I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system, It means that it conforms to the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * Perform Euclidean distance interpolation, and after splicing, the multi-layer perceptron MLP performs the final pixel space to image space decoding process, F θ A function representation of the mathematical model of a multi-layer perceptron (MLP).

[0080] The core idea of ​​the latent diffusion model (LDM) is to train a diffusion model in the latent space of a pre-trained autoencoder to achieve high-quality image synthesis with a flexible range of styles and resolutions. The entire LDM diffusion process consists of a forward process, which gradually adds random noise to the input data, and a backward process, which constructs the desired data samples from the noise. The forward process is a fixed process, and the noise latent feature vector z at time step T can be expressed as:

[0081]

[0082] Among them, z0 represents the feature vector of the original input data in the latent space; z T Represents the feature vector at time step T in the latent space; in represents α from time step 1 to T T For continuous multiplication, “:=” means it is equivalent to the symbol, α T =1-β T , β is a hyperparameter that represents the noise intensity added at each time step during the diffusion process, It is a hyperparameter sequence that represents the noise intensity in each time step i(0~T); ∈ refers to the random Gaussian noise that conforms to the standard normal distribution in the diffusion model.

[0083] On the contrary, the backward process is the reverse of the forward process. However, the backward distribution probability p(z T-1 |z T ) is more difficult because it requires knowledge of the distribution of all potential images to calculate this conditional probability. Therefore, the distribution probability p is approximated by a neural network θ You can use Bayes' theorem as follows:

[0084]

[0085] in, μ T represents the mean value under time step T, represents the variance at time step T.

[0086] The diffusion model can get better results in predicting noise at each time step T, so the diffusion model ∈ θ The objective function of (.) can be expressed as:

[0087]

[0088] Compared to traditional diffusion models that operate in pixel space, the network structure of this invention is a two-stage model, designed to speed up learning and inference. The second stage relies on the intermediate representations or features extracted in the first stage. During training, the autoencoder may introduce errors that are passed to the main diffusion network, which in turn generates additional errors. These errors, trained separately from the implicit neural decoder, can reduce decoding effectiveness.

[0089] The multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model described in the present invention is a two-stage process. The two-stage alignment strategy defines the target loss function of the entire network as:

[0090] L align =λ1L dm +λ2L recon

[0091] Where λ1 and λ2 are custom parameters for calculation; L dm represents the denoising objective loss function of the latent diffusion model; L recon Represents the reconstruction loss L of the real image obtained using a micro electron microscope SEM recon ;

[0092] The specific two-stage alignment strategy is as follows Figure 3 As shown in Figure 2, the quality of the output image is improved by reducing the alignment error between the model stages, and finally the required output image is obtained. The reconstruction loss L for the real image obtained using SEM is recon To define:

[0093]

[0094] Among them, I0 is the original input image, Represents the latent feature vector from the prediction The generated image.

[0095] Combined with the principle equation of the potential diffusion model, the denoising target loss function of the potential diffusion model can be expressed as:

[0096]

[0097] according to The above formula can be simplified to:

[0098]

[0099] Among them, L dm represents the denoising objective loss function of the latent diffusion model, and z0 represents the latent feature vector in the initial state. In combination with the principle of the diffusion model, the concept of time step is used to implement the inference process from the initial state and realize the fine-tuning of the diffusion model.

[0100] The super-resolution images processed by this method are used for scientific research and data analysis in fields such as materials science and biomedicine. They enhance the clarity and detail of microscopic electron microscope images, supporting more accurate sample observation and analysis. Furthermore, the method can optimize network models based on feedback from actual applications, improving processing efficiency and image quality. The method also collects user feedback on image processing results to further adjust and improve parameter settings.

[0101] Example 1

[0102] The present invention is based on a multi-scale zoom micro-electron microscope image processing method, using the MS-MESR network model, to achieve super-resolution processing of micro-electron microscope images through the following steps:

[0103] 1. Data collection and preparation stage

[0104] 1.1 Data Collection

[0105] Equipment: Use a scanning electron microscope (SEM) to capture images of different target samples. The equipment is capable of capturing high-resolution details of the samples.

[0106] Procedure: Take a set of images for each sample, ensuring that different viewing angles and magnifications are covered.

[0107] 1.2 Data Processing

[0108] Preprocessing: Bilinear interpolation downsampling is applied to the image to generate a low-resolution image (LR).

[0109] Data augmentation: Perform data augmentation operations such as rotating, scaling, flipping, adjusting brightness and contrast on images to improve the robustness and generalization ability of the model.

[0110] Annotation process: Annotate important details in microscopic electron microscope images to ensure the accuracy and completeness of training data.

[0111] Dataset acquisition: All preprocessed images are divided into training and test sets in a 7:3 ratio to obtain a microscopic image dataset.

[0112] 2. Model Construction

[0113] Network structure: The MS-MESR network model contains a symmetrical autoencoder-implicit neural decoder structure, an implicit neural decoder and a latent diffusion model.

[0114] Symmetrical autoencoder-implicit neural decoder structure: Through symmetric convolution and transposed convolution operations, the high resolution of the image is gradually restored, maintaining feature information.

[0115] Implicit neural decoder: Combining shallow and deep convolutional layers, it fuses multi-level feature information through group normalization layers, sigmoid activation function layers, and 3×3 convolutional layers to reduce detail loss and artifacts.

[0116] Latent Diffusion Model: A diffusion model is trained in the latent space to achieve high-quality image synthesis using a forward and backward process. The diffusion process adds noise in the forward direction and recovers data from the noise in the reverse direction.

[0117] Input low-resolution image LR, encode it in pixel space through the autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector h, w, and c are the height, width, and number of channels of the input low-resolution image LR respectively;

[0118] In the latent diffusion model LDM, random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is spliced ​​with the random input noise and used as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is spliced ​​again with the latent feature vector z obtained by the pre-trained autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state The latent feature vector z is input into the implicit neural decoder. At the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted. The latent feature vector z' in the last time step state is input into the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image.

[0119] In the multi-layer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ;

[0120]

[0121] Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U;

[0122] The decoded output image is expressed as follows:

[0123]

[0124] Among them, I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system, It means that it conforms to the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * Perform Euclidean distance interpolation, and after splicing, the multi-layer perceptron MLP performs the final pixel space to image space decoding process, F θ A function representation of the mathematical model of a multi-layer perceptron (MLP).

[0125] 3. MS-MESR network model training

[0126] When the network starts training, the network initialization parameter settings are completed first: the epoch of the training network is set to 300, the Adam optimizer is used as the optimizer, and the initial learning rate of Adam is set to 5e-5 and 1e-6. The learning rate and other hyperparameters are adjusted according to the needs of model training, and the initialization parameters λ1 and λ2 of the two calculations of the objective function are set to 1.0.

[0127] During the training phase, the LR images of the training set are input into the network from the constructed microscopic image dataset, and these images are read according to the training set storage path and then processed according to the network structure;

[0128] The input image first passes through the autoencoder to obtain the initial feature map information and extracts it as the potential feature vector z. Before inputting it into the latent diffusion model LDM denoising UNet, the latent diffusion model LDM generates an initial random Gaussian noise that conforms to the normal distribution during the forward process, splices it with the potential feature vector z, and then uses the spliced ​​result as the input of the denoising UNet. After obtaining the output feature information of the denoising UNet, it is spliced ​​again with the potential feature vector z to obtain the potential feature vector of the denoising UNet at the previous time step T-1. The decoding task is arranged in the implicit neural decoder, and the coordinate mechanism is integrated to decode the feature information, and the required super-resolution image SR is obtained by decoding from the pixel space to the image space.

[0129] When the overall network objective loss function no longer shows a significant decrease (error ±1e-5), the network training area is considered stable and the network training process is completed.

[0130] Loss function: The objective function of the model is defined by combining the reconstruction loss and the denoising target loss, and the model parameters are optimized to improve the image quality.

[0131] 4. Super-resolution processing implementation

[0132] 4.1 Model Deployment

[0133] Deployment: The trained MS-MESR network model is deployed into the image processing system for real-time processing of microscopic electron microscope images.

[0134] Processing: Input a low-resolution image LR and use the MS-MESR model to generate the corresponding high-resolution image for detail recovery and enhancement.

[0135] 4.2 Results Analysis

[0136] Quality assessment: The quality of the super-resolution image is evaluated by calculating the PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index) of the image.

[0137] Example 2

[0138] Background: Traditional methods for processing micro-electron microscope images often face high costs and long processing times. For example, processing 500 micro-electron microscope images with traditional methods takes approximately eight hours and costs up to 4,000 yuan, a significant financial burden for large-scale image processing tasks. The proposed MS-MESR network model significantly improves processing efficiency and reduces costs, providing a more economical solution for micro-electron microscope image processing.

[0139] The total number of microscopic electron microscope images obtained in the embodiment network is 5,000, which are divided into a training set and a test set in a ratio of 7:3.

[0140] The MS-MESR network model is the same as that in Example 1. The pre-trained autoencoder consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the tail, which also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv.

[0141] The implicit neural decoder includes a shallow 3×3 convolutional layer Conv and a main decoding part, which is composed of two sub-parts in series, each sub-part includes a group normalization layer, a Sigmoid activation function layer and a 3×3 convolutional layer Conv connected in sequence; the potential feature vector z' is input into the shallow 3×3 convolutional layer Conv to obtain shallow feature information, and the shallow feature information is passed through the main decoding part to obtain deep feature information, and a residual mechanism is introduced to add the shallow feature information and the deep feature information, and the addition result is spliced ​​with the coordinate information corresponding to the image pixels in the latent space coordinate system as the input of the multi-layer perceptron.

[0142] Cost-Benefit Analysis:

[0143] Traditional method: It takes about 8 hours to process 500 images and costs 4,000 yuan.

[0144] MS-MESR network model method: Using the MS-MESR network model, processing time was reduced to approximately 2.7 hours. This two-thirds reduction in processing time also significantly reduced processing costs, from 4,000 yuan to approximately 1,300 yuan. This translates to a savings of 2,700 yuan per 500 images.

[0145] By adopting the MS-MESR network model, the cost of processing microelectron microscope images has been significantly reduced. When processing 500 images, the MS-MESR method costs only one-third of traditional methods. This not only saves processing fees but also reduces the additional financial costs associated with long processing times. This cost savings is particularly significant in large-scale image processing projects.

[0146] The MS-MESR network model has achieved significant cost reduction and efficiency improvement in the field of micro-electron microscopy image processing. By improving processing efficiency and reducing processing costs, it provides a more economical solution for scientific research and industrial applications, greatly promoting the popularization and application of micro-electron microscopy image processing technology.

[0147] On the same microscopic image data set, the MS-MESR network model of the present invention was compared with existing methods. The comparison results showed that compared with the real images directly obtained by the micro-electron microscope, the method proposed by the present invention still maintained high-quality indicators, with a difference of only 0.02 PSNR (peak signal-to-noise ratio) from the real images. Compared with existing excellent diffusion model methods such as SRDiff and DDPM, the super-resolution method of the present invention has lower computational complexity and better performance. The results obtained by using multiple network models for super-resolution processing of micro-electron microscope images after training are shown in Table 1, which is the indicator comparison data under the scaling factor N = 4 times. When performing network testing, the prepared test set images are input and the network weights of the training results are imported. When the peak signal-to-noise ratio PSNR and structural similarity SSIM of the image reach 37.50 (error ± 0.03) and 0.87 (error ± 0.02) respectively, the entire network training is completed. The other models in Table 1 directly fix the scaling factor to 4.

[0148] Table 1 Comparison of indicators of different network models

[0149]

[0150] The results shown in Table 1 demonstrate that, under the same test conditions, the MS-MESR network model achieves higher peak signal-to-noise ratio and structural similarity for super-resolution micro-electron microscopy images than the existing, highly effective SRDiff and DDPM models. This significant difference is also evident when compared to the aforementioned diffusion models of the same type, demonstrating the superior performance of the present invention's super-resolution micro-electron microscopy technology.

[0151] Example 3

[0152] On the same microscopic image dataset, the MS-MESR network model of the present invention is used to realize super-resolution processing of microscopic electron microscope images under multi-scale scaling factors. The processing results under different scaling factors show that compared with the SRDiff fixed scaling factor training results, and the SRDiff model needs to be retrained when the scaling factor changes, the super-resolution method of the present invention has superior scaling performance, ensuring the quality of microscopic electron microscope images under different scaling sizes. The results are shown in Table 2.

[0153] Table 2 Comparison of different scaling factors of MS-MESR and SRDiff network models

[0154]

[0155] The MS-MESR network model of the present invention achieves significant results in reducing costs and increasing efficiency in the image processing process, provides a more efficient and economical solution, and greatly promotes the development and application of micro-electron microscope image processing technology.

[0156] Example 3

[0157] This embodiment implements the method of embodiment 1 or 2, and the hardware devices used include:

[0158] Processor: Responsible for model training and image processing, it requires high-performance computing capabilities and is suitable for deep learning tasks. Memory: Used to store datasets, model weights, and processed images, it requires high speed and high reliability.

[0159] Graphics Processing Unit (GPU): Accelerates the training and inference process of deep learning models and improves computing efficiency.

[0160] Network interface: used for data transmission and model update, ensuring high speed and stability.

[0161] Display device: used to view processing results and image quality assessment in real time.

[0162] The present invention can effectively improve the resolution and details of microscopic electron microscope images, and provide high-quality image support for scientific research.

[0163] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A microscopic electron microscope image processing method based on multi-scale scaling, characterized in that: The method comprises the following steps: Step 1: Acquire an image of the target object through a microscopic electron microscope, store the acquired microscopic electron microscope image in a file, perform bilinear difference downsampling on the image, and save the image after bilinear difference downsampling as a low-resolution image LR; rotate, scale, flip, adjust the brightness and contrast of the low-resolution image LR, and add random noise to generate additional variants; obtain a microscopic image dataset; Step 2: Constructing a multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model The multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model includes a decoding part consisting of a pre-trained autoencoder, a latent diffusion model LDM, an implicit neural decoder and a multi-layer perceptron MLP; the decoding part consisting of the implicit neural decoder and the multi-layer perceptron is used to reconstruct the latent feature vector z' in the last time step state and feed it back to the image space; Input low-resolution image LR, encode it in pixel space through the autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector h, w, and c are the height, width, and number of channels of the input low-resolution image LR respectively; In the latent diffusion model LDM, random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is spliced ​​with the random input noise and used as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is spliced ​​again with the latent feature vector z obtained by the pre-trained autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state The latent feature vector z is input into the implicit neural decoder. At the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted. The latent feature vector z' in the last time step state is input into the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image. In the multi-layer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result in, express The regularization result of , setting the parameter δ = 1e-8 to avoid the denominator of the regularization process to zero; Reshape represents the mapping function; Represents an element in the vector set U; Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ; Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U; The decoded output image is expressed as follows: Among them, I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system, It means that it conforms to the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * Perform Euclidean distance interpolation, and after splicing, the multi-layer perceptron MLP performs the final pixel space to image space decoding process, F θ Functional representation of the mathematical model of the multi-layer perceptron MLP; The pre-trained autoencoder consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the end, which also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv; The implicit neural decoder includes a shallow 3×3 convolutional layer Conv and a main decoding part. The main decoding part is composed of two sub-parts in series, each of which includes a group normalization layer, a Sigmoid activation function layer, and a 3×3 convolutional layer Conv connected in sequence. The latent feature vector z' is input into the shallow 3×3 convolutional layer Conv to obtain shallow feature information. The shallow feature information is passed through the main decoding part to obtain deep feature information. A residual mechanism is introduced to add the shallow feature information and the deep feature information. The addition result is spliced ​​with the coordinate information corresponding to the image pixels in the latent space coordinate system as the input of the multi-layer perceptron. In the decoding part including the multi-layer perceptron (MLP), flexible zoom control can be achieved, achieving more selective results of zoom factor, significantly improving the resolution and quality of microscopic electron microscope images, and the applicability under different zoom factors; Step 3: Use the microscopic image dataset to train the MS-MESR network model. The trained MS-MESR network model is used to process the microscopic electron microscope images in real time to obtain super-resolution images corresponding to the low-resolution images, and perform detail recovery and enhancement. During the training process, the quality of the super-resolution image is evaluated by calculating the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the image. When the index comparison data is compared under the scaling factor N = 4 times, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) reach 37.50 and 0.87 respectively. When the scaling factor is changed, no retraining is required. When the scaling factor is changed, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) reach 36.88 and 0.78 respectively.

2. The method according to claim 1, characterized in that The latent diffusion model (LDM) is a diffusion model trained in the latent space of a pre-trained autoencoder to achieve high-quality image synthesis. The entire LDM diffusion process consists of a forward process and a backward process. The forward process gradually adds random noise to the input data, while the backward process constructs the required data samples from the noise. The forward process is a fixed process, and the noise latent feature vector z at time step T is expressed as: Where I represents the identity matrix; z0 represents the eigenvector of the original input data in the latent space, represents 1-β from time step 1 to T T Multiplication; ":=" indicates the equivalent of the symbol, β is a hyperparameter that represents the noise intensity added at each time step during the diffusion process; It is a hyperparameter sequence that represents the noise intensity in each time step i(0~T), and ∈ refers to the random Gaussian noise that conforms to the standard normal distribution in the diffusion model; conversely, in the reverse process, it is the inversion of the forward process.

3. The method according to claim 1, characterized in that The multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model is a two-stage process. The two-stage alignment strategy defines the target loss function of the entire network as: L align =λ1L dm +λ2L recon Where λ1 and λ2 are custom parameters for calculation; L dm represents the denoising objective loss function of the latent diffusion model; L recon Represents the reconstruction loss I of the real image obtained using a micro electron microscope SEM recon ; Reconstruction loss I of real images obtained using a micro electron microscope SEM recon Defined as: Among them, I0 is the original input image, Represents the latent feature vector from the prediction The generated image; The denoising objective loss function of the latent diffusion model is expressed as: Among them, z0 represents the potential feature vector in the initial state.

4. A microscopic electron microscope image processing system based on multi-scale scaling, characterized in that: The system executes the method according to any one of claims 1 to 3.

Images