Microscopic electron microscope image processing method and system based on multi-scale zooming

Abstract

The invention relates to a microcosmic electron microscope image processing method and system based on multi-scale zooming, and the method comprises the steps: obtaining an image of a target object through a microcosmic electron microscope, storing the obtained microcosmic electron microscope image in a file, carrying out the bilinear difference value down-sampling processing of the image, and obtaining a microcosmic electron microscope image; storing the image subjected to bilinear difference downsampling processing as a low-resolution image LR; the low-resolution image LR is rotated, scaled and overturned, brightness and contrast are adjusted, random noise is added, and therefore additional variants are generated; obtaining a microscopic image data set; and a multi-scale scaling microscopic electron microscope image super-resolution MS-MESR network model is constructed. According to the method, flexible zoom multiple control can be realized, more selective results of zoom multiple are realized, and the resolution and quality of the microscopic electron microscope image and the applicability under different zoom factors are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution processing, and in particular to a micro electron microscope image processing method and system based on multi-scale scaling. Background Art

[0002] Micro-electron microscopes (SEMs) are powerful tools for observing the microstructure of materials and biological samples. These devices can display the details of samples with high resolution and are widely used in materials science, biomedicine, semiconductor manufacturing and other fields. For microscopic feature analysis, high-resolution images can provide accurate and comprehensive information, which is crucial for researchers to perform image analysis. However, although micro-electron microscopes can provide high resolution, their image quality and detail accuracy are still affected by many factors, such as image noise, image blur, resolution limitations, etc.

[0003] Traditional microscopic electron microscope images require the following series of operations inside the device to obtain high-resolution images:

[0004] Image denoising: using filters (such as Gaussian filtering, mean filtering, etc.) to reduce noise in images. However, these methods often lead to the loss of image details, especially when processing high-noise images, which may erase important structural information.

[0005] Image sharpening: Improve the contrast and details of an image by using sharpening filters (such as Laplacian filters, edge enhancement filters). However, this method may amplify noise and artifacts in the image, making the actual structure of the image difficult to discern.

[0006] Multi-frame image synthesis: Improve the resolution by synthesizing multiple images. Although this method can improve the clarity of the image to a certain extent, the synthesis process is complex and computationally intensive, the operation is cumbersome and may introduce errors.

[0007] This series of operations results in high computational complexity in the generation of high-quality SEM images, consumes a lot of computing resources, and the process is complicated, with high requirements for operators. The processing time is long, affecting the real-time analysis and application efficiency of the image. For example, the processing time of a 3840*2400 pixel image is twice as long as that of a 960*600 pixel image. The high consumption cost and expensive SEM equipment lead to high unit labor costs for the equipment, and long processing time will generate a lot of costs. Summary of the invention

[0008] In view of the shortcomings of the existing traditional microscopic electron microscope image processing methods, the purpose of the present invention is to provide a microscopic electron microscope image processing method and system based on multi-scale scaling to solve the above-mentioned obvious defects existing in the prior art.

[0009] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0010] In a first aspect, the present invention provides a microscopic electron microscope image processing method based on multi-scale scaling, the method comprising the following steps:

[0011] Step 1: Acquire an image of the target object through a microscopic electron microscope, store the acquired microscopic electron microscope image in a file, perform bilinear difference downsampling on the image, and save the image after bilinear difference downsampling as a low-resolution image LR; rotate, scale, flip, adjust the brightness and contrast of the low-resolution image LR, and add random noise to generate additional variants; obtain a microscopic image dataset;

[0012] Step 2: Constructing a multi-scale zoomed micro-electron microscope image super-resolution MS-MESR network model

[0013] The multi-scale zoom microscopic electron microscope image super-resolution MS-MESR network model includes a decoding part composed of a pre-trained autoencoder, a latent diffusion model LDM, an implicit neural decoder and a multi-layer perceptron MLP; the decoding part composed of the implicit neural decoder and the multi-layer perceptron is used to reconstruct the potential feature vector z' in the last time step state and feed it back to the image space;

[0014] Input low-resolution image LR, encode it in pixel space through autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector h, w, and c are the height, width, and number of channels of the input low-resolution image LR, respectively;

[0015] In the latent diffusion model LDM, the random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is concatenated with the random input noise as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is concatenated again with the latent feature vector z obtained by the pre-trained autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state Input to the implicit neural decoder, at the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted; the latent feature vector z' in the last time step state is input to the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image;

[0016] In the multilayer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ;

[0017]

[0018] Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U;

[0019] The decoded output image is expressed as follows:

[0020]

[0021] Where I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system. It means that it meets the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * The Euclidean distance is interpolated and the multi-layer perceptron MLP is used to decode the pixel space into the image space. θ Functional representation of the mathematical model of the multi-layer perceptron MLP;

[0022] Step 3: Use the microscopic image dataset to train the MS-MESR network model. The trained MS-MESR network model is used to process microscopic electron microscope images in real time, obtain super-resolution images corresponding to low-resolution images, and perform detail restoration and enhancement.

[0023] Furthermore, the pre-trained autoencoder consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the tail, and also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv.

[0024] Furthermore, the latent diffusion model LDM is a diffusion model trained in the latent space of the pre-trained autoencoder to achieve high-quality image synthesis; the diffusion process of the entire LDM consists of a forward process and a reverse process, the forward process gradually adds random noise to the input data, and the reverse process constructs the required data samples from the noise; the forward process is a fixed process, and the noise potential feature vector z at the time step T is expressed as:

[0025]

[0026] Where I represents the identity matrix; z0 represents the eigenvector of the original input data in the latent space, represents 1-β from time step 1 to T T Multiplication; “:=” indicates the equivalent of the symbol, β is a hyperparameter, which indicates the noise intensity added at each time step in the diffusion process; It is a hyperparameter sequence, which indicates the noise intensity in each time step i (0~T), and ∈ refers to the random Gaussian noise that conforms to the standard normal distribution in the diffusion model; on the contrary, in the reverse process, it is the inversion of the forward process.

[0027] Furthermore, the multi-scale zoomed microscopic electron microscope image super-resolution MS-MESR network model is a two-stage process, and the two-stage alignment strategy defines the target loss function of the entire network as:

[0028] L align =λ1L dm +λ2L recon

[0029] Where λ1 and λ2 are custom parameters for calculation; L dm represents the denoising objective loss function of the latent diffusion model; L recon Represents the reconstruction loss L of the real image obtained using a micro electron microscope SEM recon ;

[0030] The reconstruction loss L of the real image obtained using a micro electron microscope SEM recon Defined as:

[0031]

[0032] Among them, I0 is the original input image, Represents the latent feature vector from the prediction The generated image;

[0033] The denoising objective loss function of the latent diffusion model is expressed as:

[0034]

[0035] Among them, z0 represents the potential feature vector in the initial state.

[0036] Furthermore, the implicit neural decoder includes a shallow 3×3 convolutional layer Conv and a main decoding part, which is composed of two sub-parts in series, each of which includes a group normalization layer, a Sigmoid activation function layer and a 3×3 convolutional layer Conv connected in sequence; the potential feature vector z' is input into the shallow 3×3 convolutional layer Conv to obtain shallow feature information, and the shallow feature information is passed through the main decoding part to obtain deep feature information, and a residual mechanism is introduced to add the shallow feature information and the deep feature information, and the addition result is spliced ​​with the coordinate information corresponding to the image pixels in the latent space coordinate system as the input of the multi-layer perceptron.

[0037] Furthermore, during the training process, the quality of the super-resolution image is evaluated by calculating the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM).

[0038] In a second aspect, the present invention provides a microscopic electron microscope image processing system based on multi-scale zooming, and the system executes the described method.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The method of the present invention adopts a multi-scale zoom microscopic electron microscope image super-resolution MS-MESR (Multi-Scale Microscopic Electron microscopy image Super-Resolution) network model, and can achieve flexible zoom factor control on a decoding part including a multi-layer perceptron MLP. Compared with a traditional fixed zoom learning method, more selective results of the zoom factor are achieved, and the resolution and quality of the microscopic electron microscope image and the applicability under different zoom factors are significantly improved. More accurate and clear image support is provided for the application of the microscopic electron microscope, which is a more efficient processing method and can obtain a microscopic electron microscope image of higher quality.

[0041] The symmetrical autoencoder-implicit neural decoder structure in the present invention maintains the symmetry of the convolution part between the autoencoder and the implicit neural decoder. This design enhances the information retention and detail recovery capabilities in image super-resolution tasks. Through symmetrical convolution and transposed convolution operations, the structure gradually restores the high resolution of the image and effectively maintains the feature information extracted during the encoding process. This structure not only improves the stability of the training process, but also reduces overfitting and simplifies the model debugging and optimization process, thereby generating clearer and more accurate high-resolution images.

[0042] The implicit neural decoder in the present invention combines shallow and deep convolutional layers, and effectively integrates multi-level feature information through group normalization layers, sigmoid activation function layers, and 3×3 convolutional layers. This structure uses a residual mechanism to combine shallow and deep features, which not only enhances the ability to restore image details, but also significantly reduces detail loss and artifacts, and improves the visual effect of the image. This design enables the model to more accurately restore the details and edges of the image, improving the quality of the final generated image.

[0043] The two-stage alignment strategy in the present invention improves the speed of learning and reasoning and reduces error propagation in model training by optimizing the alignment process in the two stages of feature extraction and image generation. The objective function of the overall network is defined by comprehensive reconstruction loss and denoising target loss, thereby improving the accuracy and stability of the model. The final generated image has higher resolution and quality.

[0044] In addition, the present invention has significant advantages in improving resolution. It can recover more details from low-resolution images, breaking through the limitations of traditional resolution, achieving clearer and more detailed images, reducing detail loss, and effectively retaining and enhancing image details, reducing detail loss and artifacts caused by traditional denoising and sharpening processing.

[0045] In terms of computational efficiency, although super-resolution processing requires certain computing resources, its processing efficiency is higher than that of traditional multi-frame synthesis technology, achieving the practical purpose of reducing costs and increasing efficiency. The present invention can complete super-resolution processing in a shorter time, significantly improve image quality, improve image visual effects, and improve the accuracy of sample observation, thereby enhancing the reliability of scientific research and applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural schematic diagram of the multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model in the present invention.

[0047] Figure 2 Schematic diagram of the connection structure between the implicit neural decoder and the multi-layer perceptron in the present invention.

[0048] Figure 3 Schematic diagram of the two-stage alignment strategy in the present invention. DETAILED DESCRIPTION

[0049] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0050] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments in a suitable manner.

[0051] The present invention is based on a multi-scale zoomed microscopic electron microscope image processing method, and its overall steps are: preparation of a microscopic electron microscope image data set, construction of a multi-scale zoomed microscopic electron microscope image super-resolution MS-MESR network model, and training of the multi-scale zoomed microscopic electron microscope image super-resolution MS-MESR network model using the microscopic electron microscope image data set. The specific steps are as follows:

[0052] Step 1: Obtain a microscopic image dataset

[0053] The target object is imaged through a microscopic electron microscope (scanning electron microscope SEM), and the acquired microscopic electron microscope image is stored in a file, where the image is subjected to bilinear difference downsampling, and the image after bilinear difference downsampling is saved as a low-resolution image LR, which is regarded as the input image LR required for network training. In order to further improve the generalization ability and robustness of the model, data enhancement technology is applied. The image is rotated, scaled, flipped, brightness and contrast are adjusted, and random noise is added to generate additional variants, which helps to expand the diversity of training data, enhance the generalization ability of the model, and ensure that the model can better handle various practical situations.

[0054] According to the actual situation, the target object can be photographed with different resolutions, and after being processed by the method of the present invention, a higher resolution image can be obtained relative to the input. The types of target objects in the microscopic image data set of the present invention can be as diverse as possible, thereby improving the generalization ability of the model.

[0055] Step 2: Constructing a multi-scale zoomed micro-electron microscope image super-resolution MS-MESR network model

[0056] The multi-scale zoomed microscopic electron microscope image super-resolution MS-MESR network model has a structure diagram as shown in Figure 1 shown.

[0057] The multi-scale zoom micro-electron microscope image super-resolution MS-MESR network model includes a decoding part composed of a pre-trained autoencoder, a latent diffusion model LDM (Latent Diffusion Model), an implicit neural decoder and a multi-layer perceptron MLP.

[0058] Input low-resolution image LR, encode it in pixel space through autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector During the training process, the potential feature vector z is continuously expanded and contracted in dimension;

[0059] After the feature map information of LR is obtained by autoencoding and extracted as the latent feature vector z, in the latent diffusion model LDM, the random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is concatenated with the random input noise as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is concatenated again with the latent feature vector z obtained by the autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state Input to the implicit neural decoder, at the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted; the latent feature vector z' in the last time step state is input to the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image;

[0060] In the multilayer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ;

[0061]

[0062] Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U;

[0063] The decoded output image is expressed as follows:

[0064]

[0065] Where I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system. It means that it meets the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * The Euclidean distance is interpolated and the multi-layer perceptron MLP is used to decode the pixel space into the image space. θ Function representation of the mathematical model of the multi-layer perceptron MLP.

[0066] Step 3: Use the microscopic image dataset to train the MS-MESR network model. The trained MS-MESR network model is used to process microscopic electron microscope images in real time, obtain super-resolution images corresponding to low-resolution images, and perform detail restoration and enhancement.

[0067] The pre-trained autoencoder in the present invention follows the basic convolutional and transposed convolutional neural network structure. The pre-trained autoencoder (CNN structure) consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the tail, which also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv. The main function of the autoencoder is to convert the input image Perform encoding operation to obtain the initial feature map information and input the image Extracted as latent feature vector The subsequent decoding part consisting of an implicit neural decoder and a multi-layer perceptron reconstructs the latent feature vector z' at the last time step and feeds it back to the image space.

[0068] The convolution parts of the autoencoder and the implicit neural decoder in the network model of the present invention are symmetrical. The main advantage of using a symmetrical structure is that it can effectively enhance the information retention and detail restoration capabilities in image super-resolution tasks. Through convolution and transposed convolution operations that are symmetrical to the autoencoder, the high resolution of the image can be gradually restored, and the feature information extracted during the encoding process can be maintained and refined, thereby improving the reconstruction quality of image details and edges. This special structure not only improves the stability and generalization ability of the training process, but also simplifies the debugging and optimization process of the model, which helps to reduce overfitting and improve the reconstruction effect of the final image. In addition, it supports more natural information flow and high-quality image restoration through structural consistency and regularity.

[0069] The structure of the implicit neural decoder is as follows Figure 2As shown in the figure: the input is the potential feature vector z' in the previous time step state, which passes through a shallow 3×3 convolution layer Conv to obtain shallow feature information, and then enters the main decoding part, which consists of two sub-parts, each of which consists of a group normalization layer, a Sigmoid activation function layer and a 3×3 convolution layer Conv. The shallow feature information passes through this part to obtain the deep feature information, and the residual mechanism is introduced to add the shallow feature information and the deep feature information. The addition result is spliced ​​with the coordinate information corresponding to the image pixel in the latent space coordinate system and input into the multi-layer perceptron for the final decoding operation.

[0070] Specifically, the autoencoder-implicit neural decoder structure increases the effective information by expanding the dimension of the potential feature vector. The feature vector passes through the multi-layer perceptron MLP, transfers from the pixel space to the image space, and then decodes it into an image. In the multi-layer perceptron, its learning process is realized by mapping the RGB pixel values ​​of the image to the output coordinates.

[0071] In addition, the key to realizing the multi-scale zoom super-resolution technology of the present invention lies in the decoding part composed of an implicit neural decoder and a multi-layer perceptron, in which a zoom multiple sequence s∈(1,M] is set in the multi-layer perceptron, where M represents the maximum zoom multiple, a zoom multiple N is randomly selected in s to form a subsequence s* of zoom multiples from 1 to N, and the zoom multiple s* is mapped to a vector set When the scaling factor is adjusted in the implicit space, the modeling process of the working parameters of the multilayer perceptron at different scaling factors can be expressed as:

[0072] U=Reshape(s * )

[0073]

[0074] in, express The regularization result of , setting the parameter δ = 1e-8 to avoid the denominator of the regularization process being zero; Reshape represents the mapping function; Represents an element in the vector set U;

[0075] The working parameters f of the multilayer perceptron with different scaling factors in the subsequence corresponding to the random scaling factor N in the implicit space can be obtained. θ It is expressed as:

[0076]

[0077] The decoded output image is expressed as follows:

[0078]

[0079] Where I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system. It means that it meets the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * The Euclidean distance is interpolated and the multi-layer perceptron MLP is used to decode the pixel space into the image space. θ Function representation of the mathematical model of the multi-layer perceptron MLP.

[0080] The core idea of ​​the latent diffusion model (LDM) is to train a diffusion model in the latent space of a pre-trained autoencoder to achieve high-quality image synthesis with a flexible range of styles and resolutions. The entire LDM diffusion process consists of a forward process that gradually adds random noise to the input data, while the reverse process constructs the required data samples from the noise. The forward process is a fixed process, and the noise latent feature vector z at time step T can be expressed as:

[0081]

[0082] Among them, z0 represents the feature vector of the original input data in the latent space; z T Represents the feature vector at time step T in the latent space; in represents α from time step 1 to T T For multiplication, “:=” means it is equivalent to the symbol, α T =1-β T , β is a hyperparameter that represents the noise intensity added at each time step during the diffusion process, is a hyperparameter sequence, which indicates the noise intensity in each time step i(0~T); ∈ refers to the random Gaussian noise that conforms to the standard normal distribution in the diffusion model.

[0083] On the contrary, the backward process is the reverse of the forward process. However, the backward distribution probability p(z T-1 |z T ) is tricky because it requires knowledge of the distribution of all potential images to compute this conditional probability. Therefore, the distribution probability p is approximated by a neural network θ Bayes' theorem can be used as follows:

[0084]

[0085] in, μ T represents the mean value under time step T, represents the variance at time step T.

[0086] The diffusion model can get better results by predicting the noise at each time step T. Therefore, the diffusion model ∈ θ The objective function of (.) can be expressed as:

[0087]

[0088] Compared with the traditional diffusion model in pixel space, the network structure of the present invention is a two-stage model, which aims to improve the speed of learning and reasoning. The second stage relies on the intermediate expression or features extracted in the first stage. During the training process, the autoencoder may cause errors and pass them to the main diffusion network, which will produce additional errors and be trained separately from the implicit neural decoder. These errors will reduce the effectiveness of decoding.

[0089] The multi-scale zoomed microscopic electron microscope image super-resolution MS-MESR network model of the present invention is a two-stage process, and the two-stage alignment strategy defines the target loss function of the entire network as:

[0090] L align =λ1L dm +λ2L recon

[0091] Where λ1 and λ2 are custom parameters for calculation; L dm represents the denoising objective loss function of the latent diffusion model; L recon Represents the reconstruction loss L of the real image obtained using a micro electron microscope SEM recon ;

[0092] The specific two-stage alignment strategy is as follows: Figure 3 As shown in Figure 1, the quality of the output image is improved by reducing the alignment error between the model stages, and finally the desired output image is obtained. The reconstruction loss L for the real image obtained using SEM recon To define:

[0093]

[0094] Among them, I0 is the original input image, Represents the latent feature vector from the prediction The generated image.

[0095] Combined with the principle equation of the potential diffusion model, the denoising target loss function of the potential diffusion model can be expressed as:

[0096]

[0097] according to The above formula can be simplified to:

[0098]

[0099] Among them, L dm represents the denoising target loss function of the potential diffusion model, and z0 represents the potential feature vector in the initial state. Combining the principle of the diffusion model, the concept of time step is used to realize the reasoning process from the initial state backwards and realize the fine-tuning of the diffusion model.

[0100] The super-resolution images processed by the method of the present invention are used for scientific research and data analysis in the fields of materials science, biomedicine, etc., to improve the clarity and details of microscopic electron microscope images, support more accurate sample observation and analysis, and optimize the network model according to feedback in actual applications, improve processing efficiency and image quality. The present invention can also collect user feedback on the image processing effect, and further adjust and improve parameter settings.

[0101] Example 1

[0102] The present invention is based on a multi-scale zoomed microscopic electron microscope image processing method, using the MS-MESR network model, and implements super-resolution processing of microscopic electron microscope images through the following steps:

[0103] 1. Data collection and preparation stage

[0104] 1.1 Data Collection

[0105] Equipment: Use a scanning electron microscope (SEM) to collect images of different target samples. The equipment is able to capture high-resolution details of the samples.

[0106] Procedure: Take a set of images for each sample, making sure to cover different viewing angles and magnifications.

[0107] 1.2 Data processing

[0108] Preprocessing: Bilinear interpolation downsampling is applied to the image to generate a low-resolution image (LR).

[0109] Data augmentation: Perform data augmentation operations such as rotating, scaling, flipping, adjusting brightness and contrast on images to improve the robustness and generalization ability of the model.

[0110] Labeling process: Label the important details in the microscopic electron microscope images to ensure the accuracy and completeness of the training data.

[0111] Dataset acquisition: All preprocessed images are divided into training set and test set in a 7:3 ratio to obtain a microscopic image dataset.

[0112] 2. Model Construction

[0113] Network structure: The MS-MESR network model contains a symmetrical autoencoder-implicit neural decoder structure, an implicit neural decoder, and a latent diffusion model.

[0114] Symmetrical autoencoder-implicit neural decoder structure: Through symmetric convolution and transposed convolution operations, the high resolution of the image is gradually restored and the feature information is maintained.

[0115] Implicit neural decoder: Combining shallow and deep convolutional layers, it fuses multi-level feature information through group normalization layers, sigmoid activation function layers, and 3×3 convolutional layers to reduce detail loss and artifacts.

[0116] Latent Diffusion Model: A diffusion model is trained in latent space to achieve high-quality image synthesis using forward and reverse processes. The diffusion process adds noise in the forward direction and recovers data from the noise in the reverse direction.

[0117] Input low-resolution image LR, encode it in pixel space through autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector h, w, and c are the height, width, and number of channels of the input low-resolution image LR, respectively;

[0118] In the latent diffusion model LDM, the random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is concatenated with the random input noise as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is concatenated again with the latent feature vector z obtained by the pre-trained autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state Input to the implicit neural decoder, at the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted; the latent feature vector z' in the last time step state is input to the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image;

[0119] In the multilayer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ;

[0120]

[0121] Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U;

[0122] The decoded output image is expressed as follows:

[0123]

[0124] Where I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system. It means that it meets the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * The Euclidean distance is interpolated and the multi-layer perceptron MLP is used to decode the pixel space into the image space. θ Function representation of the mathematical model of the multi-layer perceptron MLP.

[0125] 3. MS-MESR network model training

[0126] When the network starts training, the network initialization parameter setting is completed first: the epoch of the training network is set to 300, the optimizer uses the Adam optimizer, and the initial learning rate of Adam is set to 5e-5 and 1e-6, and the learning rate and other hyperparameters are adjusted according to the needs of model training, and the two calculated initialization parameters λ1 and λ2 of the objective function are set to 1.0.

[0127] During the training phase, the LR images of the training set are input into the network from the constructed microscopic image dataset, and these images are read according to the training set storage path and then processed according to the network structure;

[0128] The input image first passes through the autoencoder to obtain the initial feature map information, and then extracts it into the latent feature vector z. Before inputting it into the latent diffusion model LDM denoising UNet, the latent diffusion model LDM generates an initial random Gaussian noise that conforms to the normal distribution during the forward process, and concatenates it with the latent feature vector z. The concatenated result is then used as the input of the denoising UNet. After obtaining the output feature information of the denoising UNet, it is concatenated with the latent feature vector z again to obtain the latent feature vector of the denoising UNet at the previous time step T-1. The decoding task is arranged in the implicit neural decoder, and the coordinate mechanism is integrated to decode the feature information, and the required super-resolution image SR is obtained from decoding from pixel space to image space.

[0129] When the overall network objective loss function no longer shows a significant decrease (error ±1e-5), the network training area is considered stable and the network training process is completed.

[0130] Loss function: The objective function of the model is defined by combining the reconstruction loss and the denoising target loss, and the model parameters are optimized to improve the image quality.

[0131] 4. Super-resolution processing implementation

[0132] 4.1 Model Deployment

[0133] Deployment: The trained MS-MESR network model is deployed into the image processing system for real-time processing of microscopic electron microscope images.

[0134] Processing: Input the low-resolution image LR, use the MS-MESR model to generate the corresponding high-resolution image, and perform detail restoration and enhancement.

[0135] 4.2 Results Analysis

[0136] Quality Assessment: The quality of the super-resolution image is evaluated by calculating the PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index) of the image.

[0137] Example 2

[0138] Background: In the field of micro-electron microscope image processing, traditional methods usually face high costs and long processing time. For example, it takes about 8 hours and costs up to 4,000 yuan to process 500 micro-electron microscope images with traditional methods, which is a significant economic burden for large-scale image processing tasks. The MS-MESR network model of the present invention provides a more economical solution for micro-electron microscope image processing by significantly improving processing efficiency and reducing costs.

[0139] The total number of microscopic electron microscope images obtained in the embodiment network is 5,000, which are divided into a training set and a test set in a ratio of 7:3.

[0140] The MS-MESR network model is the same as that in Example 1. The pre-trained autoencoder consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the tail, and also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv.

[0141] The implicit neural decoder includes a shallow 3×3 convolution layer Conv and a main decoding part, the main decoding part is composed of two sub-parts in series, each sub-part includes a group normalization layer, a Sigmoid activation function layer and a 3×3 convolution layer Conv connected in sequence; the potential feature vector z' is input into the shallow 3×3 convolution layer Conv to obtain shallow feature information, the shallow feature information is passed through the main decoding part to obtain deep feature information, and a residual mechanism is introduced to add the shallow feature information and the deep feature information, and the addition result is spliced ​​with the coordinate information corresponding to the image pixels in the latent space coordinate system as the input of the multi-layer perceptron.

[0142] Cost-Benefit Analysis:

[0143] Traditional method: It takes about 8 hours to process 500 images and costs 4,000 yuan.

[0144] MS-MESR network model method: After using the MS-MESR network model, the processing time was shortened to about 2.7 hours. Since the processing time was reduced by two-thirds, the corresponding processing cost was also significantly reduced from the original 4,000 yuan to about 1,300 yuan. This means that the processing cost of every 500 images was saved by 2,700 yuan.

[0145] By adopting the MS-MESR network model, the cost of micro-electron microscope image processing has been significantly reduced. When processing 500 images, the cost of the MS-MESR method is only one-third of that of the traditional method, which not only saves processing costs, but also reduces the additional economic costs caused by long processing time. This cost saving effect is particularly significant in large-scale image processing projects.

[0146] The MS-MESR network model has achieved significant cost reduction and efficiency improvement in the field of micro-electron microscope image processing. It provides a more economical solution for scientific research and industrial applications by improving processing efficiency and reducing processing costs, and has greatly promoted the popularization and application of micro-electron microscope image processing technology.

[0147] On the same microscopic image data set, the MS-MESR network model of the present invention is compared with the existing methods. The comparison results show that compared with the real image directly obtained by the microscopic electron microscope, the method proposed by the present invention still maintains high-quality indicators, and the difference with the real image is only 0.02PSNR (peak signal-to-noise ratio). Compared with the existing excellent diffusion model methods such as SRDiff and DDPM, the super-resolution method of the present invention has lower calculation amount and better performance. The results obtained by using multiple network models for super-resolution processing of microscopic electron microscope images after training are shown in Table 1, which is the indicator comparison data under the scaling factor N=4 times. When performing network testing, the prepared test set images are input, and the network weights of the training results are imported. When the peak signal-to-noise ratio PSNR and structural similarity SSIM of the image reach 37.50 (error ±0.03) and 0.87 (error ±0.02) respectively, the entire network training is completed. The scaling factor of other models in Table 1 is directly fixed to 4.

[0148] Table 1 Comparison of indicators of different network models

[0149]

[0150] The results shown in Table 1 show that in the super-resolution processing of micro-electron microscope images, under the same test conditions, the peak signal-to-noise ratio and structural similarity of the MS-MESR network model are higher than those of the existing SRDiff and DDPM, which have relatively good performance. Compared with the above-mentioned diffusion models of the same type, the difference is significant, which proves that the micro-electron microscope image super-resolution processing technology of the present invention has superior application performance.

[0151] Example 3

[0152] On the same microscopic image data set, the MS-MESR network model of the present invention is used to realize super-resolution processing of microscopic electron microscope images under multi-scale scaling factors. The processing results under different scaling factors show that compared with the SRDiff fixed scaling factor training results, and the SRDiff model needs to be retrained when the scaling factor changes, the super-resolution method of the present invention has superior scaling performance and ensures the quality of microscopic electron microscope images under different scaling sizes. The results are shown in Table 2.

[0153] Table 2 Comparison of results of different scaling factors of MS-MESR and SRDiff network models

[0154]

[0155] The MS-MESR network model of the present invention achieves remarkable results in reducing costs and increasing efficiency in the image processing process, provides a more efficient and economical solution, and greatly promotes the development and application of micro-electron microscope image processing technology.

[0156] Example 3

[0157] This embodiment implements the method of embodiment 1 or 2, and the hardware device used includes:

[0158] Processor: Responsible for model training and image processing, requires high-performance computing capabilities, and is suitable for deep learning tasks. Memory: Used to store data sets, model weights, and processed images, requiring high speed and high reliability.

[0159] Graphics Processing Unit (GPU): Accelerates the training and reasoning process of deep learning models and improves computing efficiency.

[0160] Network interface: used for data transmission and model update, ensuring high speed and stability.

[0161] Display device: used to view processing results and image quality assessment in real time.

[0162] The present invention can effectively improve the resolution and details of microscopic electron microscope images and provide high-quality image support for scientific research.

[0163] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A microscopic electron microscope image processing method based on multi-scale scaling, characterized in that: The method comprises the following steps: Step 1: Acquire an image of the target object through a microscopic electron microscope, store the acquired microscopic electron microscope image in a file, perform bilinear difference downsampling on the image, and save the image after bilinear difference downsampling as a low-resolution image LR; rotate, scale, flip, adjust the brightness and contrast of the low-resolution image LR, and add random noise to generate additional variants; obtain a microscopic image dataset; Step 2: Constructing a multi-scale zoomed micro-electron microscope image super-resolution MS-MESR network model The multi-scale zoom microscopic electron microscope image super-resolution MS-MESR network model includes a decoding part composed of a pre-trained autoencoder, a latent diffusion model LDM, an implicit neural decoder and a multi-layer perceptron MLP; the decoding part composed of the implicit neural decoder and the multi-layer perceptron is used to reconstruct the potential feature vector z' in the last time step state and feed it back to the image space; Input low-resolution image LR, encode it in pixel space through autoencoder, encode the input low-resolution image LR into feature map information, and extract it into a potential feature vector h, w, and c are the height, width, and number of channels of the input low-resolution image LR, respectively; In the latent diffusion model LDM, the random input noise is obtained by combining the Gaussian distribution random variables in the latent space with the noise generated by the forward diffusion process. The latent feature vector z is concatenated with the random input noise as the input of the denoising UNet, the main part of the LDM. The denoising UNet then performs the forward and reverse processes of diffusion. When the denoising UNet output is obtained, it is concatenated again with the latent feature vector z obtained by the pre-trained autoencoder to obtain the latent feature vector at the previous time step T-1. The potential feature vector obtained at the T-1 time step state Input to the implicit neural decoder, at the same time, the coordinate information corresponding to the image pixels in the latent space coordinate system is input into the decoding part composed of the implicit neural decoder and the multi-layer perceptron MLP, and the dimension of the latent feature vector z is continuously expanded and contracted; the latent feature vector z' in the last time step state is input to the decoding part, and the final decoding is performed in the image space to obtain the final output super-resolution image; In the multilayer perceptron, a scaling multiple sequence s∈(1,M] is set, where M represents the maximum scaling multiple. A scaling multiple N is randomly selected from s to form a subsequence s* of scaling multiples from 1 to N. According to U=Reshape(s * ) maps the subsequence s* to a vector set Then the elements in the vector set are regularized to obtain the regularized result Then, the working parameters f of the multilayer perceptron at different scaling factors are obtained according to the following formula: θ ; Where, j = 1, 2; i = 1, 2, ..., N; Represents an element in the vector set U; The decoded output image is expressed as follows: Where I(c) is the definition of the RGB pixel value of the image in the implicit space coordinate system. It means that it meets the definition of normal distribution, and D represents the decoding part operation; refers to the implicit neural decoder, z' is the latent feature vector in the last time step state after sampling by the latent diffusion model, and c * is the pixel coordinate associated with z'; the latent feature vector z' first passes through the implicit neural decoder Decode and then use c * The Euclidean distance is interpolated and the multi-layer perceptron MLP is used to decode the pixel space into the image space. θ Functional representation of the mathematical model of the multi-layer perceptron MLP; Step 3: Use the microscopic image dataset to train the MS-MESR network model. The trained MS-MESR network model is used to process microscopic electron microscope images in real time, obtain super-resolution images corresponding to low-resolution images, and perform detail restoration and enhancement.

2. The method according to claim 1, characterized in that: The pre-trained autoencoder consists of a first 3×3 convolutional layer Conv, a group normalization layer, an activation function ReLU, a second 3×3 convolutional layer Conv, and an activation function ReLU at the tail, and also includes a residual connection from the input of the first 3×3 convolutional layer to the second 3×3 convolutional layer Conv.

3. The method according to claim 1, characterized in that The latent diffusion model LDM is a diffusion model trained in the latent space of the pre-trained autoencoder to achieve high-quality image synthesis; the diffusion process of the entire LDM consists of a forward process and a reverse process, the forward process gradually adds random noise to the input data, and the reverse process constructs the required data samples from the noise; the forward process is a fixed process, and the noise potential feature vector z at the time step T is expressed as: Where I represents the identity matrix; z0 represents the eigenvector of the original input data in the latent space, represents 1-β from time step 1 to T T Multiplication; ":=" means equivalent to the symbol, β is a hyperparameter, which indicates the noise intensity added at each time step in the diffusion process; is a hyperparameter sequence, which indicates the noise intensity in each time step i (0~T), ∈ refers to the random Gaussian noise that conforms to the standard normal distribution in the diffusion model; on the contrary, in the reverse process, it is the inversion of the forward process.

4. The method according to claim 1, characterized in that The multi-scale zoomed microscopic electron microscope image super-resolution MS-MESR network model is a two-stage process. The two-stage alignment strategy defines the target loss function of the entire network as: L align =λ1L dm +λ2L recon Where λ1 and λ2 are custom parameters for calculation; L dm represents the denoising objective loss function of the latent diffusion model; L recon Represents the reconstruction loss L of the real image obtained using a micro electron microscope SEM recon ; The reconstruction loss L of the real image obtained using a micro electron microscope SEM recon Defined as: Among them, I0 is the original input image, Represents the latent feature vector from the prediction The generated image; The denoising objective loss function of the latent diffusion model is expressed as: Among them, z0 represents the potential feature vector in the initial state.

5. The method according to claim 1, characterized in that The implicit neural decoder includes a shallow 3×3 convolutional layer Conv and a main decoding part, which is composed of two sub-parts connected in series, each of which includes a group normalization layer, a Sigmoid activation function layer and a 3×3 convolutional layer Conv connected in sequence; The latent feature vector z' is input into the shallow 3×3 convolutional layer Conv to obtain shallow feature information. The shallow feature information is decoded by the main decoding part to obtain deep feature information. A residual mechanism is introduced to add the shallow feature information and the deep feature information. The addition result is concatenated with the coordinate information corresponding to the image pixels in the latent space coordinate system as the input of the multi-layer perceptron.

6. The method according to claim 1, characterized in that During the training process, the quality of the super-resolution image is evaluated by calculating the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM).

7. A microscopic electron microscope image processing system based on multi-scale scaling, characterized in that: The system executes the method according to any one of claims 1 to 6.

Images