A processing method and device for optimizing quality of electron microscope images

Abstract

The embodiment of the application relates to a kind of processing method and device for optimizing electron micrograph quality, the method comprises: one complete model pre-training DDPM model is recorded as first pre-training model;And by the way of fusing multiple LoRA adapters on first pre-training model, obtain first fusion model;Collect multiple spherical aberration electron micrograph to form corresponding first training atlas;The parameter set L of first fusion model is fine-tuned based on first training atlas;After fine-tuning, corresponding electron micrograph quality optimization processing is carried out to any ordinary transmission electron micrograph based on the second denoising autoencoder of first fusion model, and corresponding optimized electron micrograph is obtained.Through the application, the image quality of ordinary transmission electron micrograph can be optimized to reach the quality level of spherical aberration electron micrograph, and the model fine-tuning efficiency can be improved, and the model use cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a processing method and apparatus for optimizing the quality of electron micrographs. Background Technology

[0002] The working principle of a transmission electron microscope (TEM) is to irradiate a sample with a high-energy electron beam of extremely short wavelength. Based on the sample's physical properties and feedback characteristics under electron beam irradiation, an image is formed and output. TEM images are mainly used to observe the internal structure of materials, such as lattices, defects, and phase transitions, and are an important research tool in materials science, physics, chemistry, and other fields. The working principle of an aberration-corrected electron microscope (ACEM) is to add a spherical aberration corrector to a conventional TEM to correct for spherical aberrations and other aberrations, thereby improving the resolution of the electron microscope image. In other words, for the same sample, the quality of the electron microscope image output by an aberration-corrected electron microscope is far superior to that of a TEM image, and analysis of the sample material structure based on the electron microscope image output by an aberration-corrected electron microscope can achieve better analytical results. However, in reality, aberration-corrected electron microscopes are very expensive, often ten or even dozens of times more expensive than conventional TEM microscopes, which is unaffordable for most teaching or research institutions.

[0003] The first technical document, "Denoising Diffusion Probabilistic Models," proposes a class of large-scale visual models for image processing based on the principle of forward diffusion and backward denoising. As described in the first document, the DDPM model framework consists of two parts: diffusion models and denoising autoencoders. The diffusion models perform forward diffusion processing on the input image using a progressive noise addition method to obtain a noisy image. The denoising autoencoders then perform backward denoising processing on the noisy image using a progressive noise reduction method to obtain a final denoised image, which is then output as the generated image of the DDPM model. Training the DDPM model using the training dataset and model training method given in the first document yields a pre-trained DDPM model. This pre-trained model can be used to optimize the resolution of ordinary images, thereby improving image quality. Inspired by technical document 1, we fine-tuned the model parameter set (hereinafter referred to as parameter set W) of the pre-trained model using a series of spherical aberration electron microscopy images to optimize it for ordinary transmission electron microscopy images and achieve the image quality level of spherical aberration electron microscopy images. However, during the fine-tuning process, we found that if we fine-tuned the parameter set W by directly modifying the parameter values ​​using conventional model optimizers (such as SGD optimizer, Adam optimizer, etc.), the computational load would be large and the fine-tuning cycle would be long, resulting in low model fine-tuning efficiency.

[0004] Technical document two, "LoRA: Low-rank Adaptation of Large Language Models," presents a LoRA model designed to reduce the debugging difficulty of large target models. As described in document two, the debugging principle of the LoRA model involves embedding a corresponding LoRA adapter into each key module of the large target model. The parameters of each adapter are then fused with the original model parameters of the corresponding module to achieve the fusion of the LoRA model and the large target model, resulting in a fused model of the large target model and the LoRA model. During model debugging, the original model parameters of the large target model are not modified; only the parameters of the LoRA model are fine-tuned to achieve overall optimization of the fused model. Because the number and complexity of parameters in the LoRA model are much smaller than those in the large target model, this fine-tuning method based on the fused model can effectively reduce computation and shorten the debugging cycle. Inspired by technical document two, we further introduce the LoRA model into the fine-tuning process of pre-trained models to improve the efficiency of model fine-tuning.

[0005] Finally, through the above two-step technical attempts, we have summarized an electron microscopy image quality optimization processing mechanism based on the DDPM+LoRA fusion model to optimize the image quality of ordinary transmission electron microscopy images and achieve the quality level of spherical aberration electron microscopy images. The technical implementation scheme of this processing mechanism is given in this invention. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, apparatus, electronic device, and computer-readable storage medium for optimizing the quality of electron microscopy (TEM) images. This invention denotes a pre-trained DDPM model (composed of a first diffusion model and a first denoising autoencoder) as the first pre-trained model. A fused model (composed of a second diffusion model and a second denoising autoencoder) is obtained by fusing multiple LoRA adapters onto the first pre-trained model. The fused model parameters of the first fused model consist of a parameter set W and a parameter set L. Parameter set W is the overall model parameter set of the first pre-trained model, and parameter set L is the adapter parameter set of all LoRA adapters. Multiple spherical aberration TEM images are acquired to form a corresponding first training image set. The parameter set L of the first fused model is fine-tuned based on the first training image set. After fine-tuning, the second denoising autoencoder of the first fused model is used to perform corresponding TEM image quality optimization processing on any ordinary transmission electron microscopy (TEM) image to obtain the corresponding optimized TEM image. This invention can optimize the image quality of ordinary transmission electron microscopy (TEM) images to achieve the quality level of spherical aberration electron microscopy (SEM) images, and can also improve the efficiency of model fine-tuning and reduce the cost of model use. This invention can help ordinary teaching or research institutions reduce research costs and improve research efficiency.

[0007] To achieve the above objectives, a first aspect of the present invention provides a method for optimizing the quality of electron micrographs, the method comprising:

[0008] A pre-trained DDPM model is denoted as the first pre-trained model; a fusion model is obtained by fusing multiple LoRA adapters on the first pre-trained model, denoted as the first fusion model; wherein, the first pre-trained model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-trained model is denoted as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of parameter set W and parameter set L; the parameter set L is composed of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameters A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameters B of all the adapter parameters;

[0009] Multiple spherical aberration electron micrographs are acquired to form a corresponding first training image set; wherein, the first training image set includes multiple first training images; each first training image is an acquired spherical aberration electron micrograph.

[0010] The parameter set L of the first fusion model is fine-tuned based on the first training graph set;

[0011] After fine-tuning, the corresponding optimized electron microscope image is obtained by performing corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder.

[0012] Preferably, the step of obtaining a fusion model by fusing multiple LoRA adapters on the first pre-trained model, denoted as the corresponding first fusion model, specifically includes:

[0013] Step 21: Select multiple first key layers from the first diffusion model and the first denoising autoencoder of the first pre-trained model according to the preset key layer selection rules; and count the total number of the first key layers to obtain the corresponding total number of key layers N.

[0014] The expressions for each of the first key layers are as follows:

[0015] h i =w i x i ;

[0016] 1 ≤ key layer index i ≤ N; w i x i h i The model parameters, input vector, and output vector of the i-th first key layer are respectively, and the model parameters w i ∈ parameter set W;

[0017] Step 22, based on the preset low-rank parameter r and the model parameters w of each of the first key layers. i For each of the first key layers, a pair of corresponding low-rank matrix parameters A and B are set, denoted as low-rank matrix parameter A. i B i ; and by each pair of the low-rank matrix parameters A i B i The adapter parameters are used to form a corresponding LoRA adapter; and a corresponding LoRA adapter is fused on each of the first key layers to form a corresponding first fusion layer;

[0018] The first fusion layer corresponds one-to-one with the first key layer;

[0019] The expressions for each of the first fusion layers are:

[0020] h i =w i x i +A i B i x i ;

[0021] h'i This is the output vector of the i-th first fusion layer;

[0022] Step 23: The first diffusion model of the fused multiple LoRA adapters is used as the corresponding second diffusion model, and the first denoising autoencoder of the fused multiple LoRA adapters is used as the corresponding second denoising autoencoder; the second diffusion model and the second denoising autoencoder are combined to form the corresponding first fusion model; and the low-rank matrix parameter A of the N adapter parameters is used. i The corresponding first low-rank matrix parameter set is formed; and the low-rank matrix parameters B of the N adapter parameters are... i The first and second low-rank matrix parameter sets are used to form the corresponding parameter set L; and the parameter set W and the parameter set L are used to form the fusion model parameters of the first fusion model.

[0023] Preferably, the acquisition of multiple spherical aberration electron microscopy images to form a corresponding first training image set specifically includes:

[0024] Spherical aberration electron microscopy images are acquired through multiple publicly available data acquisition channels and experimental acquisition methods, and each acquired spherical aberration electron microscopy image is used as a corresponding first training image; and all the obtained first training images form the corresponding first training image set; wherein, the multiple publicly available data acquisition channels include at least publicly available spherical aberration electron microscopy image databases and publicly available scientific and technological literature / journals / technical papers.

[0025] Preferably, the fine-tuning of the parameter set L of the first fusion model based on the first training graph set specifically includes:

[0026] Step 41: Based on a preset first segmentation ratio, the first training image set is divided into two sub-image sets, denoted as the first image set and the second image set.

[0027] Wherein, both the first and second image sets are composed of multiple first training images; the ratio of the total number of training images in the first image set to the total number of training images in the second image set satisfies the first segmentation ratio;

[0028] Step 42: Extract the first training image from the first image set as the corresponding current label image;

[0029] Step 43: Input the current label map into the second diffusion model of the first fusion model to perform corresponding forward diffusion processing to obtain the corresponding current noise map; and input the current noise map into the second denoising autoencoder of the first fusion model to perform corresponding reverse denoising processing to obtain the corresponding current denoised map;

[0030] Step 44: Substitute the current denoised image and the current labeled image into the preset first model loss function; and, with the parameter set W frozen, perform one round of optimization on the parameter set L based on the preset first model optimizer in the direction of minimizing the first model loss function;

[0031] Wherein, the first model loss function is the loss function of the DDPM model; the first model optimizer includes at least the SGD optimizer and the Adam optimizer;

[0032] Step 45: Identify whether the current label image is the last first training image in the first image set; if yes, proceed to step 46; if no, extract the next first training image in the first image set as the new current label image and return to step 43.

[0033] Step 46: Perform a traversal of all the first training images in the second image set; during this traversal, take the currently traversed first training image as the corresponding current label image; input the current label image into the second diffusion model of the first fusion model for corresponding forward diffusion processing to obtain the corresponding current noise image; input the current noise image into the second denoising autoencoder of the first fusion model for corresponding reverse denoising processing to obtain the corresponding current denoised image; and form a corresponding first prediction-label image pair with the current denoised image and the current label image; and at the end of this traversal, input all the obtained first prediction-label image pairs into a preset first model evaluation function to calculate the corresponding first evaluation value;

[0034] The first model evaluation function includes at least the RMSE function;

[0035] Step 47: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 42 to continue training; if it meets the range, stop training and confirm that the fine-tuning is complete.

[0036] Preferably, the step of performing corresponding electron microscopy image quality optimization processing on any ordinary transmission electron microscopy image based on the second denoising autoencoder to obtain the corresponding optimized electron microscopy image specifically includes:

[0037] Receive any ordinary transmission electron microscope (TEM) image as the corresponding current TEM image; input the current TEM image into the second denoising autoencoder for corresponding reverse denoising processing to obtain the corresponding current denoised image; and use the obtained current denoised image as the corresponding optimized TEM image.

[0038] A second aspect of the present invention provides an apparatus for implementing the processing method for optimizing electron micrograph quality as described in the first aspect above, the apparatus comprising: a model fusion module, a data acquisition module, a model fine-tuning module, and a model application module;

[0039] The model fusion module is used to denote a pre-trained DDPM model as the corresponding first pre-trained model; and to obtain a fusion model by fusing multiple LoRA adapters on the first pre-trained model, denoted as the corresponding first fusion model; wherein, the first pre-trained model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-trained model is denoted as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of parameter set W and parameter set L; the parameter set L is composed of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameters A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameters B of all the adapter parameters;

[0040] The data acquisition module is used to acquire multiple spherical aberration electron micrographs to form a corresponding first training image set; wherein, the first training image set includes multiple first training images; each first training image is an acquired spherical aberration electron micrograph.

[0041] The model fine-tuning module is used to fine-tune the parameter set L of the first fusion model based on the first training graph set;

[0042] The model application module is used to perform corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder after fine-tuning to obtain the corresponding optimized electron microscope image.

[0043] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0044] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0045] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0047] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for optimizing the quality of electron microscopy (TEM) images. As described above, this invention pre-trains a DDPM model (composed of a first diffusion model and a first denoising autoencoder) and obtains a fusion model (composed of a second diffusion model and a second denoising autoencoder) by fusing multiple LoRA adapters onto the first pre-trained model. The fusion model parameters of the first fusion model consist of a parameter set W and a parameter set L, where W is the overall model parameter set of the first pre-trained model and L is the adapter parameter set of all LoRA adapters. Multiple spherical aberration TEM images are acquired to form a corresponding first training image set, and the parameter set L of the first fusion model is fine-tuned based on the first training image set. After fine-tuning, the second denoising autoencoder of the first fusion model is used to perform corresponding TEM image quality optimization processing on any ordinary transmission electron microscopy (TEM) image to obtain the corresponding optimized TEM image. The embodiments of the present invention can optimize the image quality of ordinary transmission electron microscopy images to achieve the quality level of spherical aberration electron microscopy images, and can improve the efficiency of model fine-tuning and reduce the cost of model use. The embodiments of the present invention can help ordinary teaching or research institutions reduce research costs and improve research efficiency. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a method for optimizing the quality of electron micrographs according to Embodiment 1 of the present invention;

[0049] Figure 2 This is a module structure diagram of a processing device for optimizing electron micrograph quality provided in Embodiment 2 of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] Embodiment 1 of the present invention provides a method for optimizing the quality of electron micrographs, such as... Figure 1 The schematic diagram shows a method for optimizing electron micrograph quality according to Embodiment 1 of the present invention. This method mainly includes the following steps:

[0053] Step 1: Denote a completed DDPM model as the first pre-trained model; and obtain a fusion model by fusing multiple LoRA adapters on the first pre-trained model, which is also referred to as the first fusion model.

[0054] Specifically, it includes: Step 11, recording a completed DDPM model as the corresponding first pre-trained model;

[0055] Here, the DDPM model in this embodiment of the invention is the diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM) proposed in technical document 1, "Denoising Diffusion Probabilistic Models". This DDPM model has been trained according to the training dataset and model training method given in technical document 1. Here, it is regarded as a DDPM model that has completed model pre-training and is referred to as the first pre-trained model. As can be seen from technical document 1, the model framework of the DDPM model consists of two parts: diffusion models and denoising autoencoders. Therefore, the first pre-trained model includes the first diffusion model and the first denoising autoencoder. The overall model parameter set of the first pre-trained model is denoted as parameter set W.

[0056] Step 12, and obtain a fusion model by fusing multiple LoRA adapters on the first pre-trained model, denoted as the corresponding first fusion model;

[0057] The first fusion model includes a second diffusion model and a second denoising autoencoder; the adapter parameters of each LoRA adapter consist of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model consist of parameter set W and parameter set L; parameter set L consists of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set consists of the low-rank matrix parameters A of all adapter parameters; the second low-rank matrix parameter set consists of the low-rank matrix parameters B of all adapter parameters.

[0058] Specifically, it includes: Step 121, selecting multiple first key layers from the first diffusion model and the first denoising autoencoder of the first pre-trained model according to the preset key layer selection rules; and statistically calculating the total number of first key layers to obtain the corresponding total number of key layers N.

[0059] Here, the expressions for each of the first key layers of the first pre-trained model in this embodiment of the invention are: h i =w i x i Where 1 ≤ key layer index i ≤ N; w ix i h i These are the model parameters, input vector, and output vector of the i-th first key layer, respectively, and the model parameters w are... i ∈ parameter set W;

[0060] The key layer selection rule in this embodiment of the invention is a preset rule, which can be customized based on application requirements, or it can be customized by referring to the LoRA adapter embedding methods listed in technical document 2 "LoRA: Low-rank Adaptation of Large Language Models". A common customization method under normal circumstances is to use one or more attention layers and / or linear layers in the first diffusion model and the first denoising autoencoder as key layers. Another common customization method under normal circumstances is to use one or more attention layers and / or linear layers in the first denoising autoencoder as key layers. Yet another common customization method under normal circumstances is to use all attention layers and the last one or two linear layers in the first denoising autoencoder as key layers.

[0061] Step 122, based on the preset low-rank parameter r and the model parameters w of each first key layer. i For each first key layer, set a corresponding pair of low-rank matrix parameters A and B, denoted as low-rank matrix parameter A. i B i ; and by each pair of low-rank matrix parameters A i B i The adapter parameters are used to form a corresponding LoRA adapter; and a corresponding LoRA adapter is fused on each first key layer to form a corresponding first fusion layer;

[0062] Here, the configuration of the low-rank parameter r and the low-rank matrix parameters A and B can be found in technical document 2, "LoRA: Low-rank Adaptation of Large Language Models". In this embodiment of the invention, the first fusion layer corresponds one-to-one with the first key layer, and the expression for each first fusion layer is: h' i =w i x i +A i B i x i ; where h' i This is the output vector of the i-th first fusion layer;

[0063] Step 123: The first diffusion model of the fused multiple LoRA adapters is used as the corresponding second diffusion model, and the first denoising autoencoder of the fused multiple LoRA adapters is used as the corresponding second denoising autoencoder; the second diffusion model and the second denoising autoencoder are combined to form the corresponding first fusion model; and the low-rank matrix parameter A of N adapter parameters is used. i The corresponding first low-rank matrix parameter set is formed; and the low-rank matrix parameter B of the N adapter parameters is formed. i The first and second low-rank matrix parameter sets are combined to form the corresponding parameter set L; and the parameter set W and parameter set L are combined to form the fusion model parameters of the first fusion model.

[0064] Step 2: Collect multiple spherical aberration electron microscopy images to form the corresponding first training image set;

[0065] Specifically, this includes: acquiring spherical aberration electron microscopy images through multiple publicly available data acquisition channels and experimental acquisition methods, and using each acquired spherical aberration electron microscopy image as a corresponding first training image; and forming a corresponding first training image set from all the obtained first training images.

[0066] Here, the various publicly available data acquisition channels include at least publicly available spherical aberration electron microscopy image databases and publicly available scientific and technological literature / journals / technical papers; the first training image set includes multiple first training images; each first training image is an acquired spherical aberration electron microscopy image.

[0067] Step 3: Fine-tune the parameter set L of the first fusion model based on the first training graph set;

[0068] Specifically, it includes: Step 31, dividing the first training image set into two sub-image sets based on a preset first segmentation ratio, denoted as the corresponding first image set and second image set;

[0069] The first segmentation ratio is a pre-set ratio parameter, such as 8:2; both the first and second image sets are composed of multiple first training images; the ratio of the total number of training images in the first image set to the total number of training images in the second image set satisfies the first segmentation ratio.

[0070] Step 32: Extract the first training image from the first image set as the corresponding current label image;

[0071] Step 33: Input the current label image into the second diffusion model of the first fusion model to perform the corresponding forward diffusion processing to obtain the corresponding current noise image; and input the current noise image into the second denoising autoencoder of the first fusion model to perform the corresponding reverse denoising processing to obtain the corresponding current denoised image;

[0072] Step 34: Input the current denoised image and the current label image into the preset first model loss function; and under the premise of freezing the parameter set W, optimize the parameter set L in one round based on the preset first model optimizer in the direction of minimizing the first model loss function;

[0073] Wherein, the loss function of the first model is the loss function of the DDPM model; the first model optimizer includes at least the SGD optimizer and the Adam optimizer;

[0074] Here, because the optimization only modifies the parameter set L with a small amount of data and does not need to modify the parameter set W with a large amount of data, the computational load can be significantly reduced, the training cycle can be significantly shortened, and the training efficiency can be quickly improved.

[0075] Step 35: Identify whether the current label image is the last first training image in the first image set; if yes, proceed to step 36; if no, extract the next first training image in the first image set as the new current label image and return to step 33.

[0076] Step 36: Perform a traversal of all the first training images in the second image set; during this traversal, use the currently traversed first training image as the corresponding current label image; input the current label image into the second diffusion model of the first fusion model for corresponding forward diffusion processing to obtain the corresponding current noise image; input the current noise image into the second denoising autoencoder of the first fusion model for corresponding reverse denoising processing to obtain the corresponding current denoised image; and form a corresponding first prediction-label image pair with the current denoised image and the current label image; at the end of this traversal, input all the obtained first prediction-label image pairs into the preset first model evaluation function to calculate the corresponding first evaluation value;

[0077] The first model evaluation function includes at least the RMSE function;

[0078] Step 37: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 32 to continue training; if it meets the range, stop training and confirm that the fine-tuning is complete.

[0079] Here, the first evaluation value range is a pre-set numerical range.

[0080] Step 4: After fine-tuning, perform corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder to obtain the corresponding optimized electron microscope image.

[0081] Specifically, this includes: receiving any ordinary transmission electron microscope (TEM) image as the corresponding current TEM image; inputting the current TEM image into a second denoising autoencoder for corresponding reverse denoising processing to obtain the corresponding current denoised image; and using the obtained current denoised image as the corresponding optimized TEM image.

[0082] Figure 2 This is a module structure diagram of a processing device for optimizing electron micrograph quality provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a model fusion module 201, a data acquisition module 202, a model fine-tuning module 203, and a model application module 204.

[0083] The model fusion module 201 is used to denote a pre-trained DDPM model as the corresponding first pre-trained model; and to obtain a fusion model by fusing multiple LoRA adapters on the first pre-trained model as the corresponding first fusion model; wherein, the first pre-trained model includes a first diffusion model and a first denoising autoencoder; the first fusion model includes a second diffusion model and a second denoising autoencoder; the overall model parameter set of the first pre-trained model is denoted as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of parameter set W and parameter set L; parameter set L is composed of the first low-rank matrix parameter set and the second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameters A of all adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameters B of all adapter parameters.

[0084] The data acquisition module 202 is used to acquire multiple spherical aberration electron micrographs to form a corresponding first training image set; wherein, the first training image set includes multiple first training images; each first training image is an acquired spherical aberration electron micrograph.

[0085] The model fine-tuning module 203 is used to fine-tune the parameter set L of the first fusion model based on the first training graph set.

[0086] The model application module 204 is used to perform corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder after fine-tuning to obtain the corresponding optimized electron microscope image.

[0087] The processing apparatus for optimizing electron micrograph quality provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0088] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the model fusion module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0089] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0090] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0091] Figure 3 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 3 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0092] exist Figure 3The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0093] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0094] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0095] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for optimizing the quality of electron microscopy (TEM) images. As described above, this invention pre-trains a DDPM model (composed of a first diffusion model and a first denoising autoencoder) and obtains a fusion model (composed of a second diffusion model and a second denoising autoencoder) by fusing multiple LoRA adapters onto the first pre-trained model. The fusion model parameters of the first fusion model consist of a parameter set W and a parameter set L, where W is the overall model parameter set of the first pre-trained model and L is the adapter parameter set of all LoRA adapters. Multiple spherical aberration TEM images are acquired to form a corresponding first training image set, and the parameter set L of the first fusion model is fine-tuned based on the first training image set. After fine-tuning, the second denoising autoencoder of the first fusion model is used to perform corresponding TEM image quality optimization processing on any ordinary transmission electron microscopy (TEM) image to obtain the corresponding optimized TEM image. The embodiments of the present invention can optimize the image quality of ordinary transmission electron microscopy images to achieve the quality level of spherical aberration electron microscopy images, and can improve the efficiency of model fine-tuning and reduce the cost of model use. The embodiments of the present invention can help ordinary teaching or research institutions reduce research costs and improve research efficiency.

[0096] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0097] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the quality of electron micrographs, characterized in that, The method includes: A pre-trained DDPM model is denoted as the first pre-trained model; a fusion model is obtained by fusing multiple LoRA adapters onto the first pre-trained model, denoted as the first fusion model; wherein, the first pre-trained model includes a first diffusion model and a first denoising autoencoder; the first fusion model includes a second diffusion model and a second denoising autoencoder; the overall model parameter set of the first pre-trained model is denoted as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of parameter set W and parameter set L; the parameter set L is composed of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameters A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameters B of all the adapter parameters; Multiple spherical aberration electron micrographs are acquired to form a corresponding first training image set; wherein, the first training image set includes multiple first training images; each first training image is an acquired spherical aberration electron micrograph. The parameter set L of the first fusion model is fine-tuned based on the first training graph set; After fine-tuning, the corresponding optimized electron microscope image is obtained by performing corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder.

2. The method for optimizing electron micrograph quality according to claim 1, characterized in that, The method of obtaining a fusion model by fusing multiple LoRA adapters on the first pre-trained model, denoted as the corresponding first fusion model, specifically includes: Step 21: Select multiple first key layers from the first diffusion model and the first denoising autoencoder of the first pre-trained model according to the preset key layer selection rules; and count the total number of the first key layers to obtain the corresponding total number of key layers N. The expressions for each of the first key layers are as follows: h i =w i x i ; 1 ≤ key layer index i ≤ N; w i x i h i The model parameters, input vector, and output vector of the i-th first key layer are respectively, and the model parameters w i ∈ parameter set W; Step 22, based on the preset low-rank parameter r and the model parameters w of each of the first key layers. i For each of the first key layers, a pair of corresponding low-rank matrix parameters A and B are set, denoted as low-rank matrix parameter A. i B i ; and by each pair of the low-rank matrix parameters A i B i The adapter parameters are used to form a corresponding LoRA adapter; and a corresponding LoRA adapter is fused on each of the first key layers to form a corresponding first fusion layer; The first fusion layer corresponds one-to-one with the first key layer; The expressions for each of the first fusion layers are: h’ i =w i x i +A i B i x i ; h' i This is the output vector of the i-th first fusion layer; Step 23: The first diffusion model of the fused multiple LoRA adapters is used as the corresponding second diffusion model, and the first denoising autoencoder of the fused multiple LoRA adapters is used as the corresponding second denoising autoencoder; the second diffusion model and the second denoising autoencoder are combined to form the corresponding first fusion model; and the low-rank matrix parameter A of the N adapter parameters is used. i The corresponding first low-rank matrix parameter set is formed; and the low-rank matrix parameters B of the N adapter parameters are... i The first and second low-rank matrix parameter sets are used to form the corresponding parameter set L; and the parameter set W and the parameter set L are used to form the fusion model parameters of the first fusion model.

3. The method for optimizing electron micrograph quality according to claim 1, characterized in that, The acquisition of multiple spherical aberration electron microscopy images to form a corresponding first training image set specifically includes: Spherical aberration electron microscopy images are acquired through multiple publicly available data acquisition channels and experimental acquisition methods, and each acquired spherical aberration electron microscopy image is used as a corresponding first training image; and all the obtained first training images form the corresponding first training image set; wherein, the multiple publicly available data acquisition channels include at least publicly available spherical aberration electron microscopy image databases and publicly available scientific and technological literature / journals / technical papers.

4. The method for optimizing electron micrograph quality according to claim 1, characterized in that, The fine-tuning of the parameter set L of the first fusion model based on the first training graph set specifically includes: Step 41: Based on a preset first segmentation ratio, the first training image set is divided into two sub-image sets, denoted as the first image set and the second image set. Wherein, both the first and second image sets are composed of multiple first training images; the ratio of the total number of training images in the first image set to the total number of training images in the second image set satisfies the first segmentation ratio; Step 42: Extract the first training image from the first image set as the corresponding current label image; Step 43: Input the current label map into the second diffusion model of the first fusion model to perform corresponding forward diffusion processing to obtain the corresponding current noise map; and input the current noise map into the second denoising autoencoder of the first fusion model to perform corresponding reverse denoising processing to obtain the corresponding current denoised map; Step 44: Substitute the current denoised image and the current labeled image into the preset first model loss function; and, with the parameter set W frozen, perform one round of optimization on the parameter set L based on the preset first model optimizer in the direction of minimizing the first model loss function; Wherein, the first model loss function is the loss function of the DDPM model; the first model optimizer includes at least the SGD optimizer and the Adam optimizer; Step 45: Identify whether the current label image is the last first training image in the first image set; if yes, proceed to step 46; if no, extract the next first training image in the first image set as the new current label image and return to step 43. Step 46: Perform a traversal of all the first training images in the second image set; during this traversal, take the currently traversed first training image as the corresponding current label image; input the current label image into the second diffusion model of the first fusion model for corresponding forward diffusion processing to obtain the corresponding current noise image; input the current noise image into the second denoising autoencoder of the first fusion model for corresponding reverse denoising processing to obtain the corresponding current denoised image; and form a corresponding first prediction-label image pair with the current denoised image and the current label image; and at the end of this traversal, input all the obtained first prediction-label image pairs into a preset first model evaluation function to calculate the corresponding first evaluation value; The first model evaluation function includes at least the RMSE function; Step 47: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step 42 to continue training; if it meets the range, stop training and confirm that the fine-tuning is complete.

5. The method for optimizing electron micrograph quality according to claim 1, characterized in that, The step of performing corresponding electron microscopy image quality optimization processing on any ordinary transmission electron microscopy image based on the second denoising autoencoder to obtain the corresponding optimized electron microscopy image specifically includes: Receive any ordinary transmission electron microscope (TEM) image as the corresponding current TEM image; input the current TEM image into the second denoising autoencoder for corresponding reverse denoising processing to obtain the corresponding current denoised image; and use the obtained current denoised image as the corresponding optimized TEM image.

6. An apparatus for performing the processing method for optimizing electron micrograph quality according to any one of claims 1-5, characterized in that, The device includes: a model fusion module, a data acquisition module, a model fine-tuning module, and a model application module; The model fusion module is used to denote a pre-trained DDPM model as the corresponding first pre-trained model; and to obtain a fusion model by fusing multiple LoRA adapters on the first pre-trained model, denoted as the corresponding first fusion model; wherein, the first pre-trained model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-trained model is denoted as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of parameter set W and parameter set L; the parameter set L is composed of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameters A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameters B of all the adapter parameters; The data acquisition module is used to acquire multiple spherical aberration electron micrographs to form a corresponding first training image set; wherein, the first training image set includes multiple first training images; each first training image is an acquired spherical aberration electron micrograph. The model fine-tuning module is used to fine-tune the parameter set L of the first fusion model based on the first training graph set; The model application module is used to perform corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder after fine-tuning to obtain the corresponding optimized electron microscope image.

7. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-5; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-5.

Images