Processing method and device for optimizing quality of electron microscope image

Abstract

The embodiment of the invention relates to a processing method and device for optimizing the quality of an electron microscope graph. The method comprises the steps that a DDPM model completing model pre-training is marked as a first pre-training model; obtaining a first fusion model by fusing a plurality of LoRA adapters on the first pre-training model; collecting a plurality of spherical aberration electron microscope images to form a corresponding first training image set; performing fine tuning on a parameter set L of the first fusion model based on the first training image set; and after fine tuning is finished, a second denoising auto-encoder based on the first fusion model performs corresponding electron microscope graph quality optimization processing on any common transmission electron microscope graph to obtain a corresponding optimized electron microscope graph. According to the invention, the image quality of a common transmission electron microscope image can be optimized to reach the quality level of a spherical aberration electron microscope image, the fine tuning efficiency of the model can be improved, and the use cost of the model is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, and in particular to a processing method and device for optimizing the quality of electron microscope images. Background Art

[0002] The working principle of the transmission electron microscope (TEM) is to irradiate the sample with a high-energy electron beam of extremely short wavelength, image the sample based on the feedback characteristics of the sample's physical properties under the electron beam irradiation, and output the corresponding transmission electron microscope image. The transmission electron microscope image is mainly used to observe the internal structure of the material, such as lattice, defects, phase change, etc., and is an important research tool in the fields of materials science, physics, chemistry, etc. The working principle of the spherical aberration electron microscope (ACEM) is to add a spherical aberration corrector to the ordinary transmission electron microscope to correct the spherical aberration and other aberrations of the ordinary transmission electron microscope, so as to achieve the purpose of improving the resolution of the electron microscope image. In other words, for the same sample, the quality of the electron microscope image output by the spherical aberration electron microscope is much better than that of the transmission electron microscope. The electron microscope image output by the spherical aberration electron microscope can achieve better analysis results for the structure analysis of the sample material. However, the reality is that the price of the spherical aberration electron microscope is very expensive, often ten or even dozens of times that of the ordinary transmission electron microscope, which is unaffordable for ordinary teaching or scientific research institutions.

[0003] The diffusion probability model (DDPM) proposed in the technical document "Denoising Diffusion Probabilistic Models" is a type of large visual model for image processing based on the principle of forward diffusion-reverse denoising. It can be seen from the technical document that the model framework of the DDPM model consists of two parts: diffusion models and denoising autoencoders. The diffusion model performs forward diffusion processing on the input image of the DDPM model based on a step-by-step denoising method to obtain a noisy image, while the denoising autoencoder performs reverse denoising processing on the noisy image through a step-by-step denoising method to obtain a final denoised image as the generated image of the DDPM model and outputs it. By training the DDPM model according to the training data set and model training method given in the technical document, a pre-training DDPM model (referred to as the pre-training model) can be obtained. The pre-training model can be used to optimize the resolution of conventional images to achieve the purpose of improving image quality. Inspired by the technical document 1, we fine-tune the model parameter set (referred to as parameter set W) of the pre-trained model through a series of spherical aberration electron microscopy images so that it can optimize ordinary transmission electron microscopy images and achieve the image quality level of spherical aberration electron microscopy images. However, during the fine-tuning process, we found that if the parameter set W is fine-tuned by directly modifying the parameter value based on conventional model optimizers (such as SGD optimizer, Adam optimizer, etc.), the resulting calculation amount is large and the fine-tuning cycle is long, resulting in low model fine-tuning efficiency.

[0004] Technical document 2 "LoRA: Low-rank Adaptation of Large Language Models" provides a small model LoRA for reducing the difficulty of debugging the target large model. It can be seen from technical document 2 that the debugging principle of the LoRA model is to embed a corresponding LoRA adapter in each key module of the target large model, and fuse the parameters of each adapter with the original model parameters of the corresponding module to achieve the purpose of fusing the LoRA model with the target large model and thereby obtain a fusion model of the target large model + LoRA model. When debugging the model, the original model parameters of the target large model are not modified, and only the model parameters of the LoRA model are fine-tuned to achieve the debugging effect of overall optimization of the fusion model. Because the number of parameters and parameter complexity of the LoRA model are much smaller than the number of parameters and parameter complexity of the target large model, the fine-tuning method based on the above fusion model can effectively reduce the amount of calculation and shorten the debugging cycle. Inspired by technical document 2, we further introduced the LoRA model into the fine-tuning process of the pre-trained model, thereby achieving the purpose of improving the efficiency of model fine-tuning.

[0005] Finally, through the above two-step technical attempts, we have summarized an electron microscope image quality optimization processing mechanism based on the DDPM+LoRA fusion model to optimize the image quality of ordinary transmission electron microscope images and achieve the quality level of spherical aberration electron microscope images, and the present invention provides a technical implementation solution of the processing mechanism. Summary of the invention

[0006] The purpose of the present invention is to provide a processing method, device, electronic device and computer-readable storage medium for optimizing the quality of electron microscope images in view of the defects of the prior art. The present invention records a DDPM model that has completed model pre-training as a first pre-training model (composed of a first diffusion model and a first denoising autoencoder), and obtains a fusion model by fusing multiple LoRA adapters on the first pre-training model, which is recorded as a first fusion model (composed of a second diffusion model and a second denoising autoencoder). The fusion model parameters of the first fusion model are composed of a parameter set W and a parameter set L, wherein the parameter set W is the overall model parameter set of the first pre-training model, and the parameter set L is the adapter parameter set of all LoRA adapters; and multiple spherical aberration electron microscope images are collected to form a corresponding first training atlas, and the parameter set L of the first fusion model is fine-tuned based on the first training atlas; and after the fine-tuning is completed, the second denoising autoencoder based on the first fusion model performs corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image to obtain a corresponding optimized electron microscope image. On the one hand, the present invention can optimize the image quality of ordinary transmission electron microscope images to reach the quality level of spherical aberration electron microscope images, and on the other hand, it can improve the efficiency of model fine-tuning and reduce the cost of model use; the present invention can help ordinary teaching or scientific research institutions reduce scientific research costs and improve scientific research efficiency.

[0007] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a processing method for optimizing the quality of an electron microscope image, the method comprising:

[0008] A DDPM model that has completed model pre-training is recorded as the corresponding first pre-training model; and a fusion model is obtained by fusing multiple LoRA adapters on the first pre-training model and recorded as the corresponding first fusion model; wherein, the first pre-training model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-training model is recorded as parameter set W; the adapter parameters of each of the LoRA adapters are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of the parameter set W and the parameter set L; the parameter set L is composed of the first low-rank matrix parameter set and the second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameter A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameter B of all the adapter parameters;

[0009] Collecting a plurality of spherical aberration electron microscope images to form a corresponding first training atlas; wherein the first training atlas includes a plurality of first training images; each of the first training images is a collected spherical aberration electron microscope image;

[0010] Fine-tuning the parameter set L of the first fusion model based on the first training atlas;

[0011] After the fine-tuning is completed, the quality optimization of any ordinary transmission electron microscope image is performed based on the second denoising autoencoder to obtain the corresponding optimized electron microscope image.

[0012] Preferably, the method of obtaining a fusion model by fusing multiple LoRA adapters on the first pre-trained model is recorded as the corresponding first fusion model, which specifically includes:

[0013] Step 21, according to a preset key layer selection rule, multiple first key layers are selected from the first diffusion model of the first pre-trained model and the first denoising autoencoder; and the total number of the first key layers is counted to obtain the corresponding total number of key layers N;

[0014] Among them, the expression of each of the first key layers is:

[0015] h i =w i x i ;

[0016] 1≤key layer index i≤N; w i 、x i 、h i are the model parameters, input vectors and output vectors of the first key layer i, respectively. The model parameter w i ∈ parameter set W;

[0017] Step 22: based on the preset low-rank parameter r and the model parameters w of each of the first key layers i A pair of corresponding low-rank matrix parameters A and B are set for each of the first key layers, denoted as low-rank matrix parameter A i , B i ; and by each pair of the low-rank matrix parameters A i , B i The adapter parameters of a corresponding LoRA adapter are formed; and a corresponding LoRA adapter is fused on each of the first key layers to form a corresponding first fusion layer;

[0018] Wherein, the first fusion layer corresponds to the first key layer one by one;

[0019] The expressions of the first fusion layers are:

[0020] h i =w i x i +A i B i x i ;

[0021] h'i is the output vector of the i-th first fusion layer;

[0022] Step 23, using the first diffusion model of the fused multiple LoRA adapters as the corresponding second diffusion model, and using the first denoising autoencoder fused with multiple LoRA adapters as the corresponding second denoising autoencoder; and the second diffusion model and the second denoising autoencoder form the corresponding first fusion model; and the low-rank matrix parameter A of the N adapter parameters i The first low-rank matrix parameter set corresponding to the adapter parameters is formed; and the low-rank matrix parameters B of the N adapter parameters are formed. i The corresponding second low-rank matrix parameter set is formed; and the corresponding parameter set L is formed by the first and second low-rank matrix parameter sets; and the parameter set W and the parameter set L constitute the fusion model parameters of the first fusion model.

[0023] Preferably, the collecting of a plurality of spherical aberration electron microscope images to form a corresponding first training atlas specifically includes:

[0024] Spherical aberration electron microscope images are collected through multiple types of public data collection channels and experimental collection methods, and each collected spherical aberration electron microscope image is used as a corresponding first training image; and all the first training images obtained constitute a corresponding first training atlas; wherein, the multiple types of public data collection channels include at least a public spherical aberration electron microscope image database, and public scientific and technological literature / journals / technical papers.

[0025] Preferably, the fine-tuning of the parameter set L of the first fusion model based on the first training atlas specifically includes:

[0026] Step 41, based on a preset first segmentation ratio, the first training atlas is divided into two sub-atlases, which are recorded as a corresponding first atlas and a second atlas;

[0027] Wherein, both the first and second atlases are composed of a plurality of the first training images; the ratio of the total number of training images in the first atlas to the total number of training images in the second atlas satisfies the first segmentation ratio;

[0028] Step 42, extracting the first training image from the first image set as the corresponding current label image;

[0029] Step 43, inputting the current label image into the second diffusion model of the first fusion model to perform corresponding forward diffusion processing to obtain a corresponding current noise image; and inputting the current noise image into the second denoising autoencoder of the first fusion model to perform corresponding reverse denoising processing to obtain a corresponding current denoised image;

[0030] Step 44, bringing the current denoised image and the current label image into a preset first model loss function; and, under the premise of freezing the parameter set W, performing a round of optimization on the parameter set L based on a preset first model optimizer in a direction of minimizing the first model loss function;

[0031] Wherein, the first model loss function is the loss function of the DDPM model; the first model optimizer includes at least an SGD optimizer and an Adam optimizer;

[0032] Step 45, identifying whether the current label image is the last first training image of the first atlas; if so, going to step 46; if not, extracting the next first training image of the first atlas as the new current label image and returning to step 43;

[0033] Step 46, perform a round of traversal on all the first training images of the second atlas; and in this round of traversal, use the first training image currently traversed as the corresponding current label image; and input the current label image into the second diffusion model of the first fusion model for corresponding forward diffusion processing to obtain the corresponding current noise image; and input the current noise image into the second denoising autoencoder of the first fusion model for corresponding reverse denoising processing to obtain the corresponding current denoised image; and form a corresponding first prediction-label image pair with the current denoised image and the current label image; and at the end of this round of traversal, bring all the obtained first prediction-label image pairs into the preset first model evaluation function for calculation to obtain the corresponding first evaluation value;

[0034] Wherein, the first model evaluation function at least includes an RMSE function;

[0035] Step 47, identifying whether the first evaluation value satisfies a preset first evaluation value range; if not, returning to step 42 to continue training; if satisfied, stopping training and confirming the end of fine-tuning.

[0036] Preferably, the method of performing corresponding electron microscope image quality optimization processing on any common transmission electron microscope image based on the second denoising autoencoder to obtain a corresponding optimized electron microscope image specifically includes:

[0037] Receive any common transmission electron microscope image as the corresponding current electron microscope image; input the current electron microscope image into the second denoising autoencoder for corresponding reverse denoising processing to obtain the corresponding current denoised image; and use the obtained current denoised image as the corresponding optimized electron microscope image.

[0038] A second aspect of the embodiment of the present invention provides a device for implementing the processing method for optimizing the quality of electron microscope images described in the first aspect, the device comprising: a model fusion module, a data acquisition module, a model fine-tuning module and a model application module;

[0039] The model fusion module is used to record a DDPM model that has completed model pre-training as the corresponding first pre-training model; and to obtain a fusion model by fusing multiple LoRA adapters on the first pre-training model as the corresponding first fusion model; wherein, the first pre-training model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-training model is recorded as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of the parameter set W and the parameter set L; the parameter set L is composed of the first low-rank matrix parameter set and the second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameter A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameter B of all the adapter parameters;

[0040] The data acquisition module is used to collect multiple spherical aberration electron microscope images to form a corresponding first training atlas; wherein the first training atlas includes multiple first training images; each of the first training images is a collected spherical aberration electron microscope image;

[0041] The model fine-tuning module is used to fine-tune the parameter set L of the first fusion model based on the first training atlas;

[0042] The model application module is used to perform corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder after fine-tuning to obtain a corresponding optimized electron microscope image.

[0043] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0044] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0045] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0046] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0047] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for optimizing the quality of electron microscope images. As can be seen from the above content, the embodiment of the present invention records a DDPM model that has completed model pre-training as a first pre-training model (composed of a first diffusion model and a first denoising autoencoder), and obtains a fusion model by fusing multiple LoRA adapters on the first pre-training model, which is recorded as a first fusion model (composed of a second diffusion model and a second denoising autoencoder). The fusion model parameters of the first fusion model are composed of a parameter set W and a parameter set L, wherein the parameter set W is the overall model parameter set of the first pre-training model, and the parameter set L is the adapter parameter set of all LoRA adapters; and multiple spherical aberration electron microscope images are collected to form a corresponding first training atlas, and the parameter set L of the first fusion model is fine-tuned based on the first training atlas; and after the fine-tuning is completed, the second denoising autoencoder based on the first fusion model performs corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image to obtain a corresponding optimized electron microscope image. On the one hand, the embodiments of the present invention can optimize the image quality of ordinary transmission electron microscope images to reach the quality level of spherical aberration electron microscope images, and on the other hand, it can improve the efficiency of model fine-tuning and reduce the cost of model use; the embodiments of the present invention can help ordinary teaching or scientific research institutions reduce scientific research costs and improve scientific research efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a processing method for optimizing the quality of electron microscope images provided in the first embodiment of the present invention;

[0049] Figure 2 A module structure diagram of a processing device for optimizing the quality of electron microscope images provided in the second embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1 of the present invention provides a processing method for optimizing the quality of electron microscope images, such as Figure 1 As shown in the schematic diagram of a processing method for optimizing the quality of electron microscope images provided in the first embodiment of the present invention, the method mainly comprises the following steps:

[0053] Step 1: record a DDPM model that has completed model pre-training as the corresponding first pre-training model; and obtain a fusion model by fusing multiple LoRA adapters on the first pre-training model as the corresponding first fusion model;

[0054] Specifically comprising: step 11, recording a DDPM model that has completed model pre-training as a corresponding first pre-training model;

[0055] Here, the DDPM model of the embodiment of the present invention is the diffusion probability model (Denoising Diffusion Probabilistic Models, DDPM) proposed in the technical document 1 "Denoising Diffusion Probabilistic Models". The DDPM model has completed the model training according to the training data set and model training method given in the technical document 1. It is regarded as a DDPM model that has completed model pre-training and is recorded as the first pre-training model. It can be seen from the technical document 1 that the model framework of the DDPM model consists of two parts: diffusion models and denoising autoencoders. Therefore, the first pre-training model includes a first diffusion model and a first denoising autoencoder, and the overall model parameter set of the first pre-training model is recorded as parameter set W;

[0056] Step 12, and obtain a fusion model by fusing multiple LoRA adapters on the first pre-trained model, which is recorded as the corresponding first fusion model;

[0057] Among them, the first fusion model includes a second diffusion model and a second denoising autoencoder; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of a parameter set W and a parameter set L; the parameter set L is composed of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of a low-rank matrix parameter A of all adapter parameters; the second low-rank matrix parameter set is composed of a low-rank matrix parameter B of all adapter parameters;

[0058] Specifically, it includes: step 121, according to a preset key layer selection rule, selecting a plurality of first key layers from the first diffusion model and the first denoising autoencoder of the first pre-trained model; and counting the total number of the first key layers to obtain the corresponding total number of key layers N;

[0059] Here, the expressions of the first key layers of the first pre-trained model in the embodiment of the present invention are: i =w i x i ; Where 1≤key layer index i≤N; w i、x i 、h i are the model parameters, input vectors and output vectors of the first key layer of the i-th layer respectively, and the model parameters w i ∈ parameter set W;

[0060] The key layer selection rule of the embodiment of the present invention is a preset rule, which can be customized based on application requirements, or customized with reference to the LoRA adapter embedding method listed in the second technical document "LoRA: Low-rank Adaptation of Large Language Models"; a common customization method under normal circumstances is: one or more attention layers and / or linear layers in the first diffusion model and the first denoising autoencoder are used as key layers; another common customization method under normal circumstances is: one or more attention layers and / or linear layers in the first denoising autoencoder are used as key layers; another common customization method under normal circumstances is: all attention layers and the last one or two linear layers in the first denoising autoencoder are used as key layers;

[0061] Step 122: based on the preset low-rank parameter r and the model parameters w of each first key layer i Set a pair of corresponding low-rank matrix parameters A and B for each first key layer, denoted as low-rank matrix parameter A i , B i ; and by each pair of low-rank matrix parameters A i , B i The adapter parameters of a corresponding LoRA adapter are formed; and a corresponding LoRA adapter is fused on each first key layer to form a corresponding first fusion layer;

[0062] Here, the configuration of the low-rank parameter r and the low-rank matrix parameters A and B can be connected with reference to the technical document 2 "LoRA: Low-rank Adaptation of Large Language Models"; the first fusion layer of the embodiment of the present invention corresponds to the first key layer one by one, and the expression of each first fusion layer is: h' i =w i x i +A i B i x i ; where h' i is the output vector of the i-th first fusion layer;

[0063] Step 123, the first diffusion model of the fused multiple LoRA adapters is used as the corresponding second diffusion model, and the first denoising autoencoder fused with multiple LoRA adapters is used as the corresponding second denoising autoencoder; and the second diffusion model and the second denoising autoencoder form the corresponding first fusion model; and the low-rank matrix parameter A of the N adapter parameters i The first low-rank matrix parameter set corresponding to the adapter is formed; and the low-rank matrix parameter B of the N adapter parameters i The corresponding second low-rank matrix parameter set is formed; the first and second low-rank matrix parameter sets are formed into a corresponding parameter set L; and the parameter set W and the parameter set L are formed into the fusion model parameters of the first fusion model.

[0064] Step 2, collecting multiple spherical aberration electron microscope images to form a corresponding first training atlas;

[0065] Specifically, the method includes: collecting spherical aberration electron microscope images through multiple types of public data collection channels and experimental collection methods, and taking each collected spherical aberration electron microscope image as a corresponding first training image; and forming a corresponding first training atlas from all the obtained first training images.

[0066] Here, the multiple types of public data acquisition channels include at least a public spherical aberration electron microscope image database, public scientific and technological literature / journals / technical papers; the first training atlas includes multiple first training images; each first training image is a collected spherical aberration electron microscope image.

[0067] Step 3, fine-tuning the parameter set L of the first fusion model based on the first training atlas;

[0068] Specifically, it includes: step 31, dividing the first training atlas into two sub-atlases based on a preset first segmentation ratio and recording them as corresponding first atlas and second atlas;

[0069] The first segmentation ratio is a preset ratio parameter, such as 8:2; both the first and second atlases are composed of a plurality of first training images; the ratio of the total number of training images in the first atlas to the total number of training images in the second atlas satisfies the first segmentation ratio;

[0070] Step 32, extracting the first first training image from the first image set as the corresponding current label image;

[0071] Step 33, input the current label image into the second diffusion model of the first fusion model to perform corresponding forward diffusion processing to obtain the corresponding current noise image; and input the current noise image into the second denoising autoencoder of the first fusion model to perform corresponding reverse denoising processing to obtain the corresponding current denoised image;

[0072] Step 34, bringing the current denoised image and the current label image into the preset first model loss function; and under the premise of freezing the parameter set W, performing a round of optimization on the parameter set L based on the preset first model optimizer in the direction of minimizing the first model loss function;

[0073] The first model loss function is a loss function of a DDPM model; the first model optimizer includes at least an SGD optimizer and an Adam optimizer;

[0074] Here, because only the parameter set L with a small amount of data is modified during optimization, and the parameter set W with a large amount of data does not need to be modified, the amount of calculation can be greatly reduced, the training cycle can be significantly shortened, and the training efficiency can be quickly improved;

[0075] Step 35, identifying whether the current label image is the last first training image of the first atlas; if so, go to step 36; if not, extract the next first training image of the first atlas as the new current label image and return to step 33;

[0076] Step 36, perform a round of traversal on all the first training images of the second atlas; and in this round of traversal, use the currently traversed first training image as the corresponding current label image; and input the current label image into the second diffusion model of the first fusion model for corresponding forward diffusion processing to obtain the corresponding current noise image; and input the current noise image into the second denoising autoencoder of the first fusion model for corresponding reverse denoising processing to obtain the corresponding current denoised image; and form a corresponding first prediction-label image pair from the current denoised image and the current label image; and at the end of this round of traversal, bring all the obtained first prediction-label image pairs into the preset first model evaluation function for calculation to obtain the corresponding first evaluation value;

[0077] Wherein, the first model evaluation function at least includes the RMSE function;

[0078] Step 37, identifying whether the first evaluation value meets the preset first evaluation value range; if not, returning to step 32 to continue training; if satisfied, stopping training and confirming the end of fine-tuning.

[0079] Here, the first evaluation value range is a preset value range.

[0080] Step 4: After the fine-tuning is completed, the quality of any ordinary transmission electron microscope image is optimized based on the second denoising autoencoder to obtain the corresponding optimized electron microscope image;

[0081] Specifically, the method includes: receiving any common transmission electron microscope image as the corresponding current electron microscope image; inputting the current electron microscope image into the second denoising autoencoder for corresponding reverse denoising processing to obtain the corresponding current denoised image; and using the obtained current denoised image as the corresponding optimized electron microscope image.

[0082] Figure 2 The module structure diagram of a processing device for optimizing the quality of electron microscope images provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a model fusion module 201, a data acquisition module 202, a model fine-tuning module 203 and a model application module 204.

[0083] The model fusion module 201 is used to record a DDPM model that has completed model pre-training as the corresponding first pre-training model; and obtain a fusion model by fusing multiple LoRA adapters on the first pre-training model and record it as the corresponding first fusion model; wherein the first pre-training model includes a first diffusion model and a first denoising autoencoder; the first fusion model includes a second diffusion model and a second denoising autoencoder; the overall model parameter set of the first pre-training model is recorded as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of parameter set W and parameter set L; parameter set L is composed of a first low-rank matrix parameter set and a second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of low-rank matrix parameters A of all adapter parameters; the second low-rank matrix parameter set is composed of low-rank matrix parameters B of all adapter parameters.

[0084] The data acquisition module 202 is used to collect multiple spherical aberration electron microscope images to form a corresponding first training atlas; wherein the first training atlas includes multiple first training images; each first training image is a collected spherical aberration electron microscope image.

[0085] The model fine-tuning module 203 is used to fine-tune the parameter set L of the first fusion model based on the first training atlas.

[0086] The model application module 204 is used to perform corresponding electron microscope image quality optimization processing on any common transmission electron microscope image based on the second denoising autoencoder after the fine-tuning is completed to obtain a corresponding optimized electron microscope image.

[0087] An embodiment of the present invention provides a processing device for optimizing the quality of electron microscope images, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0088] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the model fusion module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0089] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0090] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0091] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0092] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0093] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0094] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0095] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for optimizing the quality of electron microscope images. As can be seen from the above content, the embodiment of the present invention records a DDPM model that has completed model pre-training as a first pre-training model (composed of a first diffusion model and a first denoising autoencoder), and obtains a fusion model by fusing multiple LoRA adapters on the first pre-training model, which is recorded as a first fusion model (composed of a second diffusion model and a second denoising autoencoder). The fusion model parameters of the first fusion model are composed of a parameter set W and a parameter set L, wherein the parameter set W is the overall model parameter set of the first pre-training model, and the parameter set L is the adapter parameter set of all LoRA adapters; and multiple spherical aberration electron microscope images are collected to form a corresponding first training atlas, and the parameter set L of the first fusion model is fine-tuned based on the first training atlas; and after the fine-tuning is completed, the second denoising autoencoder based on the first fusion model performs corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image to obtain a corresponding optimized electron microscope image. On the one hand, the embodiments of the present invention can optimize the image quality of ordinary transmission electron microscope images to reach the quality level of spherical aberration electron microscope images, and on the other hand, it can improve the efficiency of model fine-tuning and reduce the cost of model use; the embodiments of the present invention can help ordinary teaching or scientific research institutions reduce scientific research costs and improve scientific research efficiency.

[0096] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0097] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A processing method for optimizing the quality of electron microscope images, characterized in that: The method comprises: A DDPM model that has completed model pre-training is recorded as the corresponding first pre-training model; and a fusion model is obtained by fusing multiple LoRA adapters on the first pre-training model and recorded as the corresponding first fusion model; wherein, the first pre-training model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-training model is recorded as parameter set W; the adapter parameters of each of the LoRA adapters are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of the parameter set W and the parameter set L; the parameter set L is composed of the first low-rank matrix parameter set and the second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameter A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameter B of all the adapter parameters; Collecting a plurality of spherical aberration electron microscope images to form a corresponding first training atlas; wherein the first training atlas includes a plurality of first training images; each of the first training images is a collected spherical aberration electron microscope image; Fine-tuning the parameter set L of the first fusion model based on the first training atlas; After the fine-tuning is completed, the quality optimization of any ordinary transmission electron microscope image is performed based on the second denoising autoencoder to obtain the corresponding optimized electron microscope image.

2. The method for optimizing the quality of electron microscope images according to claim 1, characterized in that: The method of obtaining a fusion model by fusing multiple LoRA adapters on the first pre-trained model is recorded as the corresponding first fusion model, specifically including: Step 21, according to a preset key layer selection rule, multiple first key layers are selected from the first diffusion model of the first pre-trained model and the first denoising autoencoder; and the total number of the first key layers is counted to obtain the corresponding total number of key layers N; Among them, the expression of each of the first key layers is: h i =w i x i ; 1≤key layer index i≤N; w i 、x i 、h i are the model parameters, input vectors and output vectors of the first key layer i, respectively. The model parameter w i ∈ parameter set W; Step 22: based on the preset low-rank parameter r and the model parameters w of each of the first key layers i A pair of corresponding low-rank matrix parameters A and B are set for each of the first key layers, denoted as low-rank matrix parameter A i , B i ; and by each pair of the low-rank matrix parameters A i , B i The adapter parameters of a corresponding LoRA adapter are formed; and a corresponding LoRA adapter is fused on each of the first key layers to form a corresponding first fusion layer; Wherein, the first fusion layer corresponds to the first key layer one by one; The expressions of the first fusion layers are: h’ i =w i x i +A i B i x i ; h' i is the output vector of the i-th first fusion layer; Step 23, using the first diffusion model of the fused multiple LoRA adapters as the corresponding second diffusion model, and using the first denoising autoencoder fused with multiple LoRA adapters as the corresponding second denoising autoencoder; and the second diffusion model and the second denoising autoencoder form the corresponding first fusion model; and the low-rank matrix parameter A of the N adapter parameters i The first low-rank matrix parameter set corresponding to the adapter parameters is formed; and the low-rank matrix parameters B of the N adapter parameters are formed. i The corresponding second low-rank matrix parameter set is formed; and the corresponding parameter set L is formed by the first and second low-rank matrix parameter sets; and the parameter set W and the parameter set L constitute the fusion model parameters of the first fusion model.

3. The method for optimizing the quality of electron microscope images according to claim 1, characterized in that: The collecting of a plurality of spherical aberration electron microscope images to form a corresponding first training atlas specifically includes: Spherical aberration electron microscope images are collected through multiple types of public data collection channels and experimental collection methods, and each collected spherical aberration electron microscope image is used as a corresponding first training image; and all the first training images obtained constitute a corresponding first training atlas; wherein, the multiple types of public data collection channels include at least a public spherical aberration electron microscope image database, and public scientific and technological literature / journals / technical papers.

4. The method for optimizing the quality of electron microscope images according to claim 1, characterized in that: The fine-tuning of the parameter set L of the first fusion model based on the first training atlas specifically includes: Step 41, based on a preset first segmentation ratio, the first training atlas is divided into two sub-atlases, which are recorded as a corresponding first atlas and a second atlas; Wherein, both the first and second atlases are composed of a plurality of the first training images; the ratio of the total number of training images in the first atlas to the total number of training images in the second atlas satisfies the first segmentation ratio; Step 42, extracting the first training image from the first image set as the corresponding current label image; Step 43, inputting the current label image into the second diffusion model of the first fusion model to perform corresponding forward diffusion processing to obtain a corresponding current noise image; and inputting the current noise image into the second denoising autoencoder of the first fusion model to perform corresponding reverse denoising processing to obtain a corresponding current denoised image; Step 44, bringing the current denoised image and the current label image into a preset first model loss function; and, under the premise of freezing the parameter set W, performing a round of optimization on the parameter set L based on a preset first model optimizer in a direction of minimizing the first model loss function; Wherein, the first model loss function is the loss function of the DDPM model; the first model optimizer includes at least an SGD optimizer and an Adam optimizer; Step 45, identifying whether the current label image is the last first training image of the first atlas; if so, going to step 46; if not, extracting the next first training image of the first atlas as the new current label image and returning to step 43; Step 46, perform a round of traversal on all the first training images of the second atlas; and in this round of traversal, use the first training image currently traversed as the corresponding current label image; and input the current label image into the second diffusion model of the first fusion model for corresponding forward diffusion processing to obtain the corresponding current noise image; and input the current noise image into the second denoising autoencoder of the first fusion model for corresponding reverse denoising processing to obtain the corresponding current denoised image; and form a corresponding first prediction-label image pair with the current denoised image and the current label image; and at the end of this round of traversal, bring all the obtained first prediction-label image pairs into the preset first model evaluation function for calculation to obtain the corresponding first evaluation value; Wherein, the first model evaluation function at least includes an RMSE function; Step 47, identifying whether the first evaluation value satisfies a preset first evaluation value range; if not, returning to step 42 to continue training; if satisfied, stopping training and confirming the end of fine-tuning.

5. The method for optimizing the quality of electron microscope images according to claim 1, characterized in that: The method of performing corresponding electron microscope image quality optimization processing on any common transmission electron microscope image based on the second denoising autoencoder to obtain a corresponding optimized electron microscope image specifically includes: Receive any common transmission electron microscope image as the corresponding current electron microscope image; input the current electron microscope image into the second denoising autoencoder for corresponding reverse denoising processing to obtain the corresponding current denoised image; and use the obtained current denoised image as the corresponding optimized electron microscope image.

6. A device for executing the processing method for optimizing the quality of electron microscope images according to any one of claims 1 to 5, characterized in that: The device comprises: a model fusion module, a data acquisition module, a model fine-tuning module and a model application module; The model fusion module is used to record a DDPM model that has completed model pre-training as the corresponding first pre-training model; and to obtain a fusion model by fusing multiple LoRA adapters on the first pre-training model as the corresponding first fusion model; wherein, the first pre-training model includes the first diffusion model and the first denoising autoencoder; the first fusion model includes the second diffusion model and the second denoising autoencoder; the overall model parameter set of the first pre-training model is recorded as parameter set W; the adapter parameters of each LoRA adapter are composed of two low-rank matrix parameters A and B; the fusion model parameters of the first fusion model are composed of the parameter set W and the parameter set L; the parameter set L is composed of the first low-rank matrix parameter set and the second low-rank matrix parameter set; the first low-rank matrix parameter set is composed of the low-rank matrix parameter A of all the adapter parameters; the second low-rank matrix parameter set is composed of the low-rank matrix parameter B of all the adapter parameters; The data acquisition module is used to collect multiple spherical aberration electron microscope images to form a corresponding first training atlas; wherein the first training atlas includes multiple first training images; each of the first training images is a collected spherical aberration electron microscope image; The model fine-tuning module is used to fine-tune the parameter set L of the first fusion model based on the first training atlas; The model application module is used to perform corresponding electron microscope image quality optimization processing on any ordinary transmission electron microscope image based on the second denoising autoencoder after fine-tuning to obtain a corresponding optimized electron microscope image.

7. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 5; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is enabled to execute the method according to any one of claims 1 to 5.

Images