A transmission electron microscope image denoising method and device based on block feature alignment

Abstract

The present invention provides a transmission electron microscope image denoising method and device based on block feature alignment, which relates to the field of microscopic image characterization and analysis technology. The method comprises: obtaining an atomic microscopic video stream, screening to obtain unpaired high-quality and low-quality data sets; using a variety of clustering methods to cluster the high-quality and low-quality data sets respectively, obtaining blocks containing atomic regions based on the clustering results and a voting mechanism, performing unsupervised reconstruction training on the autoencoder, and obtaining a pre-trained model for high-quality and low-quality data reconstruction; extracting features from the low-quality and high-quality data sets using the pre-trained model, projecting and aligning their potential representations into a common latent space, and achieving low-quality image denoising. The transmission electron microscope image denoising method based on block feature alignment of the present invention can solve the problems in the prior art of inaccurate identification of atomic regions, limited denoising effects, and lack of a high-quality and low-quality image feature alignment mechanism, thereby improving image quality and structural restoration accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of microscopic image characterization and analysis, and in particular to a transmission electron microscope image denoising method and device based on block feature alignment. Background Art

[0002] In materials science research, the size and quality of generated datasets are growing exponentially with the continuous advancement of image acquisition equipment. Currently, high-resolution modes of TEM (Transmission electron microscopy) can produce atomic-level resolution images. These images contain complex information such as atomic coefficients, lattice parameters, lattice type, sample orientation, and defects. This information can be quantitatively interpreted by localizing atomic columns. However, the quality of experimental images produced by these instruments is often affected by both random and deterministic environmental distortions, with noise being particularly pronounced at low doses or high resolution. Furthermore, scanning distortion caused by instrument drift and environmental vibrations can occur during long imaging periods, leading to structural distortion in the image. Increasing imaging speed significantly degrades the signal-to-noise ratio and image quality, hindering dynamic observation with high spatial and temporal resolution. Therefore, improving image quality is a critical step in ensuring accurate analysis, aiming to preserve image detail while improving the signal-to-noise ratio.

[0003] Traditional regularized image denoising models are divided into two categories: linear models and nonlinear models. In TEM data denoising, the preservation of edge details is crucial. Patent publication number CN117830136A provides a method and system for processing transmission electron microscope images. The patent extracts the collected original electron microscope image and performs restoration processing, calculates the pixel value of each pixel in the original electron microscope image, corresponds the coordinates of the pixel points of the target image to the coordinates in the original electron microscope image, and restores the target image through bilinear interpolation. The restored image is decomposed using wavelet transform to obtain high-frequency coefficients and low-frequency coefficients, and the high-frequency coefficients are threshold-quantized to obtain processed components. Wavelet reconstruction is performed based on the components and low-frequency coefficients to obtain a denoised image. However, this patent only performs image enhancement based on fixed rules and explicit algorithms, which makes it difficult to adapt to the complex and diverse noise types and structural distortion problems in transmission electron microscope images. It lacks modeling of the feature mapping relationship between high-quality and low-quality images, and cannot achieve more robust and generalized image denoising and structural restoration. There are certain limitations in practical applications. Summary of the Invention

[0004] To address the technical issues of existing technologies, such as inaccurate atomic region recognition, limited denoising effects, and a lack of a mechanism for aligning high-quality and low-quality image features, the present invention provides a method and apparatus for denoising transmission electron microscope images based on block feature alignment. The technical solution is as follows:

[0005] In one aspect, a transmission electron microscope image denoising method based on block feature alignment is provided. The method is implemented by a transmission electron microscope image denoising device, and the method comprises:

[0006] S1. Acquire atomic microscopic video streams through a transmission electron microscope, filter the images in the atomic microscopic video streams, and obtain unpaired high-quality and low-quality datasets.

[0007] S2. Use multiple clustering methods to cluster the high-quality dataset and the low-quality dataset respectively, and obtain the blocks containing atomic regions in the high-quality dataset and the blocks containing atomic regions in the low-quality dataset based on the clustering results and the voting mechanism.

[0008] S3. Perform unsupervised reconstruction training on the autoencoder based on the blocks containing atomic regions in the high-quality dataset and the blocks containing atomic regions in the low-quality dataset, respectively, to obtain a pre-training model for high-quality data reconstruction and a pre-training model for low-quality data reconstruction.

[0009] Among them, the pre-training model for high-quality data reconstruction includes a high-quality encoder and a high-quality decoder, and the pre-training model for low-quality data reconstruction includes a low-quality encoder and a low-quality decoder.

[0010] S4. Obtain high-quality image block features based on a high-quality dataset and a pre-trained model reconstructed from high-quality data. Obtain low-quality image block features based on a low-quality dataset and a pre-trained model reconstructed from low-quality data. Project the high-quality image block features and the low-quality image block features into a common latent space. Use a joint loss function as a constraint to gradually allow the low-quality image block features to approach the high-quality image block features, thereby obtaining a trained low-quality encoder.

[0011] S5. Denoise the low-quality image based on the trained low-quality encoder and high-quality decoder.

[0012] Optionally, S2 includes:

[0013] S21. Crop the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks.

[0014] S22. Cluster the high-quality image blocks using multiple clustering methods to obtain multiple clustering results of the high-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the high-quality data set.

[0015] S23. Cluster the low-quality image blocks using multiple clustering methods to obtain multiple clustering results of the low-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the low-quality data set.

[0016] Optionally, the multiple clustering methods include a clustering method based on grayscale features of an image, a clustering method based on statistical features of an image average grayscale value and grayscale standard deviation, and a clustering method based on image frequency domain and structural statistical features.

[0017] Optionally, in S3, unsupervised reconstruction training is performed on the autoencoder according to the blocks containing the atomic regions in the high-quality data set and the blocks containing the atomic regions in the low-quality data set, respectively, to obtain a pre-trained model for high-quality data reconstruction and a pre-trained model for low-quality data reconstruction, including:

[0018] S31. Input the blocks containing atomic regions in the high-quality data set into the U-Net with an encoder-decoder structure, and use the mean square error loss for unsupervised reconstruction training to obtain a pre-trained model for high-quality data reconstruction.

[0019] S32. Input the blocks containing atomic regions in the low-quality data set into a U-Net with an encoder-decoder structure, perform unsupervised reconstruction training using mean square error loss, and obtain a pre-trained model for low-quality data reconstruction.

[0020] Optionally, S4 includes:

[0021] S41 , cropping the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks.

[0022] S42. Perform feature extraction on the high-quality image block using a pre-trained model reconstructed using high-quality data to obtain high-quality image block features, and use the high-quality image block features as a reference distribution.

[0023] S43. Perform feature extraction on the low-quality image block using a pre-trained model reconstructed using low-quality data to obtain features of the low-quality image block.

[0024] S44. Freeze the high-quality encoder and use the low-quality encoder as a trainable model. Project the high-quality image block features and the low-quality image block features into a common latent space. Use a joint loss function to constrain the network training so that the low-quality image block features gradually approach the high-quality image block features to obtain a trained low-quality encoder.

[0025] Optionally, the joint loss function is as shown in the following formula (1):

[0026] (1)

[0027] in,

[0028] (2)

[0029] (3)

[0030] Where, represents the joint loss function, express The weight coefficient, represents the KL divergence loss function, express The weight coefficient, represents the mean square error loss function, Indicates the number of channels, represents the variance of the output feature of the encoder to be optimized, represents the variance of the frozen high-quality encoder output features, represents the mean value of the output feature of the encoder to be optimized, represents the mean of the frozen high-quality encoder output features, represents the total number of combinations of all channels and spatial positions, 、 Indicates the first Features.

[0031] Optionally, S5 includes:

[0032] Obtain a low-quality transmission electron microscope image to be denoised, input it into a trained low-quality encoder to obtain output features, and reconstruct the output features according to the high-quality decoder to obtain the denoised image.

[0033] On the other hand, a transmission electron microscope image denoising device based on block feature alignment is provided. The device is applied to a transmission electron microscope image denoising method based on block feature alignment. The device comprises:

[0034] The acquisition module is used to acquire an atomic microscopic video stream through a transmission electron microscope, and to screen images in the atomic microscopic video stream to obtain unpaired high-quality data sets and low-quality data sets.

[0035] The clustering module is used to cluster the high-quality dataset and the low-quality dataset using multiple clustering methods. Based on the clustering results and the voting mechanism, the blocks containing atomic regions in the high-quality dataset and the blocks containing atomic regions in the low-quality dataset are obtained.

[0036] The reconstruction module is used to perform unsupervised reconstruction training on the autoencoder based on the blocks containing atomic regions in the high-quality data set and the blocks containing atomic regions in the low-quality data set, respectively, to obtain a pre-trained model for high-quality data reconstruction and a pre-trained model for low-quality data reconstruction.

[0037] Among them, the pre-training model for high-quality data reconstruction includes a high-quality encoder and a high-quality decoder, and the pre-training model for low-quality data reconstruction includes a low-quality encoder and a low-quality decoder.

[0038] The training module is used to obtain high-quality image block features based on a high-quality dataset and a pre-trained model reconstructed from the high-quality data, and to obtain low-quality image block features based on a low-quality dataset and a pre-trained model reconstructed from the low-quality data. The high-quality image block features and the low-quality image block features are projected onto a common latent space, and a joint loss function is used as a constraint to gradually allow the low-quality image block features to approach the high-quality image block features, thereby obtaining a trained low-quality encoder.

[0039] The output module is used to denoise low-quality images based on the trained low-quality encoder and high-quality decoder.

[0040] Optionally, the clustering module is further configured to:

[0041] S21. Crop the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks.

[0042] S22. Cluster the high-quality image blocks using multiple clustering methods to obtain multiple clustering results of the high-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the high-quality data set.

[0043] S23. Cluster the low-quality image blocks using multiple clustering methods to obtain multiple clustering results of the low-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the low-quality data set.

[0044] Optionally, the multiple clustering methods include a clustering method based on grayscale features of an image, a clustering method based on statistical features of an image average grayscale value and grayscale standard deviation, and a clustering method based on image frequency domain and structural statistical features.

[0045] Optionally, the reconstruction module is further configured to:

[0046] S31. Input the blocks containing atomic regions in the high-quality data set into the U-Net with an encoder-decoder structure, and use the mean square error loss for unsupervised reconstruction training to obtain a pre-trained model for high-quality data reconstruction.

[0047] S32. Input the blocks containing atomic regions in the low-quality data set into a U-Net with an encoder-decoder structure, perform unsupervised reconstruction training using mean square error loss, and obtain a pre-trained model for low-quality data reconstruction.

[0048] Optionally, the training module is further configured to:

[0049] S41 , cropping the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks.

[0050] S42. Perform feature extraction on the high-quality image block using a pre-trained model reconstructed using high-quality data to obtain high-quality image block features, and use the high-quality image block features as a reference distribution.

[0051] S43. Perform feature extraction on the low-quality image block using a pre-trained model reconstructed using low-quality data to obtain features of the low-quality image block.

[0052] S44. Freeze the high-quality encoder and use the low-quality encoder as a trainable model. Project the high-quality image block features and the low-quality image block features into a common latent space. Use a joint loss function to constrain the network training so that the low-quality image block features gradually approach the high-quality image block features to obtain a trained low-quality encoder.

[0053] Optionally, the joint loss function is as shown in the following formula (1):

[0054] (1)

[0055] in,

[0056] (2)

[0057] (3)

[0058] Where, represents the joint loss function, express The weight coefficient, represents the KL divergence loss function, express The weight coefficient, represents the mean square error loss function, Indicates the number of channels, represents the variance of the output feature of the encoder to be optimized, represents the variance of the frozen high-quality encoder output features, represents the mean value of the output feature of the encoder to be optimized, represents the mean of the frozen high-quality encoder output features, represents the total number of combinations of all channels and spatial positions, 、 Indicates the first Features.

[0059] Optionally, the output module is further configured to:

[0060] Obtain a low-quality transmission electron microscope image to be denoised, input it into a trained low-quality encoder to obtain output features, and reconstruct the output features according to the high-quality decoder to obtain the denoised image.

[0061] On the other hand, a transmission electron microscope image denoising device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned transmission electron microscope image denoising methods based on block feature alignment is implemented.

[0062] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned transmission electron microscope image denoising methods based on block feature alignment.

[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0064] In this method, a transmission electron microscope (CED) camera is used to record the atomic microscopy video stream during the experiment. The video is segmented by frame number, and high-quality and low-quality data are manually selected based on image resolution, atomic clarity, and lattice structure integrity. Direct clustering, statistical feature clustering, and Fourier feature-based clustering are used on the high-quality and low-quality data sets, respectively. A global model is used to predict the quality of blocks in a single image, and the results are fused through a voting mechanism. Unsupervised training is performed on the selected low-quality and high-quality blocks using an autoencoder with a U-Net architecture to reconstruct the low-quality and high-quality data sets, and the pre-trained models are saved separately to provide a basis for subsequent feature alignment. The extracted features are aligned in a common latent space to achieve denoising of the low-quality image. In this way, the method, which combines traditional clustering with deep learning, can effectively identify areas in the image containing key atomic structures and achieve image enhancement under complex noise through the feature alignment mechanism. Compared with existing rule-driven or single-channel reconstruction methods, the present invention is particularly suitable for dealing with the problem of signal-to-noise ratio degradation in atomic-scale dynamic images caused by high-resolution imaging, low dose or sample disturbance, providing a clearer and more reliable basis for atomic-level high-resolution microscopic image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1This is a flow chart of a transmission electron microscope image denoising method based on block feature alignment provided by an embodiment of the present invention;

[0067] Figure 2 Detailed flowchart of a transmission electron microscope image denoising method based on block feature alignment provided by an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram based on the block clustering principle provided by an embodiment of the present invention;

[0069] Figure 4 This is a schematic diagram of block clustering results provided by an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the block feature alignment principle provided by an embodiment of the present invention;

[0071] Figure 6 is a schematic diagram of denoising results provided by an embodiment of the present invention;

[0072] Figure 7 This is a block diagram of a transmission electron microscope image denoising device based on block feature alignment provided by an embodiment of the present invention;

[0073] Figure 8 The figure is a schematic structural diagram of a transmission electron microscope image denoising device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0075] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0076] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0077] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0078] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0079] The embodiment of the present invention provides a TEM image denoising method based on block feature alignment, which can be implemented by a TEM image denoising device, which can be a terminal or a server. Figure 1 The flowchart of the transmission electron microscope image denoising method based on block feature alignment is shown. The processing flow of the method may include the following steps:

[0080] S1. Acquire atomic microscopic video streams through a transmission electron microscope, filter the images in the atomic microscopic video streams, and obtain unpaired high-quality and low-quality datasets.

[0081] In a feasible embodiment, the TEM image denoising method based on block feature alignment of the present invention is as follows: Figure 2 As shown, experiments were conducted in a transmission electron microscope. Experimental phenomena were observed and recorded using high-resolution mode, atomic-level high-resolution microscopic video streams were obtained, and the obtained videos were segmented into frames. For the segmented videos, high-quality data sets with high resolution, clear atomic points, and neat arrangement were selected, and low-quality data sets with low resolution, blurred atomic points, and broken and discontinuous lattice fringes were selected.

[0082] Specifically, a high-resolution transmission electron microscope recording device equipped with a slow-scan CCD camera is used to continuously record the electronic signals generated in the experiment, and the microscopic optical signals received at the bottom of the fluorescent screen are converted into digital signals to observe and record experimental phenomena and obtain the experimental data set.

[0083] For the collected video data, the parameters, reference area, clarity, and magnification must be kept consistent during the experiment. The data must be preliminarily screened to obtain key frames that meet the above conditions before transmission electron microscopy image denoising based on block feature alignment can be performed.

[0084] Furthermore, for the filtered and Zhang data high-quality and low-quality datasets, Zhang data use Indicates that .

[0085] S2. Use multiple clustering methods to cluster the high-quality dataset and the low-quality dataset respectively, and obtain the blocks containing atomic regions in the high-quality dataset and the blocks containing atomic regions in the low-quality dataset based on the clustering results and the voting mechanism.

[0086] In one feasible implementation, K-means clustering is performed on two data sets based on original data, statistical features, and Fourier transform features, respectively. A global model is used to predict the quality of blocks in a single image, and the results are fused through a voting mechanism to select blocks containing the foreground of the atomic region.

[0087] Optionally, the above step S2 may include the following steps S21-S23:

[0088] S21. Crop the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks.

[0089] In a feasible implementation, each image in the dataset is cropped into blocks of the same size, specifically 32×32 blocks.

[0090] S22. Cluster the high-quality image blocks using multiple clustering methods to obtain multiple clustering results of the high-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the high-quality data set.

[0091] S23. Cluster the low-quality image blocks using multiple clustering methods to obtain multiple clustering results of the low-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the low-quality data set.

[0092] Optionally, the multiple clustering methods include a clustering method based on grayscale features of an image, a clustering method based on statistical features of an image average grayscale value and grayscale standard deviation, and a clustering method based on image frequency domain and structural statistical features.

[0093] In a feasible implementation, Figure 3 As shown, the direct pixel values, statistical features, and Fourier transform-based features of each block are extracted to classify it as belonging to the foreground or background area. A voting mechanism is used to perform statistics on the three K-means clustering methods, and the fused results are obtained to determine the blocks containing the atomic region. Three K-means clustering models are trained on the entire dataset.

[0094] Specifically, each image is cropped into For direct clustering, each block is represented as , its eigenvector is:

[0095] (1)

[0096] After flattening each image block into a one-dimensional vector, the features of all blocks are clustered, and all flattened vectors are clustered into two categories using K-means, which are then sorted by the average pixel value after clustering.

[0097] For statistical feature clustering, the grayscale mean of each block is:

[0098] (2)

[0099] The grayscale standard deviation is:

[0100] (3)

[0101] A two-dimensional feature vector is constructed using the grayscale mean and standard deviation. All blocks are divided into two categories using K-means clustering. The mean of all feature vectors in each category is calculated, and the two clustering results are sorted in ascending order based on the combined size of the mean.

[0102] Furthermore, for the clustering method based on Fourier frequency domain features, a two-dimensional fast Fourier transform is used to perform frequency domain analysis on each image block, and the six feature dimensions of the number of significant peaks in the Fourier spectrum, the average distance between the peak and the frequency domain center, the proportion of high-frequency energy, the spectrum information entropy, the periodic information, and the grayscale space variance are combined into a six-dimensional feature vector. Cluster classification is performed based on the above feature vectors, and the number of frequency domain peaks representing the periodic structural characteristics is selected as the evaluation basis. The average value of this feature of each cluster is sorted in ascending order to determine the image quality level to which it belongs. In particular, in this embodiment, the input image block Perform a two-dimensional Fourier transform:

[0103] (4)

[0104] Where, Represents the frequency component in the horizontal direction of the image frequency space, Represents the frequency component in the vertical direction of the image frequency space.

[0105] The obtained Fourier coefficients After amplitude calculation, the amplitude spectrum is formed , and further calculate the logarithmic magnitude spectrum:

[0106] (5)

[0107] and convert it into a one-dimensional signal Then, the significance threshold is set by local extreme value detection and peak significance determination. (like ) to ensure that: All indexes of The significant peak count is used to reflect the structural information of the image block and serves as part of the frequency domain features in the subsequent clustering process.

[0108] Furthermore, the three K-means clustering results are statistically analyzed, the global model is used to predict the quality of the blocks in a single image, and the three methods are fused using a voting mechanism to determine the blocks containing the atomic regions, e.g. Figure 4 shown.

[0109] Furthermore, each block has three binary classification labels:

[0110] (6)

[0111] The final fusion result is:

[0112] (7)

[0113] S3. Perform unsupervised reconstruction training on the autoencoder based on the blocks containing atomic regions in the high-quality dataset and the blocks containing atomic regions in the low-quality dataset, respectively, to obtain a pre-training model for high-quality data reconstruction and a pre-training model for low-quality data reconstruction.

[0114] Optionally, the above step S3 may include the following steps S31-S32:

[0115] S31. Input the blocks containing atomic regions in the high-quality data set into the U-Net with an encoder-decoder structure, and use the mean square error loss for unsupervised reconstruction training to obtain a pre-trained model for high-quality data reconstruction.

[0116] S32. Input the blocks containing atomic regions in the low-quality data set into a U-Net with an encoder-decoder structure, perform unsupervised reconstruction training using mean square error loss, and obtain a pre-trained model for low-quality data reconstruction.

[0117] In one feasible implementation, the blocks of the selected atomic regions are input into a U-Net with an encoder-decoder structure, and unsupervised reconstruction training is performed in the form of an autoencoder. The encoder is used only to extract the potential features of the image for the blocks containing the atomic regions, and the decoder restores and reconstructs the features, while the rest remain unchanged.

[0118] Furthermore, the mean squared error loss is used to measure the difference between each pixel before and after reconstruction, and the reconstructed blocks are pieced back into the original image one by one. The mean squared error of the reconstructed image is compared with the original image to evaluate the image restoration effect.

[0119] Save the pre-trained models reconstructed from low-quality and high-quality datasets respectively.

[0120] S4. Obtain high-quality image block features based on a high-quality dataset and a pre-trained model reconstructed from high-quality data. Obtain low-quality image block features based on a low-quality dataset and a pre-trained model reconstructed from low-quality data. Project the high-quality image block features and the low-quality image block features into a common latent space. Use a joint loss function as a constraint to gradually allow the low-quality image block features to approach the high-quality image block features, thereby obtaining a trained low-quality encoder.

[0121] In one feasible implementation, a pre-trained model is used to extract features from high-quality and low-quality datasets respectively, and the projections of the features of the two datasets are mapped into a common latent space. The network optimization model is trained to gradually align the low-quality features with the high-quality features, thus achieving low-quality image denoising. Figure 5 shown.

[0122] Optionally, the above step S4 may include the following steps S41-S44:

[0123] S41 , cropping the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks.

[0124] In one feasible implementation, the low-quality and high-quality datasets are cut into The blocks are then loaded into memory.

[0125] Specifically, the input size of the low-quality image is cropped to ,according to The image is divided into multiple blocks of size and fed into the neural network one by one as samples.

[0126] S42. Extract features of the high-quality image block using the pre-trained model reconstructed with high-quality data to obtain high-quality image block features. The high-quality image block features are used as a reference distribution, denoted as .

[0127] S43, extract features of the low-quality image block using the pre-trained model reconstructed with low-quality data, and obtain the low-quality image block features, which are recorded as .

[0128] In one feasible implementation, a pre-trained model is used to extract features of blocks containing atomic regions in low-quality and high-quality images, respectively. The pre-trained model is believed to be able to better extract potential features of the data.

[0129] S44. Freeze the high-quality encoder and use the low-quality encoder as a trainable model. Project the high-quality image block features and the low-quality image block features into a common latent space. Use a joint loss function to constrain the network training so that the low-quality image block features gradually approach the high-quality image block features to obtain a trained low-quality encoder.

[0130] In one feasible implementation, the encoder part of the pre-trained model of the high-quality dataset is frozen, and the feature distribution obtained in the high-quality encoder is used as a reference target.

[0131] The encoder part of the pre-trained model of the low-quality dataset is used as the trainable model, with the goal of aligning it with the output features of the high-quality encoder to improve the representation ability.

[0132] The feature distributions of the two are projected into a common latent space. In particular, the pre-trained model is believed to be better able to extract the latent features of the data.

[0133] Furthermore, in the alignment model stage, the U-Net-based network is optimized, and the Kullback-Leibler divergence (KL divergence) and mean square error loss of the Gaussian distribution are used as indicators of the feature alignment mechanism to achieve alignment reconstruction with high quality as a reference.

[0134] To achieve consistent alignment of feature distribution, this embodiment of the present invention introduces the Kullback-Leibier divergence (KL Divergence) based on the Gaussian assumption as a loss function on the encoder output feature map, which is defined as follows:

[0135] (8)

[0136] In the above formula, represents the variance of the output feature of the encoder to be optimized, represents the variance of the frozen high-quality encoder output features, represents the mean value of the output feature of the encoder to be optimized, represents the mean of the frozen high-quality encoder output features, is the number of channels. This loss measures the difference between the current network output feature distribution and the target distribution, thereby achieving distribution alignment in the feature space.

[0137] Furthermore, to ensure the proximity of the encoder output features in the value range and suppress local distortion, the embodiment of the present invention also introduces the mean squared error (MSE) loss as a supplementary indicator, which is defined as follows:

[0138] (9)

[0139] in and Represents the first Features, Represents the total number of combinations of all channels and spatial positions.

[0140] The example of the present invention ultimately uses a weighted combination of KL divergence and mean square error as the joint optimization goal to simultaneously guide feature distribution alignment and feature value approximation. The overall optimization objective function is as follows:

[0141] (10)

[0142] Furthermore, the decoder of the high-quality block is used to reconstruct the features of the low-quality encoder after feature alignment to obtain the denoised image.

[0143] This method combines traditional clustering with deep learning to effectively identify regions within an image containing key atomic structures and achieves image enhancement in the presence of complex noise through a feature alignment mechanism. Compared to existing rule-driven or single-channel reconstruction methods, this method is particularly suitable for addressing the degradation of signal-to-noise ratio in atomic-scale dynamic images caused by high-resolution imaging, low dose, or sample perturbations, providing a clearer and more reliable foundation for atomic-level high-resolution microscopic image analysis.

[0144] S5. Denoise the low-quality image based on the trained low-quality encoder and high-quality decoder.

[0145] Specifically, a low-quality transmission electron microscope image to be denoised is obtained, input into a trained low-quality encoder to obtain output features, and the output features are reconstructed according to a high-quality decoder to obtain a denoised image.

[0146] In an embodiment of the present invention, a transmission electron microscope (CED) camera is used to record the atomic microscopy video stream during the experiment. The video is segmented by frame number, and high-quality and low-quality data are manually selected based on image resolution, atomic clarity, and lattice structure integrity. Direct clustering, statistical feature clustering, and Fourier feature-based clustering methods are used on high-quality and low-quality data sets, respectively. A global model is used to predict the quality of blocks in a single image, and the results are fused through a voting mechanism. Unsupervised training is performed on the selected low-quality and high-quality blocks using an autoencoder with a U-Net architecture to reconstruct the low-quality and high-quality data sets, and the pre-trained models are saved separately to provide a basis for subsequent feature alignment. The extracted features are aligned in a common latent space to achieve denoising of the low-quality image. In this way, the method of combining traditional clustering with deep learning can effectively identify areas containing key atomic structures in the image, and achieve image enhancement under complex noise through the feature alignment mechanism. Compared with existing rule-driven or single-channel reconstruction methods, the present invention is particularly suitable for dealing with the problem of signal-to-noise ratio degradation in atomic-scale dynamic images caused by high-resolution imaging, low dose or sample disturbance, providing a clearer and more reliable basis for atomic-level high-resolution microscopic image analysis.

[0147] Figure 7This is a block diagram of a transmission electron microscope image denoising device based on block feature alignment according to an exemplary embodiment. The device is used in a transmission electron microscope image denoising method based on block feature alignment. Figure 7 The device includes an acquisition module 310, a clustering module 320, a reconstruction module 330, a training module 340, and an output module 350.

[0148] The acquisition module 310 is configured to acquire an atomic microscopic video stream through a transmission electron microscope, and screen images in the atomic microscopic video stream to obtain unpaired high-quality data sets and low-quality data sets.

[0149] The clustering module 320 is used to cluster the high-quality dataset and the low-quality dataset using multiple clustering methods, and obtain blocks containing atomic regions in the high-quality dataset and blocks containing atomic regions in the low-quality dataset based on the clustering results and a voting mechanism.

[0150] The reconstruction module 330 is used to perform unsupervised reconstruction training on the autoencoder based on the blocks containing atomic regions in the high-quality data set and the blocks containing atomic regions in the low-quality data set, respectively, to obtain a pre-trained model for high-quality data reconstruction and a pre-trained model for low-quality data reconstruction.

[0151] Among them, the pre-training model for high-quality data reconstruction includes a high-quality encoder and a high-quality decoder, and the pre-training model for low-quality data reconstruction includes a low-quality encoder and a low-quality decoder.

[0152] The training module 340 is used to obtain high-quality image block features based on a high-quality data set and a pre-trained model reconstructed from the high-quality data, and to obtain low-quality image block features based on a low-quality data set and a pre-trained model reconstructed from the low-quality data, project the high-quality image block features and the low-quality image block features into a common latent space, and use a joint loss function as a constraint to gradually allow the low-quality image block features to approach the high-quality image block features, thereby obtaining a trained low-quality encoder.

[0153] The output module 350 is used to implement low-quality image denoising based on the trained low-quality encoder and high-quality decoder.

[0154] In an embodiment of the present invention, a transmission electron microscope (CED) camera is used to record the atomic microscopy video stream during the experiment. The video is segmented by frame number, and high-quality and low-quality data are manually selected based on image resolution, atomic clarity, and lattice structure integrity. Direct clustering, statistical feature clustering, and Fourier feature-based clustering methods are used on high-quality and low-quality data sets, respectively. A global model is used to predict the quality of blocks in a single image, and the results are fused through a voting mechanism. Unsupervised training is performed on the selected low-quality and high-quality blocks using an autoencoder with a U-Net architecture to reconstruct the low-quality and high-quality data sets, and the pre-trained models are saved separately to provide a basis for subsequent feature alignment. The extracted features are aligned in a common latent space to achieve denoising of the low-quality image. In this way, the method of combining traditional clustering with deep learning can effectively identify areas containing key atomic structures in the image, and achieve image enhancement under complex noise through the feature alignment mechanism. Compared with existing rule-driven or single-channel reconstruction methods, the present invention is particularly suitable for dealing with the problem of signal-to-noise ratio degradation in atomic-scale dynamic images caused by high-resolution imaging, low dose or sample disturbance, providing a clearer and more reliable basis for atomic-level high-resolution microscopic image analysis.

[0155] Figure 8 FIG. 1 is a schematic structural diagram of a transmission electron microscope image denoising device provided by an embodiment of the present invention. Figure 8 As shown, the transmission electron microscope image denoising device may include the above Figure 7 The TEM image denoising device based on block feature alignment is shown. Optionally, the TEM image denoising device 410 may include a first processor 2001 .

[0156] Optionally, the transmission electron microscope image denoising device 410 may further include a memory 2002 and a transceiver 2003 .

[0157] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0158] The following combination Figure 8 The components of the transmission electron microscope image denoising device 410 are described in detail:

[0159] The first processor 2001 is the control center of the transmission electron microscope image denoising device 410 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0160] Optionally, the first processor 2001 may execute various functions of the transmission electron microscope image denoising device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0161] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0162] In a specific implementation, as an embodiment, the transmission electron microscope image denoising device 410 may also include multiple processors, such as Figure 8 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0163] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0164] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be accessed through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0165] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0166] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0167] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the transmission electron microscope image denoising device 410 through an interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0168] It should be noted that Figure 8 The structure of the transmission electron microscope image denoising device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0169] In addition, the technical effects of the transmission electron microscope image denoising device 410 can refer to the technical effects of the transmission electron microscope image denoising method based on block feature alignment described in the above method embodiment, and will not be repeated here.

[0170] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0171] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0172] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0173] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0174] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0175] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0176] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0177] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0178] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0182] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A transmission electron microscope image denoising method based on block feature alignment, characterized in that: The method comprises: S1. Acquire atomic microscopic video streams through a transmission electron microscope, filter the images in the atomic microscopic video streams, and obtain unpaired high-quality and low-quality datasets. S2. Clustering the high-quality dataset and the low-quality dataset using multiple clustering methods, and obtaining blocks containing atomic regions in the high-quality dataset and blocks containing atomic regions in the low-quality dataset based on the clustering results and a voting mechanism; S3. Perform unsupervised reconstruction training on the autoencoder based on the blocks containing the atomic regions in the high-quality dataset and the blocks containing the atomic regions in the low-quality dataset, respectively, to obtain a pre-trained model for high-quality data reconstruction and a pre-trained model for low-quality data reconstruction; The pre-trained model for high-quality data reconstruction includes a high-quality encoder and a high-quality decoder, while the pre-trained model for low-quality data reconstruction includes a low-quality encoder and a low-quality decoder. S4. Obtain high-quality image block features based on the high-quality dataset and the pre-trained model reconstructed from the high-quality data, obtain low-quality image block features based on the low-quality dataset and the pre-trained model reconstructed from the low-quality data, project the high-quality image block features and the low-quality image block features into a common latent space, and use a joint loss function as a constraint to gradually make the low-quality image block features approach the high-quality image block features, thereby obtaining a trained low-quality encoder. S5. Denoise the low-quality image based on the trained low-quality encoder and high-quality decoder.

2. The transmission electron microscope image denoising method based on block feature alignment according to claim 1, characterized in that: Said S2 comprises: S21, cropping the images in the high-quality dataset and the low-quality dataset respectively to obtain high-quality image blocks and low-quality image blocks; S22. Clustering the high-quality image blocks using multiple clustering methods to obtain multiple clustering results of the high-quality image blocks, and fusing the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the high-quality data set; S23. Cluster the low-quality image blocks using multiple clustering methods to obtain multiple clustering results of the low-quality image blocks, and fuse the multiple clustering results using a voting mechanism to obtain blocks containing atomic regions in the low-quality data set.

3. The TEM image denoising method based on block feature alignment according to claim 2, characterized in that: The multiple clustering methods include a clustering method based on grayscale features of an image, a clustering method based on statistical features of an average grayscale value and grayscale standard deviation of an image, and a clustering method based on frequency domain and structural statistical features of an image.

4. The transmission electron microscope image denoising method based on block feature alignment according to claim 1, characterized in that: The step S3 performs unsupervised reconstruction training on the autoencoder based on the blocks containing atomic regions in the high-quality data set and the blocks containing atomic regions in the low-quality data set, respectively, to obtain a pre-trained model for high-quality data reconstruction and a pre-trained model for low-quality data reconstruction, including: S31. Input the block containing the atomic region in the high-quality data set into a U-Net with an encoder-decoder structure, and perform unsupervised reconstruction training using the mean square error loss to obtain a pre-trained model for high-quality data reconstruction; S32. Input the blocks containing atomic regions in the low-quality data set into a U-Net with an encoder-decoder structure, perform unsupervised reconstruction training using mean square error loss, and obtain a pre-trained model for low-quality data reconstruction.

5. The transmission electron microscope image denoising method based on block feature alignment according to claim 1, characterized in that: Said S4 comprises: S41, cropping the images in the high-quality data set and the low-quality data set respectively to obtain high-quality image blocks and low-quality image blocks; S42, performing feature extraction on the high-quality image block using a pre-trained model reconstructed using high-quality data to obtain high-quality image block features, and using the high-quality image block features as a reference distribution; S43, extracting features from the low-quality image block using a pre-trained model reconstructed using low-quality data to obtain features of the low-quality image block; S44. Freeze the high-quality encoder, use the low-quality encoder as a trainable model, project the high-quality image block features and the low-quality image block features into a common latent space, use a joint loss function to constrain the training of the network, and let the low-quality image block features gradually approach the high-quality image block features to obtain a trained low-quality encoder.

6. The transmission electron microscope image denoising method based on block feature alignment according to claim 5, characterized in that: The joint loss function is shown in the following formula (1): （1） in, （2） （3） Where, represents the joint loss function, express The weight coefficient, represents the KL divergence loss function, express The weight coefficient, represents the mean square error loss function, Indicates the number of channels, represents the variance of the output feature of the encoder to be optimized, represents the variance of the frozen high-quality encoder output features, represents the mean value of the output feature of the encoder to be optimized, represents the mean of the frozen high-quality encoder output features, represents the total number of combinations of all channels and spatial positions, 、 Indicates the first Features.

7. The transmission electron microscope image denoising method based on block feature alignment according to claim 1, characterized in that: Said S5 comprises: Obtain a low-quality transmission electron microscope image to be denoised, input it into a trained low-quality encoder to obtain output features, and reconstruct the output features according to the high-quality decoder to obtain the denoised image.

8. A transmission electron microscope image denoising device based on block feature alignment, wherein the transmission electron microscope image denoising device based on block feature alignment is used to implement the transmission electron microscope image denoising method based on block feature alignment according to any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module is used to acquire an atomic microscopic video stream through a transmission electron microscope, and to filter images in the atomic microscopic video stream to obtain unpaired high-quality and low-quality data sets; a clustering module, configured to cluster the high-quality dataset and the low-quality dataset respectively using a plurality of clustering methods, and obtain blocks containing atomic regions in the high-quality dataset and blocks containing atomic regions in the low-quality dataset based on the clustering results and a voting mechanism; A reconstruction module is used to perform unsupervised reconstruction training on the autoencoder based on the blocks containing atomic regions in the high-quality dataset and the blocks containing atomic regions in the low-quality dataset, respectively, to obtain a pre-trained model for high-quality data reconstruction and a pre-trained model for low-quality data reconstruction; The pre-trained model for high-quality data reconstruction includes a high-quality encoder and a high-quality decoder, while the pre-trained model for low-quality data reconstruction includes a low-quality encoder and a low-quality decoder. a training module configured to obtain high-quality image block features based on a high-quality dataset and a pre-trained model reconstructed from the high-quality data, and to obtain low-quality image block features based on a low-quality dataset and a pre-trained model reconstructed from the low-quality data, project the high-quality image block features and the low-quality image block features into a common latent space, and use a joint loss function as a constraint to gradually bring the low-quality image block features closer to the high-quality image block features, thereby obtaining a trained low-quality encoder; The output module is used to denoise low-quality images based on the trained low-quality encoder and high-quality decoder.

9. A transmission electron microscope image denoising device, characterized in that: The transmission electron microscope image denoising device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images