Method and device for registering under-focus sequence images of high-noise transmission electron microscope

Abstract

The invention discloses a high-noise transmission electron microscope under-focus sequence image registration method and device, and the method comprises the steps: employing the synergistic effect of frequency domain modulation and information entropy optimization, specifically, extracting the amplitude information of a structure signal through Fourier transform, combining with a multi-type frequency domain modulation function, and obtaining an under-focus sequence image of a high-noise transmission electron microscope. Noise spectrum energy suppression and effective signal frequency band enhancement are realized in a frequency domain; mutual information calculation is carried out on the basis of the image after frequency domain modulation, and the defect that traditional intensity matching is sensitive to noise is overcome by utilizing non-linear relevance of frequency domain signal statistical distribution, so that robust registration is driven.

Description

Technical Field

[0001] The present invention relates to the field of electron microscope image processing, and in particular to a registration method and device for high-noise transmission electron microscope defocus series images based on frequency-domain modulation information entropy, which is applicable to the automatic alignment of multiple frames of defocus images obtained by devices such as transmission electron microscopes (TEM) and scanning transmission electron microscopes (STEM) under low-dose conditions, and is widely used in high-resolution electron microscopy techniques such as low-dose imaging, exit wave reconstruction, and electron tomography. Background Art

[0002] In transmission electron microscope (TEM) and scanning transmission electron microscope (STEM) imaging, low-dose imaging technology has become a key means for studying electron beam-sensitive materials (such as biological samples, two-dimensional materials, metal-organic frameworks, etc.) by reducing electron beam irradiation damage; however, the low-dose condition leads to a significant reduction in the signal-to-noise ratio (SNR) of the image, severely limiting the accuracy of traditional registration methods that rely on gradient or feature point matching. At the same time, for applications such as exit wave reconstruction and electron tomography, it is necessary to collect defocus series images (Defocus Series Images) containing different defocus amounts, and achieve multimodal data fusion through high-precision registration. However, due to the dynamic change of the defocus amount in such defocus series images, the registration problem faces two challenges: First, the defocus amount difference will cause non-linear distortion of the apparent features of the same structure (such as edge sharpness, phase contrast drift), destroying the intensity or structural consistency assumptions relied on by traditional algorithms; Second, the defocus-induced diffraction effects further introduce local phase and amplitude distortion, resulting in the failure of the method based on frequency-domain phase matching.

[0003] In the registration task of traditional electron microscope images, common image registration methods include CrossCorrelation (CC), Geometric Phase Correction (GPC), bUnwarpJ algorithm, etc., which can achieve good results in general low-noise or optical microscope image processing, but when applied to high-noise defocus series images, their stability and accuracy both drop significantly. The main reason is that these methods rely on the global features or frequency-domain information of the image during the registration process, and the noise distribution of defocus series images is complex and the image contrast changes significantly at different focal lengths, making it difficult for traditional methods to extract stable matching features. Therefore, under low-dose conditions, the coupling interference formed by the decrease in signal-to-noise ratio and the non-linear distortion of the apparent features of the same structure caused by the defocus amount difference makes it difficult for existing registration techniques to balance robustness and sub-pixel accuracy, severely restricting the application effect of high-resolution reconstruction. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned drawbacks of the prior art and provides a method and apparatus for registering defocus series images of a high-noise transmission electron microscope based on frequency domain modulated information entropy (Frequency Domain Modulated and Information Entropy based Image Registration, FDMIE-Reg), which is used for reconstructing the outgoing wave of radiation-sensitive materials, electron tomography, and other electron microscopy techniques that require high-precision image registration to support key operations such as phase recovery.

[0005] Aiming at the problems of insufficient accuracy and stability of traditional algorithms such as cross-correlation (CC), geometric phase correlation (GPC), and bUnwarpJ in registering high-noise electron microscopy defocus series images, the present invention proposes a method for registering defocus series images of a high-noise transmission electron microscope based on frequency domain modulated information entropy. By enhancing the main structural frequency information in the frequency domain to suppress noise interference, and at the same time combining a statistical correlation optimization strategy based on information entropy to resist the gray drift caused by changes in defocus amount, the registration accuracy and robustness of high-noise TEM data are significantly improved. This method is particularly suitable for electron microscopy techniques such as outgoing wave reconstruction and electron tomography that rely on high-precision image registration, providing a reliable data basis for phase recovery of radiation-sensitive materials.

[0006] The first aspect of the present invention relates to a method for registering defocus series images of a high-noise transmission electron microscope, comprising the following steps:

[0007] (1) Obtain high-noise defocus series electron microscope images: Under low-dose shooting conditions, collect a set of transmission electron microscope images with equal defocus amounts to form a complete sequence of images from defocus to in-focus and then to over-focus; or generate simulation data using simulation software based on experimental conditions.

[0008] (2) Perform Fourier transform on the image sequence: Perform Fourier transform on each frame of the collected images, extract the amplitude components of each frame, and accumulate them to form a frequency domain amplitude pattern to enhance the periodic structure signal while suppressing random noise.

[0009] (3) Enhance the frequency domain signal: Select a modulation function to modulate the frequency domain amplitude components while retaining the original phase information.

[0010] (4) Perform inverse Fourier transform: Perform inverse Fourier transform on the modulated frequency domain signal to obtain the denoised image sequence.

[0011] (5) Registration optimization based on information entropy: Calculate the mutual information of the registered images, where: For each of the two images, count the occurrence frequencies of each pixel intensity value. Based on the frequency ratios, calculate the marginal entropy of each image, and count the joint occurrence frequency distribution of the pixel intensity pairs at corresponding positions in the two images. Calculate the joint information entropy through the joint frequency ratio; Subtract the joint entropy from the sum of the marginal entropies of the two images to obtain the mutual information.

[0012] (6) Normalize the mutual information to reduce sensitivity to the size of the overlapping region.

[0013] (7) Multi-scale optimization: Use an optimization algorithm to optimize the mutual information MI value and transformation parameters to obtain the registration parameters that maximize the MI value.

[0014] (8) Correct the registration of the global defocused sequence images based on the MI calculation with iterative optimization.

[0015] (9) Registration result evaluation: For defocused sequence electron microscope images, use the root mean square error RMS to quantitatively evaluate the registration translation parameters and verify the registration accuracy. For defocused sequence electron microscope images collected by the actual low-dose technique, through

[0016] Among them, the images described in step (2) have n frames, forming an image stack, where the i-th frame of the image is represented as , where (x, y) corresponds to the projection position of the sample in the two-dimensional space during transmission electron microscope acquisition, x is the horizontal coordinate, y is the vertical coordinate, The value of reflects the electron signal intensity detected by the detector at that position; The frequency domain amplitude mode is represented as , where u and v represent the contributions of different frequency components in the spatial frequency, u corresponds to the frequency in the horizontal direction, v corresponds to the frequency in the vertical direction, The value of reflects the result of the Fourier transform, and its mathematical meaning is to map the spatial domain to the frequency domain. The Fourier transform is calculated using the fast Fourier transform FFT, and the formula is as follows:

[0017] (1)

[0018] In formula (1), the integral part represents the continuous Fourier transform of a single-frame image , specifically, by integrating to calculate the inner product of the i-th frame image and the corresponding complex exponential wave of the i-th frame image to quantify the contribution of each frequency component in the image, that is, calculate its complex value at the frequency point ; The absolute value part represents taking the amplitude spectrum of the Fourier transform, that is, the modulus of the Fourier transform, and the summation part represents the superposition of the spectra of multiple frames of images, and finally obtain .

[0019] Among them, the modulation functions described in step (3) include: Gaussian modulation function; Wiener modulation function; soft threshold modulation function; Gaussian high-pass modulation function; median modulation function;

[0020] The formula for enhancing the frequency-domain signal described in step (3) is:

[0021] (2)

[0022] Among them, represents the phase angle of the frequency signal. The physical meaning of the phase is to determine the spatial position of the periodic structure in the signal. In formula (2), comes from the original signal in the image, that is, it means the phase information remains unchanged; is the original phase information of the frequency point in the i-th frame of the image. It encodes the phase information into a complex form for facilitating the synthesis of the frequency-domain signal; is the modulation function, which modulates the frequency-domain amplitude component; represents the enhanced signal at the frequency point in the i-th frame of the image, including the adjusted amplitude and phase information, which is used for the subsequent inverse Fourier transform to reconstruct the enhanced spatial-domain signal;

[0023] The Gaussian modulation function described in step (3) is used to enhance the low-frequency signal and suppress the high-frequency noise. Its mathematical expression is as follows:

[0024] (3)

[0025] where σ is the standard deviation of the Gaussian kernel (controlling the degree of blurring).

[0026] Among them, the Wiener modulation function described in step (3) is used to minimize the mean square error (MSE) between the signal and the noise, and the noise power spectrum needs to be known. Its mathematical expression is as follows:

[0027] (4)

[0028] where is the power spectrum of the original signal, is the power spectrum of the noise.

[0029] Among them, the soft threshold modulation function described in step (3) is used to enhance the specific frequency signal and remove the background noise. Its mathematical expression is as follows:

[0030] (5)

[0031] where the specified threshold is defined as, .

[0032] Among them, the Gaussian high-pass modulation function described in step (3) is used to remove low-frequency artifacts and enhance high-frequency features, and its mathematical expression is as follows:

[0033] (6)

[0034] Where , represents the distance from the frequency point to the center of the spectrum (M / 2, N / 2), is the cut-off frequency (radius), which controls the low-frequency suppression range.

[0035] Among them, the median modulation function described in step (3) is used to eliminate outlier noise points, and its mathematical expression is as follows:

[0036] (7)

[0037] where k is the modulation window radius;

[0038] Among them, the inverse Fourier transform described in step (4), its mathematical expression is as follows:

[0039] (8)

[0040] Where is the spatial domain signal restored from the frequency domain signal through the inverse Fourier transform, containing enhanced or corrected information, that is, the i-th frame in the denoised image sequence, represents converting each frequency component in the frequency domain signal into a plane wave in the spatial domain; the integral part in formula (8) represents recombining the separated frequency components in the frequency domain to restore the global and local features of the spatial domain image.

[0041] Among them, the edge entropy and described in step (5), its mathematical expression is as follows:

[0042] (9)

[0043] (10)

[0044] Where and represent the i-th and j-th frames of images in the total n-frame image stack after the inverse Fourier transform, calculating the edge entropy and , which needs to be based on the input , the pixel intensity values corresponding to the two pictures , The occurrence frequency of and , is the intensity value of a certain pixel in the image , and is the intensity value of the pixel at the corresponding position in the image . The sum of the number of all non - zero pixel intensity values of the two images is respectively denoted as and , and its mathematical expression is as follows:

[0045] (11)

[0046] (12)

[0047] Among them, is the total number of all non - zero pixel values in the image , is the total number of all non - zero pixel values in the image . The total number combined with the occurrence frequency is used to represent the occurrence probability of a certain intensity in a single image;

[0048] Among them, the joint information entropy described in step (5) , and its mathematical expression is as follows:

[0049] (13)

[0050] Calculate the joint information entropy of the two images , which requires the occurrence frequency of the appearance of the pixel intensity value pair corresponding to the corresponding pixel positions of the two images . and represent the number of occurrences of the pixel intensity values at the same position in the two images being respectively, and the sum of the number of all non - zero pixel intensity value pairs of the two images is denoted as

[0051] (14)

[0052] Among them, represents the total number of pairs of non - zero pixels at the same position in the two images. The total number of pairs combined with the occurrence frequency of pixel pairs is used to express the occurrence probability of the corresponding pixel intensity value pair in the two images;

[0053] Among them, the mutual information described in step (5) , and its mathematical expression is as follows:

[0054] (15)

[0055] Among them, the logarithmic term is expressed as the ratio of the joint probability to the independent probability, which is used to measure the statistical correlation between two images. The larger the value, the more dependent the pixel intensity distributions of the two images are; the summation part represents the comprehensive statistical relationship of all pixel pairs to obtain the global mutual information value; the mutual information is then used to quantify the statistical dependence between two images. The larger the value, the more correlated the pixel intensity distributions of the two images are, that is, the more similar or better registered they are.

[0056] Among them, the mutual information calculation described in step (6) uses the normalized mutual information (NMI) calculation to reduce the sensitivity of registration to the size of the image overlapping area. Its mathematical expression is as follows:

[0057] (16)

[0058] Normalized mutual information is used to normalize the mutual information to the fixed range [0,1] for facilitating the comparison of the similarity of different image pairs.

[0059] Among them, the optimization algorithms described in step (7) include differential evolution, gradient descent, and simulated annealing;

[0060] Among them, the optimization algorithms described in step (7) are used to calculate the optimal registration parameters , and its mathematical expression is as follows:

[0061] (17)

[0062] Among them, represents the image after translating the image, and represent the offsets in the horizontal and vertical directions; represents finding the set of translations that maximizes the mutual information value among all possible translation combinations, so as to obtain the optimal registration parameters , that is, the horizontal and vertical translation amounts required when the image registration is the best.

[0063] Among them, the MI calculation based on iterative optimization described in step (8) corrects the global defocused sequence image registration, and its mathematical expression is as follows:

[0064] (18)

[0065] Among them, m represents the number of image pairs used for registration in the optimization process, represents the best registration translation amount calculated for the i-th frame image relative to the k-th frame reference image, Represents the global correction displacement vector of the i-th frame image, that is, the total translation amount finally used to align this image with the global reference image, so as to improve the robustness of registration.

[0066] The second aspect of the present invention relates to a registration device for defocused sequence images of a high-noise transmission electron microscope, including a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the registration method for defocused sequence images of the high-noise transmission electron microscope of the present invention.

[0067] The third aspect of the present invention relates to a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the registration method for defocused sequence images of the high-noise transmission electron microscope of the present invention.

[0068] The working principle of the present invention is to utilize the synergistic effect of frequency domain modulation and information entropy optimization. Specifically, the amplitude information of the structural signal is extracted through Fourier transform, combined with multiple types of frequency domain modulation functions, to suppress the noise spectrum energy and enhance the effective signal frequency band in the frequency domain; then, mutual information calculation is performed based on the frequency domain modulated image, and the non-linear correlation of the statistical distribution of the frequency domain signal is utilized to overcome the defect that the traditional intensity matching is sensitive to noise, thereby driving robust registration.

[0069] The advantages of the present invention are:

[0070] The registration error of high-noise defocused sequence images is significantly reduced. Under the condition of high noise (SNR = -32.3dB), the registration success rate remains above 99%. Compared with traditional methods such as geometric phase correlation (GPC) and cross-correlation (CC), the error is reduced from 26% to 0.8%. In addition, the method of this invention eliminates the registration deviation caused by defocus difference by quantifying the statistical distribution similarity of the frequency domain modulated image, rather than relying on the intensity contrast related to defocus. The registration error rate within the drift amount of 2 - 100nm is less than 1%, showing strong robustness and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figures 1a - 1c is the drift amount before and after registration of the present invention distribution comparison diagram, where Figure 1a is the true drift amount curve, Figure 1b is the MI registration drift amount curve, Figure 1c is the registration drift amount curve of the present invention.

[0072] Figures 2a - 2d is the comparison diagram of electron microscope images before and after registration of the present invention, where Figure 2a is a single high-noise defocused image before registration, Figure 2b is the electron microscope image after CC registration, Figure 2cThe electron microscope image after registration for the present invention Figure 2d It is a superposition image of a group of defocus images with no displacement and high noise. In the small images below each image, the left image is an enlarged view of the unit structure, and the right image is an FFT image. The white arc in the FFT image indicates the resolution size.

[0073] Figure 3 It is the RMS contrast graph of the present invention, where BP-CC is the Bandpass-CC algorithm.

[0074] Figure 4a and Figure 4b It is a comparison graph of electron microscope images reconstructed after registration by different methods. Among them Figure 4a is the electron microscope image reconstructed after CC registration Figure 4b The electron microscope image reconstructed after registration of the present invention. In the small images below each image, the left image is an enlarged view of the unit structure, and the right image is an FFT image. The white circle in the FFT image indicates the resolution size. Detailed implementation mode

[0075] The present invention will be described in detail below with reference to the accompanying drawings and examples. The present invention improves the registration method for high-noise defocus sequence images, realizing precise registration of defocus sequence images collected by electron microscopy of radiation-sensitive materials at low doses. To verify the feasibility and reliability of the present invention, this example uses simulated data of the assumed sample MOF material MIL101 to perform traditional algorithms such as geometric phase correlation, cross-correlation, and bUnwarpJ, as well as the improved registration method of the present invention, and compares the results after registration. The main reason for using simulated data is that simulated data can exclude interference from external uncertain factors such as astigmatism, thereby proving the effectiveness of the registration method for high-noise defocus sequence images of the present invention.

[0076] Example 1

[0077] Referring to Figures 1a - 3 , the registration method for high-noise transmission electron microscope defocus sequence images provided in this example includes the following steps:

[0078] 1. First, simulate the projection type of MIL101 material imaging under the electron microscope to generate a data set with a size of 512×512×21 pixels. The data simulation conditions are: acceleration voltage 300 kv, the defocus amount of each image is set to 2 nm, 5 nm, 10 nm, 20 nm, 50 nm, 100 nm for each group, and each group contains a complete series of images from defocus to positive focus and then to overfocus. Add Poisson noise with an intensity of 0.5 to each simulated image, and its signal-to-noise ratio SNR is -32.3 to simulate the noise characteristics of low-dose imaging (as shown in Figure 2a and 2d ), and randomly introduce translational drift to simulate the motion error in actual TEM imaging (as shown in Figure 1aas shown).

[0079] As used in the present invention, the low dose specifically refers to the electron dose required to maintain the inherent structure of extremely sensitive materials (such as MOF, hybrid perovskite, and supramolecular crystals), which is usually much lower than 100 e - / Å 2 , in this example, SNR - 32.3, and the total electron dose of its corresponding image is 3.8 e - / Å 2 , negative indicates that the signal is less than the noise, that is, a high - noise feature.

[0080] 2. Perform Fourier transform on this series of images: For an image stack containing n frames , let the input original electron microscopy image be , perform Fourier transform (FT) on each frame in the image stack, extract the amplitude component of each frame (i.e., the intensity of the frequency - domain signal), and sum up the amplitude components of all frames to form an amplitude pattern. The formula is as follows:

[0081] (1)

[0082] By superimposing multiple frames, enhance the frequency - domain signal (such as Bragg reflection peaks) of periodic structures (such as crystal lattices), while suppressing random noise.

[0083] 3. Enhance the frequency - domain signal: Select a modulation function , enhance the frequency - domain signal of the Fourier - transform amplitude of each frame, while preserving the original phase :

[0084] (2)

[0085] In this example, a soft - threshold modulation function is selected. Its function is to accurately locate pixels in the amplitude pattern whose intensity is greater than a specified threshold to generate an enhanced frequency - domain information pattern. Its modulation function is:

[0086] (5)

[0087] where the specified threshold is defined as .

[0088] 4. Perform inverse Fourier transform on the modified frequency - domain signal. The formula is as follows:

[0089] (8)

[0090] Obtain the image with noise reduction.

[0091] 5. Entropy - based Optimization of Registration: The processed images are optimized using mutual information based on information entropy. The goal is to find the optimal transformation parameters (translation, rotation, scaling, etc.) such that the mutual information of the registered images reaches the maximum. First, assume the input , The corresponding pixel intensity values of two pictures , The occurrence frequencies are and . The sums of the number of all non - zero pixel intensity values of the two images are denoted as and respectively. The formulas are as follows:

[0092] (11)

[0093] (12)

[0094] Meanwhile, assume that the occurrence frequency of the pixel intensity values at the corresponding positions of the two pictures is , and the sum of the number of all non - zero pixel intensity value pairs of the two images is denoted as . The formula is as follows:

[0095] (14)

[0096] Then, calculate the marginal entropy of , the two images, denoted as and respectively. The formulas are as follows:

[0097] (9)

[0098] (10)

[0099] Meanwhile, the joint entropy of the two images is . The formula is as follows:

[0100] (13)

[0101] Then the formula for mutual information is:

[0102] (15)

[0103] 6. Normalized Mutual Information: Solve the sensitivity of MI to the size of the overlapping region and improve robustness. The formula is as follows:

[0104] (16)

[0105] 7. Multi-scale optimization: Use an optimization algorithm to optimize the MI value and transformation parameters, with the goal of finding the transformation parameters that maximize the mutual information. The algorithm used in this example is the differential evolution algorithm. The formula is as follows:

[0106] (17)

[0107] And visualize the displacement parameters as a two-dimensional coordinate curve, as Figure 1c shown. Another displacement parameter curve graph registered using the CC method is as Figure 1b shown.

[0108] 8. Correction of global defocus sequence image registration based on iterative optimization of MI calculation:

[0109] (18)

[0110] The registered image is as Figure 2c shown. Another result registered using the CC method is as Figure 2b shown.

[0111] 9. Registration result evaluation: Based on the root mean square error (RMS) quantization evaluation criterion, error analysis is performed on the registration transformation parameters , . The experiment applies different registration methods to six groups of data respectively, and compares the result errors after registration. The results are as Figure 3 shown. The experimental results show that this method significantly reduces the registration error of low-dose defocus sequence images. Compared with the average error of 21% in the traditional method, the average registration error of this invention method is reduced to 0.8%, which is much lower than the calibration errors of methods such as geometric phase correlation (GPC) and cross-correlation (CC). In addition, under the condition of high noise (SNR = -32dB), this method still maintains a registration success rate of more than 99%, as Figure 2c shown, demonstrating strong robustness and adaptability.

[0112] Example 2

[0113] Referring to Figure 4a and Figure 4b , the registration method for defocus sequence images of a high-noise transmission electron microscope provided in this example includes the following steps:

[0114] 1. First, collect a group of defocus sequence images in a transmission electron microscope. The acceleration voltage is 300 kv, the image size is 512×512 pixels, the collected focal lengths are ±200 nm, and one image is collected every 20 nm of focal length interval. The total dose is 5 e - / Å2 .

[0115] As used in the present invention, the low dose specifically refers to the electron dose required to maintain the inherent structure of extremely sensitive materials (such as MOF, hybrid perovskite, and supramolecular crystals), which is usually much lower than 100 e - / Å 2 . In this example, the total electron dose is 5 e - / Å 2 , and its corresponding SNR is -14.1. A negative indicates that the signal is less than the noise, which is a high-noise feature.

[0116] 2. Perform Fourier transform on this series of images: For a stack of images containing n frames , let the input original electron microscopy image be . Perform Fourier transform (FT) on each frame in the image stack, extract the amplitude component of each frame (i.e., the intensity of the frequency-domain signal), and sum up the amplitude components of all frames to form an amplitude pattern. The formula is as follows:

[0117] (1)

[0118] By superimposing multiple frames, the frequency-domain signal (such as Bragg reflection peaks) of periodic structures (such as crystal lattices) is enhanced, while random noise is suppressed.

[0119] 3. Enhance the frequency-domain signal: Select a modulation function to enhance the frequency-domain signal of the Fourier transform amplitude of each frame while retaining the original phase :

[0120] (2)

[0121] In this example, a soft-threshold modulation function is selected. Its function is to accurately locate pixels in the amplitude pattern whose intensity is greater than a specified threshold to generate an enhanced frequency-domain information pattern. Its modulation function is:

[0122] (5)

[0123] where the specified threshold is defined as .

[0124] 4. Perform inverse Fourier transform on the modified frequency-domain signal. The formula is as follows:

[0125] (8)

[0126] The denoised image is obtained.

[0127] 5. Registration Optimization Based on Information Entropy: The processed images are optimized using mutual information based on information entropy. The goal is to find the optimal transformation parameters (translation, rotation, scaling, etc.) such that the mutual information of the registered images reaches the maximum. First, assume the input , The frequency of occurrence of the corresponding pixel intensity values of the two images , is and . The sum of the number of all non-zero pixel intensity values of the two images is denoted as and respectively. The formulas are as follows:

[0128] (11)

[0129] (12)

[0130] Meanwhile, assume the frequency of occurrence of the pixel intensity values at the corresponding positions of the two images is . The sum of the number of all non-zero pixel intensity value pairs of the two images is denoted as . The formula is as follows:

[0131] (14)

[0132] Then, calculate the marginal entropy of , the two images, denoted as and respectively. The formulas are as follows:

[0133] (9)

[0134] (10)

[0135] Meanwhile, the joint entropy of the two images is . The formula is as follows:

[0136] (13)

[0137] Then the formula for mutual information is:

[0138] (15)

[0139] 6. Normalized Mutual Information: Solve the sensitivity of MI to the size of the overlapping region and improve robustness. The formula is as follows:

[0140] (16)

[0141] 7. Multi-scale optimization: Use an optimization algorithm to optimize the MI value and transformation parameters, with the goal of finding the transformation parameters that maximize the mutual information. The algorithm used in this example is the differential evolution algorithm. The formula is as follows:

[0142] (17)

[0143] 8. Iterative optimization-based MI calculation for correcting global defocus series image registration:

[0144] (18)

[0145] 9. Registration result evaluation: Process the registered defocus series images, and reconstruct the resolution to below 3.0 Å ( Figure 4b ), while the traditional cross-correlation (CC) method is limited to 5.8 Å ( Figure 4a ). It can be seen that this method is compatible with the imaging requirements of radiation-sensitive materials. When the total dose is not higher than 5 e - / Å 2 ), the high-frequency structural features of the sample are retained through frequency-domain modulation.

[0146] Example 3

[0147] This example relates to a registration device for defocus series images of a high-noise transmission electron microscope, including a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the registration method for defocus series images of the high-noise transmission electron microscope in Example 1.

[0148] Example 4

[0149] This example relates to a computer-readable storage medium with a program stored thereon. When the program is executed by a processor, it implements the registration method for defocus series images of the high-noise transmission electron microscope in Example 1.

[0150] The content described in the embodiments of this specification is only an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments. The protection scope of the present invention also extends to equivalent technical means that those skilled in the art can think of based on the inventive concept of the present invention.

Claims

1. A registration method for high-noise transmission electron microscope out-of-focus sequence images, characterized in that: The following steps are involved: (1) Obtaining high-noise underfocus sequential electron microscopy images; (2) Perform Fourier transform on the image sequence: Perform Fourier transform on the acquired images frame by frame, extract the amplitude components of each frame, and accumulate them to form a frequency domain amplitude pattern to enhance the periodic structure signal while suppressing random noise; (3) Enhance the frequency domain signal: Select a modulation function to modulate the frequency domain amplitude component while retaining the original phase information; (4) Inverse Fourier transform: Perform inverse Fourier transform on the modulated frequency domain signal to obtain a denoised image sequence; (5) Optimize registration based on information entropy: Calculate the mutual information of the registered images, where: the frequency of occurrence of each pixel intensity value in the two images is counted respectively, and the edge entropy of each image is calculated based on the frequency ratio. The joint frequency distribution of the pixel intensity pairs at the corresponding positions of the two images is counted, and the joint information entropy is calculated based on the joint frequency ratio. The mutual information is obtained by subtracting the joint entropy from the sum of the edge entropies of the two images. (6) Normalize mutual information to reduce sensitivity to the size of the overlapping area; (7) Multi-scale optimization: Use optimization algorithms to optimize mutual information values ​​and transformation parameters to obtain the registration parameters that maximize the mutual information value; (8) Correction of global underfocus sequence image registration based on iteratively optimized mutual information value calculation; (9) Evaluation of registration results: For defocused electron microscopy images, the root mean square error (RMS) is used to quantitatively evaluate the registration translation parameters and verify the registration accuracy. For defocused electron microscopy images acquired using actual low-dose technology, the registration level is verified by comparing the resolution of the reconstructed structural images.

2. The method according to claim 1, characterized in that: The image described in step (2) has n frames, forming an image stack, where the image of the i-th frame is represented as , where (x, y) corresponds to the projection position of the sample in two-dimensional space during transmission electron microscopy acquisition, x is the horizontal coordinate, and y is the vertical coordinate. The value of reflects the intensity of the electronic signal detected by the detector at that position; the frequency domain amplitude mode is expressed as , where u and v represent the contribution of different frequency components in the spatial frequency, u corresponds to the frequency in the horizontal direction, and v corresponds to the frequency in the vertical direction. The value reflects the result of Fourier transform, and its mathematical meaning is to map the spatial domain to the frequency domain. The Fourier transform is calculated using the fast Fourier transform FFT, and the formula is as follows: （1） For a single frame image Perform continuous Fourier transform, specifically by calculating the i-th frame image by integration The complex exponential wave corresponding to the i-th frame image The inner product of quantizes each frequency component in the image The contribution of The absolute value part represents the amplitude spectrum of the Fourier transform, that is, the modulus of the Fourier transform, and the summation part represents the superposition of the spectra of multiple frames of images, and finally obtains .

3. The method according to claim 1, characterized in that: The modulation function described in step (3) includes: Gaussian modulation function; Wiener modulation function; soft threshold modulation function; Gaussian high-pass modulation function; median modulation function; The formula for enhancing the frequency domain signal in step (3) is: (2) in, It is expressed as the phase angle of the frequency signal. The physical meaning of the phase is to determine the spatial position of the periodic structure in the signal. It comes from the original signal in the image, which means that the phase information remains unchanged; is the frequency point in the i-th frame image The original phase information of the , which encodes the phase information into a complex form, which is convenient for frequency domain signal synthesis; is the modulation function, which modulates the amplitude component in the frequency domain; Indicates the frequency point in the i-th frame image The enhanced signal contains the adjusted amplitude and phase information, which is used for subsequent inverse Fourier transform to reconstruct the enhanced spatial domain signal.

4. The method according to claim 1, characterized in that: The mathematical expression of the inverse Fourier transform in step (4) is as follows: （8） in, To transform the frequency domain signal from The restored spatial domain signal contains the enhanced or corrected information, that is, the i-th frame in the denoised image sequence. Indicates that the frequency domain signal Each frequency component in Converted into a plane wave in the spatial domain; the integral part in formula (8) represents the recombination of the separated frequency components in the frequency domain to restore the global and local features of the spatial domain image.

5. The method according to claim 1, characterized in that: The mutual information described in step (5) , its mathematical expression is as follows: （15） in, For images The total number of non-zero pixel values ​​in , For images The total number of non-zero pixel values ​​in , and Represents the edge entropy of the i-th and j-th frames in the image stack of a total of n frames after inverse Fourier transform. and , based on the input , The corresponding pixel intensity values ​​of the two images , The frequency of occurrence and , is an image The intensity value of a pixel in is an image The intensity value of the corresponding pixel in the two images is recorded as and ,The total number combined with the frequency of occurrence is used to express the probability of occurrence of a certain intensity in a single image, that is, the independent probability of each image; represents the total number of non-zero pixels in the same position of the two images, Indicates that in the two images, the pixel intensity values ​​at the same position are and The total logarithm combined with the frequency of occurrence of pixel pairs is used to express the probability of occurrence of a corresponding pixel intensity value pair in the two images, that is, the joint probability of the two images; the logarithmic term is expressed as the ratio of the joint probability to the independent probability, which is used to measure the statistical correlation of the two images. The larger the value, the more dependent the pixel intensity distribution of the two images is; the summation part represents the statistical relationship of all pixel pairs, and the global mutual information value is obtained; mutual information It is used to quantify the statistical dependence of two images. The larger the value, the more correlated the pixel intensity distributions of the two images are, that is, the more similar they are or the better the registration is.

6. The method according to claim 1, characterized in that: The mutual information calculation described in step (6) adopts normalized mutual information NMI calculation to reduce the sensitivity of registration to the size of image overlap area. Its mathematical expression is as follows: （16） Normalized Mutual Information Mutual Information Normalized to a fixed range [0,1] to facilitate comparison of similarities between different image pairs.

7. The method according to claim 1, characterized in that: The optimization algorithm described in step (7) includes differential evolution, gradient descent, and simulated annealing; The optimization algorithm described in step (7) is used to calculate the optimal registration parameters , its mathematical expression is as follows: (17) in, Represents the image after the image is translated. and Indicates the offset in horizontal and vertical directions; Indicates that in all possible In the translation combination, find the group of translations that maximizes the mutual information value, so as to obtain the optimal registration parameters , that is, the horizontal and vertical translation required when the image registration is best.

8. The method according to claim 1, characterized in that: The mathematical expression of the iteratively optimized MI calculation-corrected global underfocus sequence image registration described in step (8) is as follows: (18) Where m represents the number of image pairs used for registration during the optimization process, represents the optimal registration translation calculated for the i-th frame image relative to the k-th frame reference image, It represents the global corrected displacement vector of the i-th frame image, that is, the total translation amount finally used to align the image with the global reference image to improve the robustness of the registration.

9. A registration device for high-noise transmission electron microscope out-of-focus sequence images, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, the registration method of high-noise transmission electron microscope underfocus sequence images according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the registration method for high-noise transmission electron microscope underfocus sequence images according to any one of claims 1 to 8 is implemented.

Images