Registration method and device for high-noise out-of-focus transmission electron microscope image sequences

Abstract

The present invention discloses a registration method and device for high-noise transmission electron microscope out-of-focus sequence images. The method includes: utilizing the synergistic effect of frequency domain modulation and information entropy optimization, specifically extracting the amplitude information of the structural signal through Fourier transform, combining multiple types of frequency domain modulation functions, and realizing noise spectrum energy suppression and effective signal band enhancement in the frequency domain; then calculating the mutual information based on the image after frequency domain modulation, and utilizing the nonlinear correlation of the statistical distribution of the frequency domain signal to overcome the defect of traditional intensity matching being sensitive to noise, thereby driving robust registration.

Description

Technical Field

[0001] The present invention relates to the field of electron microscope image processing, and specifically to a registration method and device for high-noise transmission electron microscope underfocus sequence images based on frequency domain modulation information entropy. The method and device are suitable for automatic alignment of multiple frames of underfocus images acquired by equipment such as transmission electron microscopes (TEMs) and scanning transmission electron microscopes (STEMs) under low-dose conditions, and are widely used in high-resolution electron microscopy technologies such as low-dose imaging, exit wave reconstruction, and electron tomography. Background Art

[0002] In transmission electron microscopy (TEM) and scanning transmission electron microscopy (STEM) imaging, low-dose imaging techniques have become a key tool for studying electron-beam-sensitive materials (such as biological samples, two-dimensional materials, and metal-organic frameworks) by reducing electron beam damage. However, low-dose conditions significantly reduce the image signal-to-noise ratio (SNR), severely limiting the accuracy of traditional registration methods that rely on gradient or feature point matching. At the same time, for applications such as exit wave reconstruction and electron tomography, it is necessary to acquire defocused series images with varying amounts of defocus and achieve multimodal data fusion through high-precision registration. However, the dynamic variation in defocus amounts presents a dual challenge in the registration of such defocused series images. First, differences in defocus amount can cause nonlinear distortions in the apparent features of the same structure (such as edge sharpness and phase contrast drift), undermining the intensity or structural consistency assumptions relied upon by traditional algorithms. Second, defocus-induced diffraction effects further introduce local phase and amplitude distortions, rendering frequency-domain phase matching-based methods ineffective.

[0003] In the registration task of traditional electron microscopy images, commonly used image registration methods include cross correlation (CC), geometric phase correlation (GPC), and the bUnwarpJ algorithm. These methods achieve good results in general low-noise or optical microscopy image processing, but their stability and accuracy are significantly reduced when applied to high-noise out-of-focus image sequences. The main reason is that these methods rely on global image features or frequency domain information during the registration process. However, the noise distribution of out-of-focus image sequences is complex, and the image contrast varies significantly at different focal lengths, making it difficult for traditional methods to extract stable matching features. Therefore, under low-dose conditions, the decrease in signal-to-noise ratio and the nonlinear distortion of the apparent features of the same structure caused by the difference in out-of-focus amount form coupled interference, making it difficult for existing registration technologies to achieve a balance between robustness and sub-pixel accuracy, seriously restricting the application effect of high-resolution reconstruction. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a method and device for registration of high-noise transmission electron microscope underfocus sequence images based on frequency domain modulation information entropy (FDMIE-Reg), which is used for reconstruction of the outgoing wave of irradiated sensitive materials, electron tomography and other electron microscopy techniques that require high-precision image registration to support key operations such as phase recovery.

[0005] To address the insufficient accuracy and stability of traditional algorithms such as cross-correlation (CC), geometric phase correlation (GPC), and bUnwarpJ in the registration of high-noise defocus series electron microscopy images, this paper proposes a registration method for high-noise transmission electron microscopy defocus series images based on frequency-domain modulation information entropy. This method enhances the structural dominant frequency information through frequency-domain modulation to suppress noise interference. Simultaneously, a statistical correlation optimization strategy based on information entropy is combined to combat grayscale drift caused by changes in defocus, significantly improving the registration accuracy and robustness of high-noise TEM data. This method is particularly suitable for electron microscopy techniques such as exit wave reconstruction and electron tomography that rely on high-precision image registration, providing a reliable data foundation for phase retrieval of radiation-sensitive materials.

[0006] A first aspect of the present invention relates to a registration method for high-noise out-of-focus transmission electron microscope image sequences, comprising the following steps:

[0007] (1) Obtaining high-noise underfocused sequential electron microscopy images: Under low-dose shooting conditions, collect a set of transmission electron microscopy images with equal underfocus to form a complete sequence of images from underfocus to focus and then to overfocus; or use simulation software to generate simulated data based on experimental conditions.

[0008] (2) Perform Fourier transform on the image sequence: Perform Fourier transform on the acquired images frame by frame, extract the amplitude components of each frame, and accumulate them to form the frequency domain amplitude pattern to enhance the periodic structure signal while suppressing random noise.

[0009] (3) Enhance the frequency domain signal: Select a modulation function to modulate the frequency domain amplitude component while retaining the original phase information.

[0010] (4) Inverse Fourier transform: Perform inverse Fourier transform on the modulated frequency domain signal to obtain a denoised image sequence.

[0011] (5) Optimize registration based on information entropy: Calculate the mutual information of the registered images, where: the frequency of occurrence of each pixel intensity value in the two images is counted respectively, and the edge entropy of each image is calculated based on the frequency ratio. The joint frequency distribution of the pixel intensity pairs at the corresponding positions of the two images is counted, and the joint information entropy is calculated based on the joint frequency ratio. The sum of the edge entropies of the two images is subtracted from the joint entropy to obtain the mutual information.

[0012] (6) Normalize the mutual information to reduce the sensitivity to the size of the overlapping region.

[0013] (7) Multi-scale optimization: Use optimization algorithms to optimize the mutual information MI value and transformation parameters to obtain the registration parameters that maximize the MI value.

[0014] (8) Correction of global underfocus sequence image registration based on iteratively optimized MI calculation.

[0015] (9) Registration result evaluation: A control group with different registration methods was established. The root mean square error (RMS) of the registered defocused electron microscopy images was used to quantitatively evaluate the registration translation parameters and verify the registration accuracy. The defocused electron microscopy images acquired using actual low-dose technology were reconstructed after registration using steps (2) to (8). The resolution of the reconstructed structural images after registration with the control group was compared to verify the registration level.

[0016] The image in step (2) has n frames, forming an image stack, where the image of the i-th frame is represented as , where (x, y) corresponds to the projection position of the sample in two-dimensional space during transmission electron microscopy acquisition, x is the horizontal coordinate, and y is the vertical coordinate. The value of reflects the intensity of the electronic signal detected by the detector at that position; the frequency domain amplitude mode is expressed as , where u and v represent the contribution of different frequency components in the spatial frequency, u corresponds to the frequency in the horizontal direction, and v corresponds to the frequency in the vertical direction. The value reflects the result of Fourier transform, whose mathematical meaning is to map the spatial domain to the frequency domain. The Fourier transform is calculated using the fast Fourier transform FFT, and the formula is as follows:

[0017]

[0018] The integral part in formula (1) represents the integral of a single frame image. Perform continuous Fourier transform, specifically by calculating the i-th frame image by integration The complex exponential wave corresponding to the i-th frame image The inner product of quantizes each frequency component in the image The contribution of The absolute value part represents the amplitude spectrum of the Fourier transform, that is, the modulus of the Fourier transform, and the summation part represents the superposition of the spectrum of multiple frames of images, and finally obtains .

[0019] Wherein, the modulation function described in step (3) includes: Gaussian modulation function; Wiener modulation function; soft threshold modulation function; Gaussian high-pass modulation function; median modulation function;

[0020] The formula for enhancing the frequency domain signal in step (3) is:

[0021]

[0022] in, It is expressed as the phase angle of the frequency signal. The physical meaning of the phase is to determine the spatial position of the periodic structure in the signal. It comes from the original signal in the image, which means that the phase information remains unchanged; is the frequency point in the i-th frame image The original phase information of the , which encodes the phase information into a complex form, is convenient for frequency domain signal synthesis; is the modulation function, which modulates the amplitude component in the frequency domain; Indicates the frequency point in the i-th frame image The enhanced signal contains the adjusted amplitude and phase information, which is used for subsequent inverse Fourier transform to reconstruct the enhanced spatial domain signal;

[0023] The Gaussian modulation function described in step (3) is used to enhance low-frequency signals and suppress high-frequency noise. Its mathematical expression is as follows:

[0024]

[0025] where σ is the standard deviation of the Gaussian kernel (controls the degree of blur).

[0026] The Wiener modulation function described in step (3) is used to minimize the mean square error (MSE) between the signal and the noise, and the noise power spectrum must be known. Its mathematical expression is as follows:

[0027]

[0028] in is the original signal power spectrum, is the noise power spectrum.

[0029] The soft threshold modulation function in step (3) is used to enhance the specific frequency signal and remove background noise. Its mathematical expression is as follows:

[0030]

[0031] where the specified threshold It is defined as, .

[0032] The Gaussian high-pass modulation function in step (3) is used to remove low-frequency artifacts and enhance high-frequency features. Its mathematical expression is as follows:

[0033]

[0034] in , represents the frequency point The distance to the center of the spectrum (M / 2, N / 2), is the cutoff frequency (radius), which controls the low frequency suppression range.

[0035] The median modulation function in step (3) is used to eliminate outlier noise points, and its mathematical expression is as follows:

[0036]

[0037] Where k is the modulation window radius;

[0038] The mathematical expression of the inverse Fourier transform in step (4) is as follows:

[0039]

[0040] in, To transform the frequency domain signal from the inverse Fourier transform The restored spatial domain signal contains the enhanced or corrected information, i.e. the i-th frame in the denoised image sequence. Indicates that the frequency domain signal Each frequency component in Converted into a plane wave in the spatial domain; the integral part in formula (8) represents the recombination of the frequency components separated in the frequency domain to restore the global and local features of the spatial domain image.

[0041] Among them, the edge entropy E( ) and E( ), its mathematical expression is as follows:

[0042]

[0043] in, and Represents the image of the i-th and j-th frames in the image stack of a total of n frames after inverse Fourier transform, and calculates the edge entropy E( ) and E( ), based on the input 、 The corresponding pixel intensity values ​​of the two images Frequency of occurrence and , is an image The intensity value of a pixel in is an image The intensity value of the corresponding pixel in the two images is recorded as and , its mathematical expression is as follows:

[0044]

[0045] in, For images The total number of non-zero pixel values ​​in , For images The total number of all non-zero pixel values ​​in , combined with the frequency of occurrence, is used to represent the probability of occurrence of a certain intensity in a single image;

[0046] Among them, the joint information entropy in step (5) is , its mathematical expression is as follows:

[0047]

[0048] Calculate the joint information entropy of two images , it is necessary to base the pixel intensity value pair of the corresponding pixel positions of the two images on Frequency of occurrence , Indicates that in the two images, the pixel intensity values ​​at the same position are and The number of occurrences of , and the sum of all non-zero pixel intensity value pairs of the two images is recorded as , its mathematical expression is as follows:

[0049]

[0050] in, It represents the total number of non-zero pixels at the same position in both images. The total number of pairs combined with the frequency of pixel pairs is used to express the probability of occurrence of a corresponding pixel intensity value pair in the two images.

[0051] Among them, the mutual information described in step (5) , its mathematical expression is as follows:

[0052]

[0053] The logarithmic term is expressed as the ratio of the joint probability to the independent probability, which is used to measure the statistical correlation between the two images. The larger the value, the more dependent the pixel intensity distribution of the two images is. The summation part represents the statistical relationship of all pixel pairs to obtain the global mutual information value. Mutual information It is used to quantify the statistical dependence of two images. The larger the value, the more correlated the pixel intensity distributions of the two images are, that is, the more similar they are or the better the registration.

[0054] The mutual information calculation in step (6) is performed using normalized mutual information (NMI) to reduce the sensitivity of registration to the size of the image overlap area. Its mathematical expression is as follows:

[0055]

[0056] Normalized mutual information Mutual Information Normalized to a fixed range [0, 1] to facilitate comparison of similarities between different image pairs.

[0057] Wherein, the optimization algorithm described in step (7) includes differential evolution, gradient descent, and simulated annealing;

[0058] The optimization algorithm described in step (7) is used to calculate the optimal registration parameters , its mathematical expression is as follows:

[0059]

[0060] in, Represents the image after the image is translated. and Indicates the offset in the horizontal and vertical directions; Indicates that in all possible In the translation combination, find the set of translations that maximizes the mutual information value, so as to obtain the optimal registration parameters , that is, the horizontal and vertical translation required when the image registration is best.

[0061] The mathematical expression of the iteratively optimized MI calculation-based correction of global underfocus sequence image registration in step (8) is as follows:

[0062]

[0063] Where m represents the number of image pairs used for registration during the optimization process, It represents the optimal registration translation calculated for the i-th frame image relative to the k-th frame reference image, It represents the global corrected displacement vector of the i-th frame image, that is, the total translation amount ultimately used to align the image with the global reference image to improve the robustness of the registration.

[0064] A second aspect of the present invention relates to a registration device for a high-noise transmission electron microscope defocused sequence image, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the registration method for a high-noise transmission electron microscope defocused sequence image of the present invention.

[0065] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the registration method of the high-noise transmission electron microscope out-of-focus sequence images of the present invention.

[0066] The working principle of the present invention is to utilize the synergistic effect of frequency domain modulation and information entropy optimization. Specifically, the amplitude information of the structural signal is extracted through Fourier transform, and multiple types of frequency domain modulation functions are combined to achieve noise spectrum energy suppression and effective signal band enhancement in the frequency domain; then mutual information calculation is performed based on the image after frequency domain modulation, and the nonlinear correlation of the statistical distribution of frequency domain signals is utilized to overcome the defect of traditional intensity matching that is sensitive to noise, thereby driving robust alignment.

[0067] The advantages of the present invention are:

[0068] The registration error of high-noise out-of-focus sequence images is significantly reduced. Even under high noise conditions (SNR=−32.3dB), the registration success rate is still maintained at over 99%. Compared with traditional methods such as geometric phase correlation (GPC) and cross-correlation (CC), the error is reduced from 26% to 0.8%. In addition, this inventive method eliminates the registration deviation caused by defocus difference by quantifying the statistical distribution similarity of the image after frequency domain modulation rather than relying on the intensity contrast related to defocus. The registration error rate within the drift range of 2-100nm is less than 1%, showing strong robustness and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1a-Figure 1c is the drift before and after registration of the present invention Distribution comparison chart, where Figure 1a is the actual drift curve, Figure 1b is the MI registration drift curve, Figure 1c The drift curve is aligned with the present invention.

[0070] Figure 2a-2d is a comparison diagram of electron microscope images before and after registration of the present invention, wherein Figure 2a is a single high-noise out-of-focus image before registration. Figure 2b After CC registration, the electron microscope image is Figure 2cFor the present invention, the electron microscope image is registered. Figure 2d This is a stack of non-displacement, high-noise, out-of-focus images. The small image below each image shows a magnified image of the unit structure on the left and an FFT image on the right. The white arc in the FFT image indicates the resolution.

[0071] Figure 3 3 is a RMS comparison diagram of the present invention, where BP-CC is a Bandpass-CC algorithm.

[0072] Figure 4a and Figure 4b is a comparison of electron microscope images reconstructed after registration using different methods, where Figure 4a The electron microscope image reconstructed after CC registration, Figure 4b The electron microscope images reconstructed after registration in the present invention. In the small figure below each figure, the left figure is an enlarged view of the unit structure, and the right figure is an FFT graph. The white circle in the FFT graph indicates the resolution. Specific implementation methods

[0073] The present invention is described in detail below with reference to the accompanying drawings and examples. The present invention improves the registration method for high-noise out-of-focus sequence images, and achieves accurate registration of out-of-focus sequence images collected by electron microscopy of irradiation-sensitive materials under low doses. In order to verify the feasibility and reliability of the present invention, this example uses simulated data of a hypothetical sample MOF material MIL101 to perform traditional algorithms such as geometric phase correlation, cross-correlation and bUnwarpJ, as well as the improved registration method of the present invention, and compares the results after registration. The main reason for using simulated data is that simulated data can eliminate interference from external uncertain factors, such as astigmatism, thereby proving the effectiveness of the present invention's registration method for high-noise out-of-focus sequence images.

[0074] Example 1

[0075] Reference Figure 1a-Figure 3 The present embodiment provides a registration method for high-noise transmission electron microscope out-of-focus sequence images, comprising the following steps:

[0076] 1. First, simulate the projection imaging of MIL101 material under an electron microscope, generating a dataset of 512 × 512 × 21 pixels. The data simulation conditions were: an accelerating voltage of 300 kV, and defocus of each image set to 2 nm, 5 nm, 10 nm, 20 nm, 50 nm, and 100 nm. Each set contained a complete series of images from defocus to focus and then to overfocus. Poisson noise with an intensity of 0.5 was added to each simulated image, resulting in a signal-to-noise ratio (SNR) of −32.3 to simulate the noise characteristics of low-dose imaging (e.g., Figure 2a and 2d ), and randomly introduces translation drift to simulate the motion error in actual TEM imaging (as shown in Figure 1ashown).

[0077] The low dose referred to in this invention specifically refers to the electron dose required to maintain the intrinsic structure of extremely sensitive materials (such as MOFs, hybrid perovskites and supramolecular crystals), which is usually much lower than 100 e - / Å 2 In this example, the SNR is 32.3, which corresponds to a total electron dose of 3.8 e - / Å 2 , negative SNR means that the signal is smaller than the noise, which is a high noise feature.

[0078] 2. Perform Fourier transform on the series of images: For an image stack containing n frames ( ), let the input original electron microscopy image be , perform Fourier transform (FT) on each frame in the image stack, extract the amplitude component of each frame (i.e., the intensity of the frequency domain signal), and sum the amplitude components of all frames to form the amplitude pattern, as shown in the following formula:

[0079]

[0080] By superimposing multiple frames, the frequency domain signals (such as Bragg reflection peaks) of periodic structures (such as crystal lattices) are enhanced while random noise is suppressed.

[0081] 3. Enhance the frequency domain signal: select a modulation function , enhance the frequency domain signal of the Fourier transform amplitude of each frame while preserving the original phase :

[0082]

[0083] This example uses a soft threshold modulation function, which accurately locates pixels in the amplitude pattern whose intensity is greater than a specified threshold to produce an enhanced frequency domain information pattern. The modulation function is:

[0084]

[0085] where the specified threshold It is defined as, .

[0086] 4. Perform inverse Fourier transform on the modified frequency domain signal. The formula is as follows:

[0087]

[0088] Get the denoised image.

[0089] 5. Optimize registration based on information entropy: The processed images are optimized using mutual information based on information entropy. The goal is to find the optimal transformation parameters (translation, rotation, scaling, etc.) so that the mutual information of the registered images reaches the maximum value. First, assume that the input 、 The corresponding pixel intensity values ​​of the two images The frequency of occurrence is and , the sum of all non-zero pixel intensity values ​​of the two images is recorded as and , the formula is as follows:

[0090]

[0091] At the same time, assuming that the pixel intensity values ​​at the corresponding positions of the two images are The frequency of occurrence is , and the sum of all non-zero pixel intensity value pairs of the two images is recorded as , the formula is as follows:

[0092]

[0093] Then, yes 、 The marginal entropy of the two images is calculated as E( ) and E( ), the formula is as follows:

[0094]

[0095] At the same time, the joint entropy of the two images is , the formula is as follows:

[0096]

[0097] The mutual information calculation formula is:

[0098]

[0099] 6. Normalized Mutual Information: This solves the sensitivity of MI to the size of the overlapping region and improves robustness. The formula is as follows:

[0100]

[0101] 7. Multi-scale Optimization: Use an optimization algorithm to optimize the MI value and transformation parameters. The goal is to find the transformation parameters that maximize the mutual information. The algorithm used in this example is the differential evolution algorithm. The formula is as follows:

[0102]

[0103] And visualize the displacement parameters as a two-dimensional coordinate curve, such as Figure 1c As shown, the displacement parameter curve of CC method is shown as follows Figure 1b shown.

[0104] 8. Correction of global defocused image registration based on iterative optimization of MI calculation:

[0105]

[0106] The registered image is Figure 2c As shown, the results of CC registration are shown as Figure 2b shown.

[0107] 9. Registration result evaluation: Based on the root mean square error (RMS) quantitative evaluation standard, the registration transformation parameters The experiment applies different registration methods to six sets of data and compares the errors of the registration results. The results are as follows: Figure 3 As shown in Figure 2. Experimental results show that this method significantly reduces the registration error of low-dose out-of-focus sequence images. Compared with the average registration error of 21% of traditional methods, the average registration error of this method is reduced to 0.8%, which is much lower than the calibration error of methods such as geometric phase correlation (GPC) and cross correlation (CC). In addition, under high noise conditions (SNR = −32dB), this method still maintains a registration success rate of more than 99%. Figure 2c , showing strong robustness and adaptability.

[0108] Example 2

[0109] Reference Figure 4a and Figure 4b The present embodiment provides a registration method for high-noise transmission electron microscope out-of-focus sequence images, comprising the following steps:

[0110] 1. First, a set of defocused sequential images were collected in a transmission electron microscope with an accelerating voltage of 300 kV, an image size of 512 × 512 pixels, a focal length of ±200 nm, and an image collected every 20 nm of focal length. The total dose was 5 e - / Å 2 .

[0111] The low dose referred to in this invention specifically refers to the electron dose required to maintain the intrinsic structure of extremely sensitive materials (such as MOFs, hybrid perovskites and supramolecular crystals), which is usually much lower than 100 e - / Å 2 , the total electron dose in this example is 5 e - / Å 2 ,The corresponding SNR is − 14.1. Negative SNR means that the signal is smaller than the noise, which is a high noise feature.

[0112] 2. Perform Fourier transform on the series of images: For an image stack containing n frames ( ), let the input original electron microscopy image be , perform Fourier transform (FT) on each frame in the image stack, extract the amplitude component of each frame (i.e., the intensity of the frequency domain signal), and sum the amplitude components of all frames to form the amplitude pattern, as shown in the following formula:

[0113]

[0114] By superimposing multiple frames, the frequency domain signals (such as Bragg reflection peaks) of periodic structures (such as crystal lattices) are enhanced while random noise is suppressed.

[0115] 3. Enhance the frequency domain signal: select a modulation function , enhance the frequency domain signal of the Fourier transform amplitude of each frame while preserving the original phase :

[0116]

[0117] This example uses a soft threshold modulation function, which accurately locates pixels in the amplitude pattern whose intensity is greater than a specified threshold to produce an enhanced frequency domain information pattern. The modulation function is:

[0118]

[0119] where the specified threshold It is defined as, .

[0120] 4. Perform inverse Fourier transform on the modified frequency domain signal. The formula is as follows:

[0121]

[0122] Get the denoised image.

[0123] 5. Optimize registration based on information entropy: The processed images are optimized using mutual information based on information entropy. The goal is to find the optimal transformation parameters (translation, rotation, scaling, etc.) so that the mutual information of the registered images reaches the maximum value. First, assume that the input 、 The corresponding pixel intensity values ​​of the two images The frequency of occurrence is and , the sum of all non-zero pixel intensity values ​​of the two images is recorded as and , the formula is as follows:

[0124]

[0125] At the same time, assuming that the pixel intensity values ​​at the corresponding positions of the two images are The frequency of occurrence is , the sum of all non-zero pixel intensity value pairs of the two images is recorded as , the formula is as follows:

[0126]

[0127] Then, yes 、 The marginal entropy of the two images is calculated as E( ) and E( ), the formula is as follows:

[0128]

[0129] At the same time, the joint entropy of the two images is , the formula is as follows:

[0130]

[0131] The mutual information calculation formula is:

[0132]

[0133] 6. Normalized Mutual Information: This solves the sensitivity of MI to the size of the overlapping region and improves robustness. The formula is as follows:

[0134]

[0135] 7. Multi-scale Optimization: Use an optimization algorithm to optimize the MI value and transformation parameters. The goal is to find the transformation parameters that maximize the mutual information. The algorithm used in this example is the differential evolution algorithm. The formula is as follows:

[0136]

[0137] 8. Correction of global defocused image registration based on iterative optimization of MI calculation:

[0138]

[0139] 9. Registration result evaluation: Process the registered defocused sequence images and improve the reconstruction resolution to below 3.0Å ( Figure 4b), while the traditional cross-correlation (CC) method is limited to 5.8Å ( Figure 4a ). It can be seen that this method is compatible with the imaging requirements of radiation-sensitive materials, and the total dose is not higher than 5 e - / Å 2 When the sample is modulated in the frequency domain, the high-frequency structural characteristics of the sample are retained.

[0140] Example 3

[0141] This embodiment relates to a registration device for a high-noise transmission electron microscope defocused sequence image, comprising a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the registration method for a high-noise transmission electron microscope defocused sequence image of Example 1.

[0142] Example 4

[0143] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the registration method of high-noise transmission electron microscope out-of-focus sequence images of embodiment 1 is implemented.

[0144] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A registration method for high-noise out-of-focus transmission electron microscope image sequences, characterized in that: The following steps are involved: (1) Obtaining high-noise defocused sequential electron microscopy images; (2) Perform Fourier transform on the image sequence: Perform Fourier transform on the obtained image frame by frame, extract the amplitude component of each frame, and accumulate it to form the frequency domain amplitude pattern to enhance the periodic structure signal while suppressing random noise; (3) Enhance the frequency domain signal: Select a modulation function to modulate the frequency domain amplitude component while retaining the original phase information; (4) Inverse Fourier transform: Perform inverse Fourier transform on the modulated frequency domain signal to obtain a noise-reduced image sequence; (5) Optimize registration based on information entropy: Calculate the mutual information of the denoised image, where: the frequency of occurrence of each pixel intensity value in two images of different frames in the denoised image sequence is counted respectively, and the edge entropy of each image is calculated based on the frequency ratio, and the joint frequency distribution of the pixel intensity pairs at the corresponding positions of the two images is counted, and the joint information entropy is calculated based on the joint frequency ratio; the sum of the edge entropies of the two images is subtracted from the joint entropy to obtain the mutual information; (6) Normalize mutual information to reduce sensitivity to the size of the overlapping area; (7) Multi-scale optimization: Use optimization algorithms to optimize mutual information values ​​and registration parameters to obtain the registration parameters that maximize the mutual information value; (8) calculating a correction value based on the iteratively optimized mutual information value and registration parameters, and registering the defocused sequential electron microscope images obtained in step (1); (9) Evaluation of registration results: A control group with different registration methods was established. The root mean square error (RMS) was used to quantitatively evaluate the registration parameters for the registered defocused electron microscopy images to verify the registration accuracy. The defocused electron microscopy images acquired using actual low-dose technology were reconstructed after registration using steps (2) to (8). The resolution of the reconstructed structural images was compared with that of the control group after registration to verify the registration level.

2. The method according to claim 1, characterized in that: The image described in step (2) has n frames, forming an image stack, where the image of the i-th frame is represented as , where (x, y) corresponds to the projection position of the sample in two-dimensional space during transmission electron microscopy acquisition, x is the horizontal coordinate, and y is the vertical coordinate. The value of reflects the intensity of the electronic signal detected by the detector at that position; the frequency domain amplitude mode is expressed as , where u and v represent the contribution of different frequency components in the spatial frequency, u corresponds to the frequency in the horizontal direction, and v corresponds to the frequency in the vertical direction. The value reflects the result of Fourier transform, whose mathematical meaning is to map the spatial domain to the frequency domain. The Fourier transform is calculated using the fast Fourier transform FFT, and the formula is as follows: , the integral part in formula (1) represents the single frame image Perform continuous Fourier transform, specifically by calculating the i-th frame image by integration The complex exponential wave corresponding to the i-th frame image The inner product of quantizes each frequency component in the image The contribution of The absolute value part represents the amplitude spectrum of the Fourier transform, that is, the modulus of the Fourier transform, and the summation part represents the superposition of the spectrum of multiple frames of images, and finally obtains .

3. The method according to claim 1, characterized in that: The modulation function described in step (3) includes: Gaussian modulation function, Wiener modulation function, soft threshold modulation function, Gaussian high-pass modulation function, and median modulation function; The formula for enhancing the frequency domain signal in step (3) is: ,in, It is expressed as the phase angle of the frequency signal. The physical meaning of the phase is to determine the spatial position of the periodic structure in the signal. It comes from the original signal in the image, which means that the phase information remains unchanged; is the frequency point in the i-th frame image The original phase information of the , which encodes the phase information into a complex form, is convenient for frequency domain signal synthesis; is the modulation function, which modulates the amplitude component in the frequency domain; Indicates the frequency point in the i-th frame image The enhanced signal contains the adjusted amplitude and phase information, which is used for subsequent inverse Fourier transform to reconstruct the enhanced spatial domain signal.

4. The method according to claim 1, wherein: The mathematical expression of the inverse Fourier transform in step (4) is as follows: ,in, To transform the frequency domain signal from the inverse Fourier transform The restored spatial domain signal contains the enhanced or corrected information, i.e. the i-th frame in the denoised image sequence. Indicates that the frequency domain signal Each frequency component in Converted into a plane wave in the spatial domain; the integral part in formula (8) represents the recombination of the frequency components separated in the frequency domain to restore the global and local features of the spatial domain image.

5. The method according to claim 1, characterized in that: The mutual information described in step (5) , its mathematical expression is as follows: ,in, For images The total number of non-zero pixel values ​​in , For images The total number of non-zero pixel values ​​in , and Represents the image of the i-th and j-th frames in the image stack of a total of n frames after inverse Fourier transform, and calculates the edge entropy E( ) and E( ), based on the input 、 The corresponding pixel intensity values ​​of the two images Frequency of occurrence and , is an image The intensity value of a pixel in is an image The intensity value of the corresponding pixel in the two images is recorded as and ,The total number combined with the frequency of occurrence is used to express the probability of occurrence of a certain intensity in a single image, that is, the independent probability of each image; represents the total number of non-zero pixels in the same position of the two images, Indicates that in the two images, the pixel intensity values ​​at the same position are and The total logarithm combined with the frequency of pixel pairs is used to express the probability of occurrence of a certain pixel intensity value pair in the two images, that is, the joint probability of the two images; the logarithmic term is expressed as the ratio of the joint probability to the independent probability, which is used to measure the statistical correlation of the two images. The larger the value, the more dependent the pixel intensity distribution of the two images is; the summation part represents the statistical relationship of all pixel pairs to obtain the global mutual information value; mutual information It is used to quantify the statistical dependence of two images. The larger the value, the more correlated the pixel intensity distributions of the two images are, that is, the more similar they are or the better the registration.

6. The method according to claim 1, characterized in that: The mutual information calculation described in step (6) adopts normalized mutual information NMI calculation to reduce the sensitivity of registration to the size of image overlapping area. Its mathematical expression is as follows: , normalized mutual information Mutual Information Normalized to a fixed range [0, 1] to facilitate comparison of similarities between different image pairs.

7. The method according to claim 1, characterized in that: The optimization algorithm described in step (7) includes differential evolution, gradient descent, and simulated annealing; The optimization algorithm described in step (7) is used to calculate the optimal registration parameters , its mathematical expression is as follows: ,in, Represents the image after the image is translated. and Indicates the offset in the horizontal and vertical directions; Indicates that in all possible In the translation combination, find the set of translations that maximizes the mutual information value, so as to obtain the optimal registration parameters , that is, the horizontal and vertical translation required when the image registration is best.

8. The method according to claim 1, characterized in that: The correction value calculated based on the iteratively optimized mutual information value and registration parameter in step (8) is expressed as follows: , where m represents the number of image pairs used for registration during the optimization process, It represents the optimal registration translation calculated for the i-th frame image relative to the k-th frame reference image, It represents the global corrected displacement vector of the i-th frame image, that is, the total translation amount ultimately used to align the image with the global reference image to improve the robustness of the registration.

9. A registration device for high-noise transmission electron microscope out-of-focus sequence images, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method for registering high-noise transmission electron microscope out-of-focus sequence images according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the registration method for high-noise transmission electron microscope out-of-focus sequence images according to any one of claims 1 to 8 is implemented.

Images