Image processing method and device for iDPC-STEM image, electronic equipment and medium

Abstract

The invention relates to an image processing method and device for an iDPC-STEM image, electronic equipment and a medium, and the method comprises the steps: obtaining a to-be-processed image for a target molecular sieve, and enabling the to-be-processed image to be the iDPC-STEM image; performing noise extraction on the to-be-processed image by using a denoising model constructed based on a variational auto-encoder in the image processing model to obtain a target noise image, the target noise image being used for indicating a noise distribution condition of the to-be-processed image, and the image processing model further comprising a super-resolution model constructed based on a deep residual network; determining a target de-noised image according to the to-be-processed image and the target noise image; and performing image enhancement on the target de-noised image by using the super-resolution model to obtain a target super-resolution image. According to the method, the denoising and enhancement of the iDPC-STEM image can be realized, the image quality is remarkably improved, and a powerful tool is provided for subsequent single molecule detection and molecular conformation matching analysis.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, electronic equipment and medium for iDPC-STEM images. Background Art

[0002] Traditional electron microscopy requires extremely high electron doses to achieve atomic-level image resolution. This method is not suitable for materials that are sensitive to electron beams, such as zeolites, metal-organic frameworks (MOFs), and perovskites. In recent years, the development of integrated differential phase contrast scanning transmission electron microscopy (iDPC-STEM) has effectively solved the challenge of low electron utilization efficiency. However, under low electron dose conditions, a large amount of scattering noise makes it difficult to accurately identify the position and conformation of individual molecules in molecular sieves, which also makes iDPC-STEM limited.

[0003] Therefore, currently in the field of iDPC-STEM images, for images that may have low image quality or complex structures, there is an urgent need to develop more adaptive image processing methods to address the limitations and challenges encountered by related technologies in processing iDPC-STEM images. Summary of the invention

[0004] In view of this, the present disclosure proposes an image processing method and device, electronic device and medium for iDPC-STEM images, which can achieve denoising and enhancement of iDPC-STEM images, significantly improve image quality, and provide a powerful tool for subsequent single molecule detection and molecular conformation matching analysis.

[0005] According to one aspect of the present disclosure, there is provided an image processing method for iDPC-STEM images, comprising: obtaining an image to be processed for a target molecular sieve, wherein the image to be processed is an integral differential phase contrast scanning transmission electron microscope iDPC-STEM image; performing noise extraction on the image to be processed using a denoising model constructed based on a variational autoencoder in an image processing model to obtain a target noise image, wherein the target noise image is used to indicate the noise distribution of the image to be processed, wherein the image processing model further comprises a super-resolution model constructed based on a deep residual network; determining a target denoised image based on the image to be processed and the target noise image; performing image enhancement on the target denoised image using the super-resolution model to obtain a target super-resolution image, wherein the target super-resolution image is the iDPC-STEM image of the target molecular sieve after noise removal and image enhancement.

[0006] In a possible implementation, the method further includes: obtaining an ideal image and multiple simulated images of each sample molecular sieve, wherein the ideal image of each sample molecular sieve is used to indicate the real microstructure of the corresponding sample molecular sieve, and the multiple simulated images of each sample molecular sieve are iDPC-STEM images obtained by imaging under different electron doses; constructing a training sample set based on the ideal images and multiple simulated images of all sample molecular sieves; using the training sample set to train the original image processing model until the calculated loss function value of the image processing model meets a preset condition, thereby obtaining a trained image processing model, wherein the loss function value includes the denoising loss value of the denoising model and the super-resolution loss value of the super-resolution model.

[0007] In a possible implementation, the denoising loss value includes a first loss value and a second loss value, the first loss value is calculated based on a sample denoised image obtained by processing a simulated image according to the latest image processing model and an ideal image corresponding to the simulated image, and the second loss value is calculated based on a sample noise image obtained by processing the simulated image according to the latest image processing model and a preset normal distribution; the super-resolution loss value is calculated based on a sample super-resolution image obtained by processing the simulated image according to the latest image processing model and an ideal image corresponding to the simulated image.

[0008] In a possible implementation, the method further includes: acquiring atomic structure information of each of the sample molecular sieves, the atomic structure information including the atomic number, atomic coordinates, and atomic radius of each atom in the corresponding sample molecular sieve; determining a contrast value of the corresponding sample molecular sieve based on the atomic number of each atom in each of the sample molecular sieves, and determining a relative position of the corresponding sample molecular sieve based on the atomic coordinates of each atom in each of the sample molecular sieves; and constructing an ideal image of the corresponding sample molecular sieve based on the atomic radius of each atom in each of the sample molecular sieves and the corresponding contrast value and relative position.

[0009] In one possible implementation, the target super-resolution image includes at least one pore region; wherein the method further includes: performing single-molecule detection on all pore regions in the target super-resolution image, and marking all pore regions with single-molecule structures to obtain detection results.

[0010] In a possible implementation, the method further includes: when the size of the image to be processed exceeds a preset value, dividing the image to be processed into multiple images; wherein, the method further includes: when there are multiple images to be processed, smoothing the target super-resolution image corresponding to each of the images to be processed, and then splicing them according to the position information of the corresponding images to be processed to obtain the final target super-resolution image.

[0011] In a possible implementation manner, determining a target denoised image according to the image to be processed and the target noisy image includes: subtracting the target noisy image from the image to be processed to obtain the target denoised image.

[0012] According to another aspect of the present disclosure, an image processing device for iDPC-STEM images is provided, comprising: an acquisition module, used to acquire an image to be processed for a target molecular sieve, wherein the image to be processed is an integral differential phase contrast scanning transmission electron microscope iDPC-STEM image; a denoising module, used to perform noise extraction on the image to be processed using a denoising model constructed based on a variational autoencoder in an image processing model, to obtain a target noise image, wherein the target noise image is used to indicate the noise distribution of the image to be processed, wherein the image processing model also includes a super-resolution model constructed based on a deep residual network; a processing module, used to determine a target denoised image based on the image to be processed and the target noise image; and a super-resolution module, used to perform image enhancement on the target denoised image using the super-resolution model, to obtain a target super-resolution image, wherein the target super-resolution image is the iDPC-STEM image of the target molecular sieve after noise removal and image enhancement.

[0013] In a possible implementation, the device also includes a training module, which is used to: obtain an ideal image and multiple simulated images of each sample molecular sieve, wherein the ideal image of each sample molecular sieve is used to indicate the real microstructure of the corresponding sample molecular sieve, and the multiple simulated images of each sample molecular sieve are iDPC-STEM images obtained by imaging under different electron doses; construct a training sample set based on the ideal images and multiple simulated images of all sample molecular sieves; use the training sample set to train the original image processing model until the calculated loss function value of the image processing model meets a preset condition, thereby obtaining a trained image processing model, wherein the loss function value includes the denoising loss value of the denoising model and the super-resolution loss value of the super-resolution model.

[0014] In a possible implementation, the denoising loss value includes a first loss value and a second loss value, the first loss value is calculated based on a sample denoised image obtained by processing a simulated image according to the latest image processing model and an ideal image corresponding to the simulated image, and the second loss value is calculated based on a sample noise image obtained by processing the simulated image according to the latest image processing model and a preset normal distribution; the super-resolution loss value is calculated based on a sample super-resolution image obtained by processing the simulated image according to the latest image processing model and an ideal image corresponding to the simulated image.

[0015] In a possible implementation, the device also includes a construction module, which is used to: obtain atomic structure information of each of the sample molecular sieves, the atomic structure information including the atomic number, atomic coordinates, and atomic radius of each atom in the corresponding sample molecular sieve; determine the contrast value of the corresponding sample molecular sieve based on the atomic number of each atom in each of the sample molecular sieves, and determine the relative position of the corresponding sample molecular sieve based on the atomic coordinates of each atom in each of the sample molecular sieves; and construct an ideal image of the corresponding sample molecular sieve based on the atomic radius of each atom in each of the sample molecular sieves and the corresponding contrast value and relative position.

[0016] In one possible implementation, the target super-resolution image includes at least one pore region; wherein the device also includes a detection module, which is used to: perform single-molecule detection on all pore regions in the target super-resolution image, and mark all pore regions with single-molecule structures to obtain detection results.

[0017] In a possible implementation, the device further includes a division module, which is used to: when the size of the image to be processed exceeds a preset value, divide the image to be processed into multiple images; wherein the device further includes a splicing module, which is used to: when there are multiple images to be processed, smooth the target super-resolution image corresponding to each of the images to be processed, and then splice them according to the position information of the corresponding images to be processed to obtain the final target super-resolution image.

[0018] In a possible implementation manner, determining a target denoised image according to the image to be processed and the target noisy image includes: subtracting the target noisy image from the image to be processed to obtain the target denoised image.

[0019] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0020] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0022] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0024] Figures 1 to 4 A schematic diagram showing an image processing method for iDPC-STEM images according to an embodiment of the present disclosure.

[0025] Figure 5a to Figure 5c A schematic diagram showing the results of the original image and the target super-resolved image under different evaluation indicators.

[0026] Figure 6 Schematic diagram showing the pixel distribution of various types of images involved before and after reconstruction.

[0027] Figure 7 A block diagram of an image processing apparatus for iDPC-STEM images according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0029] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0031] In order to facilitate those skilled in the art to understand the technical solution provided by the embodiments of the present disclosure, the technical environment in which the technical solution is implemented is described below.

[0032] Scanning transmission electron microscopy (STEM) is considered a powerful tool in scientific research. Its imaging technology achieves sub-angstrom spatial resolution and can theoretically observe dynamic behavior at the atomic level with ultra-high spatial resolution. This technology plays an indispensable role in studying material properties and chemical processes. Traditional electron microscopy imaging usually uses a ring detector to receive scattered electron signals. This method has low electron utilization efficiency and requires extremely high electron doses to achieve atomic-level image resolution. However, this method is not suitable for materials that are sensitive to electron beams, such as zeolites, metal-organic frameworks (MOFs), perovskites, etc.

[0033] In recent years, the development of integrated differential phase contrast scanning transmission electron microscopy (iDPC-STEM) has effectively solved the challenge of low electron utilization efficiency by adopting a segmented annular detector located at a lower angle (4mrad to 22mrad) to capture electron signals, thereby achieving efficient imaging at a low electron dose. In addition, its imaging contrast is nearly linearly related to the atomic number, allowing simultaneous imaging of light and heavy elements. Under low electron dose conditions, iDPC-STEM has limitations, and a large amount of scattering noise makes it difficult to accurately identify the position and conformation of individual molecules. In addition, dynamically tracking the image changes of thousands of pores in a large field of view is an extremely challenging task, highlighting the urgent need for advanced analytical techniques that go beyond traditional time-consuming and expert-dependent manual interpretation.

[0034] The advent of deep learning heralds a revolutionary approach to enhancing STEM images, showing more promising performance than traditional noise reduction methods such as principal component analysis (PCA) and independent component analysis (ICA). Machine learning image recognition methods trained on large datasets have shown remarkable ability to handle the subtleties of image variations, thus excelling in atomic localization, single-atom dynamics, defect identification, and image enhancement. Although deep learning has achieved a series of achievements in the field of STEM imaging, its potential for processing images under low-dose conditions has not yet been fully explored. In addition, the extended time required to capture a specific STEM image also imposes significant limitations on the amount of data, posing challenges to the widespread application of deep learning techniques in electron microscopy imaging.

[0035] In summary, in the field of low-electron-dose STEM images, there is an urgent need to develop more adaptive algorithms for images with low quality or complex structures that can address the limitations and challenges encountered by existing technologies when processing low-dose STEM images such as iDPC-STEM images.

[0036] In order to solve the above technical problems, the embodiments of the present disclosure provide an image processing method for iDPC-STEM images, which aims to improve the quality of iDPC-STEM images through denoising and enhancement technology to meet the imaging requirements of transmission electron microscopes. Figures 1 to 4 A schematic diagram of an image processing method for iDPC-STEM images according to an embodiment of the present disclosure is shown. Figures 1 to 4 The image processing method for iDPC-STEM images provided in an embodiment of the present disclosure is schematically illustrated.

[0037] like Figure 1 As shown, the image processing method may include the following steps S101 to S104.

[0038] Step S101, obtaining an image to be processed for a target molecular sieve.

[0039] The image to be processed is an integrated differential phase contrast scanning transmission electron microscope iDPC-STEM image.

[0040] The disclosed embodiment designs a new image processing model, namely the Denoising Inferred Variational Autoencoder Super Resolution (DIVAESR) model, which is used for noise reduction and enhancement of low electron dose transmission electron microscope images, thereby achieving more effective detection and clearer visualization of molecular frameworks and single molecule regions. The image processing model may include two new key models, a denoising model (DIVAE) based on the Denoising Inferred Variational Autoencoder and a super-resolution model (EDSR, or enhancement model) based on the enhanced deep residual network. For example, the super-resolution model may adopt Figure 2 The network structure is shown.

[0041] Step S102: extract noise from the image to be processed using the denoising model constructed based on the variational autoencoder in the image processing model to obtain a target noisy image.

[0042] The target noise image is used to indicate the noise distribution in the image to be processed. An autoencoder is an unsupervised neural network, which is mainly used for data compression and feature learning. They reconstruct the input data by learning the representation of the input data. An autoencoder may include an encoder and a decoder. The encoder compresses the input data from a higher dimension into a lower dimensional representation, while the decoder reconstructs this low dimensional representation back to the original data. The denoising model of the disclosed embodiment adopts such a structure to extract the corresponding target noise image from the image to be processed.

[0043] Step S103: Determine a target denoised image according to the image to be processed and the target noisy image.

[0044] In some embodiments, step S103 may include: subtracting the target noise image from the image to be processed to obtain a target denoised image. The noise portion in the image to be processed (i.e., the target denoised image) may be removed by the denoising model to improve the clarity of the molecular structure in the iDPC-STEM image, thereby improving the accuracy of single molecule detection and visualization.

[0045] Since DIVAE loses information when converting from low dimension to high dimension, the denoised image may lack details. Therefore, the embodiment of the present disclosure performs image enhancement processing after denoising, and inputs the denoised image into the super-resolution model to further improve the image resolution and refine the image details, that is, the following step S104.

[0046] Step S104: Use the super-resolution model to perform image enhancement on the target denoised image to obtain a target super-resolution image.

[0047] The target super-resolution image is the iDPC-STEM image of the target molecular sieve after noise removal and image enhancement.

[0048] The super-resolution model of the disclosed embodiment adopts the structure of a deep residual network, which avoids the degradation problem in deep network training, effectively improves the performance of super-resolution processing, and improves the learning ability and efficiency of the overall network by expanding the number of feature maps in each residual block and removing unnecessary batch normalization layers. In this way, the image resolution is further improved through the enhanced deep super-resolution model, and more detailed image details are obtained, which is crucial for its effective use in subsequent advanced scientific analysis.

[0049] In this way, through the above steps S101 to S104, the image to be processed is processed using the image processing model. In short, image denoising is performed first and then image enhancement is performed. This can overcome the limitations and challenges faced by related models in iDPC-STEM image processing, achieve denoising and enhancement of iDPC-STEM images, significantly improve image quality, lay a solid foundation for in-depth single-molecule conformational analysis and detection, and greatly facilitate the subsequent research work.

[0050] The image processing method may also include a training process for an image processing model, which may include the following steps: obtaining an ideal image and multiple simulated images of each sample molecular sieve, wherein the ideal image of each sample molecular sieve is used to indicate the real microstructure of the corresponding sample molecular sieve, and the multiple simulated images of each sample molecular sieve are iDPC-STEM images obtained by imaging at different electron doses; constructing a training sample set based on the ideal images and multiple simulated images of all sample molecular sieves; and using the training sample set to train the original image processing model until the calculated loss function value of the image processing model meets a preset condition, thereby obtaining a trained image processing model.

[0051] The loss function consists of two parts. The first part reconstructs the noise by distinguishing the actual molecular structure from the noise artifacts through variational learning. The second part is the SR loss, which enhances the details in the reconstructed image by minimizing the loss between the upsampled reconstructed image and the ideal image. By integrating these two loss parts into a final loss function, a factor λ is introduced to balance the weights of the two loss parts, and λ is set according to actual needs. The synergistic effect of the two loss parts in the training phase is enhanced to produce high-resolution images with greatly reduced noise, which is critical for accurate molecular detection. In addition, the image processing model is rigorously iteratively trained to ensure that the generated images retain key molecular details for effective use in advanced scientific analysis.

[0052] The loss function value (loss total1 ) also includes two parts, one of which is the denoising loss value of the denoising model (loss DIVAE ), and the other part is the super-resolution loss value of the super-resolution model (loss EDSR ). In this embodiment, loss DIVAE +λloss EDSR =loss total1 .like Figure 2 As shown, the denoising loss value includes a first loss value (loss1) and a second loss value (loss2), the first loss value is calculated based on a sample denoised image (vae pic) obtained by processing a simulated image with the latest image processing model and an ideal image (label pic) corresponding to the simulated image, and the second loss value is calculated based on a sample noise image obtained by processing a simulated image with the latest image processing model and a preset normal distribution, for example, it can be calculated by a KL divergence function, wherein the sample noise image is obtained by extracting noise from a simulated image with the denoising model in the latest image processing model, and the sample denoised image is obtained by removing the sample noise image from the simulated image with the denoising model. The super-resolution loss value (loss3) is calculated based on a sample super-resolution image (HR pic) obtained by processing a simulated image with the latest image processing model and an ideal image (label pic) corresponding to the simulated image.

[0053] The image processing method may also include a process for generating an ideal image, which may include: obtaining the atomic structure information of each sample molecular sieve, the atomic structure information including the atomic number, atomic coordinates, and atomic radius of each atom in the corresponding sample molecular sieve; determining the contrast value of the corresponding sample molecular sieve based on the atomic number of each atom in each sample molecular sieve, and determining the relative position of the corresponding sample molecular sieve based on the atomic coordinates of each atom in each sample molecular sieve; constructing an ideal image of the corresponding sample molecular sieve based on the atomic radius of each atom in each sample molecular sieve and the corresponding contrast value and relative position.

[0054] One of the innovative points of the disclosed embodiment is that a generation program capable of realizing the above generation process is developed based on the approximate linear relationship between pixel value and atomic number. By identifying the atomic structure information in the crystallographic information file (CIF file), that is, the information of all atoms in the corresponding sample molecular sieve, the contrast value and relative position of each atom in the ideal image are determined according to the atomic number, atomic coordinates and atomic radius. In this way, the reconstructed ideal image ignores the noise, maintains the relative position of the atoms, and shows the contrast value that conforms to the ideal distribution. These achievements ensure the clarity and accuracy of the image and provide an accurate data basis for further analysis. Using such an ideal image generation process, it is possible to quickly generate batches of ideal images through the CIF files of molecular sieves of known structures, significantly improving the efficiency of generating ideal images, as label data for training models, that is, the ideal STEM images under the specified imaging direction. Thus, the data foundation for model training is laid, and the disclosed embodiment is more efficient in data processing and analysis, and can process large quantities and multiple types of material data, which not only reduces the cost of manpower and material resources, but also speeds up the generation speed of data.

[0055] Then, if Figure 3 As shown in the figure, the abTEM software platform can be used to simulate multiple images under different electron dose conditions as sample sets for model training. After the sample set data is ready, it is input into the original DIVAESR model for training. The hyperparameters in the training process are carefully selected to obtain the optimal model weights, and these weights are saved for subsequent model inference work.

[0056] In addition, the decoder part of the denoising model in the machine learning model of the image processing model provided by the embodiment of the present disclosure can also be used as a data generator for STEM images of specific structures, i.e., simulated images, thereby providing more data sources for the development of subsequent active learning frameworks. This means that using the method provided by the embodiment of the present disclosure, researchers and engineers can obtain more diverse data sets faster.

[0057] After the model training is completed, the image to be processed obtained from the experiment is input into the trained model to perform denoising, enhancement and reconstruction processing to obtain a clearer image, namely the target super-resolution image. Figure 4 As shown, the image processed by the DIVAESR model can be further applied to the identification of single-molecule points (for example, using Faster-RCNN as a single-molecule detection model) to obtain corresponding detection results. The target super-resolution image may include at least one pore area. The image processing method may also include: performing single-molecule detection on all pore areas in the target super-resolution image, and marking all pore areas with single-molecule structures to obtain detection results. Thereby providing a solid foundation for single-molecule conformational analysis and detection. This will provide a new path for the exploration of catalytic active sites in zeolite molecular sieves, imaging of electron-sensitive materials, and the study of intermolecular forces on a single-molecule scale. The image processed by the DIVAESR model can also be further applied to element type recognition and conformational analysis based on pixel clustering to implement in-depth image analysis and processing.

[0058] The disclosed embodiment also processes large-size images through a series of image processing steps to cope with the computational challenges of large images. The image processing method may also include: when the size of the image to be processed exceeds a preset value, dividing the image to be processed into multiple images; when there are multiple images to be processed, smoothing the target super-resolution image corresponding to each image to be processed, and then splicing according to the position information of the corresponding image to be processed to obtain the final target super-resolution image. In some embodiments, first, the large image of the image to be processed is cropped into smaller fragments by a sliding window to facilitate the separate processing of each fragment; then, each cropped small fragment is used as an image to be processed, so as to apply the image processing model for denoising and image quality enhancement to improve the image details. Each processed image fragment is then saved, and visual recognition technology (such as Faster R-CNN) is used to identify and locate objects. After completing the processing of all small fragments, the fragments are spliced ​​back to the size of the original image to obtain a complete image. This process involves not only simple splicing, but also smoothing and fusion of the edges of each fragment to ensure the coherence and overall quality of the reconstructed image. In this way, the method can effectively process large-size images, provide high-definition imaging results, and support real-time dynamic analysis in complex application scenarios.

[0059] The new artificial intelligence (AI) driven framework provided by the disclosed embodiment, namely DIVAESR, effectively achieves the quality improvement of low-dose STEM images and their subsequent image analysis and processing, and provides strong support for single-molecule conformation analysis and detection. Not only does it overcome the challenges in processing iDPC-STEM images, but it also opens up new ways for the exploration of catalytic active sites in zeolite molecular sieves, the imaging of electron-sensitive materials, and the study of intermolecular forces at the single-molecule level, providing a powerful tool for subsequent single-molecule detection and molecular conformation matching analysis, showing its huge application potential. This method provides a new perspective for fields such as materials science, chemistry, and biomedicine, and opens up new possibilities and prospects by improving image quality to explore the microscopic world in depth. For example, using this solution, it is not only possible to improve the recognition and analysis accuracy of single molecules, reveal the morphological changes of molecules under different environments, but also to analyze the conformation of single molecules based on clear iDPC-STEM images. For another example, using this solution, a new way is also opened up for real-time monitoring of the dynamic changes of intermediates in chemical reactions, thereby accelerating the understanding of chemical reaction mechanisms and the development of new materials.

[0060] The disclosed embodiment performs denoising processing on the iDPC-STEM image through a denoising model to reduce the scattering noise in the image, and then uses a super-resolution model to significantly improve the visualization clarity of the molecular skeleton and single-molecule area. The disclosed embodiment also optimizes the denoising and image enhancement process through a cleverly designed loss function, and successfully achieves the generation of high-resolution images by fusing the variational learning modeling noise removal technology with the super-resolution (SR) enhancement strategy, while significantly reducing the noise level. This improvement is crucial for the precise detection of molecules, and effectively improves the accuracy and efficiency of single-molecule detection in STEM imaging under different electron doses. After training on a simulated data set, the model has demonstrated excellent robustness and generalization capabilities in the application of real STEM images, demonstrating its effective denoising enhancement and detection capabilities on a variety of data sets. Figure 5a to Figure 5c The following is a schematic diagram showing the results of the original image and the target super-resolution image under different evaluation indicators. Figure 5a to Figure 5c As shown, the clear iDPC-STEM image reconstructed by the image processing method provided in the embodiment of the present disclosure, i.e., the target super-resolution image, shows a good improvement in peak signal-to-noise ratio (PSNR), variance expansion factor (VIF), and structural similarity (SSIM) compared to the original iDPC-STEM image. Figure 6 A schematic diagram showing the pixel distribution of various images involved before and after reconstruction. Figure 6 The pixel distribution of the image to be processed, the target super-resolution image, the ideal image, and the target noise image can be obtained.

[0061] The disclosed embodiment also provides an image processing device for iDPC-STEM images. The device includes an acquisition module for acquiring an image to be processed for a target molecular sieve, wherein the image to be processed is an iDPC-STEM image of an integral differential phase contrast scanning transmission electron microscope; a denoising module for extracting noise from the image to be processed using a denoising model constructed based on a variational autoencoder in an image processing model to obtain a target noise image, wherein the target noise image is used to indicate the noise distribution of the image to be processed, wherein the image processing model also includes a super-resolution model constructed based on a deep residual network; a processing module for determining a target denoised image based on the image to be processed and the target noise image; and a super-resolution module for performing image enhancement on the target denoised image using the super-resolution model to obtain a target super-resolution image, wherein the target super-resolution image is an iDPC-STEM image of the target molecular sieve after noise removal and image enhancement.

[0062] In a possible implementation, the device also includes a training module, which is used to: obtain an ideal image and multiple simulated images of each sample molecular sieve, wherein the ideal image of each sample molecular sieve is used to indicate the real microstructure of the corresponding sample molecular sieve, and the multiple simulated images of each sample molecular sieve are iDPC-STEM images obtained by imaging under different electron doses; construct a training sample set based on the ideal images and multiple simulated images of all sample molecular sieves; use the training sample set to train the original image processing model until the calculated loss function value of the image processing model meets a preset condition, thereby obtaining a trained image processing model, wherein the loss function value includes the denoising loss value of the denoising model and the super-resolution loss value of the super-resolution model.

[0063] In a possible implementation, the denoising loss value includes a first loss value and a second loss value, the first loss value is calculated based on a sample denoised image obtained by processing a simulated image according to the latest image processing model and an ideal image corresponding to the simulated image, and the second loss value is calculated based on a sample noise image obtained by processing the simulated image according to the latest image processing model and a preset normal distribution; the super-resolution loss value is calculated based on a sample super-resolution image obtained by processing the simulated image according to the latest image processing model and an ideal image corresponding to the simulated image.

[0064] In a possible implementation, the device also includes a construction module, which is used to: obtain atomic structure information of each of the sample molecular sieves, the atomic structure information including the atomic number, atomic coordinates, and atomic radius of each atom in the corresponding sample molecular sieve; determine the contrast value of the corresponding sample molecular sieve based on the atomic number of each atom in each of the sample molecular sieves, and determine the relative position of the corresponding sample molecular sieve based on the atomic coordinates of each atom in each of the sample molecular sieves; and construct an ideal image of the corresponding sample molecular sieve based on the atomic radius of each atom in each of the sample molecular sieves and the corresponding contrast value and relative position.

[0065] In one possible implementation, the target super-resolution image includes at least one pore region; wherein the device also includes a detection module, which is used to: perform single-molecule detection on all pore regions in the target super-resolution image, and mark all pore regions with single-molecule structures to obtain detection results.

[0066] In a possible implementation, the device further includes a division module, which is used to: when the size of the image to be processed exceeds a preset value, divide the image to be processed into multiple images; wherein the device further includes a splicing module, which is used to: when there are multiple images to be processed, smooth the target super-resolution image corresponding to each of the images to be processed, and then splice them according to the position information of the corresponding images to be processed to obtain the final target super-resolution image.

[0067] In a possible implementation manner, determining a target denoised image according to the image to be processed and the target noisy image includes: subtracting the target noisy image from the image to be processed to obtain the target denoised image.

[0068] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0069] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0070] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0071] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0072] Figure 7 1900 is a block diagram of an image processing apparatus for iDPC-STEM images according to an embodiment of the present disclosure. For example, the apparatus 1900 may be provided as a server or a terminal device. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0073] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0074] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0075] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0076] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0077] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0078] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0079] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0082] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0083] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method for iDPC-STEM images, characterized in that: include: Acquire an image to be processed for a target molecular sieve, wherein the image to be processed is an integrated differential phase contrast scanning transmission electron microscope iDPC-STEM image; Using a denoising model based on a variational autoencoder in an image processing model to extract noise from the image to be processed, to obtain a target noise image, wherein the target noise image is used to indicate the noise distribution of the image to be processed, wherein the image processing model also includes a super-resolution model based on a deep residual network; Determining a target denoised image according to the image to be processed and the target noisy image; The target denoised image is enhanced using the super-resolution model to obtain a target super-resolution image, which is an iDPC-STEM image of the target molecular sieve after noise is removed and image enhancement is performed.

2. The method according to claim 1, characterized in that The method further comprises: Acquire an ideal image and multiple simulated images of each sample molecular sieve, wherein the ideal image of each sample molecular sieve is used to indicate the real microstructure of the corresponding sample molecular sieve, and the multiple simulated images of each sample molecular sieve are iDPC-STEM images obtained by imaging at different electron doses; A training sample set is constructed based on the ideal images of all sample molecular sieves and multiple simulated images; The original image processing model is trained using the training sample set until the calculated loss function value of the image processing model meets a preset condition, thereby obtaining a trained image processing model, wherein the loss function value includes a denoising loss value of the denoising model and a super-resolution loss value of the super-resolution model.

3. The method according to claim 2, characterized in that The denoising loss value includes a first loss value and a second loss value, wherein the first loss value is calculated based on a sample denoised image obtained by processing a simulated image according to the latest image processing model and an ideal image corresponding to the simulated image, and the second loss value is calculated based on a sample noise image obtained by processing the simulated image according to the latest image processing model and a preset normal distribution; The super-resolution loss value is calculated by processing the simulated image with the latest image processing model to obtain a sample super-resolution image and an ideal image corresponding to the simulated image.

4. The method according to claim 2, characterized in that: The method further comprises: Acquiring atomic structure information of each of the sample molecular sieves, wherein the atomic structure information includes the atomic number, atomic coordinates, and atomic radius of each atom in the corresponding sample molecular sieve; Determining the contrast value of the corresponding sample molecular sieve based on the atomic number of each atom in each sample molecular sieve, and determining the relative position of the corresponding sample molecular sieve based on the atomic coordinates of each atom in each sample molecular sieve; Based on the atomic radius of each atom in each sample molecular sieve and the corresponding contrast value and relative position, an ideal image of the corresponding sample molecular sieve is constructed.

5. The method according to any one of claims 1 to 4, characterized in that The target super-resolution image includes at least one channel region; The method further comprises: performing single-molecule detection on all pore regions in the target super-resolution image, and marking all pore regions having single-molecule structures to obtain detection results.

6. The method according to any one of claims 1 to 4, characterized in that The method further includes: if the size of the image to be processed exceeds a preset value, dividing the image to be processed into a plurality of images; The method further includes: when there are multiple images to be processed, smoothing the target super-resolution image corresponding to each image to be processed, and then splicing them according to the position information of the corresponding images to be processed to obtain the final target super-resolution image.

7. The method according to any one of claims 1 to 4, characterized in that Determining a target denoised image according to the image to be processed and the target noisy image includes: The target noise image is subtracted from the image to be processed to obtain the target denoised image.

8. An image processing device for iDPC-STEM images, characterized in that: include: An acquisition module, used for acquiring an image to be processed for a target molecular sieve, wherein the image to be processed is an integrated differential phase contrast scanning transmission electron microscope iDPC-STEM image; A denoising module, used to extract noise from the image to be processed by using a denoising model constructed based on a variational autoencoder in an image processing model to obtain a target noise image, wherein the target noise image is used to indicate the noise distribution of the image to be processed, wherein the image processing model also includes a super-resolution model constructed based on a deep residual network; A processing module, used for determining a target denoised image according to the image to be processed and the target noisy image; The super-resolution module is used to perform image enhancement on the target denoised image using the super-resolution model to obtain a target super-resolution image, wherein the target super-resolution image is an iDPC-STEM image of the target molecular sieve after noise is removed and image enhancement is performed.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images