Deep learning-based method and system for reconstructing three-dimensional structure of single-walled carbon nanotubes

Abstract

The application provides a kind of single-walled carbon nanotube three-dimensional structure reconstruction method and system based on deep learning, method includes: the electron microscope simulation image of single-walled carbon nanotube is constructed as data set;Generation adversarial network model is trained;Using the trained generation adversarial network, from the electron microscope image of single-walled carbon nanotube, the amplitude image and phase image of the wave function corresponding to the image are predicted;From the wave function amplitude image and phase image, the atomic position on the upper and lower surfaces of carbon nanotube is analyzed;According to the atomic position of upper and lower layers, the tube wall atomic structure information is filled, and the three-dimensional structure of single-walled carbon nanotube is reconstructed.The method of the application is based on artificial intelligence algorithm, from high-resolution electron microscope image, the amplitude image and phase image of the exit wave function of single-walled carbon nanotube are predicted, then the atomic position is analyzed using the amplitude image and phase image of the wave function, and finally the three-dimensional atomic structure of single-walled carbon nanotube is reconstructed.The application solves the prediction problem from high-resolution electron microscope image to three-dimensional atomic structure of carbon nanotube.

Description

Technical Field

[0001] This invention belongs to the field of transmission electron microscopy image processing technology, and more specifically to the application of deep learning in electron microscope image processing, particularly relating to a method and system for reconstructing the three-dimensional atomic structure of carbon nanotubes from electron microscope images based on artificial intelligence algorithms. Background Technology

[0002] Single-walled carbon nanotubes (SWCNTs) are highly promising nanomaterials. Their unique physical structure makes them suitable for numerous applications, including batteries, catalysts, biosensors, and gas sensors. Studying the three-dimensional atomic structure of SWCNTs is crucial for understanding their properties and is essential for their future applications. However, traditional methods for determining the structure of SWCNTs cannot directly obtain the emitted wavefunction from a single transmission electron microscope (TEM) image, thus preventing the accurate determination of the three-dimensional atomic structure, especially when defects are present.

[0003] Currently, machine learning applications are limited in solving the 3D structure of carbon nuclei (CNTs). One existing method uses traditional artificial neural networks, specifically fully connected networks, to predict the atomic coordinates of CNTs. However, this method lacks precision and is extremely time-consuming. Another approach involves adding convolutional neural networks or residual neural networks, which significantly improves time efficiency compared to the former. However, both methods require chiral indices and the initial coordinates of carbon atoms as input, which greatly limits their predictive capabilities.

[0004] How to reconstruct the three-dimensional atomic structure of carbon nanotubes more efficiently and improve the resolution of the reconstructed structure is a problem that needs to be solved. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a deep learning-based method for reconstructing the three-dimensional structure of single-walled carbon nanotubes, so as to quickly and efficiently reconstruct the three-dimensional structure of single-walled carbon nanotubes.

[0006] In one aspect, the present invention provides a method for reconstructing the three-dimensional structure of single-walled carbon nanotubes based on deep learning, the method comprising the following steps:

[0007] Electron microscopy simulation images of single-walled carbon nanotubes were constructed as a dataset;

[0008] The generative adversarial network model is trained using the constructed dataset;

[0009] Using a trained generative adversarial network, the amplitude and phase images of the wave function corresponding to the image are predicted based on experimental and / or simulated electric images of single-walled carbon nanotubes.

[0010] Based on the amplitude and phase images of the wave function, the atomic positions on the upper and lower surfaces of the carbon nanotube are analyzed.

[0011] Based on the atomic positions of the upper and lower layers, the atomic structure information of the tube wall is filled in to reconstruct the three-dimensional structure of the single-walled carbon nanotube.

[0012] In some embodiments of the present invention, the electron microscopy simulation images of the constructed single-walled carbon nanotubes as a dataset include: constructing atomic structures of single-walled carbon nanotubes with different chirities and without defects; modifying the atomic structure of single-walled carbon nanotubes to construct atomic structures of single-walled carbon nanotubes with defects; simulating the outgoing wave function of high-energy electrons after the incident wave function of high-energy electrons passes through the single-walled carbon nanotubes, based on the theory of high-resolution electron microscopy, for both single-walled carbon nanotubes with different chirities and single-walled carbon nanotubes with different chirities without defects and single-walled carbon nanotubes with different chirities, and simulating high-resolution electron microscopy images based on preset experimental conditions;

[0013] Data sets of simulated transmission electron microscopy (TEM) images were obtained for various atomic structures, different electron beam incident directions, and different TEM experimental conditions. The dataset consists of four parts: a subset of high-resolution simulated TEM images of single-walled carbon nanotube structures, a subset of wavefunction amplitude images corresponding to each simulated image, a subset of wavefunction phase images corresponding to each simulated image, and a subset of complex images of the transmission transfer function corresponding to each simulated image. The dataset was preprocessed, including adding noise to the simulated high-resolution electron microscopy images and augmenting the dataset.

[0014] In some embodiments of the present invention, the generative adversarial network model includes a generator and a discriminator; the generator uses the UNet++ model as the backbone network, specifically including a convolutional module, a downsampling module, an upsampling module, and a Dropout layer.

[0015] In some embodiments of the present invention, the discriminator specifically includes a convolutional layer, a fully connected layer, and an activation function layer.

[0016] In some embodiments of the present invention, when training a generative adversarial network (GAN) model using a constructed dataset, the loss function of the generator of the GAN model includes one or more of the following: mean squared error, mean absolute error, and structural similarity; the error of the discriminator includes two aspects of mean absolute error: first, the mean absolute error between the output probability result of the generator's predicted wavefunction after resimulating a high-resolution electron microscope image under the same conditions and 0; second, the mean absolute error between the output probability result of the wavefunction used as the label input after resimulating a high-resolution electron microscope image under the same conditions and 1.

[0017] In some embodiments of the present invention, the error of the discriminator backpropagation is expressed as:

[0018] ;

[0019] in, The first term is the mean absolute error. This is the mean absolute error of the second term.

[0020] In some embodiments of the present invention, the atomic positions of the amplitude and phase images of the wave function correspond to the atomic structures of the upper and lower surfaces of the single-walled carbon nanotube relative to the incident beam during imaging.

[0021] In some embodiments of the present invention, the three-dimensional structure of a single-walled carbon nanotube is reconstructed by filling in the atomic structure information of the tube wall based on the atomic positions of the upper and lower layers, including the following steps:

[0022] Calculate the diameter of single-walled carbon nanotubes based on atomic positions and high-resolution electron microscopy images;

[0023] The single-walled carbon nanotube is approximated as a cylinder, and a center line is drawn along the length of the tube in the middle of the tube.

[0024] The distance from each atom to the center line is calculated based on the two-dimensional coordinates of each atom, and the third-dimensional height coordinates of each atom are calculated. The height of the center cross section of the single-walled tube is set to 0, and the height coordinates of the upper and lower surfaces are distinguished.

[0025] Establish a base plane that is perpendicular to the direction of electron beam incidence and has a height equal to the minimum height of all atoms in the carbon nanotube;

[0026] The cylindrical carbon nanotubes are flattened onto the base plane to obtain the atomic coordinates after unfolding into a plane;

[0027] Find the average distance between the nearest neighbor carbon atoms on the carbon nanotube;

[0028] Based on the average distance, complete the atomic positions of the two sides of the single-walled carbon nanotube where the atomic positions cannot be determined;

[0029] The three-dimensional coordinates of the sidewall atoms are calculated from the two-dimensional atomic coordinates on the base plane, based on the tube diameter and centerline information. These coordinates are then combined with the experimentally measured three-dimensional positions of all atoms to form a complete three-dimensional atomic structure.

[0030] This invention uses a generative adversarial network model from deep learning to recover the output wave function of a single-walled carbon nanotube from a single high-resolution electron microscope image, effectively removing distortion caused by lenses and recovering lost phase information.

[0031] This invention utilizes a high-resolution electron microscopy simulation method to simulate the amplitude and phase images of the wavefunction after the incident electron wavefunction passes through carbon nanotubes. It also simulates the modulation function of the transmission electron microscope lens and the high-resolution electron microscopy image. Through a trained network, the amplitude and phase images of the wavefunction are predicted from a single high-resolution electron microscopy image. The amplitude and phase images reflect the atomic structure of the lower and upper surfaces of the single-walled carbon nanotubes, respectively. Therefore, it can be used to reconstruct the three-dimensional structural information of single-walled carbon nanotubes, solving the problem in traditional methods that require a series of defocused high-resolution images or a series of tilted high-resolution images to recover the wavefunction or three-dimensional atomic structure.

[0032] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0033] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0034] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0035] Figure 1 This is an overall flowchart of the deep learning-based three-dimensional structure reconstruction method for single-walled carbon nanotubes of the present invention.

[0036] Figure 2 This is a diagram of the convolution module of the generator of the present invention.

[0037] Figure 3 This is a diagram of the downsampling module of the generator of the present invention.

[0038] Figure 4 This is a diagram of the upsampling module of the generator of the present invention.

[0039] Figure 5 This is a schematic diagram of the discriminator of the present invention.

[0040] Figure 6 This is an example of a simulated high-resolution image of the present invention.

[0041] Figure 7 This is an example of the predicted wavefunction amplitude image of the present invention.

[0042] Figure 8 This is an example of the phase image of the predicted wavefunction of the present invention.

[0043] Figure 9 This is a schematic diagram of a cylindrical carbon nanotube in one embodiment of the present invention, indicating the meaning of the upper and lower surfaces and sidewalls.

[0044] Figure 10 This is an incomplete coordinate diagram of the base plane carbon nanotubes in one embodiment of the present invention.

[0045] Figure 11 This is a completed coordinate diagram of the base plane carbon nanotubes in one embodiment of the present invention.

[0046] Figure 12 This is a three-dimensional coordinate diagram of the carbon nanotubes finally reconstructed in one embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0048] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0049] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0050] In this embodiment of the invention, the three-dimensional atomic structure of carbon nanotubes is reconstructed from an electron microscope image based on an artificial intelligence algorithm. More specifically, by using a generative adversarial network model in deep learning, the outgoing wave function of a single-walled carbon nanotube is recovered from a high-resolution electron microscope image, effectively removing the distortion caused by the lens and restoring the lost phase information, thereby reconstructing the three-dimensional atomic structure of carbon nanotubes from an electron microscope image.

[0051] The outgoing wave function refers to the outgoing wave function of an electron beam, approximately planar, after it reaches the back surface of a single-walled carbon nanotube. Since the outgoing wave function is free from distortion caused by lens magnification and contains the phase information of the wave function lost during CCD recording, its recovery can improve the interpretable resolution of the image. More importantly, the inventors discovered that the amplitude and phase components of the outgoing wave function precisely correspond to the structural information of the upper and lower surfaces of the carbon nanotube. Therefore, the near-three-dimensional atomic structure can be further solved from the recovered outgoing wave function. If the upper and lower surfaces of the carbon nanotube contain different defects, its three-dimensional atomic structure will help explain the physical properties exhibited by the carbon nanotube.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a method for reconstructing the three-dimensional structure of single-walled carbon nanotubes based on deep learning, comprising steps S1-S6:

[0053] Step S1: Construct a transmission electron microscopy (TEM) simulation image of a single-walled carbon nanotube as a dataset.

[0054] The constructed dataset can be used as a training dataset to train a generative adversarial network model. In this step, single-walled carbon nanotubes (SUVs) can be constructed first, and then corresponding transmission electron microscopy (TEM) simulation images can be generated based on the constructed SUVs. In this embodiment of the invention, the constructed SUVs can include defect-free SUVs of various chirality and defective SUVs of various chirality, with different chiral SUVs having different chiral indices. Accordingly, the TEM simulation images of the constructed SUVs are TEM simulation images of defect-free and defective SUVs of various chirality.

[0055] Step S2: Train the generative adversarial network model using the constructed dataset.

[0056] In some embodiments of this invention, mean squared error, mean absolute error, and structural similarity can be used, and one or more of these can be selected as the loss function for the generator of the generative adversarial network (GAN) model to train the GAN model. Mean squared error compares the pixel-level errors between images and is a commonly used loss function in deep learning; mean absolute error is more robust than mean squared error; structural similarity describes the differences in brightness, contrast, and structure between images, and using structural similarity as the error function can improve the overall quality of the reconstructed wavefunction amplitude and phase images and accelerate network convergence. As an example, mean absolute error is used as the loss function for the discriminator of the GAN model, and the GAN model is trained using a constructed dataset until the generator's error converges to a small value while the discriminator's error stabilizes at around 0.5.

[0057] When the generator's error includes mean square error, the amplitude and phase images of the wave function predicted by the generator are compared with the amplitude and phase images of the wave function used as the label input, respectively. This also refers to the predicted wave function being used to simulate a high-resolution electron microscope image under the same conditions and compared with the high-resolution electron microscope image used as the label input, and the mean square error between the images is calculated.

[0058] When the generator's error includes image structural similarity, the structural similarity between images is calculated by comparing the amplitude and phase images of the wave function predicted by the generator with the amplitude and phase images of the wave function used as the label input.

[0059] When the generator error includes mean absolute error, it is the error between the output probability result of the simulated image after inputting the simulated image into the discriminator and 1, which is the wave function predicted by the generator and then simulated under the same conditions.

[0060] The error of the discriminator can include two aspects of mean absolute error. One is the mean absolute error between the output probability result of the high-resolution electron microscope image after the wave function predicted by the generator is used to simulate the high-resolution electron microscope image under the same conditions and 0, and the other is the mean absolute error between the output probability result of the high-resolution electron microscope image after the wave function used as the tag input is used to simulate the high-resolution electron microscope image under the same conditions and 1.

[0061] After training, the generative adversarial network model of this invention will have the ability to predict single-walled carbon nanotubes with various chiral indices, and will also have the ability to predict defects in single-walled carbon nanotubes.

[0062] Step S3: Obtain transmission electron microscopy (TEM) images of single-walled carbon nanotubes.

[0063] Existing high-resolution imaging techniques can be used to record electron microscopic images of single-walled carbon nanotubes. The recorded electron microscopic images of carbon nanotubes must have sufficiently high resolution.

[0064] Step S4: Using the trained generative adversarial network, predict the amplitude and phase images of the wave function corresponding to the transmission electron microscopy experimental image or the transmission electron microscopy simulation image of the single-walled carbon nanotube.

[0065] In one embodiment of the present invention, a transmission electron microscope (TEM) image of a single-walled carbon nanotube can be used as input. If the input is a TEM image, it is first normalized, and then the processed image is input into a generative adversarial network (GAN). The GAN outputs the predicted amplitude and phase images of the wave function. When using an TEM image for prediction, it can be used for the analysis of real data.

[0066] In another embodiment of the invention, a transmission electron microscope (TEM) image of a single-walled carbon nanotube can also be used as input. If the input is a TEM image, noise addition and normalization processing are required. Then, the processed image is input into a Generating Adversarial Network (GAN). The GAN outputs the amplitude and phase images of the predicted wavefunction. When using a TEM image as input and performing prediction, this is mainly used to assess the network's prediction performance.

[0067] Step S5: Analyze the atomic positions on the upper and lower surfaces of the carbon nanotube based on the wave function amplitude image and phase image.

[0068] The amplitude and phase images of the wavefunction have atomic-scale resolution, allowing the position of each atom to be determined from the images. More specifically, since atoms appear as bright spots with a Gaussian intensity distribution in the amplitude and phase images of the wavefunction, the two-dimensional position of the atom can be calculated by determining the center position of the bright spot.

[0069] The atomic positions of the amplitude and phase images of the wave function correspond to the atomic structures of the upper and lower surfaces of the single-walled carbon nanotube relative to the incident beam during imaging.

[0070] Step S6: Based on the atomic positions of the upper and lower surfaces, fill in the atomic structure information of the tube wall to reconstruct the three-dimensional structure of the single-walled carbon nanotube.

[0071] In one embodiment of the present invention, step S1 may further include the following steps S13-S16:

[0072] Step S11: Construct defect-free single-walled carbon nanotube atomic structures with different chiralities.

[0073] Existing atomic structure building software can be used, or the positions of each atom in a single-walled carbon nanotube can be calculated based on the geometric relationship of the atomic structure of the single-walled carbon nanotube, to build defect-free single-walled carbon nanotube atomic structures with different chiralities.

[0074] Step S12: Modify the atomic structure of single-walled carbon nanotubes to construct atomic structures of single-walled carbon nanotubes with different chiralities and defects.

[0075] For example, several atoms can be added or deleted from the wall of a single-walled carbon nanotube. If the number of atoms added or deleted at a local location is equal to or greater than two atoms, the structure of the carbon nanotube can be further optimized using existing molecular dynamics software to construct the atomic structure of a single-walled carbon nanotube with defects.

[0076] Step S13 involves simulating the outgoing wavefunction of high-energy electrons after they pass through the single-walled carbon nanotubes (SUVs) for both defect-free and defective SUVs with different chiralities, based on the theory of high-resolution electron microscopy. High-resolution electron microscopy images are then simulated under reasonable experimental conditions. Various software programs can be used for high-resolution image simulation, such as QSTEM, Mactemps, and ToTEM. One embodiment of this invention uses ToTEM software.

[0077] In image simulation, a plane wave is incident on a certain carbon nanotube. Using the multilayer method, the carbon nanotube is divided into several thin layers. The wave function is continuously transmitted and propagated between the layers. The wave function that is emitted after reaching the bottom of the carbon nanotube is called the emitted wave function. The emitted wave function is a complex function that can be decomposed into an amplitude image and a phase image.

[0078] Then, the process of the emitted wave function being magnified and recorded by the lens is simulated. The existing imaging formula is used to simulate the image of a high-resolution transmission electron microscope (HRTEM). The simulated HRTEM image is randomly cropped to a predetermined size, such as 144×144 pixels squared. Each simulated image is randomly cropped only once.

[0079] The simulation process needs to take into account the spherical aberration and defocusing conditions of the lens. These conditions are randomly generated and should be reasonable. For example, it is necessary to consider whether the simulated high-resolution electron microscope image contains enough high-frequency information to be able to resolve individual carbon atoms. In other words, the magnitude of spherical aberration and defocusing should be reasonable. Important imaging conditions are recorded, and a complex matrix reflecting the lens transfer function is simulated. The matrix size is, for example, 144×144 pixels squared, but the present invention is not limited to this.

[0080] Step S14: Obtain a dataset of transmission electron microscopy (TEM) simulation images for various atomic structures, different electron beam incident directions, and different TEM experimental conditions.

[0081] In each set of transmission electron microscopy (TEM) simulated image data, the output wavefunction, lens transfer function (LTF) images, and high-resolution electron microscopy simulated images correspond to each other. As an example, each set of data can contain the following four parts: high-resolution TEM simulated image data of single-walled carbon nanotube structures (referred to as simulated images), wavefunction amplitude image data corresponding to each simulated image, wavefunction phase image data corresponding to each simulated image, and complex image data of the transmission transfer function corresponding to each simulated image. Therefore, the dataset contains four parts: a subset of high-resolution TEM simulated image data of single-walled carbon nanotube structures, a subset of corresponding wavefunction amplitude image data, a subset of corresponding wavefunction phase image data, and a subset of complex image data of the transmission transfer function.

[0082] After obtaining the dataset, the data in the dataset can be further preprocessed in steps S15 and S16 as follows.

[0083] Step S15: Add noise to the simulated high-resolution electron microscope image.

[0084] Adding Poisson noise to a subset of high-resolution analog image data and normalizing the image pixel values, while also normalizing the image pixel values ​​of a subset of wave function amplitude image data, can reduce the randomness of the input image pixel distribution and help accelerate network convergence.

[0085] Step S16: Augment the dataset.

[0086] In some embodiments of the present invention, a set of images can be horizontally or vertically flipped to generate a new set of image data, thereby expanding the dataset, increasing data diversity, and preventing network overfitting.

[0087] This method allows for the rapid generation of sufficient training data to train generative adversarial network models.

[0088] In some embodiments of the present invention, during the training phase of the generative adversarial network (GAN) model, the generator of the GAN model can use a UNet++ model as the backbone network, specifically including a convolutional module, a downsampling module, an upsampling module, and a Dropout layer. For example... Figure 2 As shown, the convolutional module may include a convolutional layer with a 3×3 kernel, a connected BatchNorm layer, and a LeakyReLU activation function with a negative slope angle of 0.2. Figure 3 As shown, the downsampling module may include a MaxPool layer with a 2×2 pooling kernel and a convolutional module connected to it. Figure 4 As shown, the upsampling module may specifically include a Transposed convolutional layer with a 2×2 kernel and two convolutional modules connected to it.

[0089] The generator of the generative adversarial network model is based on the Unet++ architecture, specifically consisting of four paths that each acquire the output. The generator's input is a single-channel ×144×144 image, which then passes through two convolutional modules to obtain a 64-channel ×144×144 feature matrix. This feature matrix serves as the input to all four paths of the generator.

[0090] The generator's four paths are as follows: the first path passes through a first downsampling module and an upsampling module, and finally through a convolution module to obtain the first 2-channel ×144 ×144 output image; the second path passes through the first two downsampling modules and two upsampling modules, and finally through a convolution module to obtain the second 2-channel ×144 ×144 output image; the third path passes through the first three downsampling modules and three upsampling modules, and finally through a convolution module to obtain the third 2-channel ×144 ×144 output image; the fourth path passes through all four downsampling modules and four upsampling modules, and finally through two convolution modules to obtain the fourth 2-channel ×144 ×144 output image; finally, the four output images are summed with weights of 0.1:0.2:0.3:0.4 to obtain the final reconstructed 2-channel ×144 ×144 wavefunction image.

[0091] like Figure 5 As shown, in one embodiment of the present invention, the discriminator of the generative adversarial network model may specifically include: a convolutional layer with an input of 1 channel × 144 × 144, a convolutional kernel of 3 × 3, and an output of 64 channels × 144 × 144, and a BatchNorm layer and a Softplus activation function layer connected to the convolutional layer; then connected to a convolutional layer with an input of 64 channels × 144 × 144, a convolutional kernel of 1 × 1, and an output of 1 channel × 144 × 144, and a BatchNorm layer and a Softplus activation function layer connected to the convolutional layer; finally connected to a fully connected layer with an input of a 20736-dimensional vector and an output of a single scalar value, and a Sigmoid activation function layer connected to it.

[0092] The discriminator finally outputs a probability value between 0 and 1. The closer the value is to 1, the greater the probability that the discriminator identifies the input source as a real wave function simulation image. The closer the value is to 0, the greater the probability that the discriminator identifies the input source as a generated wave function simulation image.

[0093] A generative adversarial network model is trained using a noisy and augmented dataset, with k-fold cross-validation used during training to prevent overfitting. In one embodiment of this invention, the k-fold cross-validation specifically includes:

[0094] Before inputting the data into the model, the dataset is first divided into k equal parts. The value of k can be selected according to the actual situation. Generally, k is greater than 2. In this embodiment, k is 10. Each time, one part is selected as the validation set, and the remaining k-1 parts are used as the training set. In this way, k models can be trained. Finally, the model with the smallest average validation set error is selected as the optimal model for training.

[0095] The generator was trained using a stepped adjustment method, initially using 3e. -4 The learning rate is adjusted to 2e when the training progress reaches 40%. -4 When the training progress reaches 50%, adjust the learning rate to 5e. -5 The learning rate remains constant at 2e during discriminator training. -5 .

[0096] In this embodiment of the invention, the generator error includes mean square error, which refers to the comparison between the amplitude and phase images of the wave function predicted by the generator and the amplitude and phase images of the wave function used as the label input, respectively. It also refers to the comparison between the predicted wave function and the high-resolution electron microscope image under the same conditions, and the high-resolution electron microscope image used as the label input, to calculate the mean square error between the images.

[0097] In addition, the generator's error may also include image structure similarity, which refers to comparing the amplitude and phase images of the wave function predicted by the generator with the amplitude and phase images of the wave function used as the label input to calculate the structural similarity between the images.

[0098] In addition, the generator error may also include mean absolute error, which refers to the mean absolute error between the output probability result of the high-resolution electron microscope image after the generator predicts the wave function and then simulates the high-resolution electron microscope image under the same conditions and inputs it into the discriminator and 1.

[0099] In this embodiment, the mean square error corresponding to the predicted wavefunction and the wavefunction used as the label input is calculated. The mean square error between the high-resolution electron microscope image simulated by the predicted wavefunction and the high-resolution electron microscope image used as the label input. The similarity score (ssim) between the predicted wavefunction and the wavefunction used as the label is then calculated using 1-ssim to obtain the structural similarity value. The mean absolute error between the output probability result of the predicted wavefunction re-simulated simulated image input to the discriminator and 1. The error of the final generator backpropagation for;

[0100] ;

[0101] In some embodiments of the present invention, the error of the discriminator may include two aspects of mean absolute error: the first mean absolute error refers to the mean absolute error between the output probability result of the generator predicting the wavefunction and simulating the high-resolution electron microscope image under the same conditions and inputting the simulated high-resolution electron microscope image into the discriminator and 0; the second mean absolute error refers to the mean absolute error between the output probability result of the generator predicting the wavefunction as the tag input and simulating the high-resolution electron microscope image under the same conditions and inputting it into the discriminator and 1.

[0102] In some embodiments of the present invention, the first term, mean absolute value error, is denoted as... The second term, the mean absolute error, is labeled as... The error of the final discriminator backpropagation for:

[0103] ;

[0104] The wavefunction predicted by the generator is then used to simulate a high-resolution electron microscope image under the same conditions. The specific formula is as follows:

[0105] ;

[0106] in, To record high-resolution analog images, For the spatial vector of the image, For image wave function, The transmission transfer function is... Represents the frequency space vector. This indicates that the inverse Fourier transform is performed, specifically meaning that the high-resolution analog image is equal to the product of the wave function and the transmission transfer function in the frequency domain. Then, the inverse Fourier transform is used to transform to the spatial domain, and finally, the modulus average is taken to obtain the high-resolution analog image.

[0107] Using a trained generative adversarial network, simulated electric images based on single-walled carbon nanotubes (see...) Figure 6 It can predict the amplitude image of the wave function corresponding to the image (see...). Figure 7 ) and phase image (see Figure 8 ).

[0108] The atomic positions on the upper and lower surfaces of carbon nanotubes can be analyzed from the wave function amplitude and phase images.

[0109] In one embodiment of the present invention, the two-dimensional atomic positions on the amplitude and phase images can be determined using existing CalAtom software.

[0110] The reconstructed wavefunction amplitude image reflects the atomic information of the lower surface of the carbon nanotube, while the reconstructed phase image reflects the atomic information of the upper surface of the carbon nanotube.

[0111] In embodiments of the present invention, the atomic structure information of the tube wall can be filled in according to the atomic positions of the upper and lower layers to reconstruct the three-dimensional structure of the single-walled carbon nanotube. Figure 9 The diagram illustrates the meaning of the upper and lower surfaces and sidewalls of a carbon nanotube when it is approximately cylindrical. The sidewalls are approximately parallel to the incident direction of the electron beam, while the upper and lower surfaces are approximately perpendicular to the incident electron beam.

[0112] In some embodiments of the present invention, the specific steps for reconstructing the three-dimensional structure of single-walled carbon nanotubes may include:

[0113] i) Calculate the diameter of single-walled carbon nanotubes based on atomic positions and high-resolution images.

[0114] ii) Approximate the single-walled carbon nanotube as a cylinder and draw a center line along the length of the tube in the middle.

[0115] The distance from each atom to the center line is calculated based on its two-dimensional coordinates, thereby converting the three-dimensional height coordinates of each atom. Note that the height of the upper and lower surfaces is distinguished by taking the cross-section of the center of the single-walled tube as 0.

[0116] iii) Establish a base plane that is perpendicular to the direction of electron beam incidence and has a height equal to the minimum height of all atoms in the carbon nanotube.

[0117] iv) Flatten the cylindrical carbon nanotubes onto the base plane to obtain the atomic coordinates after unfolding into a plane.

[0118] Figure 10 The atomic coordinates after unfolding into a plane are shown. The y-axis is the tube length direction, and the x-axis is exactly the circumference of the cylinder after unfolding. In this embodiment, the unfolded structure of carbon nanotubes includes defects. The hollow point set shows defect 57, and the star-shaped point set shows defect 58.

[0119] v) Calculate the average distance between nearest-neighbor carbon atoms on the carbon nanotube as accurately as possible.

[0120] This value is primarily measured by the interatomic spacing between the upper and lower surfaces perpendicular to the incident beam direction, without considering the carbon-carbon spacing near the defects.

[0121] vi) Based on the average distance, complete the atomic positions on the two sides of the single-walled carbon nanotube where the atomic positions cannot be determined. Figure 11 The diagram illustrates the atomic structure on the completed base plane.

[0122] vii) Using the two-dimensional atomic coordinates on the base plane, and based on the tube diameter and centerline information, the three-dimensional coordinates of the sidewall atoms are calculated.

[0123] viii) Substitute the three-dimensional coordinates of all atoms measured in step ii) into the three-dimensional coordinates of the atoms on the added sidewalls to construct the complete three-dimensional atomic structure of the carbon nanotube.

[0124] This embodiment is provided by Figure 12 Indication.

[0125] As described above, this invention utilizes a high-resolution electron microscope simulation method to simulate the amplitude and phase images of the wavefunction after the incident electron wavefunction passes through carbon nanotubes. It also simulates the modulation function of the transmission electron microscope lens and the high-resolution electron microscope image. Through a trained network, the amplitude and phase images of the wavefunction are predicted from a single high-resolution electron microscope image. The amplitude and phase images reflect the atomic structures of the lower and upper surfaces of the single-walled carbon nanotubes, respectively. Therefore, it can be used to reconstruct the three-dimensional structural information of single-walled carbon nanotubes, solving the problem in traditional methods that require the use of a series of defocused high-resolution images or a series of tilted high-resolution images to recover the wavefunction or three-dimensional atomic structure.

[0126] This invention leverages the powerful learning capabilities of generative adversarial networks, enabling the reconstruction of data with other chiral types even when the dataset contains a limited number of chiral types. It can handle single-walled carbon nanotube data with different chiralities in practical applications.

[0127] In this invention, the loss function of the generator in the generative adversarial network applies structural similarity, which plays a better role in reconstructing the edge data of single-walled carbon nanotubes, greatly accelerates the convergence speed of the generator network, and improves the quality of the finally reconstructed wave function, thereby improving the efficiency and accuracy of subsequent calculations of the three-dimensional structure of carbon nanotubes.

[0128] Unlike typical deep learning networks, this invention incorporates a special computational process before network input and output. For example, for the generator, the simulation method used is to simulate high-resolution electron micrographs using two wave functions, and then compare the errors between the images. This ensures that the images are not affected by noise when comparing errors. Therefore, when the network is used for prediction, the predicted wave function will also be less affected by noise.

[0129] In accordance with the above method, the present invention also provides a system for reconstructing the three-dimensional atomic structure of carbon nanotubes from an electron image based on an artificial intelligence algorithm. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device performs the steps of the method as described above.

[0130] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0131] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0132] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0133] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for reconstructing the three-dimensional structure of single-walled carbon nanotubes based on deep learning, characterized in that, Includes the following steps: Electron microscopy simulation images of single-walled carbon nanotubes were constructed as a dataset; The generative adversarial network model is trained using the constructed dataset; Using a trained generative adversarial network, the amplitude and phase images of the wave function corresponding to the image are predicted based on experimental and / or simulated electric images of single-walled carbon nanotubes. Based on the amplitude and phase images of the wave function, the atomic positions on the upper and lower surfaces of the carbon nanotube are analyzed. Based on the atomic positions of the upper and lower layers, fill in the atomic structure information of the tube wall to reconstruct the three-dimensional structure of the single-walled carbon nanotube; The dataset of simulated electron microscopy images of constructed single-walled carbon nanotubes includes: constructing atomic structures of defect-free single-walled carbon nanotubes with different chirities; modifying the atomic structure of single-walled carbon nanotubes to construct atomic structures of defective single-walled carbon nanotubes; simulating the outgoing wave function of high-energy electrons after the incident wave function passes through the single-walled carbon nanotubes, based on the theory of high-resolution electron microscopy, for both defect-free and defective single-walled carbon nanotubes with different chirities, and simulating high-resolution electron microscopy images based on preset experimental conditions; obtaining a dataset of simulated transmission electron microscopy images for various atomic structures, different incident directions of electron beams, and different electron microscopy experimental conditions; the dataset contains four parts: a subset of high-resolution simulated transmission electron microscopy images of single-walled carbon nanotube structures, a subset of wave function amplitude images corresponding to each simulated image, a subset of wave function phase images corresponding to each simulated image, and a subset of complex images of the transmission transfer function corresponding to each simulated image; preprocessing the dataset, including adding noise to the simulated high-resolution electron microscopy images and augmenting the dataset. When training a generative adversarial network (GAN) model using a constructed dataset, the loss function of the generator in the GAN model includes one or more of the following: mean squared error, mean absolute error, and structural similarity. The discriminator's error includes two mean absolute errors: the first mean absolute error is the average absolute error between the output probability result of the generator's predicted wavefunction after simulating a high-resolution electron microscope image under the same conditions and 0, and the second mean absolute error is the average absolute error between the output probability result of the generator's predicted wavefunction after simulating a high-resolution electron microscope image under the same conditions and 1, and the second mean absolute error is the average absolute error between the output probability result of the generator after simulating a high-resolution electron microscope image under the same conditions and 1, using the wavefunction as the label input. Based on the atomic positions of the upper and lower layers, the atomic structure information of the tube wall is filled in to reconstruct the three-dimensional structure of the single-walled carbon nanotube. This includes the following steps: Calculate the diameter of the single-walled carbon nanotube based on the atomic positions and high-resolution electron microscopy images; approximate the single-walled carbon nanotube as a cylinder and draw a center line along the tube's length; calculate the distance from each atom to the center line based on its two-dimensional coordinates, converting this to the third-dimensional height coordinates of each atom, setting the central cross-section of the single-walled tube as 0 height to distinguish the height coordinates of the upper and lower surfaces; establish a base plane perpendicular to the electron beam incident direction, with its height equal to the minimum height of all atoms in the carbon nanotube; flatten the cylindrical carbon nanotube onto the base plane to obtain the atomic coordinates after unfolding into a plane; calculate the average distance between the nearest neighbor carbon atoms on the carbon nanotube; based on the average distance, fill in the atomic positions on the two side walls of the single-walled carbon nanotube where the atomic positions cannot be determined; convert the two-dimensional atomic coordinates on the base plane, based on the tube diameter and the center line information, to calculate the three-dimensional coordinates of the side wall atoms, and combine this with the experimentally measured three-dimensional positions of all atoms to form a complete three-dimensional atomic structure.

2. The method according to claim 1, characterized in that, The generative adversarial network model includes a generator and a discriminator; the generator uses the UNet++ model as the backbone network, specifically including a convolutional module, a downsampling module, an upsampling module, and a Dropout layer.

3. The method according to claim 2, characterized in that, The discriminator specifically includes convolutional layers, fully connected layers, and activation function layers.

4. The method according to claim 1, characterized in that, The error of the discriminator's backpropagation is expressed as: ； in, The first term is the mean absolute error. This is the mean absolute error of the second term.

5. The method according to claim 1, characterized in that, The atomic positions of the amplitude and phase images of the wave function correspond to the atomic structures of the upper and lower surfaces of the single-walled carbon nanotube relative to the incident beam during imaging.

6. A system for reconstructing the three-dimensional atomic structure of carbon nanotubes from an electron image, comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images