A computer-implemented method for identifying organic molecules from atomic force microscope images by generating a two-dimensional colored RGB structural representation of the molecule in the form of a ball-and-stick depiction

Abstract

The present invention relates to a computer-executed method for identifying organic molecules from atomic force microscope (AFM) images and generating a two-dimensional (2D) colored RGB structural representation of the organic molecules, in particular, in the form of a ball-and-stick depiction of atoms and the distances between said atoms using a trained conditional generative adversarial network (CGAN). Accordingly, the present invention is of interest to users and manufacturers of AFM, in the field of nanotechnology, in particular in fields related to chemical reactions on surfaces.

Description

Detailed Description of the Invention

[0001] 〔Technical Field〕 The present invention relates to a computer-executed method for identifying organic molecules from atomic force microscope (AFM) images by generating a two-dimensional colored RGB structural representation of the organic molecules in the form of a ball-and-stick depiction using a trained conditional generative adversarial network (CGAN).

[0002] Therefore, the present invention is of interest in the field of nanotechnology, particularly in areas related to chemical reactions on surfaces, and is thus of interest to users and manufacturers of AFM.

[0003] 〔Background Art〕 Molecules are playing an increasingly important role in modern society. In addition to their traditional uses as important materials in the food, textile, chemical, and pharmaceutical industries, in the latest trends of nanotechnology, molecules are considered to be the ultimate miniaturized elements. Single molecules currently function as electrical and optical devices and force actuators, driving the development of entirely new research fields such as molecular electronics, energy harvesting, and molecular motors. In addition to reducing the size and energy consumption of molecules, molecules offer diversity in molecular structure and composition, and the system can be finely tuned according to the intended function. The versatility requires new synthetic routes, and in addition to traditional strategies in solution chemistry, fundamentally new concepts such as surface chemistry have been added. To date, the identification of molecular structure and composition has largely relied on spectroscopic techniques such as nuclear magnetic resonance (NMR), mass spectrometry (MS), and X-ray diffraction that measure molecular ensembles. These are clearly not suitable for characterizing reaction products on surfaces. However, there are also limitations in the case of conventional solution-based synthesis.

[0004] The atomic force microscope (AFM) has become one of the important tools for nanoscale material and biological system imaging and manipulation when combined with dynamic operation modes. The AFM operating in the frequency modulation mode (FM), generally known as non-contact AFM, achieves true atomic-scale resolution. The use of a metal probe functionalized at the tip with a CO molecule enables access to the internal structure of molecules with unprecedented resolution. The main contrast mechanism of AFM with an inert probe such as CO is the Pauli repulsion due to the overlap of the electron densities of the probe and the sample. This repulsive force causes a change in the vibration frequency of the cantilever holding the probe due to the interaction between the probe and the sample (positive frequency shift), which is observed as bright features in a constant-height AFM image on top of the positions, sizes, and bonds (interatomic distances) of the atoms reflecting the molecular structure. To explain the contrast of the observed images, the accuracy of AFM simulation models has been improved. These simulation models contribute to, for example, the role of the tilt of CO, the influence of other contributions to the interaction between the probe and the sample such as electrostatic forces, the role of the charge distribution of the CO-metal probe, the identification of bond order, and the elucidation of the short-range chemical and electrostatic interactions in intermolecular bond imaging.

[0005] High-resolution experimental (HR) AFM images have opened the way to the identification of natural products such as brightfusic acid, along with the ability to handle individual molecules. While methods such as NMR can provide information about a part of the structure, they cannot provide the overall structure (K. O. Hanssen, et al., Angew. Chem. Int. Ed. 51, 12238 (2012)). HR-AFM is also crucial for imaging intermediates (including radicals) and final products generated by reactions on the surface, shedding light on the generation process and reaction pathway (S. Kawai, et al., ACS nano 11, 8122 (2017)). This technique has been able to break down more than 100 types of molecules of asphaltenes, the solid component of crude oil (B. Schuler, G. Meyer, D. Pena, O. C. Mullins, L. Gross, J. Am. Chem. Soc. 137, 9870 (2015)). In the case of asphaltenes, since they basically deal with polycyclic aromatic hydrocarbons based on C and H atoms, molecular identification in all previous cases has been supported by important information regarding the properties of the molecules involved. Despite the wealth of information provided by HR-AFM experiments and these advances in the interpretation of the observed contrast, the complete identification of molecular systems, that is, determining the structure and composition based solely on HR-AFM images without prior information, remains an unsolved problem.

[0006] There has been little research attempting to address this problem by processing AFM images using artificial intelligence (AI) techniques (B. Alldritt, et al., Sci. Adv. 6, eaay6913 (2020) and J. Carracedo-Cosme, C. Romero-Muñiz, R. Perez, Nanomaterials 11, 1658 (2021)). Deep learning (DL) is currently routinely used for image classification, interpretation, description, and analysis, providing machines with capabilities that surpass those of humans. The ability of deep learning to recognize patterns can, in principle, be utilized to characterize the structures of molecular systems. Gordon et al. (O. M. Gordon, J. E. Hodgkinson, S. M. Farley, E. L. Hunsicker, P. J. Moriarty, Nano Lett. 20, 7688 (2020)) implemented a model to automatically detect spatially correlated patterns from various sets of AFM images of self-assembled nanoparticles. However, complete atom-by-atom identification was very difficult. This is because the effects of both geometric shape and chemical composition contribute in a very complex manner to the determination of the three-dimensional molecular charge density that ultimately causes the AFM contrast.

[0007] Due to this complexity, the task of molecular identification exceeds the capabilities of human experts, as deterministically shown in the example presented in Figure 2. The structures of molecules such as rings and side-chain groups are completely hidden by their chemical composition and non-planar arrangement. Alldritt et al. (B. Alldritt, et al., Sci. Adv. 6, eaay6913 (2020)) developed a convolutional neural network (CNN) aimed at determining molecular shape from AFM images. Its performance was excellent for the structures of quasi-planar molecules, even when using the experimental results and the algorithm directly. For three-dimensional structures, information on the positions and sizes of atoms closer to the tip could be collected in a height range of 150 pm. However, conclusive results were not obtained for the identification of functional groups.

[0008] In our previous study (J. Carracedo-Cosme, C. Romero-Muniz, R. Perez, Nanomaterials 11, 1658 (2021)), we demonstrated the feasibility of highly accurate automatic molecular classification using DL techniques for a set of 60 planar molecules containing the most common atomic species in organic chemistry, using theoretically simulated AFM images. Furthermore, to include the features of experimental AFM images in the dataset, we proposed a variational autoencoder (VAE)-based approach and significantly improved the accuracy of the model tested with experimental images. However, although this approach shows the potential to recognize both the structure and composition of molecules through AFM images, (i) ordinary classification by CNN only enables a finite-length output, that is, it can classify only a finite number of structures, and (ii) molecules with non-planar adsorption configurations need to be considered, so the overall classification problem has not been solved.

[0009] Therefore, there is a need to develop a new method for processing AFM images to identify molecules. Description of the Invention

[0010] An object of the present invention is to provide a computer-executable method for identifying an organic molecule from an atomic force microscope (AFM) image by generating a two-dimensional (2D) colored RGB structural representation of the organic molecule using a trained conditional generative adversarial network (CGAN). In the computer-executable method of the present invention, a 2D RGB structural representation of an organic molecule is generated in the form of a ball-and-stick depiction, where balls of different colors and sizes represent different chemical atoms, the sticks represent the bonds between atoms, and each ball in the representation is centered at the position occupied by the atom it represents in the AFM image. The 2D RGB structural representation of an organic molecule in the form of a ball-and-stick depiction provides complete information regarding its structure and chemical composition. In the present invention, the different chemical properties of atoms are represented by different colors and sizes proportional to the van der Waals radius of the atoms.

[0011] In the method of the present invention, a trained CGAN is used. A generative adversarial network (GAN) is a machine learning framework that relies on a generator that learns the mapping from a noisy representation (usually a vector) of an input image to an output image, and a discriminator that learns to distinguish the image generated by the generator from a target image. The generator improves the mapping during learning and produces results that cannot be distinguished from "real" images by the discriminator. The discriminator is trained to do its best to detect the fakes of the generator. A conditional GAN (CGAN) is a type of GAN that uses additional conditions to map the output image in addition to the noisy representation of the input image. In the present invention, the generator learns the mapping from the observed input image and its vector representation including random noise to the output image.

[0012] A method for training a CGAN includes the following steps: i) Providing a trained CGAN to a data processing device. The trained CGAN comprises: · A generator network (U-Net architecture) with an encoder-decoder structure, a network comprising blocks of convolutional layers, pooling layers and dropout layers. · A discriminator network comprising convolutional layers and pooling layers, and · An image data generator; ii) Supplying a plurality of AFM grayscale images of a known or predetermined organic molecule at a constant height (e.g., AFM images in the database QUAM - AFM) to the generator network. The shape, contrast, and their variations due to height of the said images indicate the three - dimensional positions and sizes (chemical properties) of the atoms within the organic molecule, and the distances between said atoms. The generator generates a two - dimensional colored RGB structural representation of the positions and sizes of the atoms within the organic molecule, and the distances between said atoms in the form of a ball - and - stick depiction. Balls of different colors and sizes represent different chemical properties of the atoms, and the sticks represent the bonds between said atoms. Each ball in the representation is centered at the position occupied by the atom it represents within the AFM image. iii) Supplying the discriminator network with the same plurality of AFM grayscale images of a known or predetermined organic molecule used in step (ii) for supply to the generator, or alternatively, either the ball - and - stick depiction obtained by the generator network in step (ii), or the actual ball - and - stick depiction of a known or predetermined molecule. The discriminator overlays and compares the segmented patches of the images and depictions, and iv) In the data processing device, the discriminator selects, from the overlaid and segmented patches of the images and depictions, preferably patches of 16×16 pixels, those colored two - dimensional colored RGB structural representations in the form of a ball - and - stick depiction that match the positions and sizes of the atoms and the distances between said atoms in step (iii), and ignores those that do not match the positions and sizes of the atoms and the distances between said atoms.

[0013] For the training of the CGAN, Quasar Science Resources S.L. - Universidad Autonoma de Madrid - Atomic Force Microscopy (QUAM - AFM) (https: / / doi.org / 10.21950 / UTGMZ7) is used.

[0014] QUAM-AFM is a dataset of 165 million AFM images theoretically generated from 686,000 isolated molecules, using 240 combinations of AFM operation parameters (known to depend on 10 types of probe-sample distances, 6 types of oscillation amplitudes, 4 values of the torsional rigidity of CO molecules, and details of the attachment of the tip of the metal probe to the molecule). QUAM-AFM also provides a ball-and-stick depiction of each molecule generated from atomic coordinates. These depictions share the same scale as the scale used in the AFM images. When two images are overlaid, each ball in the depiction is centered on the position occupied by the atom it represents in the AFM image.

[0015] The QUAM-AFM dataset contains organic molecules, excluding compounds that do not have a pure molecular form such as organic salts, inorganic compounds, and polymers. The selected molecules contain, in addition to the four basic elements of organic chemistry (carbon, hydrogen, nitrogen, oxygen), sulfur, phosphorus, halogen atoms (fluorine, chlorine, bromine, iodine), etc., which are less common but still frequent in organic compounds. The largest molecule in the QUAM-AFM database has a total of 85 atoms.

[0016] Very small molecules, i.e., molecules containing less than 8 atoms, are not suitable for identification by AFM alone because of their very high surface mobility and very diverse adsorption configurations and are not included in the QUAM-AFM database. Furthermore, very large molecules with structures that do not fit into a square-based cell with a side length of 24 Å are not included in the QUAM-AFM dataset.

[0017] The QUAM-AFM database is sp 3Since it includes an aliphatic chain having a carbon atom (methyl group) in the side chain, it is limited to a quasi-planar molecule that displays only the change in height up to 1.83 Å along the z-axis. QUAM-AFM includes aliphatic, cyclic and aromatic compounds, especially a number of hydrocarbons (alkanes, alkenes, alkynes, etc.) and all typical organic families (alcohols, thiols, ethers, aldehydes and ketones, carboxylic acids, amines, amides, imines, esters, nitriles, nitro and azo compounds, halocarbons and acyl halides, etc.).

[0018] The 686K structure of QUAM-AFM is divided into a training set, a validation set, and a test set of 581K, 24K, and 81K structures, respectively. Note that in the CGAN test, the test set is chosen to be particularly large in order to have a sufficient variety of molecular structures and chemical compositions to evaluate the performance of CGAN.

[0019] During the training of CGAN, the generator and the discriminator compete in a zero-sum game. The generator learns both to deceive the discriminator and to generate an image as close as possible to the ball-and-stick depiction, and the discriminator learns to infer whether the second input image is real or fake. This competition allows the generator and the discriminator to significantly improve their performance during training. The discriminator only serves to force the generator to improve. Therefore, once this goal is achieved, the discriminator is discarded.

[0020] The generator with an encoder-decoder structure is composed of a series of similar blocks, and the main differences are the number of kernels applied in each convolution and the size of each input (Figure 1). The input is composed of a stack of 10 grayscale AFM images (single channel), and the corresponding ball-and-stick depiction is the output. The first layer of the generator is a dropout layer with a rate of 0.5 and two 3D convolutional layers for processing the image stack. Such a dropout layer with a high rate is important to enable the CGAN to generalize and make accurate predictions when dealing with experimental images. The first 3D convolution contains 64 kernels, each kernel having a size of (4, 3, 3) and being applied with a stride of (3, 1, 1) and padding. The second 3D convolution also has 64 kernels, but in this case, the kernels have a size of (4, 4, 4) and are applied with a stride of (4, 2, 2). The output of the second convolutional layer is changed to a size of (128, 128, 64) and is activated by the Leaky ReLU (LReLU) function.

[0021] After this, the encoder is composed of seven blocks (Figure 1). Each block includes a 2D convolution, followed by batch normalization and an LReLU activation function with α = 0.2. All 2D convolution kernels have a size of (4, 4) and are applied with a stride of (2, 2). The kernels of the 2D convolutional layers are 128, 256, 512, 512, 512, 512, and 512, based on the processing direction from the one closest to the input to the one closest to the compressed representation space. The output of the activation is used for both feeding to the next block of the encoder and feeding to the decoder block of the same size. The generator-decoder blocks include the following layers: transposed convolution, batch normalization, a dropout layer with a rate of 0.2 (only the three layers closest to the compressed representation space, see Figure 1), concatenation with the output of the corresponding encoder block, and finally, a rectified linear unit activation function (ReLU) activation (except for the last block, the block closest to the output, which is activated by the hyperbolic tangent function).

[0022] The discriminator (Figure 1) consists of a series of layers that start with the concatenation of all input images (note that the 10 AFM images can be considered as a single image with 10 channels). Subsequently, it is followed by a 2D convolutional layer with 64 kernels of size (4,4) and stride (2,2), activated by LReLU. Next, there are four blocks consisting of a 2D convolutional layer, batch normalization, and LReLU activation (α = 0.2). The convolutions have 128, 256, 512, and 512 kernels respectively, with size (4,4) and stride (2,2). The last layer is a 2D convolution with a single kernel of size (4,4), activated by the sigmoid function.

[0023] During the training of the CGAN, the discriminator is supplied with the same stack as the AFM grayscale images of a certain height of known or predetermined organic molecules used to supply the generator, instead, either a 2D colored RGB structural representation in the form of a ball-and-stick depiction obtained by the generator network in step (a), or a real ball-and-stick depiction of a known or predetermined organic molecule is supplied. The discriminator predicts whether the ball-and-stick depiction of the known or predetermined molecule is the ground truth (real) or the ball-and-stick depiction generated by the generation network (fake). This prediction is achieved by comparing the data divided into 16×16 pixel patches, rather than the overall evaluation of the input data. Since the features induced by the structure and composition on the AFM image strongly depend on the local chemical environment and smoothly depend on the overall molecular arrangement, this local analysis based on small patches of the image makes the CGAN particularly powerful in AFM image analysis.

[0024] The success or failure of the predictions of both the generator and the discriminator is incorporated into the update of the parameters of the generator and the discriminator using the standard gradient backpropagation algorithm. The ball-and-stick depiction generated by the generator is compared with the actual ball-and-stick depiction, and the difference is backpropagated to the CGAN to improve the performance of both the generator and the discriminator. In the case of the discriminator, the predictions as either the image predicted by the generator (fake) or the true ball-and-stick depiction (real) are compared with the known characteristics of this input in each case, and the parameters of the discriminator network are strengthened or changed using gradient backpropagation according to the accuracy of the prediction. Through this learning, the generator is trained to generate an image as close as possible to the real ball-and-stick depiction in order to deceive the discriminator. That is, it improves the ability to determine the nature (real or fake) of the ball-and-stick depiction shown together with the stack of AFM images.

[0025] In the training of the CGAN, one of the 24 combinations of AFM operation parameters (6 types of vibration amplitudes, 4 values for the torsional rigidity of the CO molecule) available in QUAM-AFM for each input stack is randomly selected. This variability of the input data ·confirms that the parameters under which the AFM experiment was carried out do not play a decisive role for the CGAN network to succeed in structure identification, ·suppresses overfitting, and ·provides the ability to generalize to the CGAN network.

[0026] This variability is further improved by applying an Imaging Data Generator (IDG) that adds various deformations (zoom, rotation, shift, inversion, shear) to the input image (Figure 2) and normalizes pixel values to the training set. The use of IDG is motivated because experimental images have features that cannot be captured by AFM simulations and may inhibit identification. For example, experimental images do not display the perfect symmetry of organic molecules. Such differences between experimental AFM images and theoretical AFM images of certain organic molecules are due to the presence of noise that cannot be avoided in experiments, the asymmetry of the probe that is not included in the simulation of AFM images, or the fact that the simulated AFM images used in training assume an ideal gas layer structure, but the organic molecules may be relaxed or deformed due to their interaction with the substrate.

[0027] The deformations obtained by applying IDG during training mimic these effects and greatly contribute to giving the CGAN network the ability to identify organic molecules from experimental images. Selecting appropriate deformation parameters for IDG is important because appropriate selection can significantly improve the identification accuracy.

[0028] Figure 2 shows an application example of IDG and provides information on the range values used for various operations. In the actual application of IDG, it is important to remember that the ball-and-stick depiction included in QUAM-AFM is proportional to the AFM image (sharing the same scale). That is, during training, IDG must be applied to both the input AFM image and the ball-and-stick depiction. For example, when rotating the input AFM image, the corresponding ball-and-stick depiction must also be rotated at the same angle. Otherwise, the atomic positions in the ball-and-stick depiction will not match the corresponding atomic positions in the AFM image, and the CGAN will not be able to learn the local transformation (from the pixel environment) between the shape and intensity of the AFM image and the type of atoms that caused it. This applies to all operations of IDG except for shear that is not applied to the output ball-and-stick depiction. This is due to the fact that shear does not represent the movement of the sample. It is a simulation of deformations that may occur in the experiment but should not appear in the prediction.

[0029] Regarding the loss function, the generator of the CGAN was compiled with the mean absolute error (MAE), and binary cross-entropy was used for the discriminator. The model was minimized by applying a batch of 32 inputs using Adaptive Moment Estimator (Adam) optimization, and the learning rate and first moment parameters were set to 2·10 -4 and 0.5, respectively. The training of the CGAN network was performed over 6 epochs (109K iterations), and 300 predictions of the validation set were displayed every 10,000 iterations to estimate the optimal learning point.

[0030] Considering the complexity of the CGAN and the very limited available experimental data, the success of the CGAN training depends on the following points. · Very efficient training brought about by the competition between the generator and the discriminator · The design of the CGAN incorporating dropout layers that give the ability to generalize to the CGAN · By applying an image data generator (IDG) that normalizes image pixel values and incorporates various random deformations (zoom, rotation, shift, inversion, shear), it mimics the characteristics of experimental images that are not present in AFM simulations, such as the effects related to the presence of noise and the asymmetry of the probe, which may hinder identification.

[0031] Therefore, the trained CGAN of the present invention is defined to include the following: · A generator network having a U-Net-based architecture that includes a block of convolutional layers and a block of transposed convolutional layers configured to process a plurality of AFM images to generate a two-dimensional colored RGB structure representation in the form of a ball-and-stick depiction. The generator further includes a dropout layer included in a predetermined block of convolutional layers configured to prevent overfitting during training with theoretical AFM images and to enable the generator to generalize and make accurate predictions when handling experimental AFM images. · A discriminator network that includes convolutional layers configured to process together a plurality of AFM images of known or predetermined organic molecules and a colored two-dimensional RGB structure representation in the form of a ball-and-stick depiction obtained by the generator network to predict whether the ball-and-stick depiction generated by the generator network is real or a ball-and-stick depiction (fake) generated by the generator network, and · An IDG configured to apply pixel value normalization and different random deformations selected from a zoom of [-15, 15]%, a rotation of [-180, 180] degrees, vertical and horizontal shifts, random vertical and horizontal inversions, a shear of [-20, 20]%, and any combination thereof to theoretical AFM images, thereby providing the characteristics of experimental images that are not present in AFM simulations, such as the effects related to the presence of noise and the asymmetry of the probe, which may hinder identification.

[0032] Accordingly, a first aspect of the present invention refers to a computer-executed method (hereinafter referred to as the method of the present invention) for identifying an organic molecule from an AFM image and generating a two-dimensional colored RGB structure representation of the organic molecule in the form of a ball-and-stick depiction, the method being characterized by comprising the following steps: (a) Using a frequency modulation atomic force microscope (FM-AFM) and the tip of a functionalized metal probe, obtaining a plurality of constant-height AFM grayscale images of the organic molecule at different probe height distances above the organic molecule, wherein the different probe height distances are in the range of 280 pm to 370 pm, and the shape and contrast of the images and their changes with the probe height indicate the three-dimensional positions, atomic sizes (chemical properties), and distances between the atoms within the organic molecule; (b) Providing a trained CGAN to a data processing device, the trained CGAN comprising: · A generator network having an encoder-decoder structure, including a block of convolutional layers, a pooling layer, and a dropout layer; · A discriminator network including convolutional layers and a pooling layer; and, · An image data generator; (c) Supplying the AFM grayscale image obtained in step (a) to the data processing device together with the generator network of the trained CGAN network, wherein the generator of the trained CGAN network generates a two-dimensional colored RGB structure representation of the positions and sizes (chemical properties) of the atoms within the organic molecule and the distances between the atoms in the form of a ball-and-stick depiction, the balls of different colors and sizes represent different chemical species of atoms, the sticks represent the bonds between the atoms, and each ball in the representation is centered at the position occupied by the atom it represents in the AFM image.

[0033] An AFM operating in the frequency modulation dynamic mode (FM-AFM) enables the characterization and manipulation of any substance at the atomic scale by measuring the frequency change of a vibrating probe due to its interaction with an organic molecule sample. When the probe is functionalized with an inert closed-shell molecule, especially a CO molecule, the resolution is improved and access to the internal structure of the molecule becomes possible. The excellent contrast mainly originates from the Pauli repulsion between the CO probe and the sample molecule. This contribution of the repulsive force occurs because the electron densities of the probe and the sample overlap, increasing the frequency shift. The frequency shift is the change in the vibration frequency of the cantilever holding the probe due to the interaction between the probe and the sample. The shift is observed as bright features in an AFM image of a constant height above the position, size, and bond (distance between atoms) of the atoms and reflects the organic molecular structure.

[0034] Therefore, step (a) of the method of the present invention refers to acquiring a plurality of constant height AFM grayscale images of an organic molecule at different probe height distances above the organic molecule using FM-AFM with the tip of a functionalized metal probe, wherein the distance ranges from 280 pm to 370 pm, and the shape, contrast, and their changes due to the probe height of the images are controlled by the electronic charge density of the organic molecule. The charge density of the organic molecule defines the three-dimensional position and size (chemical properties) of the atoms within the organic molecule and the distances between said atoms. As used herein, "the tip of a functionalized metal probe" refers to the tip of a probe functionalized with an inert closed-shell atom or molecule, where the metal is typically Cu, although other metals such as Ag and Pt can also be used. The tip of the probe functionalized with this inert closed-shell atom or molecule dramatically improves the resolution of the AFM image and provides access to the internal structure of the organic molecule. Examples of inert closed-shell atoms and molecules include Xe atoms and CO molecules, respectively.

[0035] The tip of a CO-functionalized metal probe is preferred in the present invention for the following reasons to improve the contrast of the AFM image: · The Pauli repulsion between the lone pair of the oxygen atom in the CO molecule and the charge density of the sample molecule is highly directional because the electronic charge associated with the lone pair is preferentially distributed along the molecular axis. · The CO molecule attached to the metal probe provides a complex electric field. This electric field has a highly localized central feature, is in front of the oxygen atom, and provides an electrostatic interaction that is repulsive to the electrons of the sample molecule and changes rapidly both vertically and horizontally. · The tilt of the CO molecule significantly enhances the features within the molecule.

[0036] Therefore, in a preferred embodiment of the present invention, the tip of the functionalized metal probe used in step (a) is selected from Cu, Ag, or Pt.

[0037] In another preferred embodiment of the method of the present invention, the tip of the functionalized metal probe used in step (a) is functionalized with an inert closed-shell atom or molecule. More preferably, the functionalized metal tip apex is functionalized with an Xe atom or a CO molecule.

[0038] To obtain a plurality of AFM grayscale images of a constant height, it is necessary to use an FM-AFM microscope capable of obtaining images with different contrasts as a function of the probe height distance at different probe height distances above the organic molecule in the range from 280 pm to 370 pm. The shape, contrast, and their variations with the probe height of the images define the three-dimensional spatial distribution of the electronic charge resulting from the interaction of the chemical species, their chemical environment, and their relative height with respect to other atoms in the molecular arrangement. Therefore, since all the information regarding the three-dimensional spatial distribution of the electronic charge is there, it is essential to acquire a plurality of images. Preferably, at least 10 images should be acquired to appropriately characterize the FM-AFM contrast in the probe height distance range between 280 pm and 370 pm above the organic molecule. In this probe height range, the Pauli repulsion and electrostatic interaction between the tip and the organic molecule change significantly, resulting in a strong change in the contrast of the FM-AFM image containing features characteristic of the atom and its molecular environment. The shape, contrast, and variations with the probe height of the FM-AFM image contain all the information regarding the three-dimensional position, size (chemical nature), and interatomic distance of the atoms.

[0039] Therefore, in a preferred embodiment of the method of the present invention, in step (a), at least 10 AFM grayscale images of a constant height of the organic molecule are acquired.

[0040] In another preferred embodiment of the method of the present invention, step (a) is performed at at least 10 different height distances.

[0041] Step (b) is to provide a trained CGAN to the data processing device, and the trained CGAN comprises: · A generator network having an encoder-decoder structure comprising a block of convolutional layers, a pooling layer, and a dropout layer included in a predetermined block; · A discriminator network comprising convolutional layers and a pooling layer; and, · Image data generator.

[0042] Step (c) refers to supplying the generator network of the trained CGAN to the data processing device using the AFM grayscale image obtained in step (a). The generator of the trained CGAN generates a two-dimensional colored RGB structure representation of the positions and sizes (chemical properties) of atoms in an organic molecule and the distances between the atoms in the form of a ball-and-stick depiction. Balls of different colors and sizes represent different chemical atoms, and the sticks represent the bonds between the atoms. Each ball in the representation is centered at the position occupied by the atom it represents in the AFM image.

[0043] Another aspect of the present invention refers to an FM-AFM microscope (hereinafter referred to as the microscope of the present invention) including the tip of a functionalized metal probe and configured to perform step (a) of the method of the present invention, and a data processing device configured to perform steps (b) and (c) of the method of the present invention.

[0044] In a preferred embodiment of the microscope of the present invention, the microscope is connected to a data processing device and further includes a display device configured to display a colored two-dimensional colored RGB structure representation in the form of a ball-and-stick depiction obtained in step (c) of the method of the present invention.

[0045] The microscope of the present invention includes the tip of a functionalized metal probe. Preferably, the metal of the functionalized metal probe is selected from Cu, Ag, or Pt.

[0046] In the present specification, the tip of the functionalized metal probe is preferably functionalized with an inert closed-shell atom or molecule. More preferably, the apex of the functionalized metal tip is functionalized with an Xe atom or a CO molecule. The tip of the functionalized metal probe has dramatically improved the resolution of the AFM image.

[0047] Another aspect of the present invention refers to a computer program (hereinafter, the computer program of the present invention) that includes instructions for causing a data processing apparatus to execute steps (b) and (c) according to the method of the present invention when the program is executed by the data processing apparatus.

[0048] The last aspect of the present invention refers to a computer-readable data carrier storing the computer program of the present invention.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which the present invention belongs. Methods and materials similar or equivalent to those described herein can be used in the practice of the present invention. Throughout this specification and the claims, the word "comprise" and its variations are not intended to exclude other technical features, additives, components, or steps. Additional objects, advantages, and features of the present invention will become apparent to those skilled in the art upon examination of this specification, or can be learned by practice of the present invention. The following examples and drawings are provided by way of illustration and are not intended to limit the present invention.

Brief Description of the Drawings

[0050]

Figure 1

Figure 2

Figure 3

Figure 4

[0051] We selected a set of AFM images already available in the literature (N.J. van der Heijden, et al., ACS Nano 10, 8517 (2016)). Since this structure was not included in the training set, it was a perfect example to verify the ability of the method of the present invention to generalize for other molecules not included in the training set. In this paper, only four images corresponding to different probe-sample distances were published. Therefore, in order to complete at least a stack of 10 input images, additional images were extracted by linearly interpolating two images at a time. It is important to emphasize that the interpolated images were generated only for the purpose of completing the input dimensions, that is, they do not provide additional information to the information provided by the original images.

[0052] Figure 3 is an experimental AFM image taken at a constant height using a CO-terminated probe for the 1-azahexacyclo[11.7.1.13,19.02,7.09,21.015,20]docosa-2,4,6,9(21),10,12,15,17,19-nonen-8,14,22-trione molecule adsorbed on the Cu(111) surface. According to the literature (N.J. van der Heijden, et al., ACS Nano 10, 8517 (2016)), these experimental AFM images show an imperfect three-fold symmetry resulting from the flexibility of the CO--Cu bond and the asymmetric tip. Therefore, testing using these experimental images is extremely stringent: not only does it significantly reduce the amount of information provided to the CGAN network, but it also supplies the CGAN network with images containing characteristic features induced by the asymmetric probe that were not considered in the theoretical AFM simulations used for training.

[0053] The method of the present invention can not only clarify the molecular structure, but also predict the chemical species constituting the molecule with perfect accuracy.

[0054] Example 2: Generation of ball-and-stick depictions from theoretically simulated AFM images obtained from the QUAM-AFM dataset Example 2 demonstrates the ability of our approach to identify molecules that are not chemically or structurally flat and contain most chemical species relevant to organic chemistry. In this case, theoretically simulated AFM images obtained from the QUAM-AFM dataset are given to the model. The three molecules considered here have not been used in the training of CGAN. For each molecule, a stack of 10 constant-height AFM images calculated by randomly selecting simulation parameters from 24 possible combinations provided by QUAM-AFM is considered.

[0055] Figure 4 shows a part of the AFM image included in the stack of each molecule. Due to the steric hindrance effect of internal rotation and the presence of methyl groups, the image is very complex due to the interaction between the chemical properties of the atoms and the non-planar nature of the molecules, such as the different heights of the atoms. Considering such complexity, it would be impossible for human experts to identify these molecules from the observation of AFM images.

[0056] Figure 4(a) shows the identification of 2-(2-aminoethoxy)-N-(3,5-dimethoxyphenyl)acetamide, which is a very difficult example. The corresponding AFM image is characterized by a strong distortion of the structure caused by the accumulation of strong charges around the oxygen atoms. These strongly electronegative atoms hide the bonds with sp 3 carbons, create triangular features at the positions of the hexagonal rings, and also hide the presence of the N atoms bonded to them. Nevertheless, this model can distinguish the presence of sp 3 carbons and sp 2 carbons and identify the two amino groups, leading to a perfect prediction.

[0057] Figure 4(b) corresponds to another difficult case of 3-[2-(4-chlorophenyl)-1,3-thiazol-4-yl]-1-(5-methylfuran-2-yl)prop-2-en-1-one. Repeatedly, even by looking at individual AFM images, it is impossible to identify the combination of hexagonal and pentagonal rings that make up the non-planar structure of the molecule, or the composition such as the carbon substitution by sulfur, oxygen, and nitrogen atoms, and the presence of chlorine substituting one of the H atoms and the methyl group. However, thanks to the change in the probe height of the contrast in the AFM images of the features related to these atoms in different local chemical environments that the CGAN learned during training, the ball-and-stick pre-prediction is accurate, indistinguishable from the actual one, and provides the identification of the complete structure and composition of the molecule. In the case of N-(5-bromo-2-iodophenyl)-5-methyl-1H-imidazole-2-amine shown in Fig. 4(c), the challenge is to distinguish between two different halogen atoms, bromine and iodine, and an additional difficulty is added. Figs. 4(b) and (c) show that the halogen atom bonded to the benzene ring is replaced by an H atom and appears as a very characteristic feature in the AFM image: an elongated bright ellipse perpendicular to the halogen-carbon bond. We have shown that an elongated bright ellipse appears perpendicular to the halogen-carbon bond (J. Tschakert et al.Comm.11,5630(2020)), indicating that this special shape reflects the strong anisotropy of the charge distribution of the halogen (X) covalently bonded to the organic molecule (R). Typically, a "band" of high electron density is observed around the axis of the X-R bond, while the electron density of the halogen cap, the so-called σ-hole, is significantly low and can even become positively charged. The quantitative details of the distribution vary depending on the type of halogen and the organic residue, but the shape of the charge density distribution is common to all halogens. Looking at the electrostatic potential related to this charge distribution, it can be seen that there are negative elliptical regions around the halogen atoms. These repulsive regions coincide with the "negative belt" around the halogen. The surface potential of the halogen cap (σ-hole) is less negative (less repulsive). The relaxation of the probe enhances the asymmetry caused by the electrostatic potential, resulting in an elliptical feature in the HR-AFM image of the halogen.

[0058] These characteristic elliptical contrast features, which are a direct fingerprint of the halogen σ-hole, are very prominent in the AFM image and can be easily found even by human experts. However, it is very difficult for this expert to identify which halogen is present. Our CGAN learned during training the details of the contrast associated with each halogen atom (shown in the AFM images of Figs. 2(b) and (c)) and its variation with the probe height, leading to the complete identification of the halogens (Cl in Fig. 4(b), I and Br in Fig. 4(c)) present in these two molecules.

Claims

1. A computer-executed method for identifying organic molecules from atomic force microscope images and generating a two-dimensional colored RGB structural representation of said molecules in the form of a ball-and-stick depiction, the method comprising the following steps: (a) Using a frequency-mode atomic force microscope, with the tip of a functionalized metal probe, obtaining a plurality of constant-height atomic force microscope grayscale images of the organic molecule at different height distances above the organic molecule, wherein the different height distances are in the range between 280 pm and 370 pm, and the shape, contrast, and their variations due to probe height of the images indicate the three-dimensional positions, atomic sizes, and distances between the atoms within the organic molecule, step; (b) Providing a trained CGAN to a data processing device, the trained CGAN comprising the following, step: ・ A generator network having an encoder-decoder structure comprising a block of convolutional layers, a pooling layer, and a dropout layer; ・ A discriminator network comprising convolutional layers and a pooling layer; and, ・ An image data generator, (c) Feeding the generator network of the trained CGAN to a data processing device using the grayscale image of the atomic force microscope obtained in step (a), wherein the generator of the trained CGAN network generates a two-dimensional colored RGB structural representation of the positions, sizes, and distances between the atoms within the organic molecule in the form of a ball-and-stick depiction, wherein balls of different colors and sizes represent different chemical atoms, the sticks represent the bonds between the atoms, and each ball of the representation is centered at the position occupied by the atom it represents in the AFM image, step.

2. The method according to claim 1, wherein in step (a), at least 10 constant-height atomic force microscope grayscale images of the organic molecule are obtained.

3. Step (a) is performed at at least 10 different height distances, the method according to claim 1 or 2.

4. The tip of the functionalized metal probe used in step (a) is selected from Cu, Ag or Pt, the method according to any one of claims 1 to 3.

5. The tip of the functionalized metal probe used in step (a) is functionalized with an inert closed-shell atom or molecule, the method according to any one of claims 1 to 4.

6. The tip of the functionalized metal probe used in step (a) is functionalized with Xe atoms or CO molecules, the method according to claim 5.

7. A frequency modulation atomic force microscope (FM-AFM) comprising a tip of a functionalized metal probe configured to perform step (a) of the method according to any one of claims 1 to 6, and a data processing device configured to perform steps (b) and (c) of the method according to any one of claims 1 to 6.

8. Further comprising a display device connected to the data processing device and configured to display a two-dimensional colored RGB structure representation in the form of a ball-and-stick depiction obtained in step (c) of the method according to any one of claims 1 to 6, the FM-AFM according to claim 7.

9. The metal of the tip of the functionalized metal probe is selected from Cu, Ag or Pt, the FM-AFM according to claim 7 or 8.

10. The tip of the functionalized metal probe is functionalized with an inert closed-shell atom or molecule, and preferably, the tip of the functionalized metal probe is functionalized with Xe atoms or CO molecules, the FM-AFM according to any one of claims 7 to 9.

11. A computer program, which when executed by a data processing apparatus, causes the data processing apparatus to execute steps (b) and (c) according to the method according to claims 1 to 6.

12. A computer-readable data carrier storing the computer program according to claim 11.

Images