Computer-implemented method for identifying organic molecules from atomic force microscope images by generating two-dimensional (2D) structural representations of said molecules colored with RGB (red, green, blue) color systems in form of ball and stick images

Abstract

The invention relates to a computer-implemented method for identifying organic molecules from an atomic force microscope (AFM) image and generating a two-dimensional structural representation of the organic molecules colored with an RGB color system using a conditional generative adversarial network (CGAN), the present invention relates to a two-dimensional structure colored with an RGB color system, in particular in the form of a ball and stick image of atoms and the distance between said atoms. Thus, the present invention relates to the field of nanotechnology, in particular the field associated with surface chemical reactions, and is therefore of interest to AFM users and manufacturers.

Description

[0001] The present invention relates to a computer-implemented method for identifying organic molecules from atomic force microscopy (AFM) images by training a conditional generative adversarial network (CGAN) to generate a 2D color RGB structural representation of the organic molecules in the form of a ball-and-stick plot.

[0002] The present invention is therefore focused on the field of nanotechnology, particularly in relation to surface chemical reactions, and is therefore of interest to AFM users and manufacturers. Background Art

[0003] Molecules play an increasingly important role in modern society. In addition to their traditional uses as key materials for the food, textile, chemical and pharmaceutical industries, recent trends in nanotechnology consider them to be the ultimate miniaturized elements. Individual molecules now act as electronic and optical devices or force actuators, driving the development of entirely new research areas such as molecular electronics, energy harvesting and molecular motors. In addition to the reduction in size and energy consumption, molecules also offer versatility in molecular structure and composition to fine-tune the system to the intended function. Versatility requires new synthetic routes, and completely new concepts like surface chemistry have been added to the traditional strategies of solution chemistry. To date, the identification of molecular structure and composition has mainly relied on spectroscopic methods that measure ensembles of molecules, such as nuclear magnetic resonance (NMR), mass spectrometry (MS), and X-ray diffraction. They are obviously not suitable for characterizing the products of surface reactions, but they are also limited in traditional solution-based syntheses.

[0004] Atomic force microscopy (AFM) combined with dynamic operation modes has become one of the key tools for imaging and manipulating materials and biological systems at the nanoscale. AFM operates in frequency modulation mode (FM), often referred to as non-contact AFM, achieving true atomic-scale resolution. Utilizing metal tips functionalized with CO molecules at the tip end provides access to the internal structure of molecules with completely unprecedented resolution. The main contrast mechanism of AFM with inert tips such as CO is Pauli repulsion, due to the overlap of the electron density of the tip and the sample. This repulsion produces a positive frequency shift—a change in the oscillation frequency of the cantilever with a fixed tip due to the tip-sample interaction—which is observed in the constant-height AFM image above the atomic positions, sizes, and bonds (distances between atoms), reflecting the molecular structure. Increasingly accurate AFM simulation models have been developed to explain the observed image contrast. They help to elucidate the role of CO tilt, the influence of other factors on the tip-sample interaction, such as electrostatic forces, the role of CO-metal tip charge distribution, and the interplay of short-range chemical interactions and electrostatics in bond order discrimination and intermolecular bond imaging.

[0005] High-resolution experimental (HR) AFM images, and the ability to address individual molecules, have paved the way for the identification of natural products - such as breitfussin A, for which the structures of some fragments are known but for which methods such as NMR have failed to provide a global structure (K. Hanssen et al., Angew. Chem. Int. Ed. 51, 12238 (2012)). HR-AFM is also critical in imaging intermediates (including free radicals) and final products generated by surface reactions, revealing formation processes and reaction pathways (S. Kawai et al., ACS nano 11, 8122 (2017)). The technology has been able to distinguish hundreds of different types of molecules in asphaltene, the solid component of crude oil (B. Schuler, G. Meyer, D. O. C. Mullins, L. Gross, J. Am. Chem. Soc. 137, 9870 (2015). The molecular identification in all the above cases was supported by important information about the nature of the molecules involved, e.g. in the case of asphaltenes we were dealing mainly with PAHs based on C and H atoms. Despite the rich information provided by HR-AFM experiments and these advances in interpreting the observed contrast, the complete identification of molecular systems, i.e. the determination of structure and composition based solely on HR-AFM images without any prior information, remains an open problem.

[0006] Few efforts have attempted to use artificial intelligence (AI) techniques (B. Alldritt et al., Sci. Adv. 6, eaay6913 (2020) and J. Carracedo-Cosme, C. R. Pérez, Nanomaterials 11, 1658 (2021)) to process AFM images to solve this problem. Deep learning (DL) is now commonly used to classify, interpret, depict and analyze images, providing machines with capabilities that surpass humans. The ability of DL to recognize patterns can in principle be used to characterize the structure of molecular systems. Gordon et al. (OM Gordon, JE Hodgkinson, SM Farley, EL Hunsicker, PJ Moriarty, Nano Lett. 20, 7688 (2020)) implemented a model to automatically detect spatially correlated patterns in different groups of AFM images of self-organized nanoparticles. However, complete atom-by-atom recognition presents a major challenge because the effects of geometry and chemical composition affect the determination of 3D molecular charge density in a very complex way, which ultimately determines the contrast of AFM.

[0007] This complexity places the task of molecular recognition beyond the capabilities of even the best human experts, e.g. Figure 2 As shown in the example in. The molecular structure in terms of ring groups and side groups is completely hidden by the chemical composition and its non-planar configuration. Alldritt et al. (B. Alldritt et al., Sci. Adv. 6, eaay6913 (2020).) developed a convolutional neural network (CNN) with the aim of determining the geometry of molecules from AFM images. For the structure of quasi-planar molecules, the performance is excellent even when the algorithm with experimental results is used directly. For 3D structures, they were able to recover information on the position and size of atoms closer to the tip within a height range of 150 pm. However, the differentiation of functional groups produced inconclusive results.

[0008] In our previous work (J.Carracedo-Cosme, C. R. Pérez, Nanomaterials 11, 1658 (2021), we demonstrate the feasibility of very accurate automatic molecular classification of a set of 60 planar molecules (including the most common atomic species in organic chemistry) using DL techniques using their theoretically simulated AFM images. In addition, we propose a variational autoencoder (VAE)-based approach to include features of experimental AFM images in the dataset, significantly increasing the accuracy of the model tested with experimental images. However, although this approach shows the potential to identify the structure and composition of molecules through AFM images, it does not come close to solving the global classification problem because (i) CNN classification in the usual sense only allows outputs of finite length, i.e., only a limited number of structures can be classified, and (ii) we need to consider molecules with non-planar adsorption configurations.

[0009] Therefore, there is a need to develop new methods to identify molecules processing AFM images. Summary of the invention

[0010] The object of the present invention is to provide a computer-implemented method for identifying organic molecules from atomic force microscope (AFM) images, which generates a two-dimensional (2D) color RGB structural representation of the organic molecules by using a trained conditional generative adversarial network (CGAN). The computer-implemented method of the present invention generates a 2D RGB structural representation of organic molecules in the form of a ball-and-stick diagram, wherein balls of different colors and sizes represent different chemical atoms, and sticks represent bonds between atoms, wherein each ball represented is centered on the position occupied by the atom represented in the AFM image. The 2D RGB structural representation of organic molecules in the form of a ball-and-stick diagram provides complete information about its structure and chemical composition. In the present invention, different chemical properties of atoms are represented by different colors and sizes proportional to the van der Waals radius of the atoms.

[0011] The trained CGAN is used in the method of the present invention.

[0012] A Generative Adversarial Network (GAN) is a machine learning framework that relies on a generator that learns a mapping from a noisy representation of an input image (usually a vector) to an output image and a discriminator that learns to distinguish between images generated by the generator and target images. The generator improves the mapping during training to produce results that are indistinguishable from "real" images by the discriminator, which is trained to do its best at detecting generator forgeries. A CGAN is a GAN that uses additional conditions to map the output image in addition to the noisy representation of the input image. In the present invention, the generator learns a mapping to the output image from an observed input image and its vector representation with random noise.

[0013] The method of training CGAN includes the following steps:

[0014] i) providing a trained CGAN to a data processor device, wherein the trained CGAN comprises:

[0015] A generator network with an encoder-decoder structure (U-Net architecture), including

[0016] Convolutional layer blocks,

[0017] Pooling layers, and

[0018] Dropout layer,

[0019] a discriminator network consisting of convolutional and pooling layers, and

[0020] Image data generator;

[0021] ii) feeding a generator network with a plurality of AFM grayscale images of known or predetermined organic molecules at constant height, such as AFM images of the database QUAM-AFM, wherein the shape and contrast of the images and their variation with height show the 3D positions and sizes (chemical properties) of atoms in the organic molecules and the distances between the atoms, the generator generates a 2D color RGB structural representation of the positions and sizes of atoms in the organic molecules and the distances between the atoms in the form of a ball-and-stick diagram, wherein balls of different colors and sizes represent different chemical properties of atoms and sticks represent bonds between the atoms, wherein each ball of the representation is centered at the position occupied by the atom it represents in the AFM image,

[0022] iii) feeding the discriminator network with the same plurality of constant height AFM grayscale images of the known or predetermined organic molecule that were used to feed the generator in step (ii), or feeding the discriminator network with the ball-and-stick plot obtained by the generator network in step (iii) or a true ball-and-stick plot of the known or predetermined molecule, and the discriminator superimposes and compares the segmented blocks of the image and the plot, and

[0023] iv) An identifier in a data processor device selects, from the segmented blocks of the image and the superposition of the graph (preferably 16x16 pixels) that match the positions and sizes of the atoms and the distances between said atoms in step (iii), a colored 2D color RGB structural representation in the form of a ball-and-stick plot, and ignores those that do not match the positions, sizes and distances between said atoms.

[0024] Quasar Science Resources Ltd.-Universidad de Madrid-Atomic Force Microscope (QUAM-AFM) (https: / / doi.org / 10.21950 / UTGMZ7) was used to train the CGAN.

[0025] QUAM-AFM is a publicly available dataset that theoretically generates 165 million AFM images from 686,000 isolated molecules using 240 different combinations of AFM operating parameters (10 tip-sample distances, 6 different oscillation amplitudes, and 4 different values ​​of torsional stiffness of the CO molecule, which are known to depend on the details of how the molecule is attached to the tip of the metal tip). QUAM-AFM also provides ball-and-stick plots of each molecule generated from the atomic coordinates. These plots are in the same scale as used in the AFM images: if we superimpose the two images, each representation of the ball is centered on the position occupied by the atom it represents in the AFM image.

[0026] The QUAM-AFM dataset includes organic molecules, discarding all other compounds that may not be in pure molecular form (such as organic salts or inorganic compounds and polymers). The selected molecules contain the four basic elements of organic chemistry (carbon, hydrogen, nitrogen and oxygen) as well as some other less common elements that are still common in organic compounds (such as sulfur, phosphorus and halogen atoms (fluorine, chlorine, bromine and iodine)). The largest molecule in the QUAM-AFM database has a total of 85 atoms.

[0027] Very small molecules, i.e. those containing fewer than 8 atoms, are not good candidates for identification by AFM methods alone due to their extremely high surface mobility and the large number of different adsorption configurations and are therefore not included in the QUAM-AFM database. Very large molecules of square-cell based structures also do not form part of the QUAM-AFM dataset.

[0028] The database QUAM-AFM is limited to quasi-planar molecules, which only show up to The height of the change to include sp 3 Aliphatic chains with carbon atoms (methyl groups) as pendant groups. QUAM-AFM includes a group of molecules that includes aliphatic, cyclic and aromatic compounds, especially a large number of hydrocarbons (alkanes, alkenes, alkynes, etc.) as well as all traditional organic families (alcohols, thiols, ethers, aldehydes and ketones, carboxylic acids, amines, amides, imines, esters, nitriles, nitro and azo compounds, halogenated hydrocarbons and acyl halides, etc.).

[0029] The 686K structures in QUAM-AFM have been split into training, validation, and test sets with 581K, 24K, and 81K structures, respectively. It must be noted that for testing CGAN, the test set was chosen to be particularly large in order to have a sufficiently diverse range of molecular structures and chemical compositions to evaluate the performance of CGAN.

[0030] During the training of CGAN, the generator and the discriminator compete against each other in a zero-sum game: the generator learns to fool the discriminator and generate images as close to the ball-and-stick diagram as possible, while the discriminator learns to guess whether the second input image is real or fake. This competition forces the generator and the discriminator to significantly improve their performance during training. The discriminator is only used to force the improvement of the generator. Therefore, once this goal is achieved, the discriminator is discarded.

[0031] The generator with encoder-decoder structure consists of a series of similar blocks, where the main difference is the number of kernels applied in each convolution and the dimension of each input ( Figure 1). The input consists of a stack of 10 grayscale AFM images (single channel) and the corresponding ball-and-stick plots are the output. The first layer of the generator is a dropout layer with a ratio of 0.5 and two 3D convolutional layers for processing the image stack. Having such a high ratio of dropout layers is important for the CGAN to be able to generalize and predict accurately when processing experimental images. The first 3D convolution consists of 64 kernels, each of size (4, 3, 3), with a stride of (3, 1, 1) applied and padding. The second 3D convolution also has 64 kernels, but in this case the kernel size is (4, 4, 4) and a stride of (4, 2, 2) is applied. The output of the second convolutional layer is resized to (128, 128, 64) and activated with the Leaky ReLU (LReLU) function.

[0032] From this point on, the encoder consists of seven blocks ( Figure 1 ). Each block consists of a 2D convolution followed by batch normalization and an LReLU activation function with α=0.2. All kernels of the 2D convolution have size (4, 4) and a stride of (2, 2) is applied. The 2D convolution layers have 128, 256, 512, 512, 512, 512, and 512 kernels, with the processing direction being from the kernel closest to the input to the kernel closest to the compressed representation space. The output of the activation is used both to feed the next block of the encoder and to feed the decoder block of the same size. The generator decoder block consists of the following layers: transposed convolution, batch normalization, dropout layer with rate 0.2 (only in the three layers closest to the compressed representation space, see Figure 1 ), concatenation with the output of the corresponding encoder block, and finally a rectified linear unit (ReLU) activation (except for the last block, where the blocks closest to the output are activated with a hyperbolic tangent function).

[0033] Discriminator( Figure 1 ) consists of a series of layers, concatenated from all the input images (note that we can consider the 10 AFM images as a single image with 10 channels). It is followed by a 2D convolutional layer with 64 kernels of size (4, 4) and stride (2, 2) activated by LReLU. Then, it has four blocks consisting of a 2D convolutional layer, batch normalization, and LReLU activation (α=0.2). The convolutions have 128, 256, 512, and 512 kernels of size (4, 4) and stride (2, 2), respectively. The last layer is a 2D convolution with a single kernel of size (4, 4) activated with a sigmoid function.

[0034] During the training of CGAN, the discriminator is fed with the same stack of AFM grayscale images of constant height of known or predetermined organic molecules used to feed the generator, or with a 2D color RGB structural representation in the form of a ball-and-stick diagram obtained by the generator network in step (a), or with a real ball-and-stick diagram of a known or predetermined organic molecule. The discriminator predicts whether the ball-and-stick diagram of the known or predetermined molecule is real data (true) or a ball-and-stick diagram generated by the generator network (false). This prediction is not made by a global evaluation of the input data, but by splitting them into blocks of 16×16 pixels for comparison. This local analysis based on small patches of images makes CGAN particularly powerful in AFM image analysis, because the features on AFM images caused by structure and composition are highly dependent on the local chemical environment and depend smoothly on the global molecular configuration.

[0035] The success or failure in the predictions of the generator and discriminator is incorporated into the update of the parameters of the generator and discriminator using a standard gradient backpropagation algorithm. The ball-and-stick images generated by the generator are compared with the real ball-and-stick images, and the difference is backpropagated into the CGAN, thereby improving the performance of the generator and discriminator. In the case of the discriminator, its prediction as the generator predicted image (fake) or the real ball-and-stick image (true) is compared with the known features of the input in each case, and gradient backpropagation is used to enhance or change the parameters in the discriminator network depending on the accuracy of the prediction. Through this training, the generator is forced to learn to generate images that are as close to the real ball-and-stick images as possible to deceive the discriminator, that is, to improve its ability to determine the nature of the ball-and-stick images shown together with the stack of AFM images (true or false).

[0036] During CGAN training, one of the 24 combinations of AFM operating parameters (6 different oscillation amplitudes and 4 different values ​​of torsional stiffness of CO molecules) available for each input stack in QUAM-AFM is randomly selected.

[0037] Ensure that the parameters in which the AFM experiment is performed do not play a decisive role in the successful identification of the structure by the CGAN network,

[0038] Prevent overfitting, and

[0039] Provide generalization capabilities for the CGAN network.

[0040] This variability is further enhanced by applying an Image Data Generator (IDG) to the training set, which applies different deformations (scaling, rotation, offset, flipping, and shearing) to the input images ( Figure 2) and normalize the pixel values. The motivation for using IDG is that experimental images have some features that cannot be captured by AFM simulations, which may hinder recognition. For example, experimental images do not show the full symmetry of organic molecules. These differences between experimental and theoretical AFM images of a given organic molecule may be due to the inevitable presence of noise in the experiment, tip asymmetries that are not included in the AFM image simulations, or relaxation and deformation of the organic molecules due to interactions with the substrate, while we are considering an ideal gas-phase structure in the simulated AFM images used for training.

[0041] Applying the deformation provided by IDG during training simulates these effects and significantly endows the CGAN network with the ability to recognize organic molecules from experimental images. Choosing appropriate deformation parameters for IDG is very important, as the right choice can greatly improve the recognition accuracy.

[0042] Figure 2 A specific example of IDG application is shown, and information is provided on the range values ​​used for different operations. In practical applications of IDG, it is important to remember that the ball-and-stick diagrams contained in QUAM-AFM are proportional to (share the same scale) as the AFM image. Therefore, during training, IDG must be applied to both the input AFM image and the ball-and-stick diagram: i.e., if we rotate the input AFM image, then the corresponding ball-and-stick diagram must be rotated by the same angle. Otherwise, the atomic positions represented by the ball-and-stick will not match the corresponding atomic positions of the AFM image, and the CGAN will not be able to learn (from the pixel context) the shape and intensity of the AFM image and the local translations between the atomic types that lead to the AFM image. This applies to all operations in IDG, with the exception of shearing, which is not applied to the output ball-and-stick diagram. This is because shearing does not represent the movement of the sample, but rather is a simulation of possible deformations that may occur in the experiment but should not appear in the predictions.

[0043] Regarding the loss function, the generator of CGAN is compiled using mean absolute error (MAE), while binary cross entropy is used for the discriminator. The model is minimized by applying the adaptive moment estimator (Adam) optimizer to a batch of 32 inputs, where the learning rate and first moment parameters are set to 2×10 -4 and 0.5. The training of the CGAN network was performed in six epochs (109K iterations), and 300 predictions of the validation set were shown every 10,000 iterations to estimate the optimal training point.

[0044] Given the complexity of CGAN and the very limited experimental data available, the successful performance of CGAN training depends on

[0045] The competition between the generator and the discriminator provides very efficient training.

[0046] The design of CGAN, including the dropout layers that provide CGAN with generalization capabilities, and

[0047] Application of an image data generator (IDG) that normalizes the image pixel values ​​and incorporates different random deformations (scaling, rotation, offset, flipping and shearing) to simulate features of experimental images (such as the presence of noise and effects related to tip asymmetry) that are absent in AFM simulations and could hinder identification.

[0048] Therefore, the trained CGAN of the present invention is defined as comprising:

[0049] A generator network having a U-Net architecture, the generator network including a convolutional layer block and a transposed convolutional layer block configured to process a plurality of AFM images to generate a 2D color RGB structure representation in the form of a ball-and-stick plot. The generator also includes a dropout layer contained in a predetermined convolutional layer block, the dropout layer being configured to prevent overfitting during training with theoretical AFM images and to allow the generator to generalize and make accurate predictions when processing experimental AFM images;

[0050] a discriminator network including a convolutional layer, the discriminator network being configured to process together a plurality of AFM images of a known or predetermined organic molecule and a colored 2D color RGB structural representation in the form of a ball-and-stick plot obtained by the generator network to predict whether the ball-and-stick plot of the known or predetermined organic molecule is real (true) or a ball-and-stick plot generated by the generator network (false);

[0051] and an IDG configured to apply pixel value normalization to the theoretical AFM images, as well as different random deformations selected from [-15, 15]% scaling, [-180, 180] degrees rotation, vertical and horizontal offsets, random vertical and horizontal flips, [-20, 20]% shearing, and any combination thereof, thereby providing them with features of the experimental images (such as the presence of noise and effects related to tip asymmetry) that are absent in the AFM simulations and that might hinder identification.

[0052] Therefore, a first aspect of the present invention relates to a computer-implemented method for identifying organic molecules from AFM images and generating a 2D color RGB structural representation of said organic molecules in the form of a ball-and-stick plot (herein the method of the present invention), characterized in that it comprises the following steps:

[0053] (a) acquiring multiple constant height AFM grayscale images of the organic compound at different tip height distances above the organic molecule by frequency modulation atomic force microscopy (FM-AFM) using a functionalized metal tip, wherein the distances of the different tip heights are between 280 pm and 370 pm, wherein the shape and contrast of the images and their changes with tip height show the 3D position, size (chemical properties) and distances between atoms in the organic molecule;

[0054] (b) providing a trained CGAN to a data processor device, wherein the trained CGAN comprises:

[0055] A generator network with an encoder-decoder structure, including blocks of convolutional layers,

[0056] Pooling layers, and

[0057] Dropout layer,

[0058] a discriminator network consisting of convolutional and pooling layers, and

[0059] Imaging data generator;

[0060] and (c) feeding a generator network of a trained CGAN network using the AFM grayscale image obtained in step (a) to a data processor device, wherein the generator of the trained CGAN network generates a 2D color RGB structural representation of the positions and sizes (chemical properties) of atoms in the organic molecule and the distances between the atoms in the form of a ball-and-stick diagram, wherein balls of different colors and sizes represent different chemical types of atoms and sticks represent bonds between atoms, wherein each ball of the representation is centered at the position occupied by the atom represented in the AFM image.

[0061] AFM operated in frequency modulation dynamic mode (FM-AFM) can characterize and manipulate a variety of materials at the atomic scale by measuring the frequency changes caused by the interaction of an oscillating tip with an organic molecule sample. Resolution is improved when the tip is functionalized with inert closed-shell molecules, particularly CO molecules, allowing access to the internal structure of the molecule. The significant contrast stems primarily from the Pauli repulsion between the CO probe and the sample molecules. This repulsion occurs because the electron density of the tip and sample overlap, resulting in an increased frequency shift, which is a change in the oscillation frequency of the cantilever holding the tip due to the tip-sample interaction. The shift is observed as a bright feature in the constant height AFM image above the atomic positions, sizes, and bonds (distances between atoms), reflecting the organic molecular structure.

[0062] Therefore, step (a) of the method of the present invention refers to collecting multiple constant height AFM grayscale images of the organic molecule by FM-AFM at different tip height distances above the organic molecule using a functionalized metal tip tip, wherein the distance is between 280 pm and 370 pm, wherein the shape, contrast and its variation with tip height of the image are controlled by the charge density of the organic molecule. The charge density of the organic molecule defines the 3D position and size (chemical properties) of the atoms in the organic molecule and the distances between the atoms.

[0063] The term "functionalized metal tip" refers herein to a tip of a metal (usually Cu), but other metals (such as Ag and Pt) may also be used, which is functionalized with inert closed shell atoms or molecules, thereby significantly improving the resolution of the AFM image and providing access to the internal structure of organic molecules. Examples of inert closed shell atoms or molecules are Xe atoms and CO molecules, respectively.

[0064] The CO-functionalized metal tip is preferred in the present invention because it increases the AFM image contrast for the following reasons:

[0065] The Pauli repulsion between the lone pairs of oxygen atoms in the CO molecule and the charge density of the sample molecule is very directional, due to the preferential distribution of the electronic charge associated with the lone pairs along the molecular axis, and

[0066] The CO molecule attached to the metal tip provides a complex electric field with a very localized central feature, located just in front of the O atom and repelling the electrons in the sample molecules, which provides rapidly changing electrostatic interactions both vertically and laterally, and

[0067] The tilt of the CO molecule significantly enhances the intramolecular features.

[0068] Therefore, in a preferred embodiment of the present invention, the functionalized metal tip used in step (a) is selected from Cu, Ag or Pt.

[0069] In another preferred embodiment of the method of the present invention, the functionalized metal tip used in step (a) is functionalized with an inert closed shell atom or molecule. More preferably, the functionalized metal tip is functionalized with a Xe atom or a CO molecule.

[0070] In order to collect multiple AFM grayscale images of constant height, different tip height distances ranging from 280pm to 370pm above the organic molecule must be obtained by an FM-AFM microscope, which allows the collection of images with different contrasts as a function of the tip height distance. The shape and contrast of the image and their changes with the tip height define the 3D spatial distribution of the charge, which is caused by the interaction of the chemical substances, their chemical environment, and their relative heights relative to other atoms in the molecular configuration. Therefore, it is crucial to collect multiple images because it contains all the information about the 3D spatial distribution of the charge. Preferably, at least 10 images should be taken to correctly characterize the FM-AFM contrast within the tip height distance range between 280pm and 370pm above the organic molecule. In this tip height range, Pauli repulsion and the electrostatic interaction between the tip and the organic molecule vary greatly, resulting in a strong change in the contrast of the FM-AFM image, including the characteristics of the atoms and their molecular environment. The shape and contrast of the image and their changes with the tip height contain all the information about the 3D position and size (chemical properties) of the atoms and the distance between the atoms.

[0071] Therefore, in a preferred embodiment of the method of the present invention, a plurality of at least 10 AFM grayscale images of the organic molecules with constant height are collected in step (a).

[0072] In another preferred embodiment of the method of the present invention, step (a) is performed at at least 10 different height distances.

[0073] Step (b) involves providing a trained CGAN to a data processor device, wherein the trained CGAN comprises:

[0074] A generator network with an encoder-decoder structure, including

[0075] Convolutional layer blocks,

[0076] Pooling layers, and

[0077] Dropout layers contained in predetermined blocks,

[0078] a discriminator network consisting of convolutional and pooling layers, and

[0079] Image data generator.

[0080] Step (c) refers to feeding the generator network of the trained CGAN using the AFM grayscale image obtained in step (a) to a data processor device, wherein the generator of the trained CGAN generates a 2D color RGB structural representation of the position and size (chemical properties) of atoms in the organic molecule and the distance between the atoms in the form of a ball-and-stick diagram, wherein balls of different colors and sizes represent different chemical atoms, and sticks represent the bonds between the atoms, wherein each ball represented is centered on the position occupied by the atom represented in the AFM image.

[0081] Another aspect of the present invention relates to a FM-AFM microscope (herein the microscope of the present invention), comprising a functionalized metal tip tip configured to perform step (a) of the method of the present invention, and a data processing device configured to perform steps (b) and (c).

[0082] In a preferred embodiment of the microscope of the present invention, the microscope further comprises a display unit connected to the data processing device and configured to display the colored 2D colored RGB structure representation obtained in step (c) of the method of the present invention in the form of a ball-and-stick plot.

[0083] The microscope of the present invention comprises a functionalized metal tip. Preferably, the metal of the functionalized metal tip is selected from Cu, Ag or Pt.

[0084] Here, the functionalized metal tip is preferably functionalized with inert closed-shell atoms or molecules, and more preferably, the functionalized metal tip is functionalized with Xe atoms or CO molecules. The functionalized metal tip greatly improves the resolution of the AFM image.

[0085] Another aspect of the present invention relates to a computer program (herein the computer program of the present invention) comprising instructions which, when executed by a data processing device, cause the data processing device to perform steps (b) and (c) according to the method of the present invention in the data processing device.

[0086] A final aspect of the invention relates to a computer-readable data carrier having stored thereon the computer program of the invention.

[0087] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those of ordinary skill in the art to which the invention belongs. Methods and materials similar or equivalent to the methods and materials described herein may be used in the practice of the present invention. Throughout the specification and claims, the term "comprising" and its variations are not intended to exclude other technical features, additives, ingredients or steps. Other objects, advantages and features of the present invention will become apparent to those skilled in the art after reading the specification, or may be learned by practicing the present invention. The following examples and drawings are provided by way of illustration and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 : The scheme of training CGAN of the present invention. The generator has an encoder-decoder structure (U-Net architecture). The encoder includes a first block with a dropout layer (black) and two 3D convolutional layers (grey), and seven blocks of 2D convolutional and pooling layers. The decoder includes eight blocks with 2D transposed convolutional layers, with a final dropout layer (black) in the first three blocks. The discriminator includes a first block with a cascade layer of all input images, followed by a 2D convolutional layer, and four blocks consisting of a 2D convolutional layer, a pooling layer, batch normalization, and an LReLU activation. The last layer is a 2D convolution activated with a sigmoid function.

[0089] Figure 2 Image Data Generator (IDG) was applied to AFM input images and ball-and-stick plots of the molecule 6,14,22-triethynyltetracyclo[18.4.0.04,9.012,17]tetraconic-1(20),4(9),5,7,12(17),13,15,21,23-nonene-2,10,18-triyne. The parameters chosen for IDG during training were randomly chosen in the range of [-180, 180] degrees for rotation, [-15, 15]% for scaling and vertical and horizontal offsets, and [-20, 20]% for shearing and random vertical and horizontal flips.

[0090] Figure 3 : (From left to right) The first four images correspond to 1-azahexacyclic [11.7.1.1 3，19 .0 2，7 .0 9，21 .0 15，20] Experimental AFM images of twenty-two carbon-2,4,6,9(21),10,12,15,17,19-nonene-8,14,22-trione. The fifth image is a 2D color RGB structural representation of the molecule obtained according to the method of the present invention, and the color codes of the balls representing the chemical substances are: carbon (gray), hydrogen (white), oxygen (red) and nitrogen (blue). The sixth image is an optimized ball-and-stick diagram of the molecule. The bottom panel provides additional grayscale versions of the fifth and sixth images, in which the positions and chemical properties of atoms other than carbon and hydrogen (oxygen (O) and nitrogen (N)) are indicated by arrows.

[0091] Figure 4 : (From left to right) Four AFM images at different tip-sample distances, 2D color RGB structural representations obtained according to the method of the present invention, the real structures and height maps of (a) 2-(2-aminoethoxy)-N-(3,5-dimethoxyphenyl)acetamide, (b) 3-[2-(4-chlorophenyl)-1,3-thiazol-4-yl]-1-(5-methylfuran-2-yl)prop-2-en-1-one and (c) N-(5-bromo-2-iodophenyl)-5-methyl-1H-imidazol-2-amine. The bottom panels provide additional grayscale versions of the 5th and 6th images, where the positions and chemical natures of atoms other than carbon and hydrogen (oxygen (O), nitrogen (N), sulfur (S), chlorine (Cl), bromine (Br) and iodine (I)) are indicated by arrows. DETAILED DESCRIPTION

[0092] Example 1

[0093] We chose a set of AFM images already available in the literature (NJ van der Heijden et al., ACS Nano 10, 8517 (2016)). This structure was not part of the training set, and therefore, it is a perfect example to verify the ability of the method of the present invention to be applicable to other molecules not included in the training set. Only 4 images corresponding to different tip-sample distances are published in this paper, so we performed a pairwise linear interpolation of the images to extract additional images to complete the input, i.e. a stack of at least 10 images. It should be emphasized that the only purpose of generating the interpolated images is to complete the input dimensionality, i.e. they do not provide the additional information provided by the original images.

[0094] Figure 3Experimental AFM images of the 1-azahexacyclo[11.7.1.13,19.02,7.09,21.015,20]docosa-2,4,6,9(21),10,12,15,17,19-nonene-8,14,22-trione molecule adsorbed on a Cu(111) surface, taken at constant height and collected with a CO-terminated tip, are shown. These experimental AFM images show imperfect three-fold symmetry, which according to the literature (NJ van der Heijden et al., ACS Nano 10, 8517 (2016)) is caused by the flexibility of the CO-Cu bond coupled to the asymmetric tip. Therefore, testing these experimental images is very difficult: not only do we severely reduce the amount of information provided to the CGAN network, but we also feed it images containing characteristic features caused by the asymmetric tip that are not considered in the theoretical AFM simulations used for training.

[0095] The method of the present invention is not only able to reveal the molecular structure, but also to predict with perfect accuracy the chemical species that make up the molecule.

[0096] Example 2: Generating a ball-and-stick plot from a theoretically simulated AFM image acquired from a QUAM-AFM dataset

[0097] The second example demonstrates the ability of our approach to identify chemically and structurally non-planar molecules, including most relevant chemical classes in organic chemistry. In this case, we feed the model theoretically simulated AFM images acquired from the QUAM-AFM dataset. None of the three molecules considered in this paper were used to train the CGAN. For each molecule, we consider a stack of 10 AFM images of constant height, computed with simulation parameters randomly chosen from the 24 possible combinations provided by QUAM-AFM.

[0098] Figure 4 A partial AFM image of each molecule included in the stack is shown. These images are extremely complex due to the interplay of the chemical nature of the atoms and the non-planar nature of the molecules, with the atoms being at different heights due to internal rotation caused by steric hindrance effects and the presence of methyl groups. Given this complexity, a human expert would not be able to identify these molecules from observation of the AFM images.

[0099] Figure 4 (a) shows the identification of 2-(2-aminoethoxy)-n-(3,5-dimethoxyphenyl)acetamide, a very difficult example. The corresponding AFM image is characterized by a strong distortion of the structure caused by the strong charge accumulation around the oxygen atoms. These highly electronegative atoms hide their sp 3The bond of the carbon forms a triangular feature in the position of the hexagonal ring, also hiding the presence of the N atom attached to it. However, the model is able to distinguish sp 3 and sp 2 The presence of carbon and identification of two amino groups led to a perfect prediction.

[0100] Figure 4 (b) corresponds to another difficult case, 3-[2-(4-chlorophenyl)-1,3-thiazol-4-yl]-1-(5-methylfuran-2-yl)prop-2-en-1-one. Again, looking at each AFM image, it is impossible to identify the combination of hexagonal and pentagonal rings that make up the non-planar structure of the molecule, let alone the composition, where carbon is replaced by sulfur, oxygen, and nitrogen atoms, and chlorine replaces one hydrogen atom and one methyl group. However, due to the change in the contrast of the tip height in the AFM image that our CGAN learned during training to be associated with the characteristics of these atoms in different local chemical environments, the ball-and-stick predictions are accurate and completely indistinguishable from the true ball-and-stick predictions, providing complete structural and compositional identification of the molecule.

[0101] Figure 4 The case of N-(5-bromo-2-iodophenyl)-5-methyl-1H-imidazol-2-amine shown in (c) adds an additional difficulty, namely the identification of two different halogen atoms, bromine and iodine. Figure 4 (b) and (c) show that a halogen atom attached to a benzene ring replaces a hydrogen atom, which appears as a very striking feature in the AFM image: an elongated bright oval, perpendicular to the halogen-carbon bond. We have shown (J. Tschakert et al., Nat. Comm. 115630 (2020)) that this special shape reflects the strongly anisotropic charge distribution of the halogen (X) covalently bonded to the organic molecule (R). Typically, a "band" of high electron density is observed around the XR bond axis, while the electron density at the halogen cap (the so-called σ hole) is significantly lower and may even be positive. Although the quantitative details of the distribution vary depending on the type of halogen and organic residue, the shape of the charge density distribution is universal for all halogens. The electrostatic potential associated with this charge distribution reveals negative elliptical regions at the halogen atoms. These repulsive regions are consistent with the "negative bands" around the halogen. The surface potential at the halogen cap (the σ hole) is lower (lower repulsion). The relaxation of the probe enhances the asymmetry introduced by the electrostatic potential, resulting in the appearance of elliptical features in the HR-AFM images at the halogen.

[0102] These prominent elliptical contrast features are direct fingerprints of the halogen’s σ-vacancies and are very visible in AFM images and can be easily spotted by a human expert. However, it would be extremely challenging for this expert to identify which specific halogen is present. Our CGAN learns the details of the contrast and the evolution of its tip height associated with each halogen atom during training (e.g. Figure 2 (b) and (c) AFM images), which perfectly identify the halogens present in these two molecules ( Figure 4 (b) Chlorine and Figure 4 (c) iodine and bromine).

Claims

1. A computer-implemented method for identifying organic molecules from atomic force microscopy images and generating a 2D color RGB structural representation of the molecules in the form of a ball-and-stick plot, characterized in that it comprises the following steps: (a) collecting a plurality of constant height atomic force microscope grayscale images of the organic molecule at different height distances above the organic molecule by frequency mode atomic force microscopy using a functionalized metal tip, wherein the different height distances are between 280 pm and 370 pm, wherein the shape, contrast and its variation with tip height show the 3D position of the atoms in the organic molecule, the atomic size and the distance between the atoms; (b) providing a trained CGAN to a data processor device, wherein the trained CGAN comprises: A generator network with an encoder-decoder structure, including Convolutional layer block Pooling layer and Dropout layer, a discriminator network consisting of convolutional and pooling layers, and an image data generator; and (c) feeding a generator network of a CGAN trained using the atomic force microscope grayscale image obtained in step (a) to the data processor device, the generator of the trained CGAN network generating a 2D color RGB structural representation of the positions and sizes of atoms in organic molecules and the distances between the atoms in the form of a ball-and-stick diagram, wherein balls of different colors and sizes represent different chemical atoms, and sticks represent bonds between the atoms, wherein each ball of the representation is centered at the position occupied by the atom represented in the AFM image.

2. The method according to claim 1, wherein: In step (a), a plurality of at least 10 atomic force microscope grayscale images of the organic molecules at constant heights are collected.

3. The method according to any one of claims 1 or 2, wherein: Step (a) is performed at at least 10 different height distances.

4. The method according to any one of claims 1 to 3, wherein: The functionalized metal tip used in step (a) is selected from Cu, Ag or Pt.

5. The method according to any one of claims 1 to 4, wherein: The functionalized metal tip used in step (a) is functionalized at the tip with inert closed shell atoms or molecules.

6. The method according to claim 5, wherein: The functionalized metal tip used in step (a) is functionalized at the tip end with Xe atoms or CO molecules.

7. A frequency modulated atomic force microscope, comprising a functionalized metal tip and a data processing device, wherein the functionalized metal tip is configured to perform step (a) of the method according to claims 1 to 6, and the data processing device is configured to perform steps (b) and (c) of the method according to claims 1 to 6.

8. The FM-AFM microscope according to claim 7, further comprising a display unit connected to the data processing device and configured to display the 2D color RGB structure representation in the form of a ball-and-stick plot obtained in step (c) of the method according to claims 1 to 6.

9. The FM-AFM microscope according to any one of claims 7 and 8, wherein: The metal at the tip of the functionalized metal tip is selected from Cu, Ag or Pt.

10. The FM-AFM microscope according to any one of claims 7 to 9, wherein: The functionalized metal tip tip is functionalized with inert closed-shell atoms or molecules, preferably, the functionalized metal tip tip is functionalized with Xe atoms or CO molecules.

11. A computer program comprising instructions which, when said program is executed by a data processing device, cause the data processing device to perform steps (b) and (c) of the method according to claims 1 to 6 in said data processing device.

12. A computer-readable data carrier having stored thereon the computer program according to claim 11.

Images