Computer-implemented method for identifying molecules from atomic force microscope images and generating names of said molecules according to IUPAC naming method

Abstract

The invention relates to a computer-implemented method for identifying a molecule from an atomic force microscope image and generating a name of the molecule according to an IUPAC naming method using two trained multimode recurrent neural networks. Furthermore, the invention relates to a system configured to perform the steps of the method. Thus, the present invention concerns the field of nanotechnology, in particular the field associated with surface chemical reactions, and is therefore interested in atomic force microscope users and manufacturers.

Description

[0001] The present invention relates to a computer-implemented method for identifying molecules from atomic force microscopy images and generating names for said molecules according to IUPAC nomenclature by using two trained multimodal recurrent neural networks.

[0002] The present invention is therefore focused on the field of nanotechnology, particularly in relation to surface chemical reactions, and is therefore of interest to users and manufacturers of atomic force microscopes. Background Art

[0003] Scanning probe microscopy has played a key role in the development of nanoscience as a fundamental tool for the local characterization and manipulation of matter with high spatial resolution. In particular, atomic force microscopy (AFM) operating in frequency modulation (FM) mode allows the characterization and manipulation of a wide range of materials at the atomic scale. This is achieved by measuring the frequency changes caused by the interaction of the oscillating tip with the sample. The resolution is greatly improved when the tip is functionalized with inert closed-shell atoms or molecules (particularly CO molecules), providing access to the inner structure of the molecule.

[0004] This outstanding contrast originates from the Pauli repulsion between the CO probe and the sample molecules, which is modified by the electrostatic (ES) interaction between the potential generated by the sample and the charge distribution associated with the oxygen lone pairs on the probe. In addition, the flexibility of the molecular probe enhances the saddle line of the total potential energy surface (PES) sensed by CO. These high-resolution AFM (HR-AFM) capabilities make it possible to visualize frontier orbitals, determine bond order potentials and charge distributions, and open the door to tracking and controlling surface chemical reactions.

[0005] Despite these impressive achievements, one of the most important goals remains unrealized: molecular identification, or the ability to name a molecule simply by observing it with HR-AFM.

[0006] By combining AFM with other experimental techniques such as scanning tunneling microscopy (STM) or Kelvin probe force microscopy (KPFM), and with the support of theoretical simulations, molecules were identified (“Noncontact atomic force microscopy: Bond imaging and beyond” Q. Zhong, X. Li, H. Zhang, L. Chi, Surf. Sci. Rep. 75, 100509 (2020)).

[0007] Chemical identification of single atoms on semiconductor surface alloys by AFM has been achieved using reactive semiconductor tips. In this case, the maximum attraction between the tip tip and the probed atom on the sample carries information about the chemical species involved in the covalent interaction. However, the situation is quite different when using tips functionalized with inert CO molecules, where the main source of AFM contrast is Pauli repulsion and the image is strongly affected by probe relaxation. The few attempts to identify atoms in molecules by HR-AFM are based on differences found in the decay of the tip-sample interaction at the molecular position or on unique image features related to the chemical properties of certain molecular components. For example, substitutional N atoms on hydrocarbon aromatic rings show sharper vertices due to their lone pairs. In addition, the CO-sample interaction decays faster on these substitutional N atoms than on their neighboring C atoms. Halogen atoms can also be resolved in AFM images, which is attributed to their elliptical shape (related to their σ-holes) and to significantly stronger repulsions compared to atoms such as nitrogen or carbon. However, even these atomic features are strongly dependent on the molecular structure and cannot be related to a single species, but rather to its part in the molecule. Due to the wide variety of possible chemical environments, molecular identification is impossible with the naked eye alone.

[0008] Artificial intelligence (AI) technology is precisely optimized to handle such subtle correlations and massive amounts of data. Deep learning, with its outstanding pattern-finding capabilities, is now routinely used to classify, interpret, depict, and analyze images, providing machines with capabilities that have so far been unique to humans, even surpassing humans in some tasks. In our previous work (“A Deep Learning Approach for Molecular Classification Based on AFM Images” J. Carracedo-Cosme, C. Romero- R. Pérez, Nanomaterials 11, 1658 (2021), we restricted ourselves to essentially flat molecules and tested the potential of deep learning techniques to classify 60 different organic molecules from AFM images of constant height. Although encouraging, the apparent success of this proof-of-concept does not provide a solution to the general problem of molecular identification. Classification methods can only identify molecules contained in the training dataset. Given the rich complexity that organic chemistry offers, even a very large dataset already poses a huge computational challenge (since the output vector has the dimension of the number of molecules in the dataset) and does not allow for the classification of many molecules of interest that are known or could be synthesized.

[0009] Therefore, there is a need to develop new methods to achieve complete molecular identification (structure and composition) including non-planar structures by AFM imaging. Summary of the invention

[0010] The present invention relates to a computer-implemented method for obtaining a multimodal recurrent neural network (M-RNN) by using two different trained A and AM-RNN) generates the name of the molecule according to the IUPAC nomenclature to identify the molecule from the atomic force microscope image; each multimodal recurrent neural network (M-RNN A and AM-RNN) include convolutional neural network (CNN) components, recurrent neural network (RNN) components, and multimodal components Therefore, it is an object of the present invention to provide a text (molecule names according to IUPAC nomenclature) depicting an image (a plurality of acquired FM-AFM images).

[0011] IUPAC nomenclature is the most widely accepted and used nomenclature in the scientific community, and the IUPAC nomenclature is used as a molecular descriptor in the present invention. The IUPAC name clearly determines the molecular composition and structure. This is achieved by defining a hierarchical keyword list to name the functional groups written in a system grammar, which defines the structural position of each part or group in the molecule. In order to convert the IUPAC nomenclature into a suitable computational language, the present invention defines a set of terms, and each IUPAC name is decomposed into these terms. In the present invention, the term "set of terms into which each IUPAC name is decomposed" refers to a group of letters or symbols representing a molecular part, a ligand, or a specified atomic position used by the IUPAC nomenclature in this article. According to the IUPAC nomenclature, these terms are combined in a specific order to generate a molecular name.

[0012] In the method of the present invention, two different trained multimodal recurrent neural networks (M-RNN) are used. A The combination of M-RNN and AM-RNN generates the names of molecules according to IUPAC nomenclature. A The main chemical groups (main molecular parts) constituting the molecule are determined, a keyword (herein, IUPAC property) is defined for each part, while the second trained AM-RNN predicts the remaining IUPAC terms and compares the remaining IUPAC terms with the trained M-RNN A The determined IUPAC properties are assembled in a precise order, resulting in the IUPAC nomenclature for the molecular structure.

[0013] In the present invention, a "term" is a group of letters or symbols used by the IUPAC nomenclature to represent a specific position of a molecular part, a ligand or an atom. The combination of these terms generates the IUPAC name of the molecule.

[0014] In the present invention, the term "IUPAC property" refers to a 100-element subset of IUPAC terms that mainly describe atomic groups. Table 1 shows the terms decomposed by IUPAC. The elements above the double line are a subset of the 100 terms that are considered as properties. The gray cell does not correspond to any term, it is colored to distinguish it from the term that spells a space between the two words.

[0015] Table 1

[0016]

[0017] Quasar Science Resources SL-Universidad Autónoma de Madrid-Atomic Force Microscopy (QUAM-AFM) (https: / / doi.org / 10.21950 / UTGMZ7) was used to train the first multimodal recurrent neural network M-RNN A Network and the properties of the second trained multimodal recurrent neural network AM-RNN.

[0018] QUAM-AFM is a publicly available dataset that theoretically generates 165 million AFM images from 686,000 isolated molecules using 240 different combinations of AFM operating parameters (10 tip-sample distances, 6 different oscillation amplitudes, and 4 different values ​​of torsional stiffness of the CO molecule, which are known to depend on the details of how the molecule is attached to the tip of the metal tip). QUAM-AFM also provides ball-and-stick plots of each molecule generated from the atomic coordinates. These plots are in the same scale as used in the AFM images: if we superimpose the two images, each ball represented is centered on the position occupied by the atom it represents in the AFM image.

[0019] The QUAM-AFM dataset includes organic molecules, discarding all other compounds that may not be in pure molecular form (such as organic salts or inorganic compounds and polymers). The selected molecules contain the four basic elements of organic chemistry (carbon, hydrogen, nitrogen and oxygen) as well as some other less common elements that are still common in organic compounds (such as sulfur, phosphorus and halogen atoms (fluorine, chlorine, bromine and iodine)). The largest molecule in the QUAM-AFM database has a total of 85 atoms.

[0020] Very small molecules, i.e. those containing fewer than 8 atoms, are not good candidates for identification by AFM methods alone due to their extremely high surface mobility and the large number of different adsorption configurations and are therefore not included in the QUAM-AFM database. Very large molecules of square-cell based structures also do not form part of the QUAM-AFM dataset.

[0021] The QUAM-AFM database is limited to quasi-planar molecules, which only show up to The height of the change to include sp 3 Aliphatic chains with carbon atoms (methyl groups) as side groups. QUAM-AFM includes a group of molecules that includes aliphatic, cyclic and aromatic compounds, especially a large number of hydrocarbons (alkanes, alkenes, alkynes, etc.) as well as all traditional organic families (alcohols, thiols, ethers, aldehydes and ketones, carboxylic acids, amines, amides, imines, esters, nitriles, nitro and azo compounds, halogenated hydrocarbons and acyl halides, etc.).

[0022] The IUPAC names in QUAM-AFM images can be decomposed into a total of 199 terms. The maximum term length for IUPAC name decomposition in QUAM-AFM is 57.

[0023] Note that the class of a molecule is defined in this paper by the type of atomic species it contains and the number of repeated atoms of each species. In addition to hydrogen in the list of species, a representative number of QUAM-AFM is obtained for each class, so molecules with completely different structures such as pyrazine, pyridazine, butane-2-dicarbonitrile or butadiene cyanocyanide belong to the same class (C4N2). This leads to a total of 2339 classes of molecular structures considered in QUAM-AFM. See Figure 1 .

[0024] The multimodal recurrent neural network generates a new sentence description to explain the content of the image. In the method of the present invention, a first trained multimodal recurrent neural network M-RNN is used. A To obtain the main chemical groups that make up the molecule (i.e. the main molecular parts), and define the IUPAC property keywords for each part.

[0025] A multimodal recurrent neural network M-RNN trained first A The method comprises the following steps:

[0026] i) Provide the first trained multimodal recurrent neural network M-RNN A , the M-RNN A include

[0027] The first convolutional neural network CNN / M-RNNA A component comprising a 3D convolutional layer block and one or more dropout layers;

[0028] ·The first recurrent neural network RNN / M-RNN A a component comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a gated recurrent unit (GRU);

[0029] The first multimodal component It includes a connection layer with one or more dropout layers;

[0030] And image data generator

[0031] ii) pre-training with a set of known or predetermined AFM images of molecules fed to the first recurrent neural network RNN / M-RNN A Component's first convolutional neural network CNN / M-RNN A components, these images correspond to classes of molecules with the same chemical composition (number of different chemical species and number of atoms of each species, excluding H atoms); for example, AFM images of known or predetermined molecules corresponding to classes of molecules with the same chemical composition in the database QUAM-AFM, wherein the shape and contrast of said images and their variation with height show the 3D positions and sizes (chemical properties) of atoms in organic molecules and the distances between said atoms;

[0032] iii) using the first convolutional neural network CNN / M-RNN pre-trained in step (ii) with a plurality of AFM grayscale images of known or predetermined organic molecules at constant heights (e.g., AFM images of the database QUAM-AFM) A component, fed into the first trained multimodal recurrent neural network M-RNN A , and alternately,

[0033] Training RNN / M-RNN A component, update its weights, and fix CNN / M-RNN A The weight of the component,

[0034] Or train CNN / M-RNN A component, update its weights, and fix the RNN / M-RNN A The weights of the components are used to predict the IUPAC properties of organic molecules.

[0035] CNN / M-RNN AThe component includes a block of 3D convolutional layers and a dropout layer, and includes a modified Inception ResNet V2 model (C. Szegedy, S. Ioffe, V. Vanhoucke, AAAlemi, 31rd Proc. AAAI Conf. on Artificial Intelligence (AAAI Press, Palo Alto, CA, USA, 2017), p. 4278-4284), in which the 2D convolutional layer is replaced by a block containing two 3D convolutional layers, each with 32 filters, a kernel of size (3, 3, 3) and a stride of (2, 1, 1) in order to process 10 AFM images with different tip-sample distances, followed by a dropout layer. This dropout layer is essential for generalization to different images, such as experimental images. In addition, we remove the last fully connected layer of the model dedicated to the original classification task, thereby obtaining the output vector v.

[0036] RNN / M-RNN A It consists of an embedding layer, followed by a dropout layer and finally a recurrent layer (see Figure 2 ). Attributes are encoded by assigning an integer from 1 to 100 to each attribute. RNN / M-RNN A The input to is a vector of fixed size 19, which is the maximum number of different attributes in a molecule name from QUAM-AFM (18) plus the startseq tag. In the first step, it contains only an integer specifying the startseq tag, while in the following steps, it will include startseq plus an integer associated with the attribute predicted in the previous step. At each time step, the input is terminated with zeros until we reach a length of 19.

[0037] The embedding layer processes the input to represent each attribute in a vector space, converting the input vector into a dense vector with real values ​​that reflect the syntax and semantics of the attribute, placing similar attributes close together in the vector space after neural connections are established during training. For example, the attributes closest to bromine are chlorine, fluorine, and iodine.

[0038] The recurrent layer stores information about previous predictions in its internal state. In attribute prediction, we use a gated recurrent unit (GRU) as the recurrent layer. GRU is computationally efficient and suitable for short chains of attributes to be predicted. It is necessary to introduce a dropout layer between the embedding layer and the recurrent layer to avoid overfitting during training.

[0039] Multimodal components The CNN output v is first processed in two fully connected layers with a dropout between them. The resulting vector is concatenated with the output vector of the RNN to feed into two fully connected layers, resulting in a probability vector. This final vector has 103 components: 100 components related to the attribute, one with padding, and two with startseq and endseq tokens. The position of the larger component in the vector gives us the prediction for the new attribute.

[0040] This property predicts that only RNN / M-RNN A The startseq in the input (S0, 0, ..., 0) marks the beginning of S0. For a given time step t, we feed RNN / M-RNN A Feed input (S0, Y1 A , …, Y A t-1 , 0…, 0), which concatenates S0 with all the predictions already made in the previous time steps and pads until we have a length of 19 (see Figure 3 ). This process is repeated until the endseq marker is predicted and the closed loop is broken (see Figure 4 ).

[0041] Figure 5 Provides M-RNN A RNN / M-RNN A and Details of each layer of the component, including operators, dimensions, and activation functions.

[0042] Object M-RNN A Such a complex network is very time-consuming to train globally and is prone to overfitting in components with fewer layers. Therefore, the first multimodal recurrent neural network M-RNN trained A Training is done in different stages:

[0043] First, a set of known or predetermined molecular AFM images corresponding to the categories of molecules sharing the same chemical composition (e.g., using the AFM images of known or predetermined molecules corresponding to the categories of molecules sharing the same chemical composition in the database QUAM-AFM) is pre-trained to feed the first recurrent neural network CNN / M-RNN A Component's first convolutional neural network CNN / M-RNN A The shape and contrast of the image and its variation with height show the 3D position, size (chemical properties) and distance between atoms in the organic molecule.

[0044] Secondly, the first convolutional neural network CNN / M-RNN pre-trained with a plurality of constant height AFM grayscale images of known or predetermined organic molecules (such as AFM images of the database QUAM-AFM) A The components are fed into the first trained multimodal recurrent neural network M-RNN A , and alternately,

[0045] Training RNN / M-RNN A Components of CNN / M-RNN are fixed at the same time A Component weights, for RNN / M-RNN A Components and CNN / M-RNN A Component weighting,

[0046] Or, train a CNN / M-RNN A Components while fixing RNN / M-RNN A Component weights, for CNN / M-RNN A Components and RNN / M-RNN A Component weighting to predict IUPAC properties in organic molecules.

[0047] In the training first training multimodal recurrent neural network M-RNN A During the experiment, one of the 24 combinations of AFM operating parameters (6 different oscillation amplitudes and 4 different values ​​of the torsional stiffness of the CO molecule) available for each input stack in QUAM-AFM was randomly selected.

[0048] Ensure that the parameters of the AFM experiment do not affect the first trained multimodal recurrent neural network M-RNN A Successful identification of the structure plays a decisive role.

[0049] Prevent overfitting, and

[0050] · The first trained multimodal recurrent neural network M-RNN A Provides generalization capabilities.

[0051] This variability is further enhanced by applying an Image Data Generator (IDG) to the training set, which applies different deformations (scaling, rotation, offset, flipping, and shearing) to the input images ( Figure 6) and normalize the pixel values. The motivation for using IDG is that experimental images have some features that cannot be captured by AFM simulations, which may hinder recognition. For example, experimental images do not show the full symmetry of organic molecules. These differences between experimental and theoretical AFM images of a given organic molecule may be due to the inevitable presence of noise in the experiment, tip asymmetries that are not included in the AFM image simulations, or relaxation and deformation of the organic molecules due to interactions with the substrate, while we are considering an ideal gas-phase structure in the simulated AFM images used for training.

[0052] Applying the deformation provided by IDG during training simulates these effects and significantly improves the first trained multimodal recurrent neural network M-RNN. A Ability to identify organic molecules from experimental images. Selecting appropriate deformation parameters for IDG is very important, as the correct choice can greatly improve the accuracy of identification.

[0053] The first trained multimodal recurrent neural network M-RNN A Includes (see Figure 2 ):

[0054] Convolutional Neural Network CNN / M-RNN A Component, which includes a 3D convolutional layer block and one or more dropout layers, a convolutional neural network CNN / M-RNN A The component is configured to encode a plurality of constant-height atomic force microscope images into an output vector v,

[0055] Recurrent Neural Network RNN / M-RNN A A component comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a gated recurrent unit (GRU) for processing language processes, a recurrent neural network RNN / M-RNN A The component is configured to embed the representation of each attribute based on its semantic meaning and put similar attributes together. Secondly, the semantic temporal context is stored in the recurrent layer.

[0056] Multimodal components It includes connected layers with one or more dropout layers, multimodal components Configured to combine and process CNN / M-RNN A and RNN / M-RNN A Output to predict one of the attributes,

[0057] and an image data generator configured to apply pixel value normalization to the theoretical AFM image, as well as different random deformations selected from scaling of [-15, 15]%, rotation of [-180, 180] degrees, vertical and horizontal offsets, random vertical and horizontal flips, shearing of [-20, 20]% and any combination thereof, to provide simulated features of the experimental image (such as the presence of noise and effects related to tip asymmetry) that are not present in the AFM simulation and may hinder identification.

[0058] A second trained multimodal recurrent neural network AM-RNN is used in the method of the present invention to obtain the name of the molecule according to the IUPAC nomenclature, using QUAM-AFM to predict the different terms in the IUPAC name of the molecule in the correct order.

[0059] A method for training a second trained multimodal recurrent neural network AM-RNN, comprising the following steps:

[0060] 1) Providing a second trained multimodal recurrent neural network AM-RNN, wherein the AM-RNN includes

[0061] a second convolutional neural network CNN / AM-RNN component, which includes a block of 3D convolutional layers and one or more dropout layers,

[0062] a second recurrent neural network RNN / AM-RNN component comprising one or more embedding layers, one or more dropout layers and one or more recurrent layers, wherein at least one recurrent layer is a long short-term memory (LSTM),

[0063] Second multimodal component It includes connected layers with one or more dropout layers,

[0064] and a second image data generator;

[0065] 2) training a first convolutional neural network CNN / M-RNN with a set of known or predetermined AFM images of molecules corresponding to the categories of molecules sharing the same chemical composition (e.g., with AFM images of known or predetermined molecules corresponding to the categories of molecules sharing the same chemical composition in the database QUAM-AFM) A The second convolutional neural network CNN / M-RNN of the component A A component wherein the shape and contrast of the image and its variation with height show the 3D position, size (chemical properties) and distances between atoms in an organic molecule.

[0066] 3) Multiple constant height AFM grayscale images of known or predetermined organic molecules (e.g., AFM images of the database QUAM-AFM and the multimodal recurrent neural network M-RNN trained in the first step) are used to generate the image. A The first convolutional neural network CNN / AM-RNN component trained in step (2) is fed into the second multimodal recurrent neural network AM-RNN, and alternately

[0067] While training the RNN / AM-RNN component, fix the weights of the CNN / AM-RNN component, and weight the RNN / AM-RNN component and the CNN / AM-RNN component.

[0068] Or train the CNN / AM-RNN component while fixing the weights of the RNN / AM-RNN component, weight the CNN / AM-RNN component and the RNN / AM-RNN component to generate the IUPAC names of organic molecules.

[0069] The CNN / M-RNNA component includes a block of 3D convolutional layers and a dropout layer, and includes a modified Inception ResNet V2 model (C. Szegedy, S. Ioffe, V. Vanhoucke, AAAlemi, 31rd Proc. AAAI Conf. on Artificial Intelligence (AAAI Press, Palo Alto, CA, USA, 2017), p. 4278-4284), in which the 2D convolutional layer is replaced by a block containing two 3D convolutional layers, each with 32 filters, a kernel of size (3, 3, 3) and a stride of (2, 1, 1) to process 10 AFM images with different tip-sample distances, followed by a dropout layer. This dropout layer is essential for generalization to different images, such as experimental images. In addition, we remove the last fully connected layer of the model dedicated to the original classification task, thereby obtaining the output vector v.

[0070] The RNN / AM-RNN component consists of an embedding layer, followed by a dropout layer and finally a recurrent layer (see Figure 2 ). The terms are encoded by assigning an integer (from 1 to 199) to each term. The input to the RNN / AM-RNN is a vector of fixed size 76. This number comes from the maximum number of different attributes in a molecule name in QUAM-AFM plus the sum of startseq tokens (18+1=19), and the maximum number of terms in the IUPAC name of a molecule in QUAM-AFM is 57. Each RNN / AM-RNN input is a vector of size 76, which is obtained by first training a multimodal recurrent neural network M-RNNA It is concatenated with the predicted attribute with the startseq token (padded with zeros if less than 18) and the term predicted at each previous time step (padded until we get a vector of length 57). In the first step it will contain integers specifying the startseq token and the attribute, while in the following steps it will also include integers associated with the terms predicted in the previous steps.

[0071] The embedding layer processes the input to represent each term in a vector space, converting the input vector into a dense vector with real values ​​that reflect the syntax and semantics of the term, placing similar inputs close together in the vector space to follow the neural connections established during training. For example, terms representing numbers are represented in a close manner, i.e. the terms closest to nona are octa, deca, undeca, and dodeca.

[0072] The recurrent layer stores information about previous predictions in its internal state. In term prediction, we use a Long Short-Term Memory (LSTM) network as a recurrent layer. LSTM is more efficient than RNN / M-RNN on long time series. A The GRU used in is more accurate and suitable for predicting longer term strings, with a maximum length of 57 terms. Finally, it is necessary to introduce a dropout layer between the embedding layer and the recurrent layer to avoid overfitting during training.

[0073] Multimodal components First, process the CNN / M-RNN in two fully connected layers A Output v, there is a dropout layer between these two fully connected layers. The resulting vector is concatenated with the output of the RNN / AM-RNN to feed two fully connected layers, resulting in a probability vector. This vector has 202 components: 199 components are related to the term, one has padding, and two have startseq and endseq tokens. The position of the larger component in the vector gives us the prediction for the new term.

[0074] Term predictions start with the list of attributes predicted in step (b) (padded if necessary), plus the startseq token (Y1 A , …, Y A 18 , startseq, 0, ..., 0) as the input of RNN / AM-RNN. For a given time step t, RNN / AM-RNN is fed with input (Y1 A , …, Y A 18 , startseq, Y1 T , …, Y T t-1, 0…, 0), where , includes all term predictions that have been performed in the previous time step, padded with zeros to obtain a length of 57 (i.e., the maximum number of terms). This process is repeated until the endseq token is predicted and the closed loop is broken (see Figure 4 ). Figure 7 The input and output at each time step of the term prediction for the perylene-1,12-diol molecule using the AM-RNN are shown. Figure 3 The state of the RNN / AM-RNN and a representation of the AM-RNN input vector in the fourth time step are shown.

[0075] Figure 5 Provides RNN / AM-RNN and Details of each layer of the component, including operators, dimensions, and activation functions.

[0076] First, a second convolutional neural network CNN / AM-RNN component fed to the second recurrent neural network CNN / AM-RNN component is pre-trained with a set of known or predetermined AFM images of molecules corresponding to categories of molecules sharing the same chemical composition (for example, AFM images of known or predetermined molecules corresponding to categories of molecules sharing the same chemical composition in a database QUAM-AFM), wherein the shape and contrast of the images and their variations with height show the 3D positions and sizes (chemical properties) of atoms in organic molecules and the distances between the atoms.

[0077] Secondly, a second convolutional neural network CNN / AM-RNN component pre-trained with a plurality of AFM grayscale images of known or predetermined organic molecules at constant height (e.g., AFM images from the database QUAM-AFM) is fed to a second trained multimodal recurrent neural network AM-RNN, and alternately

[0078] While training the RNN / AM-RNN component, fix the weights of the CNN / AM-RNN component, and weight the RNN / AM-RNN component and the CNN / AM-RNN component.

[0079] Or train the CNN / AM-RNN component while fixing the weights of the RNN / AM-RNN component, weight the CNN / AM-RNN component and the RNN / AM-RNG component to generate the IUPAC names of organic molecules.

[0080] The second trained multimodal recurrent neural network AM-RNN includes (see Figure 2 )

[0081] a second convolutional neural network CNN / AM-RNN, comprising a convolutional layer block and one or more dropout layers, configured to encode the multiple constant-height atomic force microscope images obtained in step (a) into an output vector v;

[0082] a second recurrent neural network (RNN) / AM-RNN component, comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a long short-term memory (LSTM), configured to process language processes and embed the representation of each IUPAC term according to its semantics, placing similar terms close together, and secondly, storing the semantic temporal context in the recurrent layer;

[0083] Second multimodal component It includes a connection layer with one or more dropout layers responsible for combining and processing the CNN / AM-RNN and RNN / AM-RNN outputs to predict one of the IUPAC terms;

[0084] and a second image data generator configured to apply pixel value normalization to the theoretical AFM image, as well as different random deformations selected from [-15, 15]% scaling, [-180, 180] degrees rotation, vertical and horizontal offsets, random vertical and horizontal flips, [-20, 20]% shearing, and any combination thereof, to provide simulated features of the experimental image (such as the presence of noise and effects related to tip asymmetry) that are not present in the AFM simulation and may hinder identification.

[0085] A first aspect of the present invention relates to a computer-implemented method for identifying organic molecules from atomic force microscopy images and generating names for the organic molecules according to IUPAC nomenclature, the method being characterized in that it comprises the following steps:

[0086] a) acquiring a plurality of constant height atomic force microscopy images of an organic molecule at different tip height distances above the molecule by means of a frequency mode atomic force microscope (FM-AFM microscope) using a functionalized metal tip tip, wherein the distances of the different tip heights range from 280 pm to 370 pm above the molecule, wherein the shape and contrast of the images and their variation with tip height show the 3D position, size (chemical properties) and distances between atoms in the organic molecule,

[0087] b) providing the first trained multimodal recurrent neural network M-RNN to the data processor device A , where the first trained multi-mode recurrent neural network M-RNN A include

[0088] -The first convolutional neural network CNN / RNN A component, which includes a block of 3D convolutional layers and one or more dropout layers,

[0089] -The first recurrent neural network component RNN / M-RNN A A component comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a gated recurrent unit (GRU), and

[0090] -First Multimodal components comprising a connected layer with one or more dropout layers,

[0091] c) using the first trained multimodal recurrent neural network M-RNN using the atomic force microscope image obtained in step (a) A The first trained multimodal recurrent neural network M-RNN is fed to the data processor device. A Generate IUPAC properties with syntax and semantics for organic molecules;

[0092] d) providing a second trained attribute multimodal recurrent neural network AM-RNN to a data processor device, wherein the second trained attribute multimodal recurrent neural network AM-RNN includes

[0093] - a second convolutional neural network CNN / AM-RNN component comprising a block of 3D convolutional layers and one or more dropout layers,

[0094] - a second recurrent neural network component RNN / AM-RNN component comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a long short-term memory (LSTM), and

[0095] - Second Multimodality components comprising a connected layer with one or more dropout layers,

[0096] e) feeding a second trained multimodal recurrent neural network AM-RNN using the IUPAC properties obtained in step (d) and the atomic force microscope image obtained in step (a) to a data processor device, wherein the second trained multimodal recurrent neural network AM-RNN generates an IUPAC name for the organic molecule.

[0097] AFM operated in frequency modulation dynamic mode (FM-AFM) can characterize and manipulate a variety of materials at the atomic scale by measuring the frequency changes caused by the interaction of an oscillating tip with an organic molecule sample. Resolution is improved when the tip is functionalized with inert closed-shell molecules, particularly CO molecules, allowing access to the internal structure of the molecule. The significant contrast stems primarily from the Pauli repulsion between the CO probe and the sample molecules. This repulsion occurs because the electron density of the tip and sample overlap, resulting in an increased frequency shift, which is a change in the oscillation frequency of the cantilever holding the tip due to the tip-sample interaction. The shift is observed as a bright feature in the constant height AFM image above the atomic positions, sizes, and bonds (distances between atoms), reflecting the organic molecular structure.

[0098] The term "functionalized metal tip" refers herein to a tip of a metal (usually Cu), but other metals (such as Ag and Pt) may also be used, which is functionalized with inert closed shell atoms or molecules, thereby significantly improving the resolution of the AFM image and providing access to the internal structure of organic molecules. Examples of inert closed shell atoms or molecules are Xe atoms and CO molecules, respectively.

[0099] A CO-functionalized metal tip is preferred in the present invention because it increases the AFM image contrast:

[0100] The Pauli repulsion between the lone pairs of the oxygen atoms in the CO molecule and the charge density of the sample molecule is very directional, due to the preferential distribution of the electronic charge of the lone pairs along the molecular axis, and

[0101] The CO molecule attached to the metal tip provides a complex electric field with a very localized central feature, located just in front of the O atom and repelling the electrons in the sample molecule, which provides rapidly changing electrostatic interactions both vertically and laterally, and

[0102] The tilt of the CO molecule significantly enhances the intramolecular features.

[0103] Therefore, in a preferred embodiment of the present invention, the functionalized metal tip used in step (a) is selected from Cu, Ag or Pt.

[0104] In another preferred embodiment of the method of the present invention, the functionalized metal tip used in step (a) is functionalized with an inert closed shell atom or molecule. More preferably, the functionalized metal tip is functionalized with a Xe atom or a CO molecule.

[0105] In order to collect multiple AFM grayscale images of constant height, different tip height distances ranging from 280pm to 370pm above the organic molecule must be obtained by an FM-AFM microscope, which allows the collection of images with different contrasts as a function of the tip height distance. The shape and contrast of the image and their changes with the tip height define the 3D spatial distribution of the charge, which is caused by the interaction of the chemical substances, their chemical environment, and their relative heights relative to other atoms in the molecular configuration. Therefore, it is crucial to collect multiple images because it contains all the information about the 3D spatial distribution of the charge. Preferably, at least 10 images should be taken to correctly characterize the FM-AFM contrast within the tip height distance range between 280pm and 370pm above the organic molecule. In this tip height range, Pauli repulsion and the electrostatic interaction between the tip and the organic molecule vary greatly, resulting in a strong change in the contrast of the FM-AFM image, including the characteristics of the atoms and their molecular environment. The shape and contrast of the image and their changes with the tip height contain all the information about the 3D position and size (chemical properties) of the atoms and the distance between the atoms.

[0106] Therefore, in a preferred embodiment of the method of the present invention, a plurality of at least 10 AFM grayscale images of the organic molecules with constant height are collected in step (a).

[0107] In another preferred embodiment of the method of the present invention, step (a) is performed at at least 10 different tip height distances.

[0108] Step (b) of the method of the present invention involves providing a first trained multimodal recurrent neural network M-RNN to a data processor device. A , where the first trained multimodal recurrent neural network M-RNN A include

[0109] -The first convolutional neural network CNN / RNN A component, which includes a block of 3D convolutional layers and one or more dropout layers,

[0110] -The first recurrent neural network component RNN / M-RNN A A component comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a gated recurrent unit (GRU), and

[0111] -First Multimodal components comprising a connected layer with one or more dropout layers,

[0112] Step c) using the first trained multimodal recurrent neural network M-RNN using the atomic force microscope image obtained in step (a) A The first trained multi-mode recurrent neural network M-MNN is fed to the data processor device. A Generate IUPAC properties with syntax and semantics of organic molecules. As mentioned before, these properties are mentioned in Table 1.

[0113] In step (d), a second trained attribute multimodal recurrent neural network AM-RNN is provided to a data processor device, wherein the second trained attribute multimodal recurrent neural network AM-RNN includes

[0114] - a second convolutional neural network CNN / AM-RNN component comprising a block of 3D convolutional layers and one or more dropout layers,

[0115] - a second recurrent neural network component RNN / AM-RNN component comprising one or more embedding layers, one or more dropout layers and one or more recurrent layers, wherein at least one recurrent layer is a long short-term memory (LSTM), and

[0116] - Second Multimodality A component comprising a connected layer with one or more dropout layers.

[0117] Step (e) refers to feeding a second trained multimodal recurrent neural network AM-RNN using the IUPAC properties obtained in step (d) and the atomic force microscope image obtained in step (a) to a data processor device, wherein the second trained multimodal recurrent neural network AM-RNN generates the IUPAC name of the organic molecule.

[0118] Another aspect of the present invention relates to a frequency modulated atomic force microscope (FM-AFM) microscope, which comprises a functionalized metal tip tip (herein the microscope of the present invention) configured to perform step (a) of the method of the present invention as described above and a data processor device configured to perform steps (b) to (e) of the method of the present invention as described above.

[0119] Preferably, the microscope of the present invention further comprises a display unit connected to the data processor device and configured to display the IUPAC molecular names obtained in step (e) of the method according to the present invention. More preferably, the display unit connected to the data processor device is further configured to display the structural representation of the molecule identified in step (e) of the method according to the present invention in the form of a ball-and-stick diagram.

[0120] In another preferred embodiment of the microscope of the present invention, the metal of the functionalized metal tip is selected from Cu, Ag or Pt. The metal tip improves the resolution of FM-AFM images.

[0121] In another preferred embodiment of the microscope of the present invention, the functionalized metal tip tip is functionalized with inert closed shell atoms or molecules, preferably the functionalized metal tip tip is functionalized with inert closed shell Xe atoms or inert closed shell CO molecules. The functionalized metal tip tip significantly improves the resolution of FM-AFM images.

[0122] Another aspect of the present invention relates to a computer program (herein referred to as the computer program of the present invention) comprising instructions which, when executed by a data processor device, cause the data processor device to perform steps (b) to (e) of the method of the present invention as described above in the data processor device.

[0123] A final aspect of the invention relates to a computer-readable data carrier having stored thereon the computer program of the invention as described above.

[0124] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those of ordinary skill in the art to which the invention belongs. Methods and materials similar or equivalent to the methods and materials described herein may be used in the practice of the present invention. Throughout the specification and claims, the term "comprising" and its variations are not intended to exclude other technical features, additives, ingredients or steps. Other objects, advantages and features of the present invention will become apparent to those skilled in the art after reading the specification, or may be learned by practicing the present invention. The following examples and drawings are provided by way of illustration and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] Figure 1 : Atomic structures that belong to different categories depending on their chemical species.

[0126] Figure 2 M-RNN is shown A The graphical layer representation of the common structure shared by AM-RNN and its three components CNN, RNN and The CNN component follows the Inception ResNet V2 model, where the first 2D convolutional layer is replaced by two 3D convolutional layers (for processing image stacks) followed by a dropout layer (grey). The following block is just a graphical representation of the complex architecture (164 layers) of the original InceptionResNet V2 model. Note that the last fully connected layer of the model dedicated to the original classification task has been removed, resulting in an output vector (v) that will be represented by The components are further processed. The RNN components include embedding layers (white), dropout layers (grey), and recurrent layers (black). The black box represents the M-RNN A In AM-RNN, it represents the GRU layer, while in AM-RNN it represents the LSTM layer. The component processes the output vector of the CNN component with two fully connected layers (white), with a dropout layer (grey) between them, concatenating this result with the output of the RNN component and processing it through another two fully connected layers to produce a probability vector. A In the case of (AM-RNN), the position of the larger component in the vector gives us the prediction of the new attribute (term). Figure 5 Provides M-RNN A and the RNN components of AM-RNN and Further details of each component's layers, including operators, dimensions, and activation functions.

[0127] Figure 3 Shown with Figure 4 The M-RNN corresponding to the perylene-1,12-diol molecule uses the same format as in A And the representation of the RNN at the fourth time step in AM-RNN. This figure highlights the fact that the state of the RNN (especially the recurrent layers) depends on the previous predictions.

[0128] Figure 4 The architecture scheme of the method of the present invention including two different multimodal recurrent neural networks is shown, and the data flow is as follows: (a) M-RNN A and (b) AM-RNN. The rectangular boxes represent the three components of each M-RNN: convolutional neural network CNN, recurrent neural network RNN, and multimodal component In the RNN component, x t represents the input at time step t, h t represents the internal state of the loop layer at t, o t is the output of the RNN at t. The arrows indicate the information flow within the model. A Attributes are predicted at each time step until the loop is broken with the endseq marker, while the AM-RNN predicts ranked terms (one at each time step) that produce IUPAC names. Figure 7 It shows that the M-RNN A Examples of the inputs and outputs at each time step predicted by the AM-RNN network from a 3D image stack corresponding to a perylene-1,12-diol molecule (taken from the QUAM-AFM dataset). Figure 3 The M-RNN of the perylene-1,12-diol molecule can be found in ARepresentation of the RNN corresponding to the fourth time step in AM-RNN.

[0129] Figure 5 M-RNN is shown A and the RNN integrated in AM-RNN and The layer-by-layer details of the components. v represents the output vector of the CNN component. Except for the last layer of RNN, M-RNN A The operators of both models are the same, except for the GRU in CNN (attribute prediction) and the LSTM in AM-RNN (term prediction). However, due to the different sizes of inputs and outputs in each model, the units are also different.

[0130] Figure 6 The results of applying an image data generator (IDG) to the same AFM image of the N-(5-amino-4-methylpyridin-2-yl)-6-fluoro-1-benzothiophene-2-carboxamide molecule are shown. The parameters selected for the IDG during training were randomly selected within the range of [-180, 180] degrees of rotation, scaling and vertical and horizontal offsets of [-15, 15]%, shearing and random vertical and horizontal flips of [-20, 20]%. When necessary, the nearest fill was applied to points outside the input boundaries. Appropriate selection of the parameter ranges for the IDG used during training can significantly improve the accuracy of IUPAC name predictions when the method of the present invention is applied to experimental images.

[0131] Figure 7 It shows that the M-RNN A and the input and output at each time step predicted by the AM-RNN network from a 3D image stack corresponding to the case of the perylene-1,12-diol molecule (taken from the QUAM-AFM dataset). Figure 3 The t = 4 time step for both networks analyzed in detail in is highlighted in grey.

[0132] Figure 8 AFM experimental images of dibenzothiophene taken at different tip-sample distances are shown (from P. Zahl, Y. Zhang, Energy Fuels 33, 4775 (2019)). Despite the presence of strong noise and white lines running diagonally across the image, the method of the present invention provides perfect prediction of the IUPAC name.

[0133] Fig. 9 Examples of perfect predictions for molecules containing carbon, nitrogen, oxygen and different halogen atoms are shown. Each sub-figure (from left to right) shows a ball-and-stick diagram of the molecular structure (generated using the open source code Jmol) and five AFM images at different tip-sample distances. The IUPAC names predicted by the method of the present invention that perfectly match the real data are shown below the images. DETAILED DESCRIPTION

[0134] Example 1: Predicting the IUPAC name from an experimental AFM image of dibenzothiophene

[0135] This example is based on experimental AFM images of dibenzothiophene adsorbed on Au(111) measured in constant height mode using a CO functionalized tip. The measurements were performed using a Createc based low temperature (LT)-STM upgraded with an AFM and using custom GXSM control software. A Q-Plus sensor (operating at 30kHz with a typical Q factor of about 10000) with a focused ion beam (FIB) sharpened PtIr tip wire functionalized with CO molecules was used. The AFM was operated in constant height mode with feedback turned off without the need for correction for about 1 hour or more. After at least 24 hours at 5K, the XYZ drift and creep were less than 1pm / h and there were no major changes in the XYZ offset. The molecules were imaged with dual modality STM and AFM at ultra-high pressure at 5K after deposition using a home-made evaporator.

[0136] Regarding the operating conditions, the tip oscillation amplitude was about 50 pm with an applied bias voltage of 40 mV. This bias minimizes the electrostatic forces between the Au(111) surface and the actual tip tip and provides stable imaging conditions. For frequency tracking and amplitude tuning, a novel phase-amplitude convergence detector and a phase-locked loop were used. In this paper, Au(111) single crystals were used as substrates, which were typically cleaned with 3 cycles of Ar+ sputtering / annealing before use. After CO molecule functionalization, tip tuning was performed by controlled impact on the Au surface by close scanning and picking at very low bias.

[0137] Figure 8 Shown is a set of 10 AFM images of dibenzothiophene adsorbed on Au(111) taken at different tip-sample distances.

[0138] As mentioned above, the M-RNN trained with the QUAM-AFM dataset A Applied to Figure 8 A stack of 10 AFM images of constant height is shown. M-RNN A The properties benz, phen, and thi are generated correctly.

[0139] As described above, the AM-RNN trained with the QUAM-AFM dataset is fed Figure 1 The 10 AFM images with constant heights and the images generated by M-RNN are shown. AThe AM-RNN predicted the terms in the molecule’s name in the correct order: di, benz, o, thi, o, phen, and e. It is important to note that we got perfect predictions despite the strong noise and the white lines running diagonally across the image.

[0140] Example 2: Predicting IUPAC names from theoretically simulated AFM images in the QUAM-AFM dataset

[0141] The second example demonstrates the ability of our method to predict IUPAC names for non-planar molecules with complex structure and composition, which include most of the relevant chemical species in organic chemistry. In this case, we fed our model with theoretically simulated AFM images from the QUAM-AFM dataset.

[0142] The images were calculated with a torsional stiffness of 0.40 N / m and an oscillation amplitude of 40 pm. During training, none of the three molecules considered were fed to the M-RNN. A or AM-RNN shown above. For each molecule, we consider 10 AFM images of constant height calculated at different tip heights ranging from 280 pm to 370 pm, with an increment of 10 pm between images.

[0143] Fig. 9 Five images from a stack of 10 AFM images of constant height are shown for each molecule studied, along with ball-and-stick plots generated with the open source Java chemical structure viewer Jmol (http: / / www.jmol.org / ) to graphically illustrate their structure and composition.

[0144] for Fig. 9 In the case shown in a, the trained M-RNN A The properties of the molecule were correctly generated. These properties (in alphabetical order) are: azin, hydr, ide, imidin, meth, one, oxy, phen, pyr, and yl. The trained AM-RNN predicted all terms and formed the IUPAC names in the correct order in all cases. Note that this means that the method of the present invention has accurately identified all molecular parts from the image and provided the exact IUPAC name character by character.

[0145] Considering the molecules in this example, the method of the invention is able to identify planar hydrocarbons, including cyclic or aliphatic ones, but also more complex structures such as those containing nitrogen or oxygen atoms, which usually appear as faint features on the image due to their rapid charge density decay.

[0146] The halogens are represented by elliptical features on the image, whose size and intensity are proportional to their σ-hole intensity and are also correctly labeled ( Fig. 9 b, 9d and 9e).

[0147] The method of the present invention is even able to identify the presence of fluorine, which does not induce σ-vacancies and, when bound to carbon atoms, produces an AFM fingerprint very similar to that of a carbonyl group (compare Fig. 9 e and Fig. 9 f) The location of hydrogens is often left to guesswork, which is particularly striking because of the sp 2 The charge density of hydrogen atoms bound to carbon atoms is almost negligible, making them difficult to detect by high-resolution AFM (HR-AFM).

[0148] Example 3: Comparison example

[0149] A single M-RNN is fed with a stack of AFM images to predict the IUPAC name of the imaged molecule. We followed this route and used a model exactly like the AM-RNN. The only difference is that the input to the RNN component is a vector of length 58, which only includes the startseq token in the first step. The M-RNN is trained in three stages, alternating between fixing the weights of the CNN and RNN components. This M-RNN provides 26 predicted IUPAC names.

Claims

1. A computer-implemented method for identifying organic molecules from atomic force microscopy images and generating names for the organic molecules according to IUPAC nomenclature, the method comprising the following steps: a) collecting a plurality of constant height atomic force microscopy images of the organic molecule at different tip height distances above the organic molecule by frequency mode atomic force microscopy using a functionalized metal tip, wherein: The different tip height distances range between 280 pm and 370 pm, wherein the shape and contrast of the image and its variation with tip height show the 3D position of atoms in the organic molecule, the size of atoms and the distance between the atoms; b) providing the first trained multimodal recurrent neural network M-RNN to the data processor device A , wherein the first trained multimodal recurrent neural network M-RNN A include -The first convolutional neural network CNN / RNN A component, which includes a block of 3D convolutional layers and one or more dropout layers, -The first recurrent neural network component RNN / M-RNN A A component comprising one or more embedding layers, one or more dropout layers, and one or more recurrent layers, wherein at least one recurrent layer is a gated recurrent unit (GRU), and -First Multimodal / AM-RNN component, which includes a connection layer with one or more dropout layers, c) using the first trained multimodal recurrent neural network M-RNN using the atomic force microscope image obtained in step (a) A fed to the data processor device, the first trained multimodal recurrent neural network M-RNN A Generate IUPAC properties with syntax and semantics for organic molecules; d) providing a second trained attribute multimodal recurrent neural network AM-RNN to the data processor device, wherein the second trained multimodal recurrent neural network AM-RNN comprises - a second convolutional neural network CNN / AM-RNN component comprising a block of 3D convolutional layers and one or more dropout layers, - a second recurrent neural network component RNN / AM-RNN component comprising one or more embedding layers, one or more dropout layers and one or more recurrent layers, wherein at least one recurrent layer is a long short-term memory (LSTM), and - Second Multimodality / AM-RNN component, which includes a connection layer with one or more dropout layers, e) feeding a second trained multimodal recurrent neural network AM-RNN using the IUPAC properties obtained in step (d) and the atomic force microscope image obtained in step (a) to the data processing device, wherein the second trained multimodal recurrent neural network AM-RNN generates the IUPAC name of the organic molecule.

2. The method according to claim 1, wherein: In step (a), a plurality of at least 10 atomic force microscope images of the organic molecule at constant height are collected.

3. The method according to any one of claims 1 or 2, wherein: Step (a) is performed at at least 3 different height distances, preferably at at least 10 different tip height distances.

4. The method according to any one of claims 1 to 3, wherein: The functionalized metal tip used in step (a) is selected from Cu, Ag or Pt.

5. The method according to any one of claims 1 to 4, wherein: The functionalized metal tip used in step (a) is functionalized at the tip with an inert closed shell atom or molecule.

6. The method according to any one of claims 1 to 5, wherein: The functionalized metal tip used in step (a) is functionalized at the tip with Xe atoms or CO molecules.

7. A frequency modulated atomic force microscope (FM-AFM) microscope, comprising a functionalized metal tip and a data processing device, wherein the functionalized metal tip is configured to perform step (a) of the method according to any one of claims 1 to 6, and the data processing device is configured to perform steps (b) to (e) of the method according to any one of claims 1 to 6.

8. The FM-AFM microscope according to claim 7, further comprising a display unit connected to the data processing device, the display unit being configured to display the name of the molecule obtained according to IUPAC in step (e) of the method according to any one of claims 1 to 6.

9. The FM-AFM microscope according to any one of claims 7 or 8, wherein: The display unit connected to the data processing device is further configured to display a structural representation of the molecule identified in step (e) of the method according to any one of claims 1 to 6 in the form of a ball-and-stick diagram.

10. The FM-AFM microscope according to any one of claims 7 to 9, wherein: The metal at the top of the functionalized metal tip is selected from Cu, Ag or Pt.

11. The FM-AFM microscope according to any one of claims 7 to 10, wherein: The functionalized metal tip tip is functionalized with inert closed shell atoms or molecules.

12. The FM-AFM microscope according to any one of claims 7 to 11, wherein: The functionalized metal tip tip is functionalized with Xe atoms or CO molecules.

13. A computer program comprising instructions which, when said program is executed by a data processing device, cause the data processing device to perform steps (b) to (e) of the method according to claims 1 to 6 in said data processing device.

14. A computer-readable data carrier having stored thereon the computer program according to claim 13.

Images