Active learning model validation

Abstract

Methods, apparatus, and computer-implemented methods are provided for training machine learning (ML) techniques to generate a property model for predicting whether a compound has a particular property. An iterative process / feedback loop can be performed to generate a property model, the process including generating, based on the property model, a list of prediction results for a plurality of compounds and their association with the particular property, validating the property model based on compounds from the list of prediction results having an association with the particular property, and updating the property model based on the property model validation. The process / loop can be repeated using the updated property model until it is determined that the property model has been effectively trained. The property model validation can include selecting a list of compound candidates, performing simulation analysis and / or laboratory analysis on the list of compound candidates for association with the particular property, and updating the property model using the simulation and / or laboratory results.

Description

TECHNICAL FIELD

[0001] The present application relates to apparatuses, systems, and methods for active learning and model validation. BACKGROUND

[0002] Informatics is the application of computer and information technology and resources for interpreting data in one or more academic and / or scientific fields. Cheminformatics (also known as chem(o)informatics and bioinformatics includes the application of computer and information technology and resources for interpreting chemical and / or biological data. This can include solving and / or modeling processes and / or problems in the field of chemistry and / or biology. For example, these computer and information technology and resources can transform data into information and then transform the information into knowledge for use in rapidly generating compounds and / or making improved decisions in the field of drug identification, discovery, and optimization, just as examples and not limitation.

[0003] Machine learning techniques are computational methods that can be used to design complex analytical models and algorithms that help solve complex problems, such as the generation and prediction of whether a compound has one or more characteristics and / or properties. Although there are innumerable ML techniques that can be used or selected to predict whether a compound has a particular property or characteristic, there is often a lack of training data to properly train the ML techniques to generate suitable trained property models (referred to herein as property models) to predict whether a compound has a particular property. If the ML techniques are used to generate property models based on insufficient labeled training data, the resulting property models can not reliably predict whether a compound has a particular property for a wide range of compounds.

[0004] Generating labeled training datasets for training ML techniques to generate accurate and reliable property models to predict whether a compound has a particular property is expensive, time consuming, and prone to human error. The complexity of this task increases exponentially as the number of properties / characteristics that need to be predicted increases with each of a plurality of property models used to predict whether a compound has one or more of a plurality of properties and / or characteristics. There is a need to improve training and use of ML techniques to generate accurate and reliable property models to predict whether a compound has one or more particular properties to enable researchers, data scientists, engineers, and analysts to make rapid improvements in the field of drug identification, discovery, and optimization.

[0005] The embodiments described below are not limited to implementations that solve any or all of the disadvantages of the known approaches described above. SUMMARY

[0006] This summary is provided to introduce some concepts of the present application in a simplified form that are further described below in the detailed description. This summary is neither intended nor

[0007] The present disclosure provides methods and apparatus for training a machine learning (ML) technique to generate an ML model (e.g., a property model) for predicting whether a compound has a particular property. This uses an iterative process / feedback loop that can be performed to generate the ML model until it is deemed to be effectively trained. The process for each iteration of the feedback loop can include, by way of example only and not limitation, generating a list of prediction results for a plurality of compounds and their association with the particular property based on the ML model; validating the ML model based on compounds from the list of prediction results that have an association with the particular property; and updating the ML model based on the ML model validation. The process / loop can be repeated using the updated ML model until the ML model is determined to have been effectively trained. By way of example, the property model validation step can include selecting a list of compound candidates, performing simulation analysis and / or laboratory analysis on the list of compound candidates for the particular property, and updating the property model using the simulation and / or laboratory results. The simulation and / or laboratory results can be used to form further labeled training data for training the ML technique to generate an updated ML model.

[0008] In a first aspect, the present disclosure provides a computer-implemented method for generating an ML model (also referred to herein as a property model) for predicting whether a compound has a particular property. The method includes: training a ML technique to generate a property model; generating a list of prediction results for a plurality of compounds and their association with the particular property using the property model; validating the property model based on compounds from the list of prediction results that have an association with the particular property; and updating the property model based on the property model validation.

[0009] Preferably, the method includes repeating the generating and validating steps using at least the updated property model until the property model is determined to have been effectively trained. The steps of generating, validating, and updating can be part of a feedback loop that can be repeated or iterated using the updated property model of a previous iteration until the property model is determined to have been effectively trained and / or a suitable stopping criterion has been met or reached (e.g., a maximum number of iterations, a stable value of the property model score, a peak value of the property model score, etc.).

[0010] Preferably, the method further comprises generating, using the property model, a prediction result for a plurality of compounds and their association with the particular property; and validating the property model based on the compounds from the prediction result list having an association with the particular property.

[0011] Preferably, the ML technique is initially trained based on a labeled training dataset associated with a subset of the plurality of compounds related to the particular property. The subset of the plurality of compounds can be the subset of the plurality of compounds used to generate the prediction result list.

[0012] Preferably, validating the property model further comprises: validating a list of compound candidates from the prediction result list having an association with the particular property; and updating the property model further comprises: updating the property model based on training the ML technique with a labeled training dataset including the validated list of compound candidates.

[0013] Preferably, updating the property model further comprises: generating another labeled training dataset based on the validated list of compound candidates and any previous labeled training dataset associated with the particular property; and retraining the ML technique based on the generated labeled training dataset.

[0014] Preferably, validating the list of compound candidates further comprises: determining whether to perform a laboratory experiment based on the particular property and the list of compound candidates; and in response to determining to perform the laboratory experiment, using experimental results from the laboratory experiment to estimate an association of each compound on the list of compound candidates with the particular property.

[0015] Preferably, determining to perform the laboratory experiment is based on one or more of the following group: a number of validation iterations exceeding a validation iteration threshold in which simulated analysis has been performed to validate the list of candidates in succession; an indication that laboratory analysis will improve the ML score of the property model based on a previous property model score computed from a corresponding prediction result list generated after each list of compound candidates has been validated; or a combination of the number of validation iterations and an indication that the laboratory experiment will provide an improved property model.

[0016] Preferably, determining whether to perform the laboratory experiment further comprises: determining whether the selected list of compound candidates has a substantial change from a previously selected list of compound candidates; in response to determining that the selected list of compound candidates does not have a substantial change from the previously selected list of compound candidates, selecting to perform the laboratory experiment on a selected subset of compounds from the selected list of compound candidates.

[0017] Preferably, validating the candidate list further comprises: determining whether to perform a simulation analysis (or computer simulation analysis) based on the particular property and the list of compounds; and in response to determining to perform the simulation analysis, using simulation results from the simulation analysis to estimate the association of each compound on the list of compound candidates with the particular property.

[0018] Preferably, determining to perform the simulation analysis is based on one or more of the group: a number of validation iterations exceeding a validation iteration threshold in which simulation analyses have been performed in succession to validate the candidate list; an indication that the simulation analysis will improve a ML score of a property model based on a previous property model score computed from a corresponding list of prediction results generated after each list of compound candidates has been validated; or a combination of the number of validation iterations and an indication that the simulation analysis will provide an improved property model.

[0019] Preferably, the number of validation iterations in which the simulation analysis is performed in succession is greater than the number of validation iterations in which the laboratory analysis is performed.

[0020] Preferably, the laboratory analysis is performed once for each of a plurality of generation and validation iterations in which the simulation analysis is performed in succession.

[0021] Preferably, the list of prediction results comprises a prediction score of whether each compound has the particular property, the method further comprising selecting the list of compound candidates from the list of prediction results based at least in part on the prediction scores.

[0022] Preferably, validating the list of compound candidates further comprises selecting one or more compounds for the list of compound candidates from the list of prediction results based on whether the compounds have a prediction score indicative of a critical prediction score.

[0023] Preferably, the prediction score comprises a certainty score, wherein a compound known to have the particular property is given a positive certainty score, a compound known to not have the particular property is given a negative certainty score, and other compounds are given an uncertainty score between the positive certainty score and the negative certainty score.

[0024] Preferably, the certainty score is a percentage certainty score, wherein the positive certainty score is 100%, the negative certainty score is 0%, and the uncertainty score is between the positive certainty score and the negative certainty score.

[0025] Preferably, selecting the list of compound candidates from the list of prediction results further comprises selecting one or more compounds having an uncertain prediction result.

[0026] Preferably, selecting the list of compound candidates from the list of prediction results further comprises selecting one or more compounds that are dissimilar to the compounds used in any labeled training data used so far.

[0027] Preferably, selecting the list of compound candidates from the list of prediction results further comprises using a selection model to select the list of compound candidates from the list of prediction results, wherein the selection model is generated by training a reinforcement learning, RL, technique.

[0028] Preferably, generating the selection model based on the RL technique further comprises using the selection model to select a set of compounds for the list of compound candidates from the list of prediction results for validation; validating whether the selected list of compound candidates has the particular property; and updating the property model based on the ML technique and the validated list of compound candidates; generating an ML score and a further list of prediction results based on the updated property model; and determining whether to retrain the selection model to select a set of compounds for the list of compound candidates based on the ML score and a previous ML score.

[0029] Preferably, in response to determining to retrain the selection model, the method further comprises restoring the updated property model to a previous property model when the ML score does not meet a property model performance threshold compared to a corresponding previous ML score; retaining or keeping the updated property model when the ML score indicates that the property model performance threshold is met or exceeded compared to the corresponding previous ML score; and retraining the selection model to select a set of compounds from a corresponding list of prediction results based on the ML score; and repeating the selection model steps of generating at least the selecting step, validating, and updating the property model until it is determined to train the selection model.

[0030] Preferably, determining to train the selection model further comprises comparing the retained / kept property model score to a previous retained property model score; and determining that the selection model has been effectively trained based on a stable value of the property model score.

[0031] Preferably, determining whether the property model has been effectively trained further comprises determining that the property model has been effectively trained based on an indication that no further validation of the candidate list is needed. Alternatively or additionally, preferably, determining that the property model has been effectively trained further comprises comparing the retained / kept property model score to a previous retained property model score; and determining that the selection model has been effectively trained based on a stable value of the property model score.

[0032] Preferably, validating the property model further comprises generating a property model score based on the list of prediction results; and determining whether the property model has been effectively trained based on the property model score and a previous property model score.

[0033] Preferably, determining whether the property model has been effectively trained comprises determining that the property model has been effectively trained based on a stable value of the property model score.

[0034] Preferably, the ML technique comprises at least one ML technique or a combination of ML techniques from the group of: a recurrent neural network configured to predict, starting from a first compound, a second compound that exhibits a set of desired properties; a convolutional neural network configured to predict, starting from a first compound, a second compound that exhibits a set of desired properties; a reinforcement learning algorithm configured to predict, starting from a first compound, a second compound that exhibits a set of desired properties; and any neural network architecture configured for predicting, starting from a first compound, a second compound that exhibits a set of desired properties.

[0035] Preferably, the particular property comprises a property or characteristic indicative of: a compound that interfaces with another compound to form a stable complex; a ligand that interfaces with a target protein, wherein the compound is the ligand; a compound that interfaces or binds with one or more target proteins; a compound having a particular solubility or range of solubility; a compound having a particular toxicity; any other property or characteristic associated with a compound that can be simulated based on computer simulations and physical movements of atoms and molecules; any other property or characteristic associated with a compound that can be determined from an expert knowledge base; and any other property or characteristic associated with a compound that can be determined from experiments. The particular property can further comprise a property, characteristic, and / or feature indicative of: a partial coefficient (e.g., LogP), a distribution coefficient (e.g., LogD), solubility, toxicity, drug-target interactions, drug-drug interactions, off-target drug effects, cell penetration, tissue penetration, metabolism, bioavailability, excretion, absorption, drug-protein binding, drug-lipid interactions, drug-deoxyribonucleic acid (DNA) / ribonucleic acid (RNA) interactions, metabolite prediction, tissue distribution, and / or any other suitable property, characteristic, and / or feature associated with a compound.

[0036] Preferably, the method of generating a property model can be repeated until it is determined that the property model has been effectively trained. Additionally, the method can comprise further training the property model by iteratively generating, validating, and updating the property model until it is determined that the property model has been effectively trained or a stopping criterion is reached or satisfied, wherein the updated property model from a previous or current iteration is used when repeating at least the generating, validating, and updating steps in the next iteration.

[0037] In a second aspect, the disclosure provides an apparatus comprising a processor, a memory unit, and a communication interface, wherein the processor is connected to the memory unit and the communication interface, wherein the processor and the memory are configured to implement the computer-implemented method according to the first aspect, modifications thereof, and / or as described herein.

[0038] In a third aspect, the disclosure provides an ML model comprising data representing an ML model generated by training an ML technique according to the computer-implemented invention of the first aspect, modifications thereof, and / or as described herein.

[0039] In a fourth aspect, the disclosure provides a property model obtained or obtainable by the computer-implemented method according to the first aspect, modifications thereof, and / or as described herein.

[0040] In a fifth aspect, the disclosure provides an apparatus comprising a processor, a memory unit, and a communication interface, wherein the processor is connected to the memory unit and the communication interface, wherein the processor and the memory are configured to implement the ML model according to the third or fourth aspect, and / or as described herein.

[0041] In a sixth aspect, the disclosure provides a computer-readable medium comprising data or instruction code representing an ML model generated based on training an ML technique according to the computer-implemented method of the first aspect, modifications thereof, and / or as described herein, which, when executed on a processor, causes the processor to implement the ML model.

[0042] In a seventh aspect, the disclosure provides a computer-readable medium comprising data or instruction code representing the ML model according to the third or fourth aspect, and / or as described herein, which, when executed on a processor, causes the processor to implement the ML model.

[0043] In an eighth aspect, the disclosure provides a method for predicting whether a compound has a particular property using an ML model trained by the computer-implemented method according to the first aspect, modifications thereof, and / or as described herein.

[0044] In a ninth aspect, the disclosure provides a system for generating an ML model (e.g., a property model) for predicting whether a compound is associated with a particular property, the system comprising: a model generation module for training an ML learning ML technique to generate an ML model; a model testing module for generating prediction results of compounds and their association with the particular property using the ML model; a validation module for validating the ML model based on the prediction results from the compounds associated with the particular property; and a model updating module for updating the ML model based on the ML model validation.

[0045] Preferably, the system further comprises one or more features of the first aspect, modifications thereof, or as described herein. Preferably, the model generation module, the model testing module, the validation module, and / or the model updating module can be configured to implement the computer-implemented method of the first aspect, modifications thereof, and / or one or more features as described herein, etc. Preferably, the model generation module, the model testing module, the validation module, and / or the model updating module can be further configured to implement one or more functions or functionalities of one or more of the second to eighth aspects, modifications thereof, and / or as described herein, etc.

[0046] The methods described herein can be performed by software in machine readable form, for example in the form of a computer program comprising computer program code means adapted to perform all the steps of any of the methods described herein when the program is run on a computer and computer program can be embodied on a computer readable medium. Examples of tangible (or non-transitory) storage media include disks, thumb drives, memory cards and the like, and do not include propagated signals. The software can be suitable for execution on a parallel processor or a serial processor such that the method steps can be carried out in any suitable order, or simultaneously.

[0047] This application acknowledges that software and firmware can be valuable, separately tradable commodities. It is intended to encompass software, which runs on or controls "dumb" or standard hardware, to carry out the desired functions. It is also intended to encompass software which "describes" or defines the configuration of hardware, such as HDL (hardware description language) software, as is used to design silicon chips or configure generic, programmable chips, to carry out desired functions.

[0048] As will be apparent to those skilled in the art, the preferred features can be suitably combined and can be combined with any of the aspects of the application. BRIEF DESCRIPTION OF DRAWINGS

[0049] Embodiments of the application will be described, by way of example only, with reference to the following drawings, in which:

[0050] Figure 1a is a flow chart illustrating an example process for training an ML technique to generate and validate property models to predict whether a compound has a particular property according to the present application;

[0051] Figure 1b is a schematic diagram of an example apparatus illustrating an example process for implementing the method of Figure 1a according to the present application;

[0052] Figure 2 is a table illustrating an example list of prediction results output from property models of a plurality of compounds according to the present application;

[0053] Figure 3is a schematic diagram illustrating an example apparatus for validating a property model according to the present application;

[0054] Figure 4 is a schematic diagram illustrating an example apparatus for validating a list of compound candidates for training an ML technique to generate a property model according to the present application;

[0055] Figure 5 is a schematic diagram illustrating an example process for selecting a list of compounds for use in Figure 4 a and Figure 4 b according to the present application; and

[0056] Figure 6 is a schematic diagram of a computing device according to the present application.

[0057] Common reference signs are used throughout the drawings and detailed description to indicate like features. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described below by way of example only. These embodiments represent the most preferred mode of practicing the application, as presently known to the Applicant, but they are not the only mode in which the application can be practiced. The description sets forth the functions and sequences of steps for constructing and operating the examples. However, the same or equivalent functions and sequences can be accomplished by different examples.

[0059] The inventors have advantageously developed a method / mechanism that intelligently uses a combination of simulations of selected compounds and / or laboratory experiments in an iterative and semi-automated / automated manner that enhances the training of machine learning (ML) techniques to generate accurate and reliable ML models, e.g., ML models such as, by way of example only and not limitation, property models for predicting whether a compound exhibits or has a particular property. This mechanism can be particularly applicable when there is not enough labeled training data to train an ML technique to generate, by way of example only and not limitation, a property model for predicting whether a compound has a particular property. This mechanism can enhance a labeled training dataset by selecting an optimal subset of compounds that should maximize or at least improve the performance of the property model, while determining via computer simulations or via laboratory experiments when the subset is optimally validated for a particular property. The property model can be updated based on the enhanced labeled training dataset. Thereafter, this mechanism can iteratively further enhance the labeled training dataset using another selected subset of compounds primarily using simulations, and, if necessary, requiring and performing laboratory experiments on a minimal number of compounds or subset of compounds that will enhance the performance of the property model.

[0060] Although the following description of the present application is directed only by way of example and without limitation to a property model and / or ML model for predicting whether one or more compounds are associated or have a particular property (e.g., whether one or more entities are associated with a relationship), one skilled in the art will appreciate that the present application can be applied to other ML models to predict whether an entity or input data has a particular relationship with another entity, or for classifying one or more entities and / or input data according to a particular relationship, etc. The entity can include one or more compounds, drugs, proteins / genes, or other biological entities, etc.

[0061] A prediction property model (or ML model for predicting whether a compound exhibits or has a particular property) can be configured to receive a compound as input and output data representing a prediction of whether the compound has a particular property. For example, the property model can be configured to predict, by way of example only and without limitation, whether a compound will bind to a particular protein; or predict whether the compound is soluble in water; or predict whether the compound is toxic to a human or a human body part; or predict any other property of interest associated with a compound. However, a labeled training data set can only contain data for hundreds to thousands of compounds associated with a particular property. This is insufficient data to properly train an ML technique to generate a property model that can predict whether a compound exhibits and / or has a particular property.

[0062] The quality of a property model can be improved by increasing the size of the labeled training dataset. For example, a variety of compounds that are not known to be associated with a particular property can be tested in a laboratory via experiments to measure whether they exhibit or are associated with the particular property. However, this is very expensive for all but a few compounds. The inventors have developed a technique for limiting the number of compounds that need to be tested in the laboratory while improving the quality of the property model. This can be achieved by first selecting a list of compound candidates from a list of prediction results output by the property model for a variety of compounds. The candidate list is typically larger than the number of compounds that are typically sent to the laboratory for testing. Computer simulations based on molecular dynamics / interactions are used to validate the list of compound candidates for association with the particular property. The validation results from the computer simulations of the candidate list are fed back into the property model (e.g., used to augment the labeled training dataset and retrain the property model accordingly), which can output another list of prediction results based on the variety of compounds. Another candidate list can be selected, validated by computer simulations, and fed back into the property model. These steps can be repeated until it is determined that laboratory testing will further improve the quality of the property model. After laboratory testing, the laboratory results for the validated list of compound candidates can be fed back into the property model (e.g., the laboratory results are used to further augment the labeled training dataset and retrain the property model accordingly). This step can be repeated with further cycles of simulation and / or laboratory experimentation until the property model is deemed to have been properly trained.

[0063] Laboratory testing can be determined based on one or more of the following, by way of example only and without limitation: it can be determined that the simulation testing techniques have been exhausted, e.g., based on the simulations, the property model has little or no improvement; it can be observed that the list of prediction results outputs a very small candidate list of uncertain compounds; a maximum number of iterations using simulations for validating the candidate list has been reached; a minimum number of compounds have been selected for laboratory testing and it is determined that these selected compounds should yield the maximum degree of improvement in the quality of the property model; and / or an overall property model performance score of the property model stable value compared to a previous property model performance score; or the property model performance score is worse than a previous property model performance score, in which case the property model will revert to the performance optimal property model, and the candidate list selected for laboratory experimentation; any other condition or criteria that can contribute to improving the quality of the property model; and / or any combination thereof.

[0064] The compounds can be selected as a list of compound candidates for simulation and / or laboratory testing based on one or more of the following, by way of example only and without limitation: selecting those compounds that are least similar to the compounds already in the labeled training dataset; selecting those compounds for which the property model is least uncertain, whether or not the compounds exhibit a particular property (e.g., a critical case); selecting those compounds using a ML selection model that has been trained to select the optimal compounds that can result in improved ML quality; and / or any other combination thereof.

[0065] For example, the particular property can relate to docking, and a property model can be generated to predict where a compound binds to a particular point or binding site. The compounds in the selected candidate list for validation can be input to a computer docking simulation configured with respect to the binding site, which simulates whether the compounds adhere / dock to the binding site, e.g., compounds that dock with a protein. The computer simulation can output validation results, such as, by way of example only and without limitation, a docking score or data representing how well the compounds dock with the binding site. These results are fed back into the property model by using the output validation results to augment the labeled training data and retrain the ML technique using the labeled training data to generate an updated property model (e.g., a retrained property model).

[0066] A compound (also referred to as one or more molecules) can comprise or represent a chemical or biological substance composed of one or more molecules (or molecular entities) composed of atoms from one or more chemical elements (or more than one chemical element) bound together by chemical bonds. Example compounds used herein can include, by way of example only and without limitation, molecules bound together by covalent bonds, ionic compounds bound together by ionic bonds, intermetallic compounds bound together by metallic bonds, certain complexes bound together by coordinate covalent bonds, pharmaceutical compounds, biological compounds, biological molecules, biochemical compounds, one or more proteins or protein compounds, one or more amino acids, lipids or lipid compounds, carbohydrates or complex carbohydrates, nucleic acids, deoxyribonucleic acid (DNA), DNA molecules, ribonucleic acid (RNA), RNA molecules, and / or any other organization or structure of molecules or molecular entities composed of atoms from one or more chemical elements, and combinations thereof.

[0067] Each compound has or exhibits one or more properties, characteristics, or features, or a combination thereof, that can determine the usefulness of the compound for a given application. When a compound is under consideration, the properties or properties of interest of the compound can include or represent data that is representative of or indicative of a particular behavior / characteristic / feature of the compound. For example, a compound can be associated with or exhibit one or more characteristics or properties that can include, by way of example but not limitation, one or more characteristics or properties from the following group: an indication that the compound interfaces with another compound to form a stable complex; an indication that the compound is a ligand that interfaces with a target protein; an indication that the compound interfaces or binds with one or more target proteins; an indication that the compound has a particular solubility or range of solubilities; an indication that the compound has a particular electrical characteristic; an indication that the compound has a toxicity or range of toxicities; any other indication associated with the compound that can be modeled using computer simulations based on physical movement of atoms and molecules; any other indication associated with the compound that can be tested or measured experimentally. Other examples of properties, characteristics, or features of one or more compounds include, by way of example but not limitation, one or more of the following: LogP, LogD, solubility, toxicity, drug-target interactions, drug-drug interactions, off-target drug effects, cell penetration, tissue penetration, metabolism, bioavailability, excretion, absorption, drug-protein binding, drug-lipid interactions, drug-DNA / RNA interactions, metabolite prediction, tissue distribution, and / or any other suitable property, characteristic, and / or feature associated with a compound.

[0068] A property of a given compound can include data that is representative of or indicative of a particular behavior / characteristic / feature of the compound when under consideration, and the data that is representative of or indicative of the property of the compound can include, by way of example but not limitation, any continuous or discrete value / score and / or range of values / scores, series of values / scores, string of characters, or any other data representative of the property. For example, a property can be associated with, assigned, represented by, or based on, by way of example but not limitation, one or more continuous property values / scores (e.g., non-binary values), one or more discrete property values / scores (e.g., binary values), one or more continuous ranges of property values / scores, one or more discrete ranges of property values / scores, a series of property values / scores, one or more strings of property values, or any other suitable data representative of the property, etc. The property values / scores can be based on measured data or simulated data associated with the reaction and / or the particular property.

[0069] A compound can be assigned a property value / score that includes data indicative of whether the compound is associated with a particular property when the compound is subjected to a reaction associated with the particular property. The property value / score can be determined or based on, by way of example only and without limitation, laboratory measurements and / or computer simulation values / scores. The property value / score assigned to a compound gives an indication of whether the compound is associated with or exhibits a particular property. For example, a compound can be assigned a property value / score depending on whether the compound exhibits a particular property when subjected to a reaction associated with the particular property. A compound can be said to exhibit a particular property when the property value / score associated with the compound is above or below a threshold property value / score indicative of the property, within or near a range of values indicative of the property, and the like, by way of example only and without limitation.

[0070] One or more or a combination of ML techniques can be used to generate a property model generated for predicting whether a compound according to the application as described herein has one or more properties. The ML techniques can include or represent one or more or a combination of computational methods that can be used to generate analytical models and algorithms that teach themselves to solve complex problems, such as, by way of example only and without limitation, prediction and analysis of complex processes and / or compounds. The ML techniques can be used to generate ML models (e.g., property models) for use in drug discovery, identification, and / or optimization in the fields of informatics, cheminformatics, and / or bioinformatics.

[0071] For example, a labeled training dataset can be used to train an ML technique to generate an ML model (or property model) to predict whether a compound has a particular property. The labeled training dataset can include one or more compounds, each of which can be labeled with data indicative of a known property value / score or a label associated with the compound and the particular property. Thus, once the ML technique has trained the ML model based on the labeled training dataset related to the particular property, the ML model can predict whether an input compound exhibits the particular property. The ML model can output data indicative of a property value / score that indicates the association of the input compound with the particular property. The data indicative of the property value / score output by the ML model can be referred to herein as a property prediction value / score. ML model data indicative of one or more compounds can be input into the trained ML model, which can output property prediction values / scores including data indicative of one or more corresponding property values / scores that indicate whether the one or more input compounds are associated with or exhibit the particular property.

[0072] Examples of ML techniques that can be used to generate an ML model or property model for predicting whether a compound has a particular property can include, by way of example only and without limitation, at least one ML technique or a combination of ML techniques from the following group: recurrent neural networks; convolutional neural networks; reinforcement learning algorithms; and any other neural network structure configured for predicting whether a compound has a particular property.

[0073] Other examples of ML techniques that can be used in accordance with the present application as described herein can include or be based on, by way of example only and without limitation, any ML technique or algorithm / method that can be trained or adapted to generate one or more candidate compounds based on, by way of example only and without limitation, an initialization compound, a list of desired properties of the candidate compound, and / or a set of rules for modifying the compound, which can include one or more supervised ML techniques, semi-supervised ML techniques, unsupervised ML techniques, linear and / or non-linear ML techniques, ML techniques associated with classification, ML techniques associated with regression, and / or the like and / or combinations thereof. Some examples of ML techniques can include or be based on, by way of example only and without limitation, one or more of the following: active learning, multi-task learning, transfer learning, neural message passing, one-shot learning, dimensionality reduction, decision tree learning, association rule learning, similarity learning, data mining algorithms / methods, artificial neural networks (ANN), deep ANN, deep learning, deep learning ANN, inductive logic programming, support vector machines (SVM), sparse dictionary learning, clustering, Bayesian networks, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, rule-based machine learning, learning classifier systems, and / or one or more combinations thereof, and / or the like.

[0074] Some examples of supervised ML techniques can include or be based on, by way of example only and without limitation, ANNs, DNNs, association rule learning algorithms, a priori algorithms, case-based reasoning, Gaussian process regression, group method of data handling (GMDH), inductive logic programming, instance-based learning, lazy learning, learning automata, learning vector quantization, logistic model trees, minimum message length (decision trees, decision graphs, and / or the like), XGBOOST, gradient boosting machines, nearest neighbor algorithms, analogical modeling, probably approximately correct (PAC) learning, descent-based rules, knowledge acquisition methods, symbolic machine learning algorithms, support vector machines, random forests, ensembles of classifiers, bagging (bootstrap aggregating), boosting (meta-algorithm), ordinal classification, information fuzzy networks (IFN), conditional random fields, analysis of variance, quadratic classifiers, k-nearest neighbors, boosting, bagging, Bayesian networks, Naive Bayes, hidden Markov models (HMMs), hierarchical hidden Markov models (HHMMs), and any other ML technique or ML task capable of inferring a function or generating a model from labeled and / or unlabeled training data, and / or the like.

[0075] Some examples of unsupervised ML techniques can include or be based on, by way of example only and without limitation, expectation maximization (EM) algorithms, vector quantization, generative topographic maps, information bottleneck (IB) methods, and any other ML technique or ML task capable of inferring a function describing a hidden structure and / or generating a model from unlabelled data and / or a function by disregarding labels in a labelled training dataset, etc. Some examples of semi-supervised ML techniques can include or be based on, by way of example only and without limitation, one or more of the following: active learning, generative models, low-density separation, graph-based methods, co-training, transduction, or any other ML technique, task, or class of unsupervised ML techniques capable of training with unlabelled datasets and / or labelled datasets, etc.

[0076] Some examples of artificial NN (ANN) ML techniques can include or be based on, by way of example only and without limitation, one or more of the following: artificial NNs, feedforward NNs, recurrent NNs (RNNs), convolutional NNs (CNNs), autoencoder NNs, extreme learning machines, logic learning machines, self-organizing maps, and other ANN ML techniques or connected systems / computing systems inspired by biological neural networks that make up animal brains. Some examples of deep learning ML techniques can include or be based on, by way of example only and without limitation, one or more of the following: deep belief networks, deep Boltzmann machines, DNNs, deep CNNs, deep RNNs, hierarchical temporal memory, deep Boltzmann machines (DBMs), stacked autoencoders, and / or any other ML technique.

[0077] Figure 1a is a flowchart illustrating an example process 100 for training an ML technique to generate an ML model (also referred to herein as a property model) to predict whether a compound exhibits or has a particular property, in accordance with the present application. The particular property can be based on one of a plurality of properties associated with a compound. The process 100 can use an ML technique that can be trained based on a labelled training dataset that includes data representing a relationship or association of a set of compounds with a particular property. The labelled training dataset can not have a sufficient number of compound / property associations, or can not have a sufficient number of dissimilar compound / property associations to train the ML technique to generate a property model that can be used for a broad range of compounds. Accordingly, the following methods further enhance the training of the ML technique to generate an accurate and reliable property model to predict whether a broad range of compounds have a particular property. The steps of the process 100 can include one or more of the following steps:

[0078] At step 102, a list of prediction results for a plurality of compounds and their association with a particular property is generated based on an ML model (i.e., a property model). The property model can be generated by training an ML technique based on an initial labeled training dataset that includes data representing known relationships or associations of a set of compounds with the particular property. The plurality of compounds can include the set of compounds of the labeled training dataset and another set of compounds for which the association with the particular property is unknown. The plurality of compounds is input to the initially generated property model, which outputs a list of prediction results for each of the plurality of compounds that predicts whether the compound has the particular property. The list of prediction results can include a plurality of compounds each mapped to a corresponding property prediction value / score output / estimated by the ML model.

[0079] At step 104, the ML model or property model is validated based on the plurality of compounds from the list of prediction results having an association with the particular property. The initial labeled training dataset can be used to determine the degree of association between each of the plurality of compounds and the particular property predicted by the property model. This can include determining model performance statistics or an overall property model score that indicates the degree of association of the particular property with the compounds predicted by the property model. This can further include verifying or further validating the association of the selected list of compound candidates with the particular property. This can be used to augment the labeled training dataset.

[0080] At step 106, it is determined whether the ML model or property model has been sufficiently trained or whether further training of the property model is needed. This can be determined based on the property model score (or ML model score) and / or whether the predictive ability of the property model / ML model is expected to be further improved. If it is determined that the property model / ML model has not been sufficiently trained (e.g., “N”), the process 100 proceeds to step 108 to update the property model / ML model, after which steps 102-106 can be repeated using the updated property model / ML model until it is determined that the property model / ML model has been effectively trained. If it is determined that the property model / ML model is sufficiently trained (e.g., “Y”), the process 100 proceeds to step 110.

[0081] For simplicity, the term“property model” is referred to hereinafter and includes, by way of example only and without limitation, an ML model used to predict whether a compound has a particular property or is associated with a particular property (e.g., the particular property can be a property or characteristic associated with a compound, etc.). In step 108, the property model can be updated based on the results of the validation. For example, the ML scores can be used to update the property model. Additionally or alternatively, the property model can be updated based on the results of validating the list of selected compound candidates. For example, an enhanced or further labeled training dataset can be generated based on the current labeled training dataset, which includes compounds having a known association with the particular property, and based on the validation results of validating whether each of the list of compound candidates is associated with the particular property. This enhanced or further labeled training dataset can be used to train an ML technique to generate an updated property model, which can potentially replace the current property model to predict whether a compound has the particular property. Regardless, once the property model has been updated based on training the ML technique accordingly, the process 100 proceeds to step 102 to determine whether the performance of the updated property model has been improved.

[0082] In step 110, once it is determined that the property model has been effectively trained, or until such time as as much practice or possible training has been performed as possible, data representing the property model can be output for use in predicting whether a compound has a particular property. This can include storing all parameters, coefficients, weights, hyperparameters, and any other data defining the property model and / or how the property model is configured for later use. The output property model can be stored on a computer-readable medium, and when it is to be used, the output property model can be retrieved, loaded, and executed by one or more processors to predict whether one or more compounds has a particular property.

[0083] The ML technique can be initially trained based on a labeled training dataset associated with a subset of the plurality of compounds related to the particular property. The labeled training dataset can be further enhanced when the property model is validated. This can be achieved by validating the list of compound candidates from the list of predicted results having an association with the particular property. The property model can then be updated based on training the ML technique with the labeled training dataset including data representing the list of validated candidates associated with the particular property and any previously labeled training dataset associated with the particular property.

[0084] In step 108, updating the property model with the additional validated candidate list can include generating a further labeled training dataset including data representing the validated candidate list of compounds associated with the particular property and any previously labeled training dataset associated with the particular property. The ML technique can then be retrained or updated using the further labeled training dataset based on the ML technique using it.

[0085] In step 104, validating the compound candidate list can include determining, based on certain conditions, whether to perform a laboratory experiment based on the particular property and the compound candidate list, or whether to perform a computer analysis, such as, by way of example only and without limitation, a simulation analysis based on the particular property and the compound candidate list. In response to determining to perform a laboratory experiment, a request can be sent that includes the compound candidate list for a laboratory experiment related to the particular property, and experimental results are received that validate the association of each of the compound candidate list with the particular property. The experimental results from the laboratory experiment can be used to estimate data representing the association of each compound on the compound candidate list with the particular property. This can be used to augment the labeled training dataset to further update the property model. In response to determining to perform a simulation analysis instead of a laboratory experiment, the compound candidate list can be inputted for computer analysis (e.g., inputted into a molecular computer simulation related to the particular property) to determine the association of each compound candidate list with the particular property. Simulation results from the simulation analysis can be used to estimate data representing the association of each compound on the compound candidate list with the particular property. This can also be used to augment the labeled training dataset to further update the property model.

[0086] In view of the fact that laboratory experiments are generally more expensive than computer analysis / simulations, a set of conditions can need to be satisfied before sending the compound candidate list to a laboratory to determine the association of each compound with the particular property. The set of conditions can include, by way of example only and without limitation, one or more of the following set: a laboratory experiment can be selected when the number of validation iterations exceeds a validation iteration threshold in which computer / simulation analysis has been performed consecutively to validate the candidate list; a laboratory experiment can be selected when an indication that laboratory analysis will produce an improvement in the ML score of the property model based on a previous property model score that was computed from a corresponding prediction result list generated after each compound candidate list has been validated; the number m of selected compound candidate lists has a cost effective size or number for laboratory experiments (e.g., the number of m selected compound candidate lists can be less than 10), where m >= 1; or a combination of the number of validation iterations, the indication that a laboratory experiment will provide an improved property model, and the number m or size of the compound candidate list.

[0087] The computer analysis / simulation can be selected based primarily on a set of conditions associated with the compound candidate list. The computer analysis is used to determine an association of each compound with a particular property. The set of conditions can include, by way of example only and without limitation, one or more of the following set: the computer analysis is selected when a number of validation iterations is less than a validation iteration threshold in which computer / simulation analysis has been performed consecutively to validate the candidate list; the computer analysis can be selected when it is determined that the computer analysis will still produce an improvement in the ML score for the property model based on a previous property model score that was computed from a corresponding prediction result list generated after each compound candidate list has been validated; the selected compound candidate list has a size or number m of compounds that is too large to be cost effective for laboratory experiments (e.g., the number m of the selected compound candidate list can be in a range of 25 to 500), where m >= 1; or a combination of the number of validation iterations, an indication that the laboratory experiments will provide an improved property model, and the size of the selected compound candidate list.

[0088] Other conditions that can be used to determine whether to perform a laboratory experiment can include, by way of example only and without limitation, determining whether the selected compound candidate list has a substantial change from a previously selected compound candidate list; in response to determining that the selected compound candidate list does not have a substantial change from the previously selected compound candidate list, selecting to perform a laboratory experiment on a selected subset of compounds from the selected compound candidate list. The selected subset of compounds can have a size that is cost effective and / or suitable for laboratory experiments. The selected compound candidate list can be further filtered based on selecting, by way of example only and without limitation, those compounds in the candidate list that have the least uncertain scores in the prediction result list and / or are also the most dissimilar compared to the compounds in the labeled training dataset.

[0089] The property model can be used to predict whether each of a plurality of compounds has a particular property and output the results in a predicted results list. The predicted list can include one or more compounds mapped to a corresponding one or more property prediction values / scores that can be output by the property model for each compound. Each property prediction value / score assigned to each compound indicates whether the compound is associated with a particular property. This can be achieved by inputting each of a plurality of compounds into the property model and collecting the results output from the property model into a predicted results list. The predicted results list can include, by way of example only and without limitation, a property prediction score or predicted score for each of a plurality of compounds indicating whether the each compound has or exhibits a particular property. The plurality of compounds can include a subset of the compounds used to generate the property model in the labeled training data set. This allows the quality of the property model to be assessed and a ML score to be generated. The plurality of compounds also includes a set of compounds that are not in the labeled training data set used to generate the property model. The predicted results list thus includes predicted scores that predict whether each of a plurality of compounds has or exhibits a particular property.

[0090] The predicted results list can be used to select a list of compound candidates based on the predicted score (or property prediction value / score) for each compound and / or the structure of each compound. For example, one or more compounds for the list of compound candidates can be selected from the predicted results list based on whether the compounds have a predicted score that indicates a critical predicted score. A critical predicted score is a predicted score that indicates that the property model is unable to predict whether a compound has or does not have (exhibits or does not exhibit) a particular property. That is, the property model is unable to indicate a certainty that a compound is associated with a particular property.

[0091] For example, if a compound has or exhibits a particular property, the prediction score or property prediction score / value can have a positive certainty level represented as a probability in the range of 1 or a percentage score in the range of 100% (e.g., in the range of 0.85-1 or in the range of 85-100%). If a compound is known not to have or exhibit a particular property, the prediction score for that compound can have a negative certainty level represented as a probability in the range of 0 or a percentage score in the range of 0% (e.g., in the range of 0-0.15 or in the range of 0-15%). Compounds with prediction scores between the positive and negative certainty levels can be considered to have an uncertain or borderline prediction score. For example, those compounds with prediction scores in the range of 0.5 or with percentage scores in the range of 50% (e.g., between 0.45 and 0.55 or between 45-55%) can be considered the least certain or most borderline. That is, the property model is unable to determine in one way or another whether these compounds have or do not have (exhibit or do not exhibit) the particular property.

[0092] Accordingly, the prediction result list can be filtered to output compounds for which the property model is least certain or unable to predict their association with a particular property. Accordingly, a set of compounds based on the least certain or borderline cases can be generated from the prediction result list and used to select a compound candidate list. For example, the compounds with the greatest uncertainty or borderline prediction scores can be ranked and the M least certain compounds can be selected for the candidate list. Alternatively or additionally, the set of compounds based on the least certain or borderline cases can be further filtered by generating a set of the least certain dissimilar compounds. A plurality of m<=M compounds from the ranked list of uncertain or borderline compounds that are the least structurally dissimilar to the compounds with positive or negative certainty levels of prediction scores can be selected to select the compound candidate list. Alternatively or additionally, the compound candidate list can be based on selecting from the ranked list of uncertain or borderline compounds those compounds that are the most structurally dissimilar to the compounds that make up the labeled training dataset used to generate the property model. Selecting the compound candidate list based on this approach can prevent retraining or updating the property model to overfit or focus on particular types or structures of compounds and will allow the training of the ML technique to generate a property model that can predict properties for a broad range of structurally similar and dissimilar compounds.

[0093] Figure 1b is a schematic diagram illustrating a system for implementing a method according to the present application Figure 1aThe example process 100 is illustrated in the diagram of an example training apparatus or system 120. The training apparatus / system 120 includes a machine learning (ML) model generation (MLG) device 122, a model testing (MT) device 124, and a model validation (VM) device 126 coupled together in a feedback loop that can iterate or repeat until the feature model is considered to have been effectively trained. The training apparatus 120 can be configured to implement... Figure 1a Process 100. Each of the components / devices 122, 124, and 126 of the training device 120 can be configured to iteratively implement the process as described above. Figure 1a One or more steps of process 100 are used to iteratively train ML techniques to generate improved, accurate, and reliable property models for predicting whether a compound is associated with a specific property.

[0094] Initially, for the first iteration (e.g., j=1), MLG device 122 receives the labeled training dataset {T}. i} j (where 1 <= i <= N), where N is the number of training data elements (e.g., in the range of 1000 or more), where the i-th training data element includes a representation of compound C. i And its known associations with specific characteristics. The MLG device 122 uses a labeled training dataset {T}. i} j Train the ML technique (this can be pre-defined) to generate a feature model M for the j-th iteration. j Characteristic Model M j Predict whether the input compound C1 possesses a specific property. The labeled training dataset {T} i} j Further training data can be merged based on whether VM device 126 deems further training necessary {T}. k} j And output the validation results or further training data {T}. k} j It can be used to augment the labeled training dataset {T} i} j To train ML techniques to generate an updated feature model M in the next iteration. j (For example, j = j + 1).

[0095] In the j-th iteration, MT device 124 receives the generated characteristic model M j Towards the characteristic model M j Input multiple compounds {C1} j , where 1 <= 1 <= L, and L is the number of various compounds, and outputs a list of prediction results {R1}. j(where 1 <= 1 <= L), where the first prediction result R in the j-th iteration is... 1，j Includes, but is not limited to, the prediction score P for compound C1 and the j-th iteration. 1，j The data. Predicted score P 1，j M is the property model that represents the association between compound C1 and specific properties. j Predicted values. List of prediction results {R1} j Predicting multiple compounds {C1} j Does each compound in the model possess specific properties? For each iteration j, the property model M is further trained on a broader range of compounds, depending on whether further training is needed. j Multiple compounds {C1} j The quantity can be changed or it can remain unchanged.

[0096] VM device 126 receives at least the prediction result list {R1} j And use the list of results to validate the feature model M j Whether it has been effectively trained or requires further training. VM device 126 can also receive the feature model score S for the j-th iteration of the j-th feedback loop. j Alternatively or additionally, VM device 126 may base its predictions on a list {R1}. j and / or labeled training dataset {T i} j Generate the characteristic model score S for the j-th iteration of the feedback loop. j The characteristic model score S can be stored and monitored for each iteration of the feedback loop. j Feature model score S j and / or a list of predicted results {R1} j This can be used, as an example only, but not limited to, determining whether a) is as described in reference process 100 and Figure 1a The requirement is for the characteristic model M j Further training shall be conducted; b) whether it is as per reference procedure 100 and Figure 1a ;c) Whether the list of compound candidates is validated using computer analysis / simulation or laboratory experiments as described in reference procedure 100 and Figure 1a The increase or decrease of the number of compounds in the candidate list; d) whether as per reference process 100 and Figure 1a The prediction result list {R1} j The choice of compounds can be changed.

[0097] VM device 126 can be based on ML score S j and / or previous ML score {S kto determine that the property model M should be updated j and further training of the ML technique is required (e.g., step 106 of process 100). This can include selecting a list of compound candidates that can be verified using computer analysis or laboratory experiments. As a result, the VM device 126 can output further training data {T k} j and / or validation results that can be used to generate further training data {T k} j related to the selected list of compound candidates. The MLG device 122 can use the further training data {T k} j or incorporate the further training data {T k} j into the labeled training dataset {T i} j for the next iteration of the feedback loop (e.g., j = j + 1). Thus, when j = j + 1, the further training data {T k} j can be used to augment the labeled training dataset {T i} j for training the ML technique to generate an updated property model M j when j = j + 1 and the process 100 and its steps implemented by components / devices 122, 124, and 126 are repeated for the next iteration.

[0098] This iterative process 100 can continue until the VM device 126 considers that the updated property model M j has been sufficiently trained. Once the property model M j has been sufficiently trained, the property model M j can be considered a validly trained property model M v to predict whether a compound is associated with a particular property. The output device 128 can generate data representing the valid property model M v for storing the property model M v and / or for using the property model M v to predict whether a compound is associated with a particular property.

[0099] It can be seen that the process 100 can be used to train the ML technique to generate a property model based on a labeled training dataset. This can also be referred to as training or updating a property model. The property model is a model artifact embodying the data created by the training process 100 that results in a property model M v that is validly trained M vconfigured to predict whether a compound (e.g., a new compound) is associated with a particular property. A prediction score for the compound can indicate whether the compound has the particular property, or the degree of uncertainty of the prediction of the property model about whether the compound is associated with the particular property.

[0100] The output device 128 can output data representing the property model M v , which can include, by way of example only and without limitation, hyperparameters used to train the ML technique, weights, coefficients, parameters generated during training of the ML technique, data defining the structure of the property model M v , or any other data required to implement the property model M v on one or more devices, computing systems, apparatuses, and / or processors, etc., to enable the property model M v to predict whether a compound is associated with a particular property. The property model M v may be stored for retrieval and used to predict whether a compound is associated with a particular property.

[0101] The training device or system 120 for generating a property model for predicting whether a compound is associated with a particular property can be based on functional or modular components / modules that can be implemented in software and / or hardware. The system 120 can include a model generation module for training an ML technique to generate a property model, a model testing module for using the property model to generate prediction results for compounds and their association with a particular property, a validation module for validating the property model based on compounds from which the prediction results have an association with the particular property, and a model update module for updating the property model based on the property model validation. These modules can be further modified and / or configured to implement the method / process 100 and / or multiple methods / multiple processes as described herein.

[0102] Figure 2 is a table showing an example list of prediction results {R1} j 200 output from a property model for predicting whether a plurality of compounds {C1} (where 1 <= 1 <= L) are associated with a particular property. The property prediction values / scores indicating the association of the compounds with the particular property C1 can include data representing the prediction scores P1. The list of prediction results {R1} j 200 includes data representing a plurality of compounds {C1} 202 and their corresponding prediction scores {P1} 204 (e.g., property prediction values / scores) (where 1 <= 1 <= L). The plurality of compounds {C1} includes compounds C1, C2,..., C1,..., G L-1 L ​The corresponding multiple predicted scores {P1}204 include predicted scores P1, P2, ..., P1, ..., P2. L-1 P L Each prediction score P1 indicates whether each compound C1 possesses or is associated with a specific property. Verification step 106 can be based at least in part on the prediction scores from the prediction result list {R1}. j Select from a list of 200 candidate compounds.

[0103] As previously stated, the prediction score includes or represents data representing a value that indicates or indicates whether a compound has or does not have a specific property according to the ML model. The prediction score can be a value, for example only but not limited to, probability values, deterministic values ​​or scores, percentage scores, or any other value indicating whether a compound has or does not have a specific property, or whether a compound exhibits or does not exhibit a specific property, and / or how a compound is associated with a specific property; and / or any other values, scores, or statistics that can be used to assess or classify whether a compound is associated with a specific property.

[0104] For example, the prediction score P1 regarding whether compound C1 is associated with a specific property can be represented as a deterministic score value. Compounds known to possess a specific property are given a value representing a "definite" deterministic score (e.g., P). CP Compounds known not to possess a specific property are given a value representing a "negative" certainty score (e.g., P). CN Other compounds were given a score representing "uncertainty" (P1 = X1, where P...). CN <X1<P CP The "uncertainty" score can be a continuous real value representing the level of uncertainty in the ML model regarding whether the compound is associated with a particular property. The "uncertainty" score can have continuous values ​​between values ​​representing positive certainty and values ​​representing negative certainty (e.g., P...). CN <X1<P CP In this example, the certainty score is expressed as a percentage certainty score, where the positive certainty score is 100%, the negative certainty score is 0%, and the uncertainty score lies between the positive and negative certainty scores, i.e., between 0% and 100%.

[0105] exist Figure 2 In the middle, the prediction results list {R1} j200 rank a plurality of compounds {C1} 202 based on their predicted scores {P1} 204. For example, if a compound has or exhibits a particular property, the predicted score can have a positive certainty level expressed as a probability in the range of 1 or a percentage score in the range of 100% (e.g., in the range of 0.85-1 or in the range of 85-100%). In Figure 2 , C1 and C2 have a positive certainty score expressed as a percentage score of P CP = 100%, which means that the ML model has 100% confidence that these compounds C1 and C2 have the particular property. Likewise, C L-1 and C L have a negative certainty score expressed as a percentage score of P CN = 0%, which means that the ML model has 100% confidence that these compounds G L-1 and C L do not have the particular property. There can be one or more or more compounds {C1} where the predicted score has a value of P1 = X1 between P CN < P1 < P CP , where the ML model has a continuous degree of confidence as to whether these compounds are associated with the particular property. Of interest are those compounds that lie in the middle range (e.g., 45% < P1 < 55%) between P CN and P CP , which include the compounds for which the property model is predicted to be the least certain as to whether these compounds are associated with the particular property. These compounds can be of interest to select a shortlist of compounds that can be validated with respect to the particular property.

[0106] As an example, if it is reasonably known that the compound has or does exhibit a specific property, the prediction score P1 for that compound can have a positive certainty level, expressed as a probability in the range of 1 or a percentage score in the range of 100% (e.g., a probability in the range of 0.85–1 or a percentage score in the range of 85–100%). If it is reasonably known that the compound does not have or does not exhibit a specific property, the prediction score P1 for that compound can have a negative certainty level, expressed as a probability in the range of 0 or a percentage score in the range of 0% (e.g., a probability in the range of 0–0.15 or a percentage score in the range of 0–15%). Compounds with prediction scores between the positive and negative certainty levels can be considered to have uncertain or critical prediction scores. For example, those compounds with prediction scores in the range of 0.5 or in the range of 50% (e.g., between 0.45 and 0.55 or between 45–55%) can be considered the most uncertain or most critical. In other words, the property model cannot determine in one way or another whether these compounds have or do not have (exhibit or not exhibit) a specific property. It is of interest to validate these compounds relative to specific properties and thereby generate further labeled training datasets to update the property model as described in this paper.

[0107] Figure 3 This is a schematic diagram illustrating an example verification apparatus 300 according to the present invention for verifying a characteristic model in each iteration j of process 100. The verification apparatus 300 receives a list of prediction results {R1}. j 200, the prediction result list {R1} j 200 can be used by the score generator 302, the model validator 304, and the candidate list validator 306. The score generator 302 is based on the received list of prediction results {R1}. j Calculate the characteristic model score S from 200. j Model validator 304 can use the feature model score S. j Based on the feature model score S j and any previously generated feature model score {S k} (where 1 <= k < j), to determine whether the feature model has been effectively trained. Feature model score S j This is an indicator of the extent to which the property model predicts whether a compound is associated with a specific property. If the model validator 304 deems further training necessary, i.e., the property model has not been effectively trained (e.g., 'N'), then the candidate list validator 306 selects a list of compound candidates from which the property model should be enhanced (e.g., as discussed in this paper regarding...). Figure 1a-2The candidate list validator 306 outputs a validation result, which in this example is a further training data element {T k} j The ML technique can use this training data element to generate / update the property model in the next iteration j = j + 1 of the process 100.

[0108] The score generator 302 can use the labeled training data set {T i} j and the received list of prediction results {R1} j 200 to calculate a property model score S j indicative of the performance of the property model for the jth iteration. The property model score S j may be calculated based on model performance statistics that can be estimated from the labeled training data set {T i} j and / or the received list of prediction results {R1} j 200. The model performance statistics can include or represent an indication of the performance of the property model based on the labeled training data set {T i} j and / or the received list of prediction results {R1} j 200. The model performance statistics for the property model can be based on, by way of example only and without limitation to one or more of the following group: the precision of positive predictions or the recall of the property model; the sensitivity, true prediction rate, or call of the property model; the receiver operating characteristic (ROC) plot associated with the property model; the area under the precision and / or recall ROC curve associated with the property model; any other function associated with the precision and / or recall of the property model; and any other model performance statistics used to generate the property model score S j indicative of the performance of the property model.

[0109] The model validator 304 can use the property model score S i to determine whether the property model has been effectively trained or whether the property model requires further training. The model validator 304 can use previous or historical property model scores {S k}(where 1 <= k < j) to determine whether further improvement in the quality of the property model is possible. The model validator 304 can also, by way of example only and without limitation, track the number of iterations j that have been completed; track the number of consecutive times that the candidate list is validated using in silico analysis methods; track the number of times that the candidate list is validated using laboratory experiments; track the number of times that the candidate list is validated using the received list of prediction results {R1} jThe number of uncertain compounds is tracked in 200. These measures can be used to determine whether further improvement in the quality of the property model is possible.

[0110] For example, if the property model score S j and {S k} (where 1 <= k < j) have stabilized; the number of consecutive times the selected candidate list is verified using computer analysis / simulation is greater than a predetermined threshold; and no verification of the selected compound candidate list using laboratory experiments has been performed; then the model verifier 304 can determine that further improvement is possible if the selected compound candidate list is verified using laboratory experiments. It can therefore indicate to the candidate list confirmer 306 that further training is required and select a candidate list for use when verified using laboratory experiments rather than computer analysis / simulation.

[0111] In another example, if the property model score S j and {S k} (where 1 <= k < j) have not stabilized but appear to be increasing; the number of consecutive times the selected candidate list is verified using computer analysis / simulation is less than a predetermined threshold; and no verification of the selected compound candidate list using laboratory experiments has been performed; then the model verifier 304 can determine that further improvement is still possible using the selected compound candidate list verified using computer analysis / simulation. It can therefore indicate to the candidate list verifier 306 that further training is required and select a candidate list for use when verified using computer analysis / simulation.

[0112] In another example, if the property model score S j and {S k} (where 1 <= k < j) are decreasing; the number of consecutive times the selected candidate list is verified using computer analysis / simulation is less than a predetermined threshold; and no verification of the selected compound candidate list using laboratory experiments has been performed; then the model verifier 304 can determine that further improvement is possible if the selected compound candidate list is verified using laboratory experiments. It can therefore indicate to the candidate list confirmer 306 that further training is required and select a candidate list for use when verified using laboratory experiments rather than computer analysis / simulation.

[0113] The candidate list verifier 306 can receive an indication from the model verifier 302 that further training is required. The candidate list verifier 306 can also, by way of example only and without limitation, track the number of completed iterations j; track the number of consecutive times a candidate list is verified using computer analysis methods; track the number of times a candidate list is verified using laboratory experiments; track the received list of predicted results {R1} jThe number of uncertain compounds in the 200. These measures can be sent to the model validator 302 to help it make decisions about the validity of the property model during iteration j. They can also help determine the type and / or number of candidate compounds to be selected to maximize the chances of enhancing or improving the quality of the updated property model based on the validation results. Alternatively or additionally, the candidate list validator 306 may receive instructions that the validation of the candidate list should be based on computer analysis / simulation or performed via laboratory experiments.

[0114] Candidate list validator 306 can select from those described in this article or related to Figure 1a to 2 A suitable candidate list of compounds 4a-5 was compiled, and the selected candidate list was validated relative to specific properties via computer analysis or laboratory experiments using a chosen validation method. As a result, the candidate list validator 306 can use the validation results as further training data {T}. k} j Output. As described, further training data {T} k} j The labeled training dataset {T} can be used or incorporated. i} j In this process, the feature model is updated using ML techniques in the next iteration of the feedback loop (e.g., j = j + 1).

[0115] Figure 4 This is a schematic diagram illustrating an example verification apparatus 400 according to the present invention, which can be used to replace the candidate list verifier 306 for selecting and verifying a list of compound candidates used to train ML techniques to generate or update a property model. The verification apparatus 400 includes a candidate list selector 402, a verification selector 404, a computer analysis verifier 406, and a laboratory verifier 408. The verification apparatus 400 receives at least a list of predicted results {R1}. j 200, and the candidate list selector 402 predicts the result list {R1} from the prediction result list. j Select from 200 candidate compounds {C k} j When validated relative to a specific feature, the prediction result list {R1} is given. j 200 should enhance the feature model M on the next iteration of the training process 100. i Update.

[0116] For reference Figure 2 The aforementioned list of candidate compounds {C} k} jThis may include compounds that require further validation relative to specific properties and, if chosen correctly or wisely, can be used to enhance the accuracy and reliability of the property model. The prediction results list {R1} may be derived at least in part based on the prediction score {P1}. j Select a candidate compound from 200. Prediction results list {R1} j The 200 compounds of interest were considered the most uncertain or critical compounds based on their prediction scores. For these compounds, the property model could not determine, in one way or another, whether these compounds possessed or did not possess (exhibited or did not exhibit) a specific property (e.g., prediction scores were typically between 0.45 and 0.55 or between 45-55%). However, satisfying P... CN <P1<P CP Any other prediction score P1 can also be used as part of the list of candidate compounds to be selected.

[0117] Candidate list selector 402 can select from the ranked list of predicted results {R1} j Choose compounds from 200 such that the top compound in the list is the one whose property model is least certain. Generating a ranked list of compounds that the property model cannot predict as having or not having a specific property will help in selecting the candidate compound list {C}. k} j The candidate list of compounds {C k} j Enhanced ML training will generate more accurate and reliable feature models. Ranking lists can be generated in the following ways.

[0118] Assuming characteristic model M j The maximum prediction score that can be given for all compounds predicted to have a specific property is X (e.g., a positive certainty score, a percentage score with a probability of 1 or 100%), while the minimum prediction score for all compounds predicted to have absolutely no specific property is Y (e.g., a negative certainty score, a percentage score with a probability of 0 or 0%), where X > Y. For input to the property model M... j For each compound C1, it is also assumed that the property model outputs a prediction score P1 in the range Y ≤ P1 ≤ X. This provides a deterministic indication of whether the property model has or does not have a specific property in its predictions. Prediction results list {R1} j 200 can be used to generate a ranking list of compounds with the least certainty in the characteristic model, ranked from the most uncertain prediction score to the most certain prediction score with either a positive or negative level of certainty. Let P1 be the list of prediction results {R1}. jThe predicted score of the first compound out of 200, where 1 <= 1 <= L. Compounds with a predicted score P1 > (X+Y) / 2 are assigned a ranking score S by subtracting their predicted score P1 from X. R1 S R1 =X-P1. Compounds with a prediction score P1 <= (X+Y) / 2 can be assigned a ranking score S. R1 =P1. Therefore, the first compound C1 in the prediction list has a ranking score R1 = X - P1 when P1 > (X + Y) / 2, or a ranking score R1 = P1 when Pi <= (X + Y) / 2. Therefore, with ranking score S R1 The list of prediction results in descending order {R1} j Ranking the 200 compounds will produce a ranked list of compounds, with the top compound being the one with the least certain property model.

[0119] Candidate list selector 402 can select from the prediction result list {R1} based on whether a compound has a prediction score that indicates a critical prediction score. j Select one or more compounds from the 200 compound candidate list. In the above case, from the prediction result list {R1} j A ranking list of 200 generated compounds helps determine the least uncertain compounds that should be in the compound candidate list. This ranking list ranks the top compounds with the least uncertainty in the property model. These top compounds can be used to select one or more compounds for the compound candidate list, meaning that from the list of predictions with uncertain predictions {R1}... j Choose one or more compounds from 200.

[0120] While compounds at the top of the compound ranking list can help enhance the training of ML techniques and the generation / updating of feature models, some of them may be structurally similar to those already used for training ML techniques and generating / updating feature models. j The compounds are too similar. Besides selecting the top-ranked uncertain compound from the compound ranking list, or as an alternative, the candidate list can be generated by: selecting one or more compounds that are structurally dissimilar to those used in the training data with any labels used so far; or selecting one or more compounds that are structurally dissimilar to each other from the top-ranked compounds in the uncertain compound ranking list. Furthermore, the candidate list can be generated by selecting one or more top-ranked compounds that are structurally dissimilar to those used in the training data with any labels used so far from the compounds in the ranking list.

[0121] The verification selector 404 can be configured to select a verification technique for verifying a selected list of compound candidates associated with a specific property. (See reference...)Figure 3 The described verification selector can also, by way of example only and without limitation, track the number of compounds selected in the compound candidate list {C k} j ; track the type or number of dissimilar compounds in the compound candidate list; track the number of iterations j that have been completed; track the number of consecutive times that computer analysis / simulation was used to validate the candidate list; track the number of times laboratory experiments were used to validate the candidate list; track the number of received prediction result lists {R1} j 200; and track the property model score S j These measures can be used to determine whether to select computer analysis / simulation to validate the candidate list or to select laboratory experiments to validate the candidate list. They can also help determine the type and / or number of compound candidate lists {C k} j that can be selected to maximize the opportunity that the quality of the updated property model based on the validation results can be enhanced or improved.

[0122] For example, the verification selector 404 can determine to perform computer analysis / simulation based on one or more of the following group: a number of validation iterations that exceeds a validation iteration threshold in which simulation analysis has been performed consecutively to validate the candidate list, where the number of validation iterations in which simulation analysis is performed consecutively is greater than the number of validation iterations in which laboratory analysis is performed; an indication that simulation analysis will improve the ML score of the property model based on a previous property model score computed from a corresponding prediction result list generated after each compound candidate list has been validated; or a combination of the number of validation iterations and an indication that simulation analysis will provide an improved property model.

[0123] Further, the number of compounds that can be validated against a particular property using computer analysis / simulation depends significantly on the available computational resources. Generally, the number of compounds that can be simulated in a reasonable amount of time can be between 50-500 compounds (e.g., 50-100). It should be understood that the number of compounds that can be simulated against a particular property depends on the available computational resources, and the number of compounds that can be simulated will increase as computational resources increase, and become cheaper and faster. Generally, the number of compounds that can be validated against a particular property using laboratory experiments, m, is on the order of 4 to 10 compounds (e.g., 6-8 experiments). This is because laboratory time to run experiments is expensive, and the required expense is expensive. Thus, if validation is being performed using computer analysis / simulation, the number of compounds, m, in the compound candidate list can be selected to be one, two, or several orders of magnitude larger than the number of compounds, m, in the compound candidate list that can be used if validation is performed using laboratory experiments. Thus, validation selector 404 and candidate list selector 402 can communicate with each other to determine the maximum size of the compound candidate list that can be validated. Alternatively, candidate list selector 402 can simply send the compound candidate list to validation selector 404, and based on the selected validation method, validation selector 404 can truncate the compound candidate list, if necessary, to ensure that an appropriate number of compounds are validated by the selected validation method (e.g., computer analysis / simulation or laboratory experiments). k} j k} j to ensure that an appropriate number of compounds are validated by the selected validation method (e.g., computer analysis / simulation or laboratory experiments).

[0124] For example, validation selector 404 can be configured to instruct selector V T or some other technique / method to select computer analysis / simulation, such that the compound candidate list {C k} j is directed / requested to be processed by computer analysis validator 406 for validation of the compound candidate list. Computer analysis validator 406 can be connected to one or more computer analysis / simulation systems (e.g., molecular dynamics (MD) (RTM) molecular simulator) that can simulate at the atomic scale whether a compound has or exhibits a particular property. For example, MD simulators use the atoms and / or physics of a molecule to simulate the properties of a compound / molecule. The types of properties of compounds that can be simulated by MD simulations include, by way of example only and not limitation, docking simulations including proteins that dock with the compound, and / or any other property or compound that can be simulated to determine whether a compound has a particular property.

[0125] ​The computer analysis / simulator validator 406 validates the candidate list by sending the candidate list to a computer analysis / simulation system that performs computer analysis / simulation analysis based on the particular property and the compound candidate list {C k} j The computer analysis / simulator validator 406 can receive computer analysis / simulation results from the computer analysis / simulation system. The computer analysis / simulation results can be used to estimate the association of each compound on the compound candidate list {C k} j with the particular property. The computer analysis / simulation results associated with the compound candidate list {C k} j C can be output in the form of a labeled training dataset {T k} j C which can be used to generate a further training dataset {T k} j for the ML technique to generate / update the property model M j for the next iteration of the process 100 as described herein. The selector V T can be used to select the labeled training dataset {T k} j C as the further training dataset {T k} j for the ML technique to generate / update the property model M j for the next iteration of the process 100.

[0126] In another example, the validation selector 404 can be configured to instruct a laboratory experiment via the selector V T or some other technique / method such that the compound candidate list {C k} j is directed / requested to be processed by a laboratory validator 408 for validation of the compound candidate list. The laboratory validator 408 can be connected to one or more computer systems associated with one or more laboratories that can receive the compound candidate list and perform laboratory experiments as to whether each compound in the candidate list has or exhibits the particular property. The experimental results associated with the compound candidate list {C k} j can be output in the form of a labeled training dataset {T k} j L .

[0127] Alternatively, the lab validator 408 can inform an operator of the compound candidate list and the particular property for the lab experiment. The operator can send the compound candidate list and ask the lab to perform the experiment to determine whether each compound in the compound candidate list has or exhibits the particular property. At the end of the experiment, the experimental results and / or further training data associated with the compound candidate list and whether each compound has the particular property or the experimental results and / or further training data associated with the particular property can be sent to the lab validator 408.

[0128] The lab validator 408, upon receiving the experimental results or training data associated with the compound candidate list and its association with the particular property, can be configured to output a labeled training dataset {T k} j L . The labeled training dataset {T k} j L may be used as further training data {T k} j for the ML technique to generate / update the property model M j for the next iteration of the process 100 (e.g., j = j + 1) as described herein. The selector V T may be used to select the labeled training dataset {T k} j L as further training data {T k} j for the ML technique to generate / update the property model M j for the next iteration of the process 100.

[0129] Although the selector V T is shown as a switching circuit that switches between the computer analysis / simulator validator 406 and the lab validator 408, this is merely exemplary and the present application is not limited thereto, and it is to be understood that any other method, technique, device or hardware / software can be used by the skilled person to select and / or direct / request the compound candidate list to be processed between the computer analysis / simulator validator 406 and / or the lab validator 408 with respect to the particular property.

[0130] Further considerations for the validation selector 404 to determine whether to perform a laboratory experiment can be based on one or more of the following group: a number of validation iterations exceeding a validation iteration threshold in which simulated analysis has been performed in succession to validate a candidate list; an indication that laboratory analysis will improve the ML score of a property model based on a previous property model score computed from a corresponding predicted results list generated after each compound candidate list has been validated; and / or a combination of the number of validation iterations and an indication that laboratory experiments will provide an improved property model.

[0131] While a set of selection and / or validation rules can be derived to select a compound candidate list and / or to select a validation method described herein for validating a compound candidate list, a selection model can alternatively be generated based on training a reinforcement learning technique. The selection model is used to predict a compound candidate list that is suitable for validation with respect to a particular property. Thus, instead of using a set of selection rules to select an appropriate candidate list of compounds for which a property model is uncertain, an RL technique can be trained over time to make this selection. Once the RL technique learns to select a compound candidate list for enhancing a property model, the generated selection model can be used to train the property model that is used to predict whether a compound exhibits or has a property that is different from a particular property. This is because the selection model does not depend on the type of property that each property model is to model a prediction of.

[0132] The RL technique can be trained to learn from the results prediction list which compounds to select in order to maximize the quality of the selection and generate a selection model. The quality of the selection will be maximized when the selected compound candidate list is the most optimized compound chosen from that particular results prediction list that can maximize the quality of the resulting updated property model when validated with respect to a particular property. The RL technique can be used to iteratively train a selection model that is robust enough to select the most suitable or optimized compound candidate list from the results prediction list to validate with respect to a particular property. The training process for the selection model can be based on:

[0133] Initially, in the first iteration (e.g., j = 1) of the ML training process, a property model can be generated by training an ML technique based on a first labeled training dataset. The first labeled training dataset can be used to train the ML technique to generate a property model, while a second labeled training dataset can be held aside to evaluate the quality of the property model. Once the property model is trained by the ML technique, the second labeled training dataset is input into the property model and outputs a predicted results list. Likewise, a property model score S jto evaluate the quality of the property model. The RL technique can be taught which compounds in the list of predicted results can be optimally selected for validation and thereby generate a selection model. Initially, the selection model trained by the RL technique can select a set of "random" compounds from the list of results predictions as a list of candidate compounds. The selection model training process proceeds to the next iteration (e.g., j = j + 1).

[0134] In the second iteration (e.g., j = 2), the property model can be retrained based on the first set of labeled training dataset and a selected portion of the second set of labeled training dataset corresponding to the selected list of candidate compounds selected by the selection model trained by the RL technique in the previous iteration. Once the property model is retrained or updated by the ML technique, the second set of labeled training dataset is input to the property model and outputs a list of predicted results. Another property model score Sj can be derived based on the list of predicted results and / or the second set of labeled training dataset to evaluate the quality of the property model. The property model scores {S k} from previous iterations (e.g., k = j - 1) can be compared with the property model score S j of the current iteration. The retrained or updated property model can then be retained / preserved for another iteration of training the selection model. If the performance of the property model has improved in quality / accuracy, this can be fed back to the RL technique as a reward. The selection model associated with the RL technique can be updated / retrained based on the reward. Then, another set of compounds is selected from the list of results predictions by the selection model as a list of candidate compounds to be validated. The selection model training process proceeds to the next iteration (e.g., j = j + 1).

[0135] However, if the comparison results result in the performance of the property model not improving in quality / accuracy, this is fed back to the RL technique as a penalty. The selection model associated with the RL technique can be updated / retrained based on the penalty. Given that the performance of the property model has deteriorated, it can be restored to the previous retained / preserved property model before the property model has poor performance. Then, the selection model can be used to select another set of compounds from the list of results predictions as a list of candidate compounds to be validated. The selection model training process proceeds to the next iteration (e.g., j = j + 1).

[0136] Once the ML scores {S k}(1 <= k <= j) indicate that the performance of the ML technique has stabilized, it can be assumed that the selection model has been trained. Then, the selection model can be used to select a set of compounds from the list of results predictions as a list of candidate compounds to be validated as described with reference to Figure 1a-4The property model is further trained, and in this figure a plurality of compounds that the property model has never seen before can be input to the property model to generate a list of prediction results in which the selection model can be used to select a list of compound candidates for validation. As described, the validation results can be used to further update the property model, and thus iteratively further improve the property model. In this process (e.g., process 100), the selection model can also be further trained based on the training selection process described above, but in which each selected list of compound candidates is validated using computer analysis / simulation and / or in rare cases using laboratory experiments. An ML score can be computed to allow the RL technique to reward or penalize the selection model during retraining.

[0137] Figure 5 is a flowchart illustrating another example process 500 for training a selection model to select a list of compound candidates for validation in Figure 1a-4 The selection model can initially be trained by the RL technique as previously described, in which a first portion of the labeled training dataset is used to train the property model, and a second portion of the labeled training dataset is used to evaluate the property model to generate a list of prediction results and a property model score S j for initially training the RL technique to generate / train the selection model.

[0138] The process 500 can include the following steps for training or retraining the RL technique to generate a selection model that can select a set of compounds from the prediction results list output by the property model M j and / or the property model score S j better predict the list of compound candidates. In step 502, the selection model can be used to select a set of compounds from the prediction results list output by the property model M j for the list of compound candidates to validate the list of compound candidates. In step 504, the selection model sends the selected list of compound candidates for validation.

[0139] Computer analysis / simulation can be used to validate whether each selected list of compound candidates has the particular property. Sometimes, as described herein, it can be determined to validate some or all of the selected list of compound candidates via laboratory experiments. The property model can be updated based on the ML technique, the labeled training dataset, and the validated list of compound candidates. That is, the validated list of compound candidates can be represented as further labeled training dataset associated with the list of compound candidates, which can be used to further train the ML technique to produce / update the property model. A plurality of compounds {Ci} (1 <= i <= L) can be input to the updated property model, and a list of prediction results {Ri} (1 <= i <= L) and an ML score S j and the ML score S jThat is, the ML score S can be generated based on a plurality of compounds {C1} (1 <= 1 <= L) input to the updated property model j and a further list of predicted outcomes {R1} j .

[0140] In step 506, the list of predicted outcomes {R1} for the current iteration j is received by the RL technique / selection model j and the ML score S j In step 508, based on the ML score S j and previous ML scores {S k} (where 1 <= k < j), a determination is made whether to retrain the selection model to select a set of compounds for the compound candidate list. For example, the property model scores {S k} (where 1 <= k < j) from a previous iteration (e.g., k = j - 1) can be compared to the property model score S j for the current iteration. If the performance of the property model has improved in quality / accuracy, this can be fed back to the RL technique as a reward, and the selection model can be retrained (e.g., “Y”). The updated property model can then be retained / preserved for use in another iteration of training the selection model. In step 510, the selection model associated with the RL technique can be updated / retrained based on the reward. The selection model training process 500 proceeds to the next iteration (e.g., j = j + 1), and the retrained selection model can then be used in step 502 to select another set of compounds from the list of outcomes predictions as the compound candidate list for validation.

[0141] In step 508, if the comparison between the ML score S j and previous ML scores {S k} (where 1 <= k < j) results in the performance of the property model in the current iteration not improving in quality / accuracy, this can be fed back to the RL technique as a penalty, and the selection model can be retrained (e.g., “Y”). In step 510, the selection model associated with the RL technique can be updated / retrained based on the penalty. In view of the performance of the property model having worsened, the property model can be restored to the previously retained / preserved property model before the property model has poor performance. The selection model training process 500 can proceed to the next iteration (e.g., j = j + 1), and the retrained selection model can then be used in step 502 to select another set of compounds from the list of outcomes predictions as the compound candidate list for validation.

[0142] In step 508, it can be determined that the selection model has been fully trained and that further training will not necessarily improve the selection of the list of compound candidates. For example, if there is no improvement seen in the predictive property model, then the selection model can be considered trained and can not require further training. For example, one method of determining whether the selection model is sufficiently trained can include checking whether the list of selected compound candidates sent into the laboratory for testing and / or through computer simulation does not make any subsequent predictive property model generated by retraining the ML technique based on the laboratory or computer simulation results worse and / or the same. Comparing the previous property model score to the current retrained property model score can be useful in determining whether the selection model can be considered sufficiently trained. For example, when comparing the updated property model score to the previous retained / kept property model score indicates a stable value in the property model score, the selection model can be considered trained,

[0143] Other modifications to the process 500 can include, in response to determining in step 510 to retrain the selection model, the updated property model can be restored to the previous property model when the ML score does not meet a property model performance threshold compared to the corresponding previous ML score. Alternatively or additionally, in step 510, the updated property model can be retained instead of replaced by the previously trained property model when the ML score indicates that the property model performance threshold is met or exceeded compared to the corresponding previous ML score.

[0144] Further modifications can be made to allow the selection model to be trained by the RL technique to not only select the list of compound candidates, but also to select a validation method using computer analysis / simulation and / or laboratory experiments. Given the cost of performing laboratory experiments, it can be desirable to include a rule that penalizes the RL technique when the selection model selects a validation method as a laboratory experiment too early in the training process, or when further improvement using computer analysis / simulation is still needed.

[0145] Figure 6 is a schematic diagram of a computing system 600 including a computing device or apparatus 602 in accordance with the present application. The computing device or apparatus 602 can include a processor unit 604, a memory unit 606, and a communication interface 608. The processor unit 604 is connected to the memory unit 606 and the communication interface 608. The memory unit 406 can include an operating system (OS) and a data store (DS) that can include other applications and / or software such as, by way of example only and without limitation, computer-implemented methods, processes, and / or for implementing the references Figure 1a to Figure 5Instruction code for the methods and / or processes as described herein. The processor unit 604 and memory 606 can be configured to implement one or more processes 100, 500 and / or one or more steps as described herein. The processor unit 604 can include one or more processors, controllers, or any suitable type of hardware for implementing computer executable instructions of the control device 602 in accordance with the present application. The computing device 602 can be connected to a network 612 via the communication interface 608 for communicating with other computing devices / systems (not shown) and / or operations to implement the present application accordingly.

[0146] The computing system 600 can be a server system that can include a single server or a network of servers configured to implement the present application as described herein. In some examples, the functionality of the server can be provided by a network of servers distributed across a geographic region, such as a globally distributed network of servers, and a user can connect to an appropriate one of the network of servers based on the user’s location.

[0147] Further modifications or examples can include a computer-implemented method or a method for generating a property model for predicting whether a compound has a particular property in accordance with any one or more of the processes 100, 130, 500 and / or devices / systems 120, 300, 400, 600 and / or any of the methods / processes, modifications thereof, as described and / or as described herein. Figure 1a to 6 Further modifications or examples can include a computer-implemented method or a method for generating a property model for predicting whether a compound has a particular property in accordance with any one or more of the processes 100, 130, 500 and / or devices / systems 120, 300, 400, 600 and / or any of the methods / processes, modifications thereof, as described and / or as described herein. Figure 1a to Figure 6 Further modifications or examples can include a computer-implemented method or a method for generating a property model for predicting whether a compound has a particular property in accordance with any one or more of the processes 100, 130, 500 and / or devices / systems 120, 300, 400, 600 and / or any of the methods / processes, modifications thereof, as described and / or as described herein.

[0148] A device or computing apparatus 602 includes a processor 604 (or processor unit), a memory unit 606, and / or a communication interface 608, where the processor 604 can be connected to the memory unit 606 and / or the communication interface 608, where the processor 604, the communication interface 608, and / or the memory unit 606 are configured to implement a computer-implemented method for predicting whether a compound has a particular property using a model (e.g., a property model). Alternatively or additionally, the processor 604, the communication interface 608, and / or the memory unit 606 of the device or computing apparatus 602 can be configured to implement a computer-implemented method for generating or training a property model for predicting whether a compound has a particular property.

[0149] Other modifications or examples can include a system for generating a property model based on an ML technique (e.g., an RL technique or any other ML technique), the property model configured to predict whether a compound is associated with a particular property. The system can include: a model generation module, device, or apparatus configured to train an ML technique to generate a property model, as described with reference to any one or more of Figure 1a to Figure 6 the processes 100, 130, 500, and / or the apparatus / systems 120, 300, 400, 600, and / or any method / process, steps of these processes, modifications thereof, as described with reference to any one or more of

[0150] The system can include, as described with reference to any one or more of Figure 1a to Figure 6 one or more further modifications, features, steps, and / or features of the processes 100, 130, 500, and / or the apparatus / systems 120, 300, 400, 600, computer-implemented methods thereof, and / or modifications thereof, as described and / or as described herein. For example, the model generation module / device, the model testing module / device, the validation module / device, and / or the model update module / device can be configured to, as described with reference to any one or more of Figure 1a to Figure 6 one or more further modifications, features, steps, and / or features of the processes 100, 130, 500, and / or the apparatus / systems 120, 300, 400, 600, computer-implemented methods thereof, and / or modifications thereof, as described and / or as described herein.

[0151] Further, the processes 100, 130, 500, and / or the apparatus / systems 120, 300, 400, 600, and / or any method / process, steps of these processes, modifications thereof, as described with reference to any one or more of Figure 1a to Figure 6 the processes 100, 130, 500, and / or the apparatus / systems 120, 300, 400, 600, and / or any method / process, steps of these processes, modifications thereof, can be implemented in hardware and / or software. For example, methods and / or processes for training and / or implementing a property model and / or for using a property model described with reference to one or more of the figures in Figure 1a-6 may be implemented in hardware and / or software, such as, by way of example only and without limitation, as a computer-implemented method by one or more processors / processor units or according to application requirements. Such apparatuses, systems, processes, and / or methods can be used to generate an ML model that includes a representation of training an ML technique, as described with respect to the processes 100, 130, 500, and / or the apparatus / systems 120, 300, 400, 600, and / or any method / process, steps of these processes, as described with reference to any one or more of Figure 1a to 6and / or as described herein, etc. Thus, ML models or characteristic models can be obtained from the apparatuses, systems, and / or computer-implemented processes, methods as described herein.

[0152] Further, as referenced to any one or more of Figure 1a to 6 and / or as described herein, etc., ML selection and / or validation models can also be obtained from the processes 100, 130, 500 and / or apparatuses / systems 120, 300, 400, 600 and / or any methods / processes, steps of these processes, and modifications thereof, some of which can be implemented in hardware and / or software, such as, by way of example only and without limitation, computer-implemented methods that can be executed on a processor or processor unit or as needed for the application, as referenced to one or more of Figure 1a-6 and / or as described herein, etc. In another example, a computer-readable medium comprising data or instruction code representing ML models and / or characteristic models generated based on training ML techniques described with respect to the processes 100, 130, 500 and / or apparatuses / systems 120, 300, 400, 600 and / or any methods / processes, steps of these methods, and modifications thereof, as referenced to any one or more of Figure 1a to 6 and / or as described herein, etc., that when executed on a processor cause the processor to implement the ML models and / or characteristic models.

[0153] For the sake of clarity, the above description discusses embodiments of the application with reference to a single user. It will be appreciated that in practice the system can be shared by multiple users and can be shared by a very large number of users at the same time.

[0154] The above described embodiments are fully automated. In some examples, a user or operator of the system can manually instruct some steps of the processes / methods to be performed.

[0155] In the described embodiments of the application, the system can be implemented as any form of computing and / or electronic device. Such a device can include one or more processors, which can be microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to collect and record routing information. In some examples, for example when using a system on a chip architecture, the processor or processors can include one or more fixed function blocks (also referred to as accelerators) hard-wired to perform part of the method in hardware (rather than software or firmware). Platform software comprising an operating system or any other suitable platform software can be provided at the computing-based device to enable application software to be executed on the device.

[0156] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media can include, for example, computer-readable storage media. Computer-readable storage media can include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer-readable storage media can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks and blu-ray discs (BD). Further, a propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates the transfer of computer program from one place to another. A network connection, for example, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Combinations of the above should also be included within the scope of computer-readable media.

[0157] Alternatively, or in addition, the functions described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0158] Although illustrated as a single system, it will be understood that the computing device can be a distributed system. Thus, for example, multiple devices can communicate over a network and can collectively perform tasks described as being performed by a computing device.

[0159] Although shown as a local device, it should be understood that a computing device can be located remotely and can be accessed via a network or other communication link (e.g., using a communication interface). As used herein, the term "computer" refers to any device with processing power that enables it to execute instructions. Those skilled in the art will recognize that such processing power is incorporated into many different devices, and therefore the term "computer" includes PCs, servers, mobile phones, personal digital assistants, and many other devices.

[0160] Those skilled in the art will recognize that storage devices used to store program instructions can be distributed across a network. For example, a remote computer can store examples describing a process of software. A local or terminal computer can access the remote computer and download part or all of the software to run the program. Alternatively, a local computer can download the software as needed, or execute some software instructions on a local terminal while executing other software instructions on a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to them, all or part of the software instructions can be executed by dedicated circuitry such as DSPs, programmable logic arrays, etc.

[0161] It will be understood that the above benefits and advantages may apply to one embodiment or several embodiments. The embodiments are not limited to those that solve any or all of the described problems or have any or all of the described benefits and advantages. Variations should be considered to be included within the scope of this invention.

[0162] Any reference to “one” an item means one or more of those items. The term “comprising” is used herein to mean including the identified method steps or elements, however, such steps or elements are not included in an exclusive list, and a method or apparatus may include additional steps or elements. As used herein, the terms “component” and “system” are intended to cover a computer-readable data storage configured with computer-executable instructions that, when executed by a processor, cause certain functions to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may reside on a single device or be distributed across multiple devices. Furthermore, as used herein, the term “exemplary” is intended to mean “serving as an illustration or example of something.”

[0163] Furthermore, regarding the extent to which the term "comprising" is used in the detailed description or claims, such terms are intended to be included in a similar manner to the term "including," since they are interpreted as "including" when used as transitional words in the claims.

[0164] The accompanying drawings illustrate exemplary methods. While the method is illustrated and described as a series of acts building on one another, it will be appreciated that the method is not limited by the order of the sequence. For example, some acts can occur in different orders than shown. In addition, one act can occur concurrently with another act. Further, in some instances, not all acts can be required to implement the method described herein.

[0165] Also, the acts described herein can comprise computer-executable instructions that can be implemented by one or more processors and / or stored on one or more computer-readable media. The computer-executable instructions can include routines, sub-routines, programs, threads of execution, and / or the like. Still further, results of acts of the methods can be stored in computer-readable media, displayed on a display device, and / or the like.

[0166] The order of the steps of the methods described herein is exemplary only, and the steps can be executed in any suitable order, or simultaneously, where appropriate. Additionally, steps can be added or substituted, or individual steps can be deleted from any of the methods, without departing from the scope of the subject matter described herein. Any of the aspects of any of the examples described above can be combined with aspects of any of the other examples described to form further examples without losing the effect sought.

[0167] It will be understood that the above description of preferred embodiments is given by way of example only and that various modifications can be made by those skilled in the art. What has been described above includes examples of one or more embodiments. The above description does not include all of the aspects of the application, and does not limit the scope, applicability, or configurations of the application in any way. Rather, the preceding description will provide a thorough understanding of the application to the skilled artisan, along with a description of the aspects of the application that can be implemented as well as the best modes contemplated for carrying out the application. It will be understood that various modifications can be made to the embodiments described above, and that the modifications, if within the scope of the application as claimed, are to be embraced by the application. While the application has been described in some embodiments and illustrative modifications have been described above, other modifications within the scope of the application can be employed. It can be noted that, as used in the specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" or "the component" can include a plurality of such components, and so forth.

Claims

1. A computer-implemented method for generating a property model for predicting whether a compound has a particular property, the method comprising: training a machine learning (ML) technique to generate the property model; generating, using the property model, a prediction result for one or more compounds and whether they have the particular property, and wherein the property model outputs a list of prediction results; using a selection model to select a list of compound candidates from the list of prediction results for validation, wherein the selection model is generated by training a reinforcement learning (RL) technique to learn from the list of prediction results which compounds to select in order to maximize the quality of the selection; validating the property model based on whether the one or more compounds from the prediction results have the particular property, wherein validating the property model further comprises: validating the list of compound candidates from the list of prediction results that have the particular property; and updating the property model based on the validation of the property model, wherein updating the property model further comprises: updating the property model based on training the ML technique with a labeled training dataset comprising the validated list of compound candidates, wherein generating the selection model based on the RL technique further comprises: selecting, using the selection model, a set of compounds for the list of compound candidates from the list of prediction results for validation; validating whether the selected list of compound candidates have the particular property; and updating the property model based on the ML technique and the validated list of compound candidates; generating an ML score and a further list of prediction results based on the updated property model; and determining, based on the ML score and a previous ML score, whether to retrain the selection model to select a set of compounds for the list of compound candidates.

2. The computer-implemented method of claim 1, further comprising: repeating at least the generating and validating steps using the updated property model until it is determined that the property model has been effectively trained.

3. The computer-implemented method of claim 1, the method further comprising: generating, using the property model, a prediction result for a plurality of compounds and their association with the particular property; and validating the property model based on the compounds from the list of prediction results that have an association with the particular property.

4. The computer-implemented method of claim 1, wherein, initially training the ML technique based on a labeled training dataset associated with a subset of the plurality of compounds related to the particular property.

5. The computer-implemented method of claim 1, wherein, updating the property model further comprises: generating another labeled training dataset based on the validated list of compound candidates and any previous labeled training dataset associated with the particular property; and retraining the ML technique based on the generated labeled training dataset.

6. The computer-implemented method of claim 1, wherein, validating the list of compound candidates further comprises: determining, based on the particular property and the list of compound candidates, whether to perform a laboratory experiment; and in response to determining to perform a laboratory experiment, using experimental results from the laboratory experiment to estimate an association of each compound on the list of compound candidates with the particular property.

7. The computer-implemented method of claim 6, wherein, Determining to perform a laboratory experiment is based on one or more of the following group: a number of validation iterations exceeding a validation iteration threshold in which simulated analysis has been performed in succession to validate the candidate list; an indication that laboratory analysis will improve the ML score of the property model based on a previous property model score computed from a corresponding list of prediction results generated after each list of compound candidates has been validated; or a combination of a number of validation iterations and an indication that laboratory experiment will provide an improved property model.

8. The computer-implemented method of claim 6, wherein, Determining whether to perform a laboratory experiment further comprises: determining whether the selected list of compound candidates has a substantial change from a previously selected list of compound candidates; in response to determining that the selected list of compound candidates does not have a substantial change from the previously selected list of compound candidates, selecting to perform a laboratory experiment on a selected subset of compounds from the selected list of compound candidates.

9. The computer-implemented method of claim 1, wherein, Validating the list of compound candidates further comprises: determining whether to perform simulated analysis based on the particular property and the list of compound candidates; and in response to determining to perform simulated analysis, using simulated results from the simulated analysis to estimate an association of each compound on the list of compound candidates with the particular property.

10. The computer-implemented method of claim 9, wherein, Determining to perform simulated analysis is based on one or more of the following group: a number of validation iterations exceeding a validation iteration threshold in which simulated analysis has been performed in succession to validate the candidate list; an indication that simulated analysis will improve the ML score of the property model based on a previous property model score computed from a corresponding list of prediction results generated after each list of compound candidates has been validated; or a combination of a number of validation iterations and an indication that simulated analysis will provide an improved property model.

11. The computer-implemented method of claim 10, wherein, The number of validation iterations in which simulated analysis is performed in succession is greater than the number of validation iterations in which laboratory analysis is performed.

12. The computer-implemented method of claim 11, wherein, For each of a plurality of generation and validation iterations in which simulated analysis is performed in succession, a laboratory analysis is performed once.

13. The computer-implemented method of claim 1, wherein, The list of prediction results includes a prediction score of whether each compound has the particular property, the method further comprising selecting the list of compound candidates from the list of prediction results based at least in part on the prediction scores.

14. The computer-implemented method of claim 13, wherein, Validating the list of compound candidates further comprises selecting one or more compounds for the list of compound candidates from the list of prediction results based on whether the compounds have a prediction score indicative of a threshold prediction score.

15. The computer-implemented method of claim 14, wherein, The prediction score includes a certainty score, wherein a compound known to have the particular property is given a positive certainty score, a compound known to not have the particular property is given a negative certainty score, and other compounds are given an uncertainty score between the positive and negative certainty scores.

16. The computer-implemented method of claim 15, wherein, The certainty score is a percentage certainty score, wherein the positive certainty score is 100%, the negative certainty score is 0%, and the uncertainty score is between the positive and negative certainty scores.

17. The computer-implemented method of claim 1, wherein, Selecting the shortlist of compounds from the list of prediction results further comprises selecting one or more compounds with inconclusive prediction results.

18. The computer-implemented method of claim 1, wherein, Selecting the shortlist of compounds from the list of prediction results further comprises selecting one or more compounds that are dissimilar to the compounds used in any labeled training data used to date.

19. The computer-implemented method of claim 1, in response to determining to retrain the selection model, the method further comprising: restoring the updated property model to a previous property model when the ML score does not meet a property model performance threshold compared to the corresponding previous ML score; retaining the updated property model as a previously trained property model when the ML score indicates that the property model performance threshold is met or exceeded compared to the corresponding previous ML score; and retraining the selection model to select a set of compounds from the corresponding list of prediction results based on the ML score; and repeating the steps of claim 1 until it is determined that the selection model is trained.

20. The computer-implemented method of claim 19, wherein, Determining to train the selection model further comprises: comparing the retained property model score to a previous retained property model score; and determining that the selection model has been effectively trained based on a stable value of property model scores.

21. The computer-implemented method of claim 20, wherein, Determining whether the property model has been effectively trained further comprises determining that the property model has been effectively trained based on an indication that no further validation of shortlists is required.

22. The computer-implemented method of claim 1, wherein, Validating the property model further comprises: generating a property model score based on the list of prediction results; determining whether the property model has been effectively trained based on the property model score and a previous property model score.

23. The computer-implemented method of claim 22, wherein, Determining whether the property model has been effectively trained comprises determining that the property model has been effectively trained based on a stable value of property model scores.

24. The computer-implemented method of any one of claims 1-23, wherein, The ML technique comprises at least one ML technique or a combination of ML techniques from the group of: a recurrent neural network configured to predict a second compound exhibiting a set of desired properties from a first compound; a convolutional neural network configured to predict a second compound exhibiting a set of desired properties from a first compound; a reinforcement learning algorithm configured to predict a second compound exhibiting a set of desired properties from a first compound; and any neural network architecture configured to predict a second compound exhibiting a set of desired properties from a first compound.

25. The computer-implemented method of any one of claims 1-23, wherein, The particular property comprises a property or characteristic indicative of one or more of: a compound that interfaces with another compound to form a stable complex; a ligand that interfaces with a target protein, wherein the compound is the ligand; a compound that interfaces or binds with one or more target proteins; a compound having a particular solubility or range of solubilities; a compound having a particular toxicity; any other property or characteristic associated with a compound that is simulated based on computer simulations and physical motion of atoms and molecules; any other property or characteristic associated with a compound determined from an expert knowledge base; and any other properties or characteristics associated with the compound determined from the experiment.

26. The computer-implemented method of any one of claims 1-23, further comprising: further training the property model by iterating the steps of generating, validating and updating the property model until it is determined that the property model has been effectively trained, wherein the updated property model from a previous iteration is used in the generating, validating and updating steps of the current iteration.

27. An apparatus comprising a processor, a memory unit, and a communication interface, wherein, the processor is connected to the memory unit and the communication interface, wherein the processor and memory are configured to implement the computer-implemented method according to any of claims 1 to 26.

28. A method of implementing a machine learning (ML) model, wherein, the machine learning, ML, model comprises data representing an ML model generated from training an ML technique according to the computer-implemented method of any of claims 1 to 26.

29. A method of implementing a machine learning, ML, model, wherein the machine learning, ML, model is obtained by the computer-implemented method according to any of claims 1 to 26.

30. An apparatus comprising a processor, a memory unit, and a communication interface, wherein, the processor is connected to the memory unit and the communication interface, wherein the processor and memory are configured to implement the method of implementing a machine learning, ML, model according to claim 28 or 29.

31. A computer readable medium comprising data or instruction code representing a machine learning, ML, model generated based on training an ML technique according to the computer-implemented method of any of claims 1 to 26, the data or instruction code when executed on a processor causing the processor to implement the ML model.

32. A method for predicting whether a compound has a certain property using a model trained according to the computer-implemented method of any of claims 1 to 26.

33. An apparatus for generating a property model for predicting whether a compound has a certain property, the apparatus comprising: a model generation module for training a machine learning, ML, technique to generate the property model; a model testing module for using the property model to generate a prediction of a compound and whether it has the certain property, and wherein the property model outputs a list of predictions, wherein the model testing module is configured to use a selection model to select a list of candidate compounds from the list of predictions for validation, wherein the selection model is generated by training a reinforcement learning, RL, technique to learn from the list of predictions which compounds to select so as to maximize the quality of the selection; a validation module for validating the property model based on whether the compounds from the predictions have the certain property, wherein the validation module is configured to validate the list of candidate compounds from the list of predictions that have the certain property; and a model updating module for updating the property model based on the validation of the property model, wherein the model updating module is configured to update the property model based on training the ML technique on a labelled training data set comprising the list of validated candidate compounds, wherein generating the selection model based on the RL technique further comprises: selecting, using the selection model, a set of compounds from the list of predicted results for validation for the list of compound candidates; validating the selected list of compound candidates for the specific property; and updating the property model based on the ML technique and the validated list of compound candidates; generating an ML score and a further list of predicted results based on the updated property model; and determining, based on the ML score and a previous ML score, whether to retrain the selection model to select a set of compounds for the list of compound candidates.

34. The apparatus of claim 33, wherein, The model generation module, model testing module, validation module, and / or model updating module are configured to implement the computer-implemented method of any one of claims 1 to 26.

Images