COMPUTERIZED METHOD FOR DETERMINING THE CONFIDENCE OF A PREDICTION OUTPUT OF A P

Abstract

A computerized method for determining reliability of a prediction output of at least one first predictive model developed by machine learning or deep learning includes a predictive model computerization module executing at least one first predictive model to obtain a prediction output according to the at least one first predictive model and at least one other predictive model to obtain a prediction output according to the at least one other predictive model, a similarity computerization module executing a similarity scoring method to determine a similarity score between the prediction output of the at least one first predictive model and the prediction output of the at least one other predictive model, and a reliability computerization module executing a reliability model to determine reliability of the prediction output of the at least one first predictive model based on the determined similarity score.

Description

[Technical field]

[0001] The present invention relates to the field of predictive algorithms or models executed by computerized modules, in particular to the field of machine-learned or deep-learned predictive models.

[0002] The present invention finds particular application in the field of medicine, although not exclusively so. [Background technology]

[0003] Machine-learned or deep-learned networks or models are powerful black boxes that have recently achieved superior performance across a wide range of tasks.

[0004] Those black boxes are increasingly used in various applications in industry thanks to their excellent results in prediction. However, it is necessary to understand the meaning of those results and / or the process of the models to obtain those results. Therefore, the operational acceptability of such applications is mainly conditioned by the ability of engineers and decision makers to understand the meaning and properties of the results generated by those models.

[0005] Moreover, beyond the issue of interpretability and explainability of models, the uncertainty of the results of those models is also a key point: indeed, machine-learned or deep-learned networks or models are poor at quantifying the uncertainty of their predictions and tend to be overconfident in their predictions.

[0006] Therefore, in addition to achieving high accuracy, it is also important to obtain reliable uncertainty estimates, which help to determine whether a model's predictions can be trusted, even if they are not fully understood. Quantifying the uncertainty of predictions or forecasts in machine-learned or deep-learned networks or models is a difficult and unsolved problem. Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, there is a need to evaluate the reliability of the results of machine-learned or deep-learned models. [Means for solving the problem]

[0008] Thus, the present invention relates to a computerized method for determining the reliability of a predictive output of at least one first machine learned or deep learned predictive model, the method comprising: - a predictive model computerization module, - at least one first prediction model for obtaining a prediction output according to the at least one first prediction model; - at least one other predictive model for obtaining a predictive output according to the at least one other predictive model; and - a similarity computerized module, a prediction output of at least one first predictive model; and with the predicted output of at least one other predictive model; performing a similarity scoring method to determine a similarity score between - a reliability computerized module executes a reliability model to determine reliability of a predictive output of the at least one first predictive model based on the determined similarity scores.

[0009] The first prediction model and at least one other prediction model are performed on the input image.

[0010] Thanks to this method, it is possible to determine the reliability of the output of a machine-learned model (here, the first predictive model) even if it is not possible to explain and / or interpret this machine-learned model.

[0011] The method allows one to estimate the reliability of a model subject to epistemic uncertainty in addition to random uncertainty.

[0012] Thus, the method makes it possible to identify the limits of a machine-learned model's capabilities by determining the reliability of its output.

[0013] The trust determination helps decision makers decide whether or not to use the output of a machine-learned model.

[0014] Indeed, as an example, the output of a machine-learned model may be the input of another computerized module, knowing that even if the computerized module into which the input data is fed is well thought out, flawed or meaningless input data will produce a meaningless output, so the method provides an improvement to any system that runs the computerized module.

[0015] Furthermore, during the training phase of a machine-learned model, the method can warn against overfitting (settling on an inappropriate and overly complex theory designed to fit all past training data, known as overfitting).

[0016] Preferably, the model is run on a medical image.

[0017] These medical images may represent measurements of biological features of the patient, whereby the measurements are the input measurements and the corresponding biological features are the input features.

[0018] In other cases, the medical image may represent a biological sample, in which case this sample is the input biological sample.

[0019] Thus, the model has as input an input image representing an input biological property or an input measurement of an input sample.

[0020] Preferably, the output of the at least one first prediction model and the at least one other prediction model is a prediction vector of one of the following types: - Simplex probability vector (multi-class), - a one-dimensional probability vector, or - N-dimensional probability vector.

[0021] Preferably, the similarity scoring method is one of the following methods: - Cosine distance method, - Kullback-Leibler Information Method, - Equivalence of the highest probability class method, - the variance of the estimated Dirichlet method, or - Information radius method.

[0022] The present invention also provides - at least one first predictive model; - at least one other predictive model, the other predictive model being a machine-learned or deep-learned individual predictive model; In a method for determining the reliability of a predicted output, the method comprising: - a reliability computerized module executes a reliability model for determining the reliability of the predicted output based on the determined similarity scores.

[0023] If all predictive models are considered to determine similar predictions, i.e., essentially, all predictive models are considered to answer the same question based on the same input dataset, the method can determine the reliability of the output even if the machine-learned model is not explainable and / or interpretable, and then use (or not) the output or make (or not) a decision based on the output according to the determined reliability.

[0024] Preferably, the at least one first predictive model and the at least one other predictive model have different architectures. A model's architecture refers to the underlying mathematical and / or software structure of the model. Each model architecture has certain advantages and disadvantages, biases, limitations related to the advantages and limitations of the mathematical and / or software structure.

[0025] Therefore, each model has a different architecture and the outputs obtained are not subject to the same limitations / biases.

[0026] Preferably, the aggregation computerized module aggregates the prediction outputs into an aggregated prediction output.

[0027] Thus, there is a unique output for determining reliability or credibility. Furthermore, if the at least one first predictive model and the at least one other predictive model have different architectures, the aggregated output combines all the advantages and limitations of the predictive outputs obtained by running the multiple different models.

[0028] Preferably, the aggregation computerized module comprises: - majority vote, - weighted voting, - Bayesian method, or - How to search for information A truth discovery method such as is used to aggregate the predicted outputs into an aggregated predicted output.

[0029] Preferably, the at least one first predictive model and the at least one further predictive model are performed on input data from measurements or acquisitions made by one or more measurement or acquisition systems.

[0030] Preferably, the reliability model comprises: - Multinomial logistic regression model, - A polynomial linear regression model, or - Naive Bayesian model These are machine learning or deep learning predictive models such as:

[0031] The present invention also relates to a computerized method for training a reliability model, the reliability model being a machine-learned or deep-learned predictive model, the method comprising the steps of: i. a predictive computerized module executing a plurality of predictive models on a training data set to obtain a predictive output; ii. a similarity computerized module executing a similarity scoring method to determine a similarity score between the obtained prediction outputs; iii) repeating steps i and ii using a different training dataset in each iteration to obtain a training dataset for a reliability model comprising the determined similarity scores, wherein each similarity score is labeled as reliable or not; iv. a calibration computerized module calibrating the reliability model at a population level and / or at a single image level using the reliability model training dataset according to supervised training.

[0032] Preferably, all training datasets in step i are distinct from any datasets used to train the first predictive model and, if applicable, distinct from any datasets used to train any of the multiple predictive models.

[0033] Preferably, the different training data set of step iii is obtained by data destruction to the training data set of step i, such as noise, blur and / or uncontrast.

[0034] Data corruption is not a classical augmentation method, but a new method for generating new data that is different from the data measured or captured by an acquisition system such as a sensor or image capture system, particularly to allow training of calibration methods based on a wider range of performance scores.

[0035] Indeed, the objective of the present invention is to assess the reliability of the forecast output according to a first forecast model.

[0036] However, in order to obtain sufficient accuracy on this training data set, the prediction model is trained on the training data set. Therefore, the prediction output by the first prediction model is expected to be accurate when performed on a data set close to the training data set. Therefore, in such a case, the method for determining reliability according to the present invention allows determining that the prediction output is reliable.

[0037] However, we would like to be able to detect moments when performance degrades (which cannot be tracked a priori since the first predictive model was trained to provide optimal performance).To solve this paradox, data corruption artificially manufactures low-quality data that degrades performance (in other words, low-quality data is manufactured from the captured measurement data).

[0038] This allows the calibration method to be trained based on a wider range of performance scores.

[0039] Additionally, the data destruction or degradation method provides two additional benefits.

[0040] First, we calibrate based on a data population rather than on data samples. To do this, multiple impairments are run on a given data set with different impairment amplitudes to represent an entire data population.

[0041] This allows a similarity score obtained using data from the data population to be linked to the accuracy of the predictive output of the first predictive model run on the same data from the data population, thereby allowing deviations in the data of the first predictive model to be detected.

[0042] In other words, systematic deviations in the reliability level (which are only detectable in a data population and not in a data sample) are detected thanks to this calibration using corrupted data.

[0043] This is a more robust and accurate method than applying an uncertainty analysis for each sample and then averaging over the entire data population.

[0044] Second, calibrate by data sample: Instead of having a single reliability model that links the similarity score (between the predicted output of at least one first predictive model and the predicted output of at least one other predictive model) to the reliability of the predicted output of the first predictive model, a reliability model can be designed or generated for each data sample.

[0045] Specifically, for a new data sample (for which the accuracy of the first predictive model is unavailable), a set of match points by impairment amplitude is generated, and a set of similarity scores is generated by running the first predictive model and at least one other predictive model on the new data sample for all amplitude levels of impairment.

[0046] Based on this set and the corresponding amplitude levels of impairments, the relationship between the amplitude levels of impairments and the similarity scores can be known.

[0047] In the simplest case, there is a linear relationship between the amplitude level of degradation and the similarity score, so when this method is used when a new data sample appears, a regression line is calculated and used to determine an estimate of the amplitude of zero.

[0048] This method makes it possible to detect overestimation and underestimation of confidence.

[0049] Preferably, the training method further comprises the step of a bootstrap computerisation module performing a bootstrap and / or regularisation method on the reliability model.

[0050] The invention also relates to a computerized method for controlling a system implementing the method according to the invention.

[0051] The present invention also provides - a predictive computerized module, - at least one first prediction model for obtaining a prediction output according to the at least one first prediction model; - at least one other predictive model for obtaining a predictive output according to the at least one other predictive model; A predictive computerized module adapted to perform the following: a similarity computerisation module, a prediction output of at least one first predictive model; and with the predicted output of at least one other predictive model; an affinity computerized module adapted to perform an affinity scoring method to determine an affinity score between - a reliability computerization module adapted to execute a reliability model that determines the reliability of the prediction output of the at least one first predictive model based on the determined similarity score.

[0052] The present invention also relates to a computer program for determining the reliability of a prediction output of at least one first machine learned or deep learned predictive model, the computer program being adapted, when executed on a processor, to cause the processor to perform any method according to the present invention.

[0053] Embodiments of the invention are described below with reference to the following drawings. [Brief description of the drawings]

[0054] [Figure 1] FIG. 1 is a schematic diagram of the general architecture of the computerized method. [Diagram 2] FIG. 1 is a schematic diagram of a training method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] In the various drawings, the same reference numbers indicate the same or similar objects.

[0056] General Overview Machine Learning Machine learning (ML) is the study of computer algorithms that can improve automatically through experience and with the use of data. It is viewed as a part of artificial intelligence. Machine learning algorithms build models based on sample data, known as training data or training datasets, to make predictions, decisions, or classifications without being explicitly programmed. Machine learning algorithms are used in a variety of applications where it is difficult or infeasible to develop traditional algorithms to perform the required tasks, such as healthcare, email filtering, speech recognition, and computer vision.

[0057] The term "prediction" refers to the output of an algorithm, after being trained with a historical data set and applied to new data, in predicting the likelihood of a particular outcome. In some cases, this means actually predicting future outcomes, but in other cases, "prediction" may relate to categorizing data (rather than future data), in which case the term "classification" may also be used. In other words, the term "prediction" should be understood in a broader sense in the field of machine learning. As mentioned before, it may refer to the classification output, the categorization output, or any output of a machine learning model.

[0058] Machine learning techniques are traditionally classified into four broad categories according to the nature of the "signals" or "feedback" available to the learning system. - Supervised learning: the computer is presented with example inputs and their desired outputs given by a "teacher" and the goal is to learn general rules that map inputs to outputs. - Unsupervised learning: The learning algorithm is not given labels, so it finds structure in the input on its own. Unsupervised learning can be an end in itself (discovering hidden patterns in the data) or a means towards an end (feature learning). - Reinforcement learning: A computer program interacts with a dynamic environment in which it has to perform a specific objective (such as driving a vehicle, or playing a game against an opponent). When navigating the problem space, the program is provided with reward-like feedback, which it tries to maximize. - Semi-supervised learning: Semi-supervised learning sits between unsupervised learning (no labeled training data) and supervised learning (fully labeled training data). Although some of the training examples lack training labels, many machine learning researchers have found that using unlabeled data in combination with a small amount of labeled data can significantly improve learning accuracy. In weakly supervised learning, the training labels are noisy, limited, or inaccurate. However, these labels are often cheaper to obtain, which can result in a larger effective training set.

[0059] A supervised learning algorithm builds a mathematical model of a set of data that contains both inputs and desired outputs. This data is known as training data and consists of a set of training examples. Each training example has one or more inputs and a desired target output, also known as a supervisory signal. In the mathematical model, each training example is represented by an array or vector, sometimes called a feature vector, and the training data is represented by a matrix. Through iterative optimization of an objective function, a supervised learning algorithm learns a function that can be used to predict the output associated with a new input. The optimal function allows the algorithm to accurately determine the output for inputs that were not part of the training data. An algorithm that improves the accuracy of its outputs or predictions over time is said to have learned how to perform its task.

[0060] Types of supervised learning algorithms include active learning, prediction, and regression. Prediction algorithms are used when the output is restricted to a limited set of values, while regression algorithms are used when the output can have any numeric value within a range. For example, for a prediction algorithm to filter emails, the input would be the incoming email and the output would be the name of the folder in which to file the email.

[0061] Similarity learning is an area of ​​supervised machine learning closely related to regression and prediction, but where the goal is to learn from examples using a similarity function that measures how similar or related two objects are. Applications include ranking, recommender systems, visual identity tracking, face recognition, and speaker verification.

[0062] Machine Learning Models - Examples Performing machine learning involves creating a trained model based on some training data, which can then process additional data to make predictions.

[0063] There are many different types of models that are used and studied in machine learning or deep learning systems. Below is a list of examples of these different types of models:

[0064] Artificial Neural Network Model An artificial neural network (ANN), or connectionist system, is a computing system loosely inspired by the biological neural networks that make up animal brains. Such systems typically "learn" to perform tasks by examining examples, without being programmed with task-specific rules.

[0065] An ANN is a model based on a collection of connected units or nodes called "artificial neurons", loosely modeled on neurons in a biological brain. Each connection can transmit information, or a "signal", from one artificial neuron to another, similar to a synapse in a biological brain. An artificial neuron that receives a signal can process it and then send the signal to additional artificial neurons that are connected to it. In a typical ANN implementation, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated by some nonlinear function of the sum of its inputs. The connections between artificial neurons are called "edges".

[0066] Artificial neurons and edges typically have weights that are adjusted as learning progresses. Weights increase or decrease the strength of the signal in the connection. Artificial neurons may have a threshold, and a signal is sent only if the aggregate signal exceeds that threshold. Artificial neurons are typically aggregated into layers. Different layers may perform different types of transformations on the input. The signal travels from the first layer (input layer) to the last layer (output layer), possibly after passing through the layer multiple times.

[0067] Certain ANN models are deep learning models. They consist of multiple hidden layers in an artificial neural network. This technique attempts to model the way the human brain processes light and sound into vision and hearing. Learning can be supervised, semi-supervised, or unsupervised.

[0068] Deep learning architectures, including deep neural networks, deep belief networks, deep reinforcement learning, recurrent neural networks, and convolutional neural networks, have been applied to areas such as computer vision, speech recognition, natural language processing, machine translation, bioinformatics, drug design, medical image analysis, materials testing, and board game programs, producing results that match and in some cases exceed the performance of human experts.

[0069] Decision Tree Model Decision tree learning uses a decision tree as a predictive model to go from observations about items (represented in the branches) to conclusions about the item's target value (represented in the leaves). It is one of the predictive modeling techniques used in statistics, data mining, and machine learning. Tree models where the target variable can take on a discrete set of values ​​are called predictive trees, and in these tree structures, the leaves represent class labels and the branches represent the combinations of features that lead to those class labels. Decision trees where the target variable can take on continuous values ​​(usually real numbers) are called regression trees. In decision analysis, decision trees can be used to visually and explicitly represent decisions and decision making. In data mining, decision trees describe the data, but the resulting predictive trees can be the input for decision making.

[0070] Support Vector Machine Model Support vector machines (SVMs), also known as support vector networks, are a set of related supervised learning methods used for prediction and regression. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that predicts whether a new example will fall into one category or the other. SVM training algorithms are non-probabilistic binary linear classifiers, although methods such as Platt scaling exist for using SVMs in probabilistic prediction settings. In addition to performing linear prediction, SVMs can efficiently perform non-linear predictions using the so-called kernel trick to implicitly map their inputs into a high-dimensional feature space.

[0071] Regression analysis model Regression analysis encompasses a wide variety of statistical methods for estimating relationships between input variables and their associated features. The most common form is linear regression, where a single line is drawn that best fits the given data according to a mathematical criterion such as ordinary least squares. The latter is often extended by regularization (mathematical) methods to mitigate overfitting and bias, as in the case of ridge regression. When dealing with nonlinear problems, go-to models include polynomial regression (used, for example, in Microsoft Excel for trend line approximation), logistic regression (often used in statistical forecasting), and even kernel regression, which introduces nonlinearity by utilizing kernel tricks to implicitly map input variables into a high-dimensional space.

[0072] Bayesian Network Model A Bayesian network, belief network, or directed acyclic graphical model, is a probabilistic graphical model that represents a set of random variables and their conditional independence using a directed acyclic graph (DAG). For example, a Bayesian network can represent the probabilistic relationship between diseases and symptoms. The network can be used to calculate the probability of the presence of various diseases given the symptoms. Efficient algorithms exist to perform inference and learning. Bayesian networks that model sequences of variables, such as speech signals or protein sequences, are called dynamic Bayesian networks. A generalization of Bayesian networks that can represent and solve decision problems under uncertainty is called an influence diagram.

[0073] In statistics, naive Bayes classifiers are a family of simple "probabilistic classifiers" based on the application of Bayes' theorem, which assumes strong (simple) independence between features. They are one of the simplest Bayesian network models, but can achieve higher accuracy levels by combining them with kernel density estimation.

[0074] Multinomial logistic regression is used to predict categorical placement in a dependent variable or the probability of categorical membership in a dependent variable based on multiple independent variables. The independent variables can be either dichotomous (i.e., binary) or continuous (i.e., interval or ratio scale). Multinomial logistic regression is a simple extension of binary logistic regression that allows for more than two categories of the dependent or outcome variable. Like binary logistic regression, multinomial logistic regression uses maximum likelihood estimation to assess the probability of category membership.

[0075] Multiple linear regression (MLR), also known simply as multiple regression, is a statistical technique that uses multiple explanatory variables to predict the outcome of a response variable. The goal of multiple linear regression is to model a linear relationship between the explanatory (independent) variables and the response (dependent) variables. In essence, multiple regression is an extension of ordinary least squares (OLS) regression since multiple explanatory variables are involved.

[0076] There are other models, but those are the main ones.

[0077] Training the model Machine learning models typically require large amounts of data to perform well. Typically, when training a machine learning model, a large representative sample of data must be collected from a training set. The data from the training set can be a variety of data, such as a corpus of text, a collection of images, and data collected from individual users of the service.

[0078] Model training is critical to the model's performance and, in particular, to the reliability of the model's results.

[0079] Overfitting is one of the biases to watch out for when training machine learning models. A trained model derived from biased data can lead to distorted or undesirable predictions. Algorithm bias can result from data that is not well prepared for training.

[0080] Overfitting is "the generation of analyses that correspond so closely or accurately to a particular data set that they may fail to fit additional data or reliably predict future observations." An overfitted model is a statistical model that includes more parameters than can be justified by the data. The essence of overfitting is that we unwittingly extract some of the residual variation (i.e., noise) as if that variation represents the underlying model structure.

[0081] The potential for overfitting exists because the criteria used for model selection are not the same as those used to judge the fitness of the model: for example, a model may be selected by maximizing its performance on some set of training data, while its fitness may be determined by its ability to perform well on unseen data; overfitting occurs when a model begins to "memorize" the training data rather than "learning" to generalize from trends.

[0082] In the field of medicine, a concrete example is histology: the data are image data of a biological sample digitized by scanner X. We want to quantify a given marker, the coloring of which is performed by automaton Y. When training is performed, the machine-learned model was trained with scanner X and automaton Y, but the dye has been modified and the amount of this dye spread on the slide may differ from the training phase. However, these differences may lead to a poor performance of the model if the model has overfitted the training data.

[0083] Understanding the model Beyond the biases that can be introduced during the training phase, machine learning models can be difficult to understand. This understanding can be related to the "inference" of the model (how the model determines its outcome), but also to why the model determines such an outcome given the problem it is thought to answer (why does the model determine such an outcome?). These questions are called the interpretability of the model (how) and the explainability of the model (why).

[0084] Interpretability involves providing information representing both the inferences of the machine learning algorithm and the internal representation of the data in a form that can be interpreted by ML or data experts. The results provided are strongly linked to the data used and require knowledge of these data as well as the model.

[0085] Explainability, on the other hand, consists in providing information in a self-contained, complete semantic format that is accessible to the user, for example a decision maker or a field engineer, whether they are a novice or a tech enthusiast, and regardless of their machine learning expertise.

[0086] These two questions are important for assessing the reliability of the model results.

[0087] Objective of the invention A general objective of the present invention is to assess, determine, or evaluate the reliability of the output of a machine learning or deep learning model, even when interpretability and explainability of the model are not achievable.

[0088] For example, one of the objectives of the present invention may be to detect the problem of overfitting of a machine-learned model. In other words, the present invention aims to verify whether the performance of a machine-learned or deep-learned model deviates due to the introduction of data different from that used during training (domain shift).

[0089] Following the same histology example, the present invention aims to detect whether a model is reliable and / or overly reliant on the training data.

[0090] The objective is not just to calibrate the output of the model, i.e. the probabilistic estimate of a particular outcome determined by the model (or the likelihood of a particular outcome as explained above), with the actual probability of this particular outcome, following the classical calibration of machine-learned models.

[0091] Here, the objective is to estimate the reliability (and, in particular, the performance) of a model for a given sample, independent of the probability.

[0092] In other words, the objective is to estimate the reliability of the model according to epistemic uncertainty in addition to random uncertainty, not to estimate the probability of the model output according to random uncertainty only.

[0093] Random uncertainty is what gives rise to the concept of probability. For example, if a very simple machine-learned model is trained to predict the outcome of a dice throw according to the number on the dice, its output is p(X=1)=1 / 6 for a 6-sided die and p(X=2)=1 / 10 for a 10-sided die. On the other hand, if the dice are manipulated without knowing how, it is impossible to give the probability of landing on a 1, but to be precise, we can say that it is impossible for a very simple model to answer the question, which is fundamentally different.

[0094] Therefore, the objective of the present invention is to identify the limits of a model's capabilities, i.e., when the model's predictive output can be trusted.

[0095] The data on which the model runs All the models described hereafter are designed to be run on images, and therefore images are their input images, meaning that their input data is an image, which is their input image.

[0096] Henceforth, the terms "data" or "dataset" refer to an input image, a set of input images, or an input image dataset.

[0097] The input images of the model are technical data acquired by an image acquisition system according to an image acquisition method and / or digitally processed by a digital image processing system according to a digital image processing method. The acquisition system and method and / or the digital processing system and method define a digital acquisition pipeline for the input images.

[0098] Furthermore, this input image represents technical data as this input measurement or this input sample is acquired by an acquisition system according to an acquisition method and / or biologically processed by a biological processing system according to a biological processing method, which measurement systems and methods and / or biological processing systems and methods define a measurement pipeline of the input measurements or input samples.

[0099] Principles of the invention One of the main features of the present invention is the comparison of the output of a tested model with the output of a reference model.

[0100] Thus, first, a model is run and its reliability (or at least the reliability of one output of the tested model) is determined on a particular dataset. This model can be called the tested model.

[0101] Another model, called a reference model, is then run on the same data set and the outputs of the tested model and the reference model are then compared.

[0102] The tested model and the other model are different models, they are not two blocks of the same model.

[0103] However, the tested model and the other model are run on the same input image dataset, and furthermore, the output of the tested model and the output of the other model relate to the same labels, i.e., the tested model and the other model are designed and trained according to the same question and / or to solve the same technical problem (e.g., the tested model and the other model are designed and trained on a training image dataset to help experts answer the same medical question).

[0104] The reliability of the tested model is determined based on the similarity between the two outputs using the principle: the higher the similarity, the more reliable the tested model.

[0105] Finally, a reliability model can be implemented based on this principle, which determines the reliability of the tested model based on the similarity between the outputs of the tested model and the outputs of a reference model.

[0106] Furthermore, another main feature of the present invention is the comparison of multiple outputs of multiple models run on the same dataset together (comparable to each other in terms of both the model's purpose and the form of the output). In this case, there is no reference model to compare to. The idea is that the more consistent the multiple outputs are with each other, the less likely it is that all models do not provide reliable outputs.

[0107] Show me how The method is based on the above principles.

[0108] Computerized Module The method is implemented by a computerized system comprising various computerized modules each responsible for a step of the method.

[0109] The predictive model computerized module or predictive module can execute or is adapted to execute any machine learning model (e.g., the machine learning models described in the general overview).

[0110] This module is capable of, or is adapted to, run any model on a given dataset, which is called an entry dataset since it represents one or more entries of the model, while the prediction or predictions of the model are the output of the model, as explained in the general overview.

[0111] The obtained output of the model is a prediction vector of one of the following types: - Simplex probability vector (multi-class), - a one-dimensional probability vector, or - N-dimensional probability vector.

[0112] The similarity computerized module is capable of, or is adapted to, determine a similarity score between various (at least two) outputs of a prediction or predictive model and at least one machine-learned model.

[0113] To determine the similarity score, the similarity module runs a similarity scoring method on the output.

[0114] That is, the entries of a similarity scoring method are the outputs of a machine-learned model, and the outputs of a similarity scoring method are similarity scores.

[0115] The similarity scoring method can be one of the following methods. - Also known as the cosine distance method or cosine similarity method, - Kullback-Leibler Information Method, - Equivalence of the highest probability class method, - the variance of the estimated Dirichlet method, or - Information radius method.

[0116] Cosine similarity is a measure of similarity between two nonzero vectors in the dot product space. It is defined to be equal to the cosine of the angle between them, which is also the same as the dot product of the same vectors normalized so that both lengths are 1. From the latter definition, it follows that cosine similarity depends only on the angle between two nonzero vectors, not on their magnitudes. Cosine similarity is bounded to be in the range [-1,1] for any angle theta. For example, the cosine similarity of two vectors with the same orientation is 1, the similarity of two vectors oriented at right angles to each other is 0, and the similarity of two vectors with opposite orientations is -1. Cosine similarity is particularly used in positive spaces, where the results are neatly bounded in [0,1]. The name comes from the term "directional cosine", where unit vectors are maximally "similar" if they are parallel, and maximally "dissimilar" if they are orthogonal (perpendicular). This is similar to a cosine and is one (maximum) when the segment angle is zero and zero (uncorrelated) when the segment is perpendicular.

[0117] These restrictions apply to any number of dimensions, and cosine similarity is most commonly used in high-dimensional positive spaces. For example, in information retrieval and text mining, each term is conceptually assigned a different dimension, and documents are characterized by vectors, with the values ​​in each dimension corresponding to the number of occurrences of the term in the document. Cosine similarity then provides a useful measure of how similar two documents are with respect to their subject matter.

[0118] This technique is also used in the field of data mining to measure cohesion within a cluster.

[0119] One advantage of cosine similarity is its low complexity, especially for sparse vectors: only the non-zero dimensions need to be considered.

[0120] Other names for cosine similarity are Orchini similarity and Tacca congruence coefficient, while Otsuka-Ochiai similarity is a cosine similarity applied to binary data.

[0121] In mathematical statistics, the Kullback-Leibler divergence (also called relative entropy) is a statistical distance, a measure of how different one probability distribution Q is from a second, reference probability distribution P. A simple interpretation of the divergence of Q from P is the excessive surprise to be expected from using Q as a model when the actual distribution is P. It is a distance, but not a metric, the best known kind of distance, which is asymmetric (as opposed to the information variance) in the two distributions and does not satisfy the triangular inequality. Instead, in terms of information geometry, it is a generalization of divergence, the squared distance, and for certain classes of distributions (in particular the exponential family), it satisfies the generalized Pythagorean theorem (applied to squared distance).

[0122] The formative radius method, also known as the Jensen-Shannon divergence, is a method derived from the symmetric Kullback-Leibler divergence and is based on the sum of the Kullback-Leibler divergence over all distributions and the mean of all distributions.

[0123] The equivalence method for the highest probability class outputs the proportion of predictions for the most predicted class. For example, if the predicted classes are [1,3,3,2,3], the proportion of predictions is 3 / 5 = 0.6.

[0124] The variance of the Dirichlet method is the variance of the Dirichlet distribution, which describes the distribution of predictions. The Dirichlet distribution is a K-simplex distribution (e.g., the classification output of the model is a K-simplex, where the sum of the K class output probabilities is 1, e.g. [0.1, 0.3, 0.6]).

[0125] For example, models provide K-simplexes that can be approximated with a Dirichlet distribution with maximum likelihood or with Bayesian inference using a given prior distribution.

[0126] The reliability computerized module is capable of or adapted to execute a reliability model.

[0127] The reliability model can determine, or is adapted to determine, a reliability score based on the similarity scores, such that the entries in the reliability model are similarity scores (e.g., as determined by a similarity computerized module) and the outputs are reliability ratings.

[0128] The reliability assessment can take a variety of forms. - A confidence score between a lower and upper bound, - Binary predictions.

[0129] The reliability model is - Multinomial logistic regression model, - A polynomial linear regression model, or - Naive Bayesian model The model may be a machine-learned or deep-learned predictive model such as

[0130] The aggregating computerized module is capable of or adapted to aggregate the various outputs into one aggregated output.

[0131] The aggregate computerized module may, for example, - majority vote, - weighted voting, - Bayesian method, or - How to search for information A truth discovery method such as may be used or adapted to aggregate the predicted outputs into an aggregated predicted output.

[0132] The calibration computerized module can be or is adapted to calibrate or train any machine-learned model at a population level and / or at a single image level using the training dataset.

[0133] This training may be supervised, unsupervised, or reinforced training, as described in the general overview.

[0134] Calibration can be done per population or per sample: per population calibration consists of adjusting the overall performance according to the intensity of the degradation and then predicting the performance from the similarity of the predictions of the sub-models.

[0135] Sample-by-sample calibration is more accurate and consists in generating from a sample a batch, i.e. several copies of this sample with the same degradation intensity (the intensity is the same, but the results are different because it is a random process).

[0136] Batches with different intensities are generated and then a calibration similar to the population method is performed, with multiple calibrations being performed for each different intensity sample.

[0137] The bootstrap computerization module is capable of, or adapted to, perform bootstrap and / or regularization methods on any predictive model, for example a machine-learned model.

[0138] Furthermore, the computer programs, when executed on a processor, cause the processor to execute the modules of the computerized system to perform the methods.

[0139] The reliability of one output of a machine learning model Here, as shown in Figure 1, the objective of the method is to determine the reliability of sub-model 1. This sub-model 1 can be called an individual model, a first model 1, or simply model 1.

[0140] Sub-model 1 can be any machine-learned model, which is pre-trained on a given training dataset.

[0141] First, the predictive model computerization module runs sub-model 1 on a given dataset, different from the training dataset, to obtain a predicted output 10 according to sub-model 1. It is this output for which reliability should be determined.

[0142] The output 10 is typically a vector of one of the following types: - Simplex probability vector (multi-class), - a one-dimensional probability vector, or - N-dimensional probability vector.

[0143] A given data set is formed by measurement data, i.e. data that is measured or captured by an acquisition system, such as a sensor or an image capture system.

[0144] Then, or in parallel, the predictive model module runs at least one other predictive model on the same given data set. For example, as shown in FIG. 1, this at least one other model can be sub-model 2.

[0145] This sub-model 2 is a predictive model adapted to determine predictions, but does not necessarily have to be an artificial intelligence, in particular a machine learning model, but may for example be a rule-based model, an expert system model, etc.

[0146] It may be an artificial model, in particular a machine-learned model. In this case, submodel 2 may have a different architecture than submodel 1, but is designed to solve the same technical problem or answer the same question. In other words, submodel 2 has the same prediction or forecasting objective and, as mentioned before, is run on the same given dataset. So output 10 can be compared with output 20. Furthermore, submodel 2 is trained on a training dataset that may be different from the training dataset of submodel 1 and different from the given dataset on which submodels 1 and 2 are run.

[0147] In another embodiment, sub-model 2 has the same architecture as sub-model 1, but is trained on a different training dataset.

[0148] Thus, the predictive model module determines the output 20 by executing sub-model 2. Similarly, the output 20 is also a vector of one of the following types: - Simplex probability vector (multi-class), - a one-dimensional probability vector, or - N-dimensional probability vector.

[0149] Preferably, outputs 10 and 20 are of the vector output variety.

[0150] At this point, the similarity module determines a similarity score between output 10 and output 20. Output 20 can be considered as a reference for output 10 whose reliability should be determined.

[0151] As noted above, any similarity method can be used to determine this similarity score.

[0152] Next, when the similarity score between output 10 and output 20 is known, the question is, how reliable (or trustworthy) is output 10 knowing the similarity score between output 10 and output 20?

[0153] A reliability module has been devised to answer this question. Based on the similarity scores, the reliability module executes a reliability model that determines the reliability of the output 10. This reliability can be a reliability score between a lower and upper bound. It can also be a "yes" or "no" answer based on the reliability score (Boolean output). The reliability model is described below.

[0154] The method may be performed with reference to multiple sub-models.

[0155] For example, there may be four sub-models as shown in Figure 1. In this case, the prediction module further executes sub-models 3 and 4 to obtain outputs 30 and 40.

[0156] Preferably, sub-models 2, 3, and 4 have different architectures (may be different blocks or different sizes, etc.). As an example, one sub-model may use an end-to-end trained ResNet50, while another sub-model may use a self-supervised variational autoencoder with a multi-layer perceptron on top. A set of reference models is selected to globally remove systematic errors introduced by each model architecture.

[0157] The similarity module determines similarity scores between output 10, whose reliability is being evaluated, and outputs 20, 30, and 40, according to any of the similarity scoring methods described above.

[0158] The reliability module then determines the reliability of output 10 by running a reliability model on the similarity scores between output 10 and outputs 20, 30, and 40.

[0159] Reliability of multiple outputs of machine-learned models According to this embodiment, the reliability of multiple outputs of the machine-learned models is determined. As mentioned above, all machine-learned models may have different model architectures, but are designed to answer the same question or solve the same technical problem.

[0160] However, all the machine learned models were run on the same input image dataset and the outputs of all the machine learned models are associated with the same labels, i.e., all the machine learned models were designed and trained according to the same question and / or to solve the same technical problem (e.g., all the machine learned models were designed and trained on a training image dataset to help experts answer the same medical question).

[0161] For example, as shown in Figure 1, the reliability of submodels 1 to 4 needs to be determined.

[0162] In other words, compared to the previous embodiment, there is no reference sub-model to which the output is compared.

[0163] Apart from this, the methods are similar: the predictive model module runs multiple machine-learned models, e.g., the four shown in Figure 1, on a given dataset to obtain multiple predictions or forecast outputs 10-40.

[0164] The outputs 10-40 can be aggregated into an aggregate output by an aggregation computerized module to have a single output for which reliability is determined.

[0165] The multiple outputs 10-40 can then be compared to each other by a similarity module according to the similarity methods described above, resulting in a similarity score between the outputs 10-40.

[0166] The reliability module then runs a reliability model on the outputs 10-40 similarity scores to determine the reliability of these outputs.

[0167] This part of the method corresponds to the upper part of FIG.

[0168] When outputs 10-40 are aggregated, the reliability module determines the reliability of the aggregated output by running a reliability model on the similarity scores.

[0169] Furthermore, multiple sub-models, for example the four sub-models 1-4 shown in FIG. 1, can be combined into a general model called the performance model shown in FIG. 1 because multiple sub-models are run on the same given data set by the predictive model module.

[0170] In other words, the predictive model module runs the performance model on a given data set and determines as many outputs as there are sub-models collected in the performance model.

[0171] Additionally, the performance model can be combined with an aggregation module, which then aggregates the outputs of the performance models into one aggregate output. This portion of the method corresponds to the top of FIG.

[0172] In fact, it is generally more convenient to have only one output. Note that the sub-models preferably have different model architectures, all with their own biases, and thus aggregation to one aggregate output allows for having only one output that takes into account all sub-model architectures. The reliability module can then determine the reliability of the aggregated output (of the outputs) of the performance model, as described above.

[0173] Reliability Model As explained above, the trustworthiness model is responsible for the link between the similarity scores and the trustworthiness ratings.

[0174] In one embodiment, the reliability model may be a model that is not a machine-learned model, for example, a rule-based model or an expert system model.

[0175] For example, a simple way to link similarity scores and trust ratings is to predetermine a trust threshold for the similarity score, below which no trust is assigned and above which trust is assigned.

[0176] In another embodiment, the reliability model is a machine-learned model that needs to be trained so that it can link, or is adapted to link, similarity scores and reliability ratings.

[0177] The purpose of training the reliability model is to determine whether the outputs of the sub-models, if applicable, that have been collected into the performance model can be trusted, i.e., whether reliability can be assigned to those outputs, and not to determine the performance or accuracy of the sub-models or, if applicable, the performance model.

[0178] In fact, as explained in more detail in the general overview, reliability machine-learned models can be trained following four main approaches: - Supervised learning - Unsupervised learning - Reinforcement learning - Semi-supervised learning

[0179] The reliability model is trained using training data that has not been used to train any of the sub-models or individual models, whether or not combined into a performance model. Thus, the one or more training data sets of the reliability model are different from the one or more training data sets used to train the one or more sub-models. To more clearly distinguish it from the training data sets of the sub-models, the training data set of the reliability model is called a calibration data set, as shown in FIG. 2. The calibration data set is different from the one or more data sets used to train the one or more sub-models.

[0180] In this calibration dataset, degradations such as brown noise, pixel removal, blurring, etc. can be applied with the aim of increasing the number of data, but above all to allow autonomous performance tuning in order to fully calibrate the reliability model.

[0181] Since the reliability model has similarity scores for the entry inputs, a calibration data set for the reliability model needs to be created with the similarity scores. To obtain them, the prediction module runs multiple sub-models, for example four sub-models 1-4, on the data of the calibration data set to obtain the outputs of the multiple sub-models. The similarity computerization module then determines similarity scores between the obtained outputs.

[0182] These steps are repeated for other data in the calibration dataset to obtain other similarity scores, each of which can be labeled as reliable or not, since the data from the calibration dataset is known and the performance of the sub-models on these data is known.

[0183] As explained above, degradation of the calibration dataset allows a wider range of similarity scores to be determined to better train the reliability model. For example, running a submodel on low-quality data where the calibration module knows it cannot assign output reliability will result in similarity scores that are labeled as unreliable similarity scores for the reliable model.

[0184] One of the advantages is the use of reliability models that are machine-learned models: as explained in the general overview, reliability models can be difficult to interpret and even explain, but thanks to supervised learning they provide a reliability estimate or a reliability assessment of the outputs of the sub-models.

[0185] The calibration module then calibrates the confidence model at a population level and / or at a single image level using the confidence model training dataset formed by all the labeled similarity scores.

[0186] Finally, the bootstrap module performs a bootstrap and / or regularization method on the reliability model, i.e. the calibration is repeated using several subsamples of the training data of the calibration dataset to validate the reliability model parameters to know if they are robust or not. If they are not robust, they can be set to their average value or set to zero.

[0187] Advantages of the method As explained in the general overview, machine learning models can be difficult to interpret and / or explain. In this context, it is difficult to know whether the model's output can be trusted.

[0188] Thanks to the method, decision makers can avoid problems that are still unsolved for most machine-learned models, but still be able to assess their confidence in the results of the tested models.

[0189] Furthermore, as seen in certain embodiments, the method allows one to assess the reliability or confidence in the results of a model even in the absence of a reference model. This is very useful when models have different architectures, since each architecture has its own biases or limitations. Furthermore, each sub-model can be difficult to understand (difficult to interpret and / or explain), making it even more difficult to understand multiple different outputs (or answers to the same question).

[0190] When the outputs from the different models (which are compiled into a performance model) are aggregated into one aggregate output, the method allows one to determine a level of trust, confidence or reliability in this aggregate output, even if obtaining the aggregate output is difficult to understand.

[0191] The aggregated output can be seen as a unique result or response (for a question or technical problem) of the performance model, making the method very useful for decision makers as decisions can be made based on the output if good confidence is determined by the method.

[0192] Furthermore, the method allows decision makers to detect biases introduced during the training phase of a machine-learned model. An example is overfitting of a machine-learned model. Thanks to the method, an overfitted model run on data different from the training dataset will have a lower similarity score with other models run on the same data. The reliability module will flag this.

[0193] Purpose From a most general point of view, the method can be used in all industries and applications where machine-learned models are used to make predictions or decisions. In particular, the method can be used to control a system based on the reliability of the model's output or an evaluation of the reliability of the model. The control of the system can be direct or immediate, i.e. the trusted output is related to the general characteristics of the controlled system (speed of delivery, quantity delivered, location positioned, speed of movement, etc.), or it can be indirect, where the trusted output is calculated by another computerized module that then leads to the general characteristics of the controlled system.

[0194] However, machine learning and machine-learned models can also be used for other purposes, such as discovering unknown features in the data on which the models are run; this field is called data mining. While machine learning focuses on prediction, based on known features learned from training data, data mining focuses on discovering (previously) unknown features in the data (this is the analytical step of knowledge discovery in databases). The method can flag falsely detected features (such as overfitting, as mentioned above) and can also compare different machine-learned models to assess their reliability in detection.

[0195] Another application area of ​​machine-learned models is optimization, since many learning problems are formulated as the minimization of a loss function on a training set of examples. The loss function represents the discrepancy between the predictions of the trained model and real problem instances. This method makes it possible to evaluate the reliability of the results of optimized technical parameters. Optimization of technical parameters based on unknown data from measurements (or measurement data) is a common application of machine-learned models.

[0196] Field Medicine A specific field of application could be medicine.

[0197] In the art, one may wish to interpret measurement data with the understanding that the data from the measurements can be analyzed or interpreted without requiring the presence of the subject on whom the measurements were made, and performance of each step of the method does not require the bodily presence of the subject.

[0198] By way of example, medical imaging and histology are application areas for this method.

[0199] Medical imaging establishes a database of normal anatomy and physiology and enables the identification of abnormalities.

[0200] Histology is the branch of biology that studies the microscopic anatomy of biological tissues. Histology is the microscopic counterpart of gross anatomy, which examines larger structures visible without a microscope.

[0201] In those fields, the measurement data on which the sub-model (and / or, by extension, the performance model) can be executed are images captured by an acquisition system. They may be images of organs or tissues. In other words, they are the entry inputs of the sub-model (and / or, by extension, the performance model).

[0202] Measurement and recording techniques not primarily designed to produce images, such as electroencephalography (EEG), magnetoencephalography (MEG), electrocardiography (ECG), etc., represent other techniques that produce data that are amenable to representation as parameter graphs versus time or maps, including data regarding measurement locations. These techniques can be thought of as forms of medical imaging, and their representations or "images" can be the input or entry data against which the sub-models (and / or, by extension, the performance models) are run.

[0203] For example, a two-dimensional image comprises a set of pixels spanning the plane of the image. Each pixel represents a square subset of the image plane. This square subset has a centroid. In the Cartesian coordinate system (x;y), each square can be located according to the coordinates (x;y) of its centroid. The coordinates x and y represent the two degrees of freedom of an image pixel.

[0204] The position is therefore the first parameter of each pixel forming the image, and of course there is also the intensity and color of the layer, even if they do not distinguish between the pixels of the layer, since they apply to the whole layer.

[0205] The intensity varies from 0 to 255 according to the RGB system. It can also be in grey levels.

[0206] Thus, the input data from an "image" of a representation on which the sub-models (and / or, by extension, the performance model) can be run are the position (x;y), intensity and color of the layers from the "image".

[0207] The same applies mutatis mutandis to three-dimensional images.

[0208] Following the specific examples given in the general overview, the method can warn of overfitting of the model.

[0209] Just to be clear, in this concrete example, the image data (entry data) on which the histology submodel is being run is digitized on scanner X. We want to quantify a given marker that has been colored by automaton Y. When the training is performed, it is possible that, although the submodel was trained with scanner X and automaton Y, the dye has been modified and the amount of this dye spread on the slide is different from the training phase. Thanks to the similarity score and the high confidence model, the method implemented in the training phase of the histology submodel warns that the considered histology submodel suffers from overfitting. Here, the overfitting is performed based on unknown parameters related to the dye in the training phase.

[0210] Following the same example, when various histology sub-models, preferably with different architectures and preferably grouped together in a common performance model, are run on a given dataset of images to be classified, the method allows for assessing the reliability of the prediction outputs aggregated into one aggregated prediction output, allowing for a determination to be made whether the aggregated prediction output can be trusted.

[0211] If the aggregated prediction output is then trusted according to the method, the decision maker can then make a decision based thereon. For example, if the sub-models refer to predicting whether or not a subject's body tissue will develop a disease state, the method allows the decision maker to determine whether or not the aggregated output of the performance models can be trusted and whether or not a disease state is present based on the output.

[0212] In another example, the input data may be a patient's temperature (e.g., 40.1° C.) and the aggregated output of the performance model may be a hint for the diagnosis of influenza. For example, the output may be the probability of having influenza, or a Boolean output of "yes" or "no."

[0213] Four different sub-models forming the performance model are adapted to determine whether a patient is likely to have influenza if their temperature is 40.1°C.

[0214] For example, each of these sub-models determines an output, which may not be in the exact same data format, about the probability that a patient has influenza, and a similarity scoring module determines a similarity score between the outputs of the sub-models.

[0215] The method assesses the confidence in the aggregated output based on the similarity scores.

[0216] Specific Examples A specific use case in the medical field is described below, where the performance model aims to detect cancer on digitized patient tissue.

[0217] Here, the model is run on a medical image.

[0218] These medical images may represent measurements of biological features of the patient, whereby the measurements are the input measurements and the corresponding biological features are the input features.

[0219] In other cases, the medical image may represent a biological sample, in which case this sample is the input biological sample.

[0220] Thus, the model has as input an input image representing an input biological property or an input measurement of an input sample.

[0221] The case is divided into two parts: preparation of the model and then use of the model.

[0222] training In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element becomes the input of the next. The elements of a pipeline are often performed in parallel or time-sliced ​​fashion. Some amount of buffer storage is often inserted between the elements.

[0223] Here, the complete data pipeline converts images of biological tissue into an indicator of "cancerous" or "non-cancerous" according to the set of data processing defined above. In other words, the predicted output of the data pipeline is the answer "cancerous" or "non-cancerous".

[0224] This output prediction is associated with a probability.

[0225] The pipeline architecture consists of two distinct deep learning model cores that each convert an input image into a "cancerous" score from 0 to 1. In other words, the performance model here consists of two sub-models, which are two distinct deep learning models that convert an input image into a "cancerous" score from 0 to 1.

[0226] The first submodel is a classical CNN, EfficientNetB7, and the second submodel is a combination of Fourier transform layers whose outputs are sequentially connected to a Visual Transformer.

[0227] The data required to build and train all sub-models of the pipeline are a training dataset, a calibration dataset, and a dataset for regulatory clinical validation.

[0228] These data sets comprise images of biological tissue.

[0229] The first sub-model is trained on a random 90% of the training dataset, and the second sub-model is trained on another random 90% of the training dataset.

[0230] A truth-finding technique is applied to aggregate the outputs of the two sub-models, which is the absolute difference between both scores.

[0231] The similarity scoring method used measures the squared difference of both scores.

[0232] The reliability model is a regression model that is a linear regression model with bias parameter b and coefficient parameters c.

[0233] To calibrate the performance model, a function is used that adds noise of intensity I to the images. 1000 images are generated for different intensities and different images, then both the accuracy of the performance model and the corresponding calculated similarity are evaluated.

[0234] This evaluated accuracy and similarity are the data used to build a linear regression model, where similarity is the input and accuracy is the output. Now, the output of the regression model is called the confidence. Typical examples for b and c would be 1 and -0.5, respectively.

[0235] After training, the performance model ends up with 98% accuracy, which means that it gets the right answer 98% of the time (based on the training data).

[0236] Production Use For clinical reasons, we require at least 99% performance of the previous data pipeline. In other words, a minimum confidence score is set such that 99% of trusted or reliable predictions made by a performing model are accurate predictions.

[0237] Here it is simply a similarity score with a minimum value of 0.99.

[0238] The trusted model assigns no confidence to the performance model's prediction if the similarity score, as determined by the comparison scores, is:

[0239] It sends a message to doctors that the results are not reliable enough to be accepted at face value.

[0240] Other fields In another technical field, an application could be braking control based on recognition of pedestrian trajectories on video from a vehicle's on-board camera.

[0241] In this example, the input data may be the distance between the vehicle and the pedestrian previously measured by the measurement module based on images captured by the on-board camera, the speed of the vehicle calculated using the measured speed of the engine and / or the position of the vehicle measured by any position system (e.g., GPS), the position of the pedestrian on the road based on images captured by the on-board camera.

[0242] Several given state parameters of the situation can also be considered as entries or parameters of the sub-model, such as parameters relating to the road (bumps, slope), the tire wheels (wear), the weather (amount of rainfall), the position at the very moment of execution and / or a few days ago, etc.

[0243] Each of the sub-models is adapted to determine whether a pedestrian is struck by a vehicle. The output may be, for example, a probability of being struck, or a Boolean output "yes" or "no".

[0244] The performance model determines an aggregate output based on the outputs of the sub-models, and thus this aggregate output is the answer to the question "Will a pedestrian be struck by a vehicle?"

[0245] The method allows one to assess the confidence in the answer, ie the output of the performance model.

[0246] For example, the method can be applied in systems aimed at detecting nearby objects in order to avoid collisions.

[0247] This assessment allows the method to control the braking of the vehicle, and if the aggregated output classifies a collision and this output is reliable according to the method, the vehicle brakes are activated to avoid the collision.

[0248] The system consists of a camera, a RADAR, and a LiDAR associated with a deep learning model adapted to determine distances. Each of these subsystems returns an estimate of the distance to the nearest object in all directions.

[0249] Deep learning models need to be trained based on range data, and RADAR and LiDAR need to be calibrated.

[0250] In this use case, the truth discovery method is the average of the three estimates, and the similarity score is the coefficient of variation of the three estimates.

[0251] For calibration, perturbations such as fog and metal dust are also used in the same process.

[0252] Here, three sub-models predict collisions according to distances determined by the camera, RADAR, and LiDAR, which are associated with a deep-learned model adapted to determine distances. The aggregated output is the average of the three estimates, ranging from 0 to 1.

[0253] The similarity score is the coefficient of variation of the three predictions ranging from 0 to 1.

[0254] In another aspect of the invention, the method can also be implemented on three systems adapted to determine distance, i.e., a camera, a RADAR, and a LiDAR associated with a deep learned model adapted to determine distance, where the computerized method determines the reliability of the predicted output of the deep learned model.

[0255] The predictive model computerized module executes the deep learned model adapted to determine distance to obtain a predicted distance output according to the model, and other systems adapted to determine distance are also executed, such as RADAR and LiDAR, which output distance.

[0256] The three distances are aggregated according to the truth-finding method defined above.

[0257] The similarity computerized module then executes a similarity scoring method to determine a similarity score between: - the predicted distance output of the model; - RADAR and LiDAR distance output

[0258] The similarity score is the coefficient of variation of the three estimates.

[0259] Finally, a reliability computerized module executes a reliability model that determines reliability of a predictive output of the at least one first predictive model based on the determined similarity scores. [Explanation of symbols]

[0260] 1,2,3,4 Submodel 10,20,30,40 Output

Claims

1. 1. A computerized method for determining reliability of a predictive output of at least one first machine learned or deep learned predictive model, comprising: - a predictive model computerization module, - at least one first prediction model for obtaining a prediction output according to the at least one first prediction model; - at least one other predictive model for obtaining a predictive output according to said at least one other predictive model; and - a similarity computerized module, - the prediction output of the at least one first predictive model; and - the prediction output of the at least one other predictive model; performing a similarity scoring method to determine a similarity score between - a reliability computerized module executes a reliability model to determine reliability of the prediction output of the at least one first predictive model based on the determined similarity score; The method comprising:

2. The prediction output of the at least one first predictive model and the at least one other predictive model is - simplex probability vector, - a one-dimensional vector, or - N-dimensional vector The method of claim 1 , wherein the predictive vector is one of the following types:

3. The similarity scoring method comprises: - Cosine distance method, - Kullback-Leibler Information Method, - Equivalence of the highest probability class method, - the variance of the estimated Dirichlet method, or - Information radius method 3. The method according to claim 1 or 2, which is one of the methods.

4. - said at least one first predictive model; - the at least one other predictive model being a machine-learned or deep-learned individual predictive model; and and determining a reliability of the predicted output, the reliability being determined by: The method according to any one of claims 1 to 3, further comprising: - executing the reliability computerisation module a reliability model for determining the reliability of the predicted output based on the determined similarity score.

5. The method of claim 4 , wherein the at least one first predictive model and the at least one other predictive model have different model architectures.

6. The method of claim 4 or 5, further comprising the step of an aggregation computerized module aggregating the prediction outputs into an aggregated prediction output.

7. said aggregation computerized module comprising: - majority vote, - weighted voting, - Bayesian method, or - How to search for information The method of claim 6 , further comprising aggregating the predicted outputs into an aggregated predicted output using a truth discovery method such as

8. 8. The method of claim 1, wherein the at least one first predictive model and the at least one other predictive model are run on input data from measurements or acquisitions performed by one or more measurement or acquisition systems.

9. 1. A computerized method for training a reliability model, the reliability model being a machine-learned or deep-learned predictive model, the training method comprising: i. a predictive computerized module executing a plurality of predictive models on a training data set to obtain a predictive output; ii. a similarity computerized module executing a similarity scoring method to determine a similarity score between the obtained prediction outputs; iii) repeating steps i and ii using a different training dataset in each iteration to obtain a reliability model training dataset comprising the determined similarity scores, wherein each similarity score is labeled as reliable or not; iv. a calibration computerized module calibrating said reliability model at a population level and / or at a single image level using said reliability model training data set according to supervised training; The training method includes:

10. 10. The method of claim 9, wherein all of the training datasets of step i are distinct from any datasets used to train the first predictive model in the method of claim 1 and, if applicable, distinct from any datasets used to train any of the plurality of predictive models in the method of claim 4.

11. 11. The training method according to claim 9 or 10, wherein the different training data set of step iii is obtained by data destruction on the training data set of step i, such as noise, blurring and / or uncontrast.

12. 12. The method of training according to any one of claims 9 to 11, further comprising the step of a bootstrap computerisation module performing a bootstrap and / or regularisation method on said reliability model.

13. 13. A computerized method for controlling a system implementing the method of any one of claims 1 to 12, further comprising the step of controlling the system based on the predicted output.

14. 1. A computerized system for determining reliability of a predictive output of at least one first machine learned or deep learned predictive model, comprising: - a predictive computerized module, - at least one first prediction model for obtaining a prediction output according to the at least one first prediction model; - at least one other predictive model for obtaining a predictive output according to said at least one other predictive model; A predictive computerized module adapted to perform the following: a similarity computerisation module, - the prediction output of the at least one first predictive model; and - the prediction output of the at least one other predictive model; an affinity computerized module adapted to perform an affinity scoring method to determine an affinity score between - a reliability computerized module adapted to execute a reliability model for determining reliability of the prediction output of the at least one first predictive model based on the determined similarity score; A computerized system comprising:

15. 14. A computer program for determining the reliability of a prediction output of at least one first machine learned or deep learned predictive model, the computer program being adapted, when executed on a processor, to cause the processor to perform the method according to any one of claims 1 to 13.

Images