Computer-implemented method for providing a trained function, computer-implemented method for evaluating input data, training system, evaluation device, computer program and electrically readable data carrier

Abstract

The invention relates to a computer-implemented method for providing a trained function that determines output data from input data, comprising the following steps: - Compiling a multiple training datasets by, for each training dataset, - Providing training input data and multiple associated individual output data sets from different evaluators of the training input data, - Determination of training input data by statistical combination, in particular averaging, of the individual input data and a consensus value describing the degree of agreement among the raters, - Assembling the training dataset from the training input data, the training output data, and the match value, - Training the function using the training data with a loss function determined specifically for each training data set, wherein the loss function is weighted for each training data set with a weight value dependent on the agreement value of the training data set, in particular in the interval from zero to one.

Description

[0001] The invention relates to a computer-implemented method for providing a trained function that determines output data from input data. The invention also relates to a computer-implemented method for evaluating input data, a training system, an evaluation device, a computer program, and an electrically readable data carrier.

[0002] In the field of machine learning, functions implement machine learning models. These functions determine output data from input data and can be trained through machine learning (training). The trained function can then be used in applications.

[0003] Trained functions, especially functions trained through deep learning, can be used in particular to serve as classifiers, i.e., to assign class labels to input data for evaluation. Such trained functions can also be referred to as trained classifying functions.

[0004] Classifying functions, in particular, are trained using supervised learning. To obtain suitable training data sets, each observation—i.e., each set of training input data—must be assigned a set of training output data as the "ground truth." For example, in the case of a classifying function, this would be an assignment to available classes. This assignment can be performed using so-called "hard labels," which involve binary decisions, meaning that probabilities of belonging to classes can be as low as zero (0%) and one (100%). If several conceivable alternative classes exist, a probability of one can be assigned to only one class.An assignment using so-called “soft labels” is also possible, where probabilities between zero and one can be used, whereby for alternatives the sum of these probabilities should of course be one.

[0005] It often happens that human raters assign training input data, i.e. observations, to the corresponding training output data, in particular a classification with “hard labels” or “soft labels”. However, the problem here is that a lack of agreement can arise when different human raters evaluate the same set of training input data. The reason for this can lie in the difficulty of the task, different training and experience backgrounds and / or an intrinsic subjectivity of the task. Difficult tasks arise in particular in medical technology, for example when evaluating medical image data of a patient. This can involve assessing, for example, whether or not signs of illness are present in an X-ray image. Subjectivity can arise, for example, when assessing whether or not two images are similar.An example of a concrete application in which differing assessments can frequently occur between different assessors, i.e. in which there is a high degree of intervariability, is the classification of breast density in mammography or the assessment of an abnormality as a pathology.

[0006] The standard procedure in such a case, when individual output data from different raters differ for a set of training input data, is a human consensus decision. Converting "hard labels" of the individual output data into "soft labels" of the training output data has also been proposed in the prior art; see the article by Nguyen et al., "Learning classification models with soft label information," J Am Med Inform Assoc., 21 (2014), pages 501 to 508.

[0007] In a paper by Peterson et al., "Human uncertainty makes classification more robust," IEEE / CVF International Conference on Computer Vision (ICCV 2018), pages 9616 to 9625, they propose using "soft labels" that contain information about human confusion as a replacement for so-called "hot-label encodings" when training classifying features using deep learning. Here, too, they demonstrated that "soft labels" offer certain advantages over "hard labels."

[0008] While "soft labels" already offer a certain advantage and can represent uncertainties regarding evaluation, robust training in applications with high intervariability usually requires a large number of training datasets, which makes the training process complex and, in some cases, difficult to implement. Another approach to eliminating uncertainties and making training data more robust would be the objective determination of ground truths, in the example of medical technology, for example, through biopsies or other additional examinations. However, this has proven impractical and, in some cases where an objective ground truth cannot be determined by measurement, is simply not possible.

[0009] The invention is therefore based on the object of providing an improved possibility for training, in particular, classifying functions with high intervariability of evaluators.

[0010] To achieve this object, the invention provides a computer-implemented provision method according to claim 1, a computer-implemented evaluation method according to claim 10, a training system according to claim 15, an evaluation device according to claim 16, a computer program according to claim 17, and an electronically readable data carrier according to claim 18. Advantageous further developments emerge from the subclaims.

[0011] A computer-implemented method for providing a trained function that determines output data from input data, i.e. a provision method (training method), comprises the following steps according to the invention: - Compiling a plurality of training data sets by, for each training data set, - Providing training input data and several associated individual output data from different evaluators of the training input data, - Determination of training output data by statistical combination, in particular averaging, of the individual output data and determination of an agreement value describing the degree of agreement between the evaluators, and - Compiling the training dataset from the training input data, the training output data and the agreement value; and - Training the function using the training data using a loss function determined specifically for the training data set, wherein the loss function is weighted for each training data set with a weighting value dependent on the match value of the training data set, in particular in the interval from zero to one.

[0012] In general, a trained function replicates cognitive functions that people associate with other human brains. Through training based on training data (machine learning), the trained function is able to adapt to new circumstances and detect and extrapolate patterns.

[0013] Generally speaking, parameters of a trained function can be adjusted through training. In particular, supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or active learning can be used, with the present invention focusing on supervised learning. The parameters of the trained function can, in particular, be adjusted iteratively through multiple training steps.

[0014] For example, a trained function may include a neural network, a support vector machine (SVM), a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, and / or mapping rules. In particular, a neural network may be a deep neural network, a convolutional neural network (CNN), or a deep CNN. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network (GAN).

[0015] Within the scope of the present invention, the function can be, in particular, a deep neural network, preferably a convolutional neural network. Functions in which cross-entropy is used to determine the loss during training are particularly useful. Thus, it can be provided that a cross-entropy function is used as the loss function.

[0016] In particular, the invention therefore primarily relates to classifying functions that output probabilities for at least one class as output data. In other words, the output data comprises at least one item of classification information, in particular a probability for at least one class. It should be noted at this point that a binary classification, i.e., a pure "yes-no statement" regarding the membership of the input data or a feature of the input data to a class, or a "hard label," can also be understood as a probability specification (limited to values ​​of zero (0%) and one (100%)). Preferably, however, the probabilities are specified continuously, at least during training, i.e., as "soft labels."

[0017] Classification functions that use a cross-entropy function as a loss function can include a DenseNet and / or an AlexNet. Of course, other architectures, particularly classifying ones, are also conceivable, implemented by the function to be trained.

[0018] Generally speaking, it is proposed to adapt the loss function for training a function, in particular a classifying function, in such a way that it also takes into account the agreement between the raters, described by the agreement value. First, the ground truth, i.e., the training input data, is generated by combining the ratings of different raters, thus using a "soft label" as the ground truth. Furthermore, a agreement value is determined as part of the training data set, which describes the agreement between the different raters. A novel design of the loss function then makes it possible to combine the "soft label" and the agreement between the raters ("inter-rater agreement", IRA). The weighting value can preferably lie in an interval [0, 1], although other intervals, in particular other lower and upper limits for the interval, are also possible.

[0019] The inventive pretreatment of the training data and the inventive modification of the loss function ensure that the obvious cases (agreement among all raters) are correctly evaluated, while non-obvious cases may exhibit lower accuracy compared to the obvious cases. This behavior increases a user's confidence in the trained function.

[0020] Furthermore, the weighting, which can thus give more weight to obvious cases with a high degree of agreement than to difficult, non-obvious cases with a lower degree of agreement, makes it possible to achieve a robust, reliable, and trustworthy execution of the trained function even with a smaller amount of training data. Thus, the present invention allows a reduction in the amount of training data by adapting the training method. If the ground truth can also be objectively determined, the present invention can also save effort related to its metrological determination.

[0021] First, training data sets are compiled in advance, with the input training data having been evaluated by a specific number of, in particular, human, evaluators. Each evaluator can independently assign individual output data, whereby for classifying functions, probabilities (in the range from zero to one) can be assigned independently for each class. Both the use of "hard labels", i.e. only probabilities of zero or one, and the use of continuous probabilities ("soft labels") are possible. This also covers the case in which each evaluator selects only one of several alternative classes for each set of training input data (probability of one), and all other classes are thus assigned a probability of zero.

[0022] For each set of input training data, the group score (i.e., the training output data) and the agreement value are then determined based on the individual output data. The individual output data are determined continuously, i.e., specifically as "soft labels," independent of any "hard labels" used in the individual output data. For the classification function, continuous probabilities in the range from zero to one are thus determined for each class as training output data. The agreement value is preferably also a value in the range from zero to one.

[0023] The function is then trained based on the training data sets with the input training data and the corresponding output training data (as a group score). The loss function determined for each training data set, specifically the cross-entropy function, is weighted with a weighting value determined from the agreement score.

[0024] The present invention can be applied whenever high intervariability occurs among the evaluators, for example, when 100% agreement exists only for a small proportion, such as 25% or less, of training input data sets. The provisioning method is particularly advantageous for medical technology, where extremely difficult questions often arise when evaluating medical information, especially medical image data. Different evaluators, especially different examiners, can produce different results.

[0025] Particularly advantageously, it can be provided that the input data comprise medical image data, in particular mammography data, of a patient and / or the trained function is a diagnostic support function that provides usable output data for making a diagnosis. For example, in mammography and / or tomosynthesis applications, image data can be difficult to interpret, so that differing assessments by different assessors can occur more frequently, thus resulting in a high degree of intervariability. In particular, in the case of mammography data, the output data can comprise a classification into breast density classes, in particular the common breast density classes a, b, c, and d.

[0026] However, the procedure can also be applied particularly advantageously to the evaluation of medical image data in other areas of medical technology, particularly CAD (Computer-Aided Diagnosis). Generally speaking, the image data can include, for example, X-ray data, particularly computed tomography data and / or tomosynthesis data, and / or magnetic resonance data and / or ultrasound data. The trained, classifying function can, for example, determine initial data that describes the probabilities for the presence of at least one pathology, such as a tumor, and / or a classification on risk scales and / or a score.

[0027] However, applications outside of medical technology are also conceivable where high intervariability among raters occurs, and the proposed training variant can ensure increased robustness without massively increasing the amount of training data. For example, the emotion recognition function can be used in a scenario where human raters, as the basis for machine learning, assess the emotional state of a speaking and / or imaged person, specifically classifying / annotating it.

[0028] In a particularly advantageous development of the present invention, the training output data can be determined by weighted averaging of the individual output data using an evaluator weighting for each evaluator. In other words, instead of a simple mean value being calculated, each set of individual output data is weighted according to additional information about the corresponding evaluator, which may correspond to prior knowledge or may be at least partially determined automatically. In this way, information about the evaluators can also be incorporated into the process to improve the robustness, reliability, and trustworthiness of the resulting trained function. Various approaches are therefore conceivable for determining the evaluator weightings.

[0029] For example, the weightings can be determined based on prior knowledge of the assessors. For example, if one assessor is more experienced than another, the weighting can be set higher for that assessor than for the other. A similar approach can be used for educational level.

[0030] However, in advantageous embodiments, it is also possible for the rater weights to be determined based on an automatically determined measure of agreement between the rater and other raters for each set of training input data. If the output data includes multiple subcomponents, for example, multiple probabilities for multiple classes, the measure of agreement is also determined across all subcomponents. For example, one concrete way to determine the rater weights could be to calculate the agreement between all pairs of raters and to sum all values ​​involving a rater in order to determine the rater weight for that rater. The pairwise measure of agreement can be any generally common statistical measure in this regard.A simple measure of agreement, for example, is given by an overlap of ratings with those of another rater (number of agreements divided by the total number of ratings). However, it is particularly preferred that an intraclass correlation coefficient (ICC) or a kappa coefficient be determined as the measure of agreement. Such measures of agreement integrate knowledge of the underlying statistics into the calculation process and therefore provide a reliable basis for comparison with other raters.

[0031] An expedient development of the invention can provide that an evaluator function is trained for at least one selected human evaluator, limited to its individual output data, wherein the trained evaluator function - is used to evaluate input training data not evaluated by the selected human evaluator to determine missing output training data, or - is applied as an additional evaluator to all input training data to generate additional output training data.

[0032] In this way, a type of semi-supervised learning environment is created, by means of which additional ratings can be generated or missing ratings can be supplemented. For this purpose, an evaluator function is trained individually for at least one, possibly several, or all, evaluators and is subsequently applied to at least one training input data set, in particular several training input data sets. Preferably, the at least one evaluator function is used to generate ratings, i.e., individual output data, for training input data sets that have not been rated or at least not been rated by this evaluator. If such evaluator functions are created for all evaluators, it is possible to incorporate a large amount of additional sets of training input data and to generate individual output data for these.It is also conceivable, but less preferred, to consider the rater function as a new rater, since it does not necessarily represent the rater for which it was individually generated in a completely identical manner. In both cases, the data set for training the function can be easily expanded.

[0033] To specifically determine the agreement value, it can be provided that the agreement value is determined as the proportion of raters who rated the training input data according to the majority opinion. When using "hard labels", i.e. values ​​for the probability of zero or one, the agreement value is then simply calculated as the number of raters who agree with the majority opinion divided by the total number of raters. If probabilities are already continuously specified by the raters, i.e. general "soft labels", various possibilities are conceivable. On the one hand, it is possible to define membership in the majority opinion using a threshold for the corresponding probability and to count the majority opinion raters as a natural number and then divide them by the total number of raters.It is of course also possible to calculate probability sums, but this is less preferred. For continuous probability values ​​with balanced assessments and / or unclear majority opinions, for example, an additional rater can be consulted and / or prior knowledge can be used to select at least one rater to prioritize, for example, a particularly experienced rater.

[0034] In a particularly preferred development of the present invention, it can be provided that the weighting value is determined as a function of the agreement value in such a way that it lies in a range from a lower limit that is greater than zero up to an upper limit, in particular one, and / or increases more quickly towards the upper limit, in particular one, with increasing degree of agreement. The lower limit can be determined empirically and ensures that each set of training input data develops at least a certain relevance. In this way, all training data sets are also meaningfully taken into account. If the weighting value increases with increasing degree of agreement, in particular with increasing gradient, special emphasis is placed during training on cases in which the human evaluators show high agreement in their assessments.In other words, the loss function imposes particularly high penalty costs for cases incorrectly assessed by the function, where the raters are more likely to agree, and thus aims for particularly high accuracy in cases where there is a high degree of agreement among the raters (obvious cases). For example, a specific training may stipulate that the weighting value be set as a minimum. - the maximum of the lower limit, in particular a value in the range from 0.2 to 0.3, and the nth, in particular second, power of the agreement value in the range from zero to one, and - One is determined. A second-order (or higher-order) polynomial is preferably chosen here. This emphasizes the desired accuracy more strongly in obvious cases. However, it is also conceivable to choose, for example, a first-order polynomial (linear dependence). In an embodiment where 0.25 is chosen as the lower limit, a match value of one (100%) leads to the highest possible weighting value of one, while for a second-order polynomial, a match value of 0.5 (50%) or less leads to the lowest possible weight of 0.25. In between, the square of the match value is used.

[0035] It should be noted that, within the scope of the present invention, further components can also be included in the loss function, for example a weighting function based on the class. For example, if the output data describe a classification for several classes, the loss function can be additionally weighted with a class-related class weight, so that classes represented fewer times in the training data sets (i.e., classes determined as most likely according to the training output data) are given a higher weight than classes represented more often in the training data sets. In particular, the class weights can at least substantially reflect the inverse class frequencies and / or be determined as a function of these. In this way, classes underrepresented in the training data can receive a higher weight than overrepresented classes.

[0036] In addition to the provision method, the present invention also relates to a computer-implemented method for evaluating input data to determine output data (evaluation method), comprising the following steps: - Providing a trained function according to a provision method according to the invention, and - Applying the trained function to the input data to obtain the output data.

[0037] In other words, the evaluation method relates to the application of a trained function provided with the provision method, and all statements regarding the provision method can be transferred analogously to the evaluation method according to the invention and vice versa.

[0038] In classifying functions trained using the provisioning method according to the invention, the probabilities contained in the output data for the respective classes are fundamentally independent when there are several alternative classes, except that their sum always equals one (100%). However, output data satisfying this sum condition do not necessarily represent meaningful results. In other words, some combinations of probabilities of the output data may make no sense at all from an application perspective. Such implausibilities can always be detected when a similarity between classes can be determined or even quantified. In this case, it can be said that distributed probabilities in the output data of the trained classifying function should only occur for classes that are closest in terms of similarity.In the example of breast density classification, for example, a set of initial data in which breast density class a is assigned a probability of 50%, breast density class d a probability of 50%, and the remaining breast density classes are assigned probabilities of zero makes no sense, since the classes with the highest probabilities represent significantly different breast density categories. In contrast, 50% for class a and 50% for class b, with the remainder being zero, would be perfectly fine, since the actual value would then presumably lie between these two breast density classes.

[0039] Implausible initial data are not necessarily avoided by the training mechanisms in this case (and in other cases), particularly due to the independent probabilities (except for the sum condition). Therefore, a suitable refinement of the evaluation procedure, i.e., within the framework of interference, can provide that if the initial data includes probabilities for several classes and similarity information can be determined for the classes, the initial data, if they indicate a probability of one for more than one class, are evaluated depending on the probabilities and the similarity information of at least two classes with the highest probabilities to determine a reliability value.A plausibility assessment is therefore performed for the initial data based on the knowledge of the similarity of the classes in order to quantify the certainty regarding the output of the model implemented by the trained function. Reliability increases with greater similarity between the at least two classes with the highest probabilities. The reliability value preferably lies in the range from zero to one and can be understood as a "reliability probability." Additional information, namely the reliability value, is therefore provided, which makes it possible to identify deficiencies or less reliable results that could arise, particularly in the provisioning method according to the invention, using measures based on this value, and to act accordingly.

[0040] It should be noted at this point that such a determination of a reliability value is also advantageous independently of the provision method according to the invention, i.e., the consideration of the evaluation behavior of multiple evaluators when training a function. Thus, a computer-implemented method for evaluating input data by a trained, classifying function to determine output data is conceivable, wherein the output data comprise probabilities for multiple classes and similarity information can be defined for the classes. This method is characterized in that the output data, if they indicate a probability of one for more than one class, are evaluated depending on the probabilities and the similarity information of the at least two classes with the highest probabilities to determine a reliability value.All of the following statements can of course also be applied to this conceivable procedure.

[0041] Similarity information can be determined particularly easily for classes assigned an ordinal order. In this case, the ordinal distance between the classes can be used as similarity information. An example of such classes are the aforementioned breast density classes a, b, c, and d. Here, for example, breast density classes b and c are spaced by an ordinal distance of one, while breast density classes a and c are spaced by an ordinal distance of two. For example, again generally speaking, plausibility and thus a reliability value other than zero can only be given if the two classes with the highest probabilities are immediately adjacent.

[0042] In a concrete, appropriate design in this regard, it may be provided that in a first case distinction: - if the two classes with the highest probabilities are not immediately adjacent, the reliability is set to zero; - if the two classes with the highest probabilities are immediately adjacent, - the sum of the two highest probabilities is determined as the calculated value, and in a second case distinction: - if the calculated value lies in at least a first predetermined interval, in particular from zero to 0.5, the reliability value is set to zero, and / or if the calculated value lies in at least a second predetermined interval, the reliability value is set to one, and - if the calculated value lies outside all of the at least one interval, the reliability value is determined between zero and one depending on the calculated value, in particular as twice the calculated value minus one.

[0043] In other words, the two classes with the highest probabilities are first determined. Then, it is checked whether they are directly adjacent, thus corresponding to adjacent categories. If this is not the case, the reliability value is set to zero. If, however, they are directly adjacent, the sum of the two highest probabilities is determined. If, in a specific example, this lies in the first predetermined interval [0.00, 0.50], this still indicates a very high degree of uncertainty, as the probabilities are very widely distributed, which is why the reliability value can still be set to zero. Continuing this specific example, it can be checked whether the sum lies in the second predetermined interval [0.99, 1.00], which indicates a very high degree of certainty, so that the reliability value can be set to one. Between these two intervals, i.e., in the range [0.50, 0].99] the reliability value can, for example, be reasonably calculated as twice the sum minus one.

[0044] In a first example, the initial data for breast density classes can be 0%, 30%, 70%, and 0%, which, since the non-zero probabilities are distributed between two immediately adjacent breast density classes, results in a reliability value of 1. In another example, the initial data can be 15%, 30%, 50%, and 5%, whereby the reliability value, since two immediately adjacent breast density classes still have the highest probabilities, is then 2 * (30% + 50%) - 100% or 2 * (0.3 + 0.5) - 1, resulting in 60% and 0.6, respectively.

[0045] As already mentioned, the reliability value can be usefully employed, for example, to output a message to a user and / or for visualization. Specifically, if the reliability value is less than or equal to a threshold value, a non-determinability of the output data for the input data is output to a user and / or, in the case of a scale of classes in an ordinal order, the reliability value is output, in particular by means of an extension of an arrow in the longitudinal direction of the scale, determined as a function of the reliability value, visualizing the output data, in particular the class with the highest probability and / or an averaged result. The threshold value for determining non-determinability can expediently be set to zero.It is also conceivable to display the reliability value in a user interface to inform the user about the self-assessment of the trained classifying function and to raise awareness that the result may be incorrect. In particular, when using a scale, such as for breast density classes, it is useful to indicate the reliability value by changing the extent of an arrow along the scale. The arrow can indicate the class with the highest probability or a result that corresponds to the average of the classes, weighted by the respective probabilities. The lower the certainty and thus the reliability value, the longer the arrow.

[0046] A training system according to the invention comprises a training computing device with at least one processor and at least one storage means, which comprises: - a first training interface for receiving training input data and several associated individual output data from different evaluators of the training input data, - a determination unit for determining, for each set of training input data, training output data by statistical combination, in particular averaging, of the individual output data and for determining an agreement value describing the degree of agreement between the evaluators, - a compilation unit for compiling training data sets from the training input data, the training output data and the agreement value, - a training unit for training the function using the training data using a loss function determined specifically for the training data set, wherein the loss function is weighted for each training data set with a weighting value dependent on the match value of the training data set, in particular in the interval from zero to one, and - a second training interface to provide the trained function.

[0047] The statements regarding the methods according to the invention can also be applied analogously to the training system. In particular, the aforementioned or further functional units can be designed or provided for implementing further advantageous embodiments of the methods.

[0048] An evaluation system according to the invention comprises an evaluation computing device with at least one processor and at least one storage means, wherein the evaluation computing device comprises - a first evaluation interface for receiving a set of input data, - an evaluation unit for evaluating the input data to determine output data by a trained function provided according to a method according to one of claims 1 to 9, and - a second evaluation interface for outputting the output data.

[0049] Here, too, the statements regarding the other aspects of the invention continue to apply accordingly. In particular, the evaluation computing device can also have a third evaluation interface for receiving the trained function from the second training interface.

[0050] A computer program according to the invention can be loaded into a storage means of a computing device, in particular a training computing device or an evaluation computing device, and has program means such that when the computer program is executed on the computing device, the latter is caused to carry out the steps of a method according to the invention, in particular a provision method in the training computing device or an evaluation method in the evaluation computing device.The computer program can be stored on an electronically readable data carrier according to the invention, which thus comprises control information stored thereon, which comprises at least one computer program according to the invention and is designed such that, when the data carrier is used in a computing device, in particular a training computing device or an evaluation computing device, the latter is designed to execute a method according to the invention, in particular a provision method in the training computing device or an evaluation method in the evaluation computing device. The data carrier can in particular be a non-transient data carrier, for example a CD-ROM.

[0051] Further advantages and details of the present invention will become apparent from the exemplary embodiments described below and from the drawings. In the drawings: Fig.1 an embodiment of a neural network, Fig. 2 an embodiment of a Convolutional Neural Network (CNN), Fig. 3 a flowchart of an embodiment of the provision method according to the invention, Fig. 4 a flow chart of an embodiment of the evaluation method according to the invention, Fig. 5 a flow chart for a more detailed explanation of a step of the embodiment of the Fig. 4, Fig. 6 a view of a user interface for outputting output data, Fig. 7 the functional structure of a provision system according to the invention, and Fig. 8 the functional structure of an evaluation system according to the invention.

[0052] Fig.1 shows an embodiment of an artificial neural network 1. English terms for the artificial neural network 1 are “artificial neural network”, “neural network”, “artificial neural net” or “neural net”.

[0053] The artificial neural network 1 comprises nodes 6 to 18 and edges 19 to 21, where each edge 19 to 21 is a directed connection from a first node 6 to 18 to a second node 6 to 18. In general, the first node 6 to 18 and the second node 6 to 18 are different nodes 6 to 18, but it is also conceivable that the first node 6 to 18 and the second node 6 to 18 are identical. For example, in Fig.1 the edge 19 is a directed connection from the node 6 to the node 9 and the edge 21 is a directed connection from the node 16 to the node 18. An edge 19 to 21 from a first node 6 to 18 to a second node 6 to 18 is called an ingoing edge for the second node 6 to 18 and an outgoing edge for the first node 6 to 18.

[0054] In this embodiment, nodes 6 to 18 of artificial neural network 1 can be arranged in layers 2 to 5, wherein the layers can have an intrinsic order introduced by edges 19 to 21 between nodes 6 to 18. In particular, edges 19 to 21 can only be provided between adjacent layers of nodes 6 to 18. In the illustrated embodiment, there is an input layer 2 that only has nodes 6, 7, and 8, each without an incoming edge. Output layer 5 comprises only nodes 17, 18, each without an outgoing edge, with hidden layers 3 and 4 furthermore lying between input layer 2 and output layer 5. In the general case, the number of hidden layers 3, 4 can be chosen arbitrarily.The number of nodes 6, 7, 8 of the input layer 2 usually corresponds to the number of input values ​​to the neural network 1, and the number of nodes 17, 18 in the output layer 5 usually corresponds to the number of output values ​​of the neural network 1.

[0055] In particular, a (real) number can be assigned to nodes 6 to 18 of the neural network 1. Here, x denotes (n) i the value of the i-th node 6 to 18 of the n-th layer 2 to 5. The values ​​of nodes 6, 7, 8 of input layer 2 are equivalent to the input values ​​of neural network 1, while the values ​​of nodes 17, 18 of output layer 5 are equivalent to the output values ​​of neural network 1. In addition, each edge 19, 20, 21 can be assigned a weight in the form of a real number. In particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1,]. Here, w denotes (m,n) i,jthe weight of the edge between the i-th nodes 6 to 18 of the m-th layer 2 to 5 and the j-th nodes 6 to 18 of the n-th layer 2 to 5. Furthermore, the abbreviation wi,j(n) for the weight wi,j(n,n+1) defined.

[0056] To calculate output values ​​of neural network 1, the input values ​​are propagated through neural network 1. In particular, the values ​​of nodes 6 to 18 of the (n+1)-th layer 2 to 5 can be calculated based on the values ​​of nodes 6 to 18 of the n-th layer 2 to 5 by xj(n+1)=f(∑ixi(n)⋅wi,j(n)).

[0057] Here, f is a transfer function, which can also be referred to as an activation function. Common transfer functions include step functions, sigmoid functions (e.g., the logistic function, the generalized logistic function, the hyperbolic tangent, the arctangent, the error function, the smoothstep function), or rectifier functions. The transfer function is primarily used for normalization purposes.

[0058] Specifically, the values ​​are propagated layer by layer through neural network 1, with values ​​of input layer 2 being given by the input data of neural network 1. Values ​​of the first hidden layer 3 can be calculated based on the values ​​of input layer 2 of neural network 1, values ​​of the second hidden layer 4 can be calculated based on the values ​​in the first hidden layer 3, and so on.

[0059] To the values wi,j(n) for edges 19 to 21, neural network 1 must be trained using training data. In particular, training data includes training input data and training output data, which are referred to below as t i For a training step, neural network 1 is applied to the training input data to determine calculated output data. Specifically, the training output data and the calculated output data comprise a number of values, where the number is determined as the number of nodes 17, 18 of output layer 5.

[0060] In particular, a comparison between the calculated output data and the training output data is used to recursively adjust the weights within the neural network 1 (back propagation algorithm). In particular, the weights can be adjusted according to w'i,j(n)=wi,j(n)−γ⋅δj(n)⋅xi(n) where γ is a learning rate and the numbers δj(n) can be calculated recursively as δj(n)=(∑kδk(n+1)⋅wj,k(n+1))⋅f'(∑ixi(n)⋅wi,j(n)) based on δj(n+1), if the (n+1)-th layer is not the output layer 5, and δj(n)=(xk(n+1)−tj(n+1))⋅f'(∑ixi(n)⋅wi,j(n)) if the (n+1)-th layer is the output layer 5, where f' is the first derivative of the activation function and yj(n+1) is the comparison training value for the j-th node 17, 18 of the output layer 5.

[0061] In the following, with regard to Fig.2 also provides an example of a convolutional neural network (CNN). Note that the term "layer" is used in a slightly different way than for classical neural networks. For a classical neural network, the term "layer" refers only to the set of nodes that form a layer, i.e., a specific generation of nodes. For a convolutional neural network, the term "layer" is often used to refer to an object that actively modifies data—in other words, a set of nodes of the same generation and either the set of incoming or outgoing edges.

[0062] Fig.2 shows an embodiment of a convolutional neural network 22. In the illustrated embodiment, the convolutional neural network 22 comprises an input layer 23, a convolutional layer 24, a pooling layer 25, a fully connected layer 26, and an output layer 27. In alternative embodiments, the convolutional neural network 22 may contain multiple convolutional layers 24, multiple pooling layers 25, and multiple fully connected layers 26, as well as other types of layers. The order of the layers can be chosen arbitrarily, with fully connected layers 26 typically forming the last layers before the output layer 27.

[0063] In particular, within a convolutional neural network 22, the nodes 28 to 32 of one of the layers 23 to 27 can be understood as being arranged in a d-dimensional matrix or as a d-dimensional image. In particular, in the two-dimensional case, the value of a node 28 to 32 with the indices i, j in the n-th layer 23 to 27 can be represented as x (n) [i,j]. It should be noted that the arrangement of nodes 28 to 31 of a layer 23 to 27 has no effect on the calculations within the convolutional neural network 22 as such, since these effects are determined exclusively by the structure and weights of the edges.

[0064] A convolutional layer 24 is particularly characterized in that the structure and weights of the incoming edges form a convolution operation based on a specific number of kernels. In particular, the structure and weights of the incoming edges can be chosen such that the values xk(n) the node 29 of the convolutional layer 24 as a convolution xk(n)=Kk*x(n−1) based on the values ​​x (n-1) the node 28 of the previous layer 23 can be determined, where the convolution * in the two-dimensional case can be defined as xk(n)[i,j]=(Kk*x(n−1))[i,j]=∑i,∑j' Kk[i',j']⋅x(n−1)[i−i',j−j'].

[0065] In it, the k-th kernel K ka d-dimensional matrix, in this embodiment a two-dimensional matrix, which is usually small compared to the number of nodes 28 to 32, for example a 3x3 matrix or a 5x5 matrix. In particular, this implies that the weights of the incoming edges are not independent, but are chosen to generate the above convolution equation. In the example for a kernel forming a 3x3 matrix, only nine independent weights exist (where each entry of the kernel matrix corresponds to an independent weight), regardless of the number of nodes 28 to 32 in the corresponding layer 23 to 27. In particular, for a convolutional layer 24, the number of nodes 29 in the convolutional layer 24 is equivalent to the number of nodes 28 in the preceding layer 23 multiplied by the number of convolution kernels.

[0066] If the nodes 28 of the preceding layer 23 are arranged as a d-dimensional matrix, the use of the plurality of kernels can be understood as the addition of a further dimension, also referred to as the depth dimension, so that the nodes 29 of the convolutional layer 24 are arranged as a (d+1)-dimensional matrix. If the nodes 28 of the preceding layer 23 are already arranged as a (d+1)-dimensional matrix with a depth dimension, the use of a plurality of convolutional kernels can be understood as an expansion along the depth dimension, so that the nodes 29 of the convolutional layer 24 are likewise arranged as a (d+1)-dimensional matrix, wherein the size of the (d+1)-dimensional matrix in the depth dimension is larger than in the preceding layer 23 by the factor formed by the number of kernels.

[0067] The advantage of using convolutional layers 24 is that the spatially local correlation of the input data can be exploited by creating a local connection pattern between nodes of adjacent layers, in particular by each node having connections only to a small range of the nodes of the previous layer.

[0068] In the illustrated embodiment, the input layer 23 comprises thirty-six nodes 28 arranged as a two-dimensional 6x6 matrix. The convolutional layer 24 comprises seventy-two nodes 29 arranged as two two-dimensional 6x6 matrices, each of which is the result of convolving the values ​​of the input layer 23 with a convolution kernel. Similarly, the nodes 29 of the convolutional layer 24 can be understood as being arranged in a three-dimensional 6x6x2 matrix, with the latter dimension being the depth dimension.

[0069] A pooling layer 25 is characterized in that the structure and weights of the incoming edges as well as the activation function of its nodes 30 define a pooling operation based on a nonlinear pooling function f. For example, in the two-dimensional case, the values ​​x (n) the node 30 of the pooling layer 25 based on the values ​​x (n+1) the node 29 of the previous layer 24 as x(n)[i,j]=f(x(n−1)[id1,jd2]),...,x(n−1)[id1+d1−1, jd2+d2−1]) In other words, by using a pooling layer 25, the number of nodes 29, 30 can be reduced by replacing a number of d1 × d2 neighboring nodes 29 in the preceding layer 24 with a single node 30, which is calculated as a function of the values ​​of said number of neighboring nodes 29. In particular, the pooling function f can be a maximum function, an averaging function, or the L2 norm. In particular, for a pooling layer 25, the weights of the incoming edges can be fixed and not modified by training.

[0070] The advantage of using a pooling layer 25 is that the number of nodes 29, 30 and the number of parameters are reduced. This leads to a reduction in the amount of computation required within the convolutional neural network 22 and thus to a control of overfitting.

[0071] In the illustrated embodiment, pooling layer 25 is a max-pooling layer in which four neighboring nodes are replaced with a single node whose value is the maximum of the values ​​of the four neighboring nodes. Max-pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max-pooling is applied to each of the two two-dimensional matrices, reducing the number of nodes from seventy-two to eighteen.

[0072] A fully connected layer 26 is characterized in that a majority, in particular all, of the edges are present between the nodes 30 of the previous layer 25 and the nodes 31 of the fully connected layer 26, wherein the weight of each of the edges can be individually adjusted. In this embodiment, the nodes 30 of the previous layer 25 and the fully connected layer 26 are shown both as two-dimensional matrices and as non-connected nodes (shown as a row of nodes, with the number of nodes reduced for clarity). In this embodiment, the number of nodes 31 in the fully connected layer 26 is equal to the number of nodes 30 in the previous layer 25. In alternative embodiments, the number of nodes 30, 31 may be different.

[0073] Furthermore, in this embodiment, the values ​​of the nodes 32 of the output layer 27 are determined by applying the softmax function to the values ​​of the nodes 31 of the preceding layer 26. By applying the softmax function, the sum of the values ​​of all nodes 32 of the output layer 27 is one, and all values ​​of all nodes 32 of the output layer are real numbers between 0 and 1. When the convolutional neural network 22 is used to classify input data, the values ​​of the output layer 27, in particular, can be interpreted as the probability that the input data falls into one of the different classes.

[0074] A convolutional neural network 22 can also have a ReLU layer, where ReLU is an acronym for "rectified linear units." In particular, the number of nodes and the structure of the nodes within a ReLU layer are equivalent to the number of nodes and the structure of the nodes in the previous layer. The value of each node in the ReLU layer can be calculated, in particular, by applying a rectifier function to the value of the corresponding node in the previous layer. Examples of rectifier functions are f(x)=max(0,x), the hyperbolic tangent, or the sigmoid function.

[0075] Convolutional neural networks 22 can be trained, in particular, based on the backpropagation algorithm. To avoid overfitting, regularization methods can be used, such as dropout of individual nodes 28 to 32, stochastic pooling, use of artificial data, weight decay based on the L1 or L2 norm, or maximum norm constraints.

[0076] A special case of deep neural networks, especially CNNs 22, are classification networks, which can be implemented using classification functions. This is the case when, as described above, probabilities for membership in (especially mutually alternative) classes are determined in the output layer 27. If these probabilities can only assume values ​​of zero and one, they are also referred to as "hard labels"; if the probabilities are continuous, they are referred to as "soft labels."

[0077] When training classification functions, loss functions defined by cross-correlation are often used. When backpropagation is performed, the accuracy of the predictions must be determined. Loss functions are typically used for this purpose. The loss function essentially describes the difference between the initial data obtained at the current training stage and the training output data (ground truth).

[0078] The following embodiments will deal with classification functions, which in particular implement a CNN 22. Training is based on cross-correlation functions that can be determined for each training dataset. The input data comprises a medical image dataset.

[0079] In principle, however, the present invention can also be applied to applications outside of medical technology, for example, emotion recognition and the like. However, medical technology in particular involves many difficult assessment tasks, particularly diagnostic assessment tasks, where different human assessors can arrive at different results, thus resulting in high intervariability. Therefore, application in this area is particularly advantageous. The trained, classifying function provides diagnostic support in the sense of additional information through the initial data; the diagnosis itself is performed downstream and outside the exemplary embodiments described here by a human assessor. The trained, classifying function is therefore a diagnostic support function.

[0080] The following table lists the names of quantities and parameters used in the formulas and explanations given below. symbol definition c Class index i Index for training input data sets r Index for (human) evaluators I i Training input dataset, in particular medical image dataset, with index i N classes Number of classes N images Number of training input data sets Niraters Number of raters who used the training input dataset I i evaluate α Inter-rater agreement (IRA) in the interval [0, 1] α i Agreement value for the training input dataset I i pi,c,rindiv Probability given by the evaluator r to the training input data set I i was assigned that it belongs to class c (as part of the individual output data) pi,cgroup Probability given by a group of evaluators to the training input data set I i was assigned to class c (as part of the training output data) pi,cmdl Probability that the trained classifying function assigns to the training input data set I i was assigned that it belongs to class c (as part of the output data) wheel Weight value derived from the agreement value for a training input dataset I i wrindiv Rater weighting for an individual rater r βimdl Reliability value of the trained classifying function for an input dataset I i

[0081] To train the classifying function, sentences I i of training input data by a plurality Niraters evaluated by human evaluators, which means that each evaluator examines (in the medical context in particular assesses) the training input data, in particular the medical image data set contained therein, and assigns individual output data, i.e. complete sets of probabilities pi,c,rindiv The evaluations can include selecting one of the classes (i.e. “hard labels”, the selected class receives probability one, the remaining zero), but continuous values ​​for the probabilities are also possible. pi,c,rindiv conceivable.

[0082] It should be noted at this point that it is fundamentally conceivable to train an implementation of the model underlying the classifying function (in particular the CNN) as an evaluator function based on the individual ratings of a specific evaluator, i.e., the corresponding individual input data, in order to supplement missing ratings with this trained evaluator function. In particular, evaluator functions can be determined for all evaluators and additional training input data sets I i provided with the majority of evaluations, i.e., individual initial data. This results in a type of semi-supervised learning.

[0083] Fig.3 shows, as a first step S1 of an embodiment of a provision method according to the invention for training the classifying function, the receipt of the annotated training input data, i.e., provided with individual output data. For these, in a step S2, common training output data for the group of evaluators, i.e., sets of probabilities, are now generated. pi,cgroup and for each training input data set I i a match value α i determined.

[0084] The probabilities pi,cgroup (which are given as “soft labels” in continuous) are calculated from the probabilities wrindiv of the individual input data is determined through statistical analysis. One possible approach is to use a simple, unweighted mean: pi,cgroup=∑rNiraterspi,c,rindivNiraters

[0085] Preferably, and in the present embodiment, the training output data are, however, created by forming a wrindiv weighted mean pi,cgroup=∑rNiraters(wrindiv∗pi,c,rindiv)Niraters∗∑rNiraterswrindiv The evaluator weightings wrindiv can include background knowledge about the evaluator, for example, their experience and / or level of education. The more experienced or better educated the evaluator, the higher the evaluator weighting wrindiv.

[0086] However, a preferred automatic analysis is also conceivable by determining the agreement between all pairs of raters and summing up all values ​​relating to a particular rater in order to determine the rater weighting. wrindiv to determine r for this evaluator: wrindiv=∑s≠rk(pi,c,rindiv,pi,c,sindiv) k(.,.) is a measure of agreement between the ratings of two raters across all i and all c. It can be calculated, for example, using intraclass correlation (ICC) or as a kappa coefficient.

[0087] The agreement value for each sentence I i of training input data, α i , is defined as the number of evaluators who are of the majority opinion among the individual initial data divided by the total number of evaluators Niraters In the case of "soft labels", i.e. continuous probability information in the individual initial data, the number of evaluators who agree with the majority opinion is determined using a threshold value; in the case of unclear majorities or opinions, another evaluator can be consulted or an evaluator with a higher weighting can be selected. wrindiv be prioritized.

[0088] In step S2, the training data sets are then compiled from the training input data and the training output data as well as the agreement values.

[0089] In a step S3, the classifying function is then trained on the basis of the training input data I i and the training output data, i.e. the sets of probabilities pi,cgroup. The classifying function can, for example, implement a DenseNet. The loss function is adapted in such a way that it determines the agreement measure α i taken into account.

[0090] The (combined) loss function Lcmb,i with respect to an observation i in a training batch is defined as Lcmb,i=wiira∗LCE(pi,cgroup,pi,cmdl) with a cross entropy function LCE,i=−∑c(pi,cgroup∗log(pi,cmdl)) which uses “soft labels”.

[0091] The weighting value wiira is defined in the present embodiment as wiira=min(max(0.25,αi2),1.0) This means that a match value of one or 100% results in the highest possible weight wiira=1 while a match value of 0.5 or 50% corresponds to the lowest possible weight wiira=0.25 This weighting in the loss function causes the largest possible penalty for input training data I that is incorrectly classified by the classifying function at the current training stage. i , for which there is high inter-rater agreement among human raters (so-called obvious cases). This means that the loss function aims to provide high accuracy in classifying the obvious cases.

[0092] It should be noted that the loss function “hard labels” for pi,cgroup where one class would have a probability of one and all other classes would have probabilities of zero, but it was specifically designed to accept "soft labels" where each class can be assigned a probability in the range [0, 1].

[0093] Furthermore, it should be noted that additional weightings or components of the loss function are also conceivable, for example, a weighting function based on class c. In this case, underrepresented classes can receive a higher weight than overrepresented classes as the most likely outcome. Such a class weighting can, for example, describe the inverse class frequencies.

[0094] In a step S4, the classified function trained in this way is then provided.

[0095] Fig.Figure 4 shows a flowchart of an evaluation method according to the invention, in which the trained classifying function is used to evaluate a set of input data received in step S5. This evaluation, i.e., the application of the trained classifying function to the input data to obtain output data, i.e., the so-called inference, takes place in step S6.

[0096] Step S7 serves to perform a self-assessment regarding the reliability of the result within the context of using the trained classifying function. This is because, in principle, the probabilities for the various classes are independent during training, except that they should sum to one (due to the mutually exclusive nature of the classes). This makes implausible or uncertain results possible, particularly in extremely complex cases where the evaluators' assessments would also differ. Thus, step S7 takes advantage of the fact that, in the present embodiment, a similarity relationship exists between the classes; in the case under consideration, an ordinal order of these classes can even be defined.In a concrete, purely exemplary case, breast density classes a, b, c, d can be defined such that each of them covers an interval of breast densities, namely increasing values ​​from a to d. Ordinal distances, in this example a maximum of two between a and d, can then be determined as similarity information.

[0097] Fig. Figure 5 explains in more detail a possible, concrete determination of a reliability value in the interval [0, 1] for this concrete example of four classes with an ordinal order. In step S7a, after the initial data from step S6 are available, the two class indices c1, c2 are first determined, whose classes have the two highest probabilities. pi,cmdl are assigned.

[0098] In step S7b, a check is performed to determine whether these classes are immediately adjacent in the ordinal order, i.e., whether they belong to adjacent categories, such as breast density intervals. If this is not the case (|c1- c2| > 1), the reliability value is set to zero in step S7c. βimdl=0. Because such a result seems implausible.

[0099] However, if according to step S7b the classes are immediately adjacent (|c1- c2| = 1), the sum of the two highest probabilities is determined in step S7d, SHP=pi,c1mdl+pi,c2mdl

[0100] Based on this sum, a further case distinction is then made in step S7e. If the two highest probabilities are very low, so that their sum lies within a first interval from zero to a threshold value, in this case 50%, thus SHP ∈ [0.00, 0.50], then a broad distribution of probabilities exists that does not provide any real certainty. If four values ​​are present, the distribution is exactly uniform and does not contain any information. Therefore, the process branches to step S7c, and the reliability value is set to zero.

[0101] However, if the sum is extremely high, i.e. it lies in a second interval above a second limit value up to one, thus SHP ∈ ]0.99,1.00], then in a step S7f the reliability value is set to one, thus indicating maximum reliability, βimdl=1.

[0102] For all sums lying outside the first and second intervals, thus SHP ∈ [0.50, 0.99], the reliability value is calculated from the sum in a step S7g, in this case according to the formula βimdl=2∗SHP−1

[0103] If, in an example, the initial data is [40%, 60%, 0%, 0%], the result for the reliability value is one or 100% (since the sum of SHP = 100% and the classes are adjacent). For example, with initial data [6%, 10%, 35%, 49%], the reliability value is 2 * (84%) - 100% = 68%.

[0104] Returning to Fig.4, an output is then sent to the user in step S8. If the reliability value is very low, or at least if the reliability value is zero, the output data determined in step S6 is not output, but only a note indicating that a calculation (of meaningful output data) for this input data is not possible. In all other cases, the reliability value can be output with the output data. When visualized on a user interface, for example, a diagnostic device serving as an evaluation computer, the reliability value can also be visualized together with the output data.

[0105] A possible visualization 49 is in Fig.6 is shown schematically. A scale 50 is used, on which the initial data is visualized by arrows 51, 52, as well as the assigned reliability value, whereby in this case, arrow 51 is used for the right breast, and arrow 52 for the left breast of a patient. The initial data, i.e. the probabilities, are visualized (position indicated by the arrow) as a class mean weighted by the probabilities, which is why "intermediate positions" can also occur, as in the case of arrow 52, ​​where breast density class d also occurs with a significant probability. The extension of the arrows 51, 52 in the longitudinal direction of the scale (in Fig. 6 vertical) describes the reliability, whereby with decreasing reliability value a larger width is chosen to visualize the higher uncertainty.

[0106] For both the visualization and the determination of the reliability value, other variants than those in Fig. 5 and Fig. 6 shown possible.

[0107] Fig. Figure 7 shows a functional schematic diagram of a training system 33 according to the invention. The training system 33 comprises a training computing device 34, which, in addition to at least one processor, also has at least one storage device 35. Training input data and associated individual output data from various evaluators can be received via a first training interface 36 according to step S1. The determination unit 37a and the compilation unit 37b are preparation units for obtaining training data sets.

[0108] In the determination unit 37a, the training output data and the agreement value are determined according to step S2. In the compilation unit 37b, the training data sets are compiled, further according to step S2, which contain the training input data and the training output data as well as the agreement values. On this basis, training can then take place in a training unit 38 according to step S3. The trained classifying function can then be output via a second training interface 39 according to step S4.

[0109] Fig. Figure 8 shows the functional structure of an evaluation system 40 according to the invention, which comprises an evaluation computing device 41. This also initially has a storage device 42 in addition to at least one processor.

[0110] The trained classifying function can be received via a third evaluation interface 43, which can be at least temporarily connected to the second training interface 39. Input data to be evaluated is received via a first evaluation interface 44 (step S5).

[0111] In an evaluation unit 45, the trained classified function is applied to the input data according to step S6 to determine the output data. In this case, the evaluation computing device 41 also has a self-assessment unit 46, in which the reliability value can be determined according to step S7. The output data with the reliability value can be output via a second evaluation interface 47 and / or visualized by an output unit 48 (see step S8).

[0112] It should be noted that the evaluation system 40 can be implemented as part of a diagnostic device, but also as part of a medical imaging device that can acquire the medical image data of the input data. In particular, in the latter case, it is possible to obtain an (initial) evaluation of the acquired medical image data directly at the imaging device.

[0113] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited to the disclosed examples and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention.

[0114] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Nguyen et al., “Learning classification models with soft label information,” J Am Med Inform Assoc., 21 (2014), pages 501 to 508

[0006] Peterson et al., “Human uncertainty makes classification more robust,” IEEE / CVF International Conference on Computer Vision (ICCV 2018), pages 9616 to 9625

[0007]

Claims

[1] Computer-implemented method for providing a trained function that determines output data from input data, comprising the following steps: - Compiling a plurality of training data sets by, for each training data set, - Providing training input data and several associated individual output data from different evaluators of the training input data, - Determination of training output data by statistical combination, in particular averaging, of the individual output data and an agreement value describing the degree of agreement between the evaluators, - Compiling the training dataset from the training input data, the training output data and the agreement value, - Training the function using the training data using a loss function determined specifically for the training data set, wherein the loss function is weighted for each training data set with a weighting value dependent on the match value of the training data set, in particular in the interval from zero to one. [2] Method according to claim 1, characterized by that a cross-entropy function is used as the loss function and / or the function comprises a DenseNet and / or an AlexNet and / or the output data comprise at least one item of classification information, in particular a probability for at least one class. [3] Method according to claim 1 or 2, characterized by that the input data comprise medical image data, in particular mammography data, of a patient and / or the trained function is a diagnosis support function which provides output data which can be used to make a diagnosis. [4] Method according to one of the preceding claims, characterized by that the training output data are determined by a weighted average of the individual output data using an evaluator weight for each evaluator. [5] Method according to claim 4, characterized by that the evaluator weights are determined for each training input data depending on prior knowledge about the at least one evaluator and / or on the basis of an automatically determined measure of agreement between the evaluator and other evaluators. [6] Method according to claim 5, characterized by that an intraclass correlation coefficient or a kappa coefficient or an overlap is determined as a measure of agreement. [7] Method according to one of the preceding claims, characterized bythat an evaluator function is trained for at least one selected human evaluator, limited to its individual output data, wherein the trained evaluator function - is used to evaluate input training data not evaluated by the selected human evaluator to determine missing output training data, or - is applied as an additional evaluator to all input training data to generate additional output training data. [8] Method according to one of the preceding claims, characterized by that the weighting value is determined as a function of the agreement value in such a way that it lies in a range from a lower limit which is greater than zero to an upper limit, in particular one, and / or increases more quickly towards the upper limit with increasing degree of agreement. [9] Method according to claim 8, characterized bythat the weighting value is the minimum - the maximum of the lower limit, in particular 0.2 to 0.3, and the nth, in particular second, power of the agreement value in the range from zero to one, and - One is determined. [10] Computer-implemented method for evaluating input data to determine output data, comprising the following steps: - Providing a trained function according to a method according to one of the preceding claims, - Applying the trained function to the input data to obtain the output data. [11] Method according to claim 10, characterized bythat if the output data comprise probabilities for several classes and similarity information can be determined for the classes, the output data, if they indicate a probability of one for more than one class, are evaluated depending on the probabilities and the similarity information of the at least two classes with the highest probabilities to determine a reliability value. [12] Method according to claim 11, characterized by that for classes that are assigned an ordinal order, the ordinal distance between the classes is used as similarity information. [13] Method according to claim 12, characterized by that in a first case distinction then: - if the two classes with the highest probabilities are not immediately adjacent, the reliability is set to zero, - if the two classes with the highest probabilities are immediately adjacent, - the sum of the two highest probabilities is determined as the calculated value, and in a second case distinction - if the calculated value lies in at least a first predetermined interval, in particular from zero to 0.5, the reliability value is set to zero, and / or if the calculated value lies in at least a second predetermined interval, the reliability value is set to one, and - if the calculated value lies outside all of the at least one interval, the reliability value is determined between zero and one depending on the calculated value, in particular as twice the calculated value minus one. [14] Method according to one of claims 11 to 13, characterized bythat, if the reliability value is less than or equal to a threshold value, a non-determinability of the output data for the input data is output, and / or the reliability value is output, in the case of a scale (50) of classes of an ordinal order, in particular by means of an extension, determined as a function of the reliability value, of an arrow (51, 52) visualizing the output data, in particular the class with the highest probability, in the longitudinal direction of the scale (50). [15] Training system (33), comprising a training computing device (34) with at least one processor and at least one storage means (35), which comprises: - a first training interface (36) for receiving training input data and a plurality of associated individual output data from different evaluators of the training input data, - a determination unit (37a) for determining, for each set of training input data, training output data by statistical combination, in particular averaging, of the individual output data and a match value describing the degree of agreement between the evaluators, - a compilation unit (37b) for compiling training data sets from the training input data, the training output data and the agreement value, - a training unit (38) for training the function using the training data using a loss function determined specifically for the training data set, wherein the loss function is weighted for each training data set with a weighting value dependent on the match value of the training data set, in particular in the interval from zero to one, and - a second training interface (39) for providing the trained function. [16] Evaluation system (40), comprising an evaluation computing device (41) with at least one processor and at least one storage means (42), which comprises: - a first evaluation interface (44) for receiving a set of input data, - an evaluation unit (45) for evaluating the input data to determine output data by a trained function provided according to a method according to one of claims 1 to 9, and - a second evaluation interface (47) for outputting the output data. [17] A computer program which, when executed on a computing device (34, 41), causes the computing device (34, 41) to perform the steps of a method according to any one of claims 1 to 14. [18] An electronically readable data carrier on which a computer program according to claim 17 is stored.

Images