Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for quantifying uncertainty in medical image assessment

Abstract

Methods, systems and uncertainty quantifier for quantifying uncertainty of medical image assessment. In one aspect, the invention relates to a computer-implemented method for providing an uncertainty prediction for a medical assessment of imaging data (I) provided by a machine learning system (M). The method comprises receiving (1) an input dataset comprising imaging data (I) and non-imaging data that has been provided to the machine learning system (M), each non-imaging data being represented as a signal (S) with a certain degree of noise, said noise being quantified as an uncertainty, in particular an aleatoric uncertainty; providing (2) an information fusion algorithm; and applying (3) the received input dataset to the provided information fusion algorithm, while modelling a propagation of the uncertainty by the information fusion algorithm based on the provided input dataset, to predict an uncertainty of the medical assessment provided by the machine learning system (M) as a result (r).

Description

Technical Field

[0001] This invention relates to medical image processing, and more particularly to the processing of uncertainties in medical and health-related data, especially medical imaging data. Background Technology

[0002] Generally, there are several different imaging modalities used for acquiring medical images, among others, such as radiography, computed tomography (CT), and magnetic resonance imaging (MRI). Imaging procedures can be particularly well-suited for imaging specific organs or body parts to detect and / or assess clinical abnormalities and / or diseases and / or injuries. For example, a computed tomography (CT) scan or chest radiography (CXR) is typically performed to classify malignant tumors of the lung or pneumonia.

[0003] Acquired medical images can be subject to automated procedures for medical evaluation, such as initiating further measurements and / or acquiring further sensor data and / or initiating clinical procedures. The automated procedures used to evaluate medical images are machine learning programs. Machine learning systems can, for example, be configured to classify between healthy tissue and lesions (or abnormalities). Typically, machine learning systems can be configured to evaluate the provided medical images.

[0004] However, the computer-based implementation and automated evaluation of medical images are subject to uncertainty, particularly aleatoric uncertainty. Aleatoric uncertainty, also known as statistical uncertainty, represents the different unknowns each time the same experiment is run. The word "aleatoric" comes from the Latin "alea" or "dice," referring to a game of chance. Aleatoric uncertainty should be distinguished from cognitive uncertainty, which is a systematic uncertainty arising from things that the system could theoretically know or calculate but actually cannot. This could be due to inaccurate measurements or noise in the measurement or the measurement signal.

[0005] For example, the evaluation of chest radiographs (CXR) images, particularly in outpatient settings, is an inherently ambiguous task. Internal studies have revealed that the level of inter-rater agreement is 60–70% for the detection of, for example, lung nodules, and 50–60% for the detection of consolidation / gas space opacity. This level of inconsistency can often be attributed to a lack of clarity in deciding whether an abnormal area indicates abnormality A (e.g., lung nodule / mass) or abnormality B (e.g., consolidation). Current machine learning systems based solely on CXR assessment and evaluation studies are designed to force this decision-making process, whereas in clinical practice, radiologists would not make such a decision and would record the uncertainty between the two classes in the report (most likely requesting follow-up with another CXR or CT scan to obtain a clear answer). Furthermore, machine learning systems used for CXR assessment typically do not use any auxiliary non-imaging information to guide the decision (between abnormality A and B). This contrasts with radiologists, who, for example, would use the fact that the patient in question has fever to guide the decision on abnormality B / consolidation, which is a consequence of pneumonia / infection, thus explaining the fever. This leads to systems that perform poorly or unexpectedly in such ambiguous situations, resulting in limited performance and directly impacting user trust.

[0006] While providing more accurate information than CXR (e.g., for subsequent or later differential diagnosis), similar ambiguity can exist in high-resolution chest CT. Radiologists typically refer to additional information from electronic health records (EHRs), including but not limited to: the reason for scheduling the examination, the patient's medical and physical history, serological results, biomarkers from laboratory diagnostics, etc., to obtain clarity. A common scenario currently is differentiating COVID-19 in patients with respiratory diseases susceptible to interstitial lung disease (ILD) from patients with underlying lung malignancies. Summary of the Invention

[0007] Accordingly, the object of the present invention is to provide means for improving the expressiveness and / or robustness of machine learning system results based on imaging data, and / or to enable the combination of imaging data and non-imaging data to improve statements inferred from imaging data.

[0008] This objective is achieved by the appended patent claims, particularly by computer-implemented methods and uncertainty quantifiers, medical systems, and computer program products.

[0009] In a first aspect, the present invention relates to a computer-implemented method for providing uncertainty prediction for medical assessment, particularly automated (computational) medical assessment, of imaging data published or provided by a machine learning system. The method comprises the following steps: - Receive an input dataset, which includes imaging and non-imaging data that have been provided to the machine learning system. Each non-imaging data is represented as a signal with a certain degree of noise, which is quantified as uncertainty, particularly accidental uncertainty and / or cognitive uncertainty. - Provide information fusion algorithms in the storage device; - Apply the received input dataset to the provided information fusion algorithm (i.e., perform the information fusion algorithm using the received input dataset), and model the propagation of uncertainty through the information fusion algorithm based on the provided input dataset to predict the uncertainty of the medical assessment provided by the machine learning system as an outcome.

[0010] The term "non-imaging data" refers to medical or healthcare data in digital format or representation, excluding image data acquired from imaging modalities. Non-imaging data can reflect non-imaging knowledge. Some non-imaging data requires structuring before further processing. As will be explained in more detail later, this invention particularly recommends using graph neural networks for data processing. In this regard, certain non-imaging data needs to be structured before being passed to a graph neural network, such as EHR text. Therefore, preprocessing can be performed on non-imaging data. Preprocessing can include restructuring the data in memory in a processable format (e.g., normalization and normalization for processing in graph neural networks and / or information fusion models). Therefore, the storage of preprocessed data differs from the storage of raw non-imaging data (also known as signals).

[0011] In this regard, "noise" refers to the portion of a signal or data that does not include the payload signal. Noise can be quantified as uncertainty, particularly random or cognitive uncertainty. While random uncertainty is the most common, distributed uncertainty and other types of uncertainty can also be addressed. In a preferred embodiment, deep representation learning, such as a variational autoencoder (VAE), can be used and applied, for example, to encode information into a compact representation via a variational autoencoder and / or to denoise some of the collected input data. For more details on variational autoencoders, see Kingma, DP, & Welling, M. (2013), Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114.

[0012] The input data is digital data, or data converted from analog to digital using a converter. Input data can include digital signals or more complex datasets, such as signals evolving over time. Input data can include imaging and non-imaging data from different imaging modalities. Non-imaging data includes, for example, biomarkers, laboratory values, electronic health records (EHRs), and measurement signals such as physiological signals like blood pressure, body temperature, and heart rate.

[0013] An information fusion algorithm is an algorithm used to combine different datasets (including, for example, imaging data and non-imaging data) provided in different formats. The information fusion model does not need to be "deep," but it can be. The information fusion model can be a deep fusion model optimized by a mutual information criterion. The information fusion algorithm can be, or can use (applied) an information fusion model and / or a graph neural network, optimized to maximize the entropy in the non-imaging data. Typically, more than one non-imaging signal is used to improve the quality of uncertainty prediction. Preferably, an information fusion model is used, through which the propagation of uncertainty is passed. Alternatively or additionally, a graph neural network can be used. In yet another embodiment, only a graph neural network (without an information fusion model) can be used, wherein the graph neural network contains both imaging data and non-imaging data.

[0014] The result of information fusion is the uncertainty in estimates or predictions for medical assessments. Uncertainty can be expressed quantitatively. Uncertainty can be based on pre-configured measures. Uncertainty can be provided as a percentage.

[0015] Machine learning systems are used to provide automated evaluations by considering imaging data. Machine learning systems can be based on, for example, artificial neural networks (ANNs), deep neural networks (DNNs), convolutional neural networks (CNNs), and can utilize different learning algorithms, such as reinforcement learning, supervised learning, semi-supervised learning, or even unsupervised learning. Typically, machine learning models can be used to detect structures in images, classify anomalies in images, and perform other tasks.

[0016] According to a preferred embodiment of the invention, the entropy of the information fusion model and / or graph neural network, particularly the von Neumann entropy, is optimized by a greedy algorithm or another optimization algorithm (e.g., dynamic programming, grid search, and / or divide-and-conquer techniques).

[0017] According to another preferred embodiment, the method may further include: - Apply a selection algorithm to select a subset of the provided input data, which minimizes the cost function and / or reduces uncertainty by using a reinforcement learning model.

[0018] According to another preferred embodiment, input data from the input source set may or may not be present. The latter is more likely if providing input data proves too expensive (e.g., from a time / performance perspective, or from a monetary perspective). Furthermore, the method can provide a suggested result dataset that encodes guiding decisions on which of the missing input data sources will reduce uncertainty and / or minimize the cost function. Thus, input data is prioritized in terms of reducing uncertainty and / or minimizing the cost function. The cost function can be preconfigurable via user input on a user interface.

[0019] According to another preferred embodiment, the input data providing the input data source set may include data measured and / or acquired from imaging modalities and / or from medical databases (e.g., electronic health records, HER, laboratory values, radiological information systems, RIS, image archives and communication systems, PACS, etc.).

[0020] According to another preferred embodiment, non-imaging data includes (but is not limited to) biomarkers, clinical records, image annotations, medical report dictations, measurements, laboratory values, diagnostic codes, data from the EHR database, and / or patient memory data.

[0021] According to another preferred embodiment, the reinforcement learning model is based on a decision-making process, particularly a non-Markov decision-making process. , Where S denotes the state space, A denotes the action space, T denotes the random transition process, and R denotes the reward function, and Indicate the discount factor, where the action indicates that additional input data sources are provided.

[0022] According to another preferred embodiment, the reward function is defined as minimizing cost and / or minimizing prediction uncertainty. Cost can be configured by the user during the configuration phase; for example, cost can be financial and / or time / efficiency / performance related, or other detrimental factors.

[0023] According to another preferred embodiment, an uncertainty propagation model including a Bayesian deep model and / or Q-learning and / or actor-critic learning can be used for reinforcement learning.

[0024] According to another preferred embodiment, an uncertainty propagation model, particularly a Bayesian deep model, is used in the information fusion model.

[0025] According to another preferred embodiment, the information fusion model is able to handle situations where a subset of the input data sources is unavailable or only available at a specific cost.

[0026] According to another preferred embodiment, the uncertainty of the prediction is patient-specific. Alternatively or cumulatively, the uncertainty of the prediction may be specific to the imaging data. Alternatively or cumulatively, the uncertainty of the prediction may be signal-specific.

[0027] According to another preferred embodiment, a set of interactive buttons is provided on the user interface, allowing the user to indicate that the input data source is unavailable during inference, or that the action space of the non-Markov decision process is limited to the data source. The availability of the interactive button set allows the user to select the optimization type, especially if he or she wants to minimize prediction uncertainty or cost.

[0028] The present invention has been described to date with respect to the claimed method. Features, advantages, or alternative embodiments described herein may be assigned or transferred to other claimed objects (e.g., computer programs or devices, i.e., uncertainty quantizers or computer program products), and vice versa. In other words, the apparatus or device may be modified using features described or claimed in the context of the method, and vice versa. In this case, the functional features of the method are embodied by the structural units of the apparatus or device or system, and vice versa. Generally, in computer science, software implementations and corresponding hardware implementations (e.g., as embedded systems) are equivalent. Thus, for example, method steps for “storing” data may be performed using storage units and corresponding instructions for writing data to the storage device. To avoid redundancy, although the apparatus may also be used in alternative embodiments described with reference to the method, these embodiments are no longer explicitly described with respect to the apparatus.

[0029] In another aspect, the present invention relates to an uncertainty quantizer for medical evaluation of imaging data provided by a machine learning system, which is adapted to perform the method described above. The uncertainty quantizer includes: - An input interface for connecting to an input dataset source to receive an input dataset comprising imaging and non-imaging data that has been provided to the machine learning system, each non-imaging data being represented as a noisy signal, the noise being quantified as uncertainty, particularly random or cognitive uncertainty; - Storage device for storing information fusion algorithms; - A processing unit configured to apply the received input dataset to a provided information fusion algorithm, and simultaneously model the propagation of uncertainty through the information fusion algorithm based on the provided input dataset to predict the uncertainty of the medical assessment provided by the machine learning system; - Output interface, used to provide the uncertainty of the prediction as a result.

[0030] In another aspect, the present invention relates to a medical system for medical evaluation of imaging data, the medical system being provided by a machine learning system having a set of medical data sources and an uncertainty quantizer as described above.

[0031] In another aspect, the present invention relates to a computer program product comprising program elements that, when loaded into the memory of the computer or executed on the computer, cause the computer to perform steps of a method according to any one of the preceding method claims for providing uncertainty prediction for machine learning-based medical assessment of imaging data.

[0032] In another aspect, the present invention relates to a computer program that can be loaded into a memory unit of a computer system, including a program code segment, which, when executed in the computer system, causes the computer system to perform the method described above for providing uncertainty prediction for medical evaluation of imaging data.

[0033] In another aspect, the present invention relates to a computer-readable medium having stored or stored thereon a program code segment of a computer program, said program code segment being loadable into and / or executable in a computing unit, wherein when said program code segment is executed in the computing unit, the computing unit performs the method described above for providing uncertainty prediction for medical evaluation of imaging data. The computing unit may include a processing unit.

[0034] The features, characteristics, and advantages of the invention described above, as well as the ways in which they are implemented, become clearer and more understandable in light of the following description and embodiments, which will be described in more detail in the context of the accompanying drawings. The following description does not limit the invention to the included embodiments. In different figures, the same components or parts may be labeled using the same reference numerals. Generally, the figures are not to scale.

[0035] It should be understood that the preferred embodiments of the present invention may also be any combination of the dependent claims or the above embodiments with the corresponding independent claims.

[0036] These and other aspects of the invention will become clear from the embodiments described below and will be illustrated with respect to the embodiments described below. Attached Figure Description

[0037] Figure 1 It is a structured block diagram illustrating typical application scenarios for machine learning models in existing technologies; Figure 2 This is an overview of the structure and architecture of the uncertainty quantizer according to a preferred embodiment of the present invention; Figure 3It is a schematic representation of a graph neural network with minimum entropy, and Figure 4 This is another schematic representation of a graph neural network with maximum entropy; Figure 5 This is a flowchart of a method according to a preferred embodiment of the present invention; Figure 6 This is an example of processing a chest CXR image using an uncertainty quantizer to provide the result r. Detailed Implementation

[0038] Current solutions for chest radiographic evaluation focus on image-level classification of findings without precise localization, or provide approximate localization without investigating the inherent uncertainty at the instance level. The term "instance level" refers to a specific location within an image. Several methods have been proposed for uncertainty quantification. However, they do not explicitly address this type of accidental uncertainty and / or natural class overlap. Furthermore, such methods do not utilize any non-imaging information to enhance and improve classification accuracy.

[0039] If it is possible Figure 1 As seen in the diagram, a typical state of existing machine learning systems is that they are configured to process input data in the form of imaging data I to provide a classification result r', without any additional information about the quality and accuracy of the inferred classification. Therefore, the user is not informed how he can trust the results provided by the machine learning model M.

[0040] Figure 2 A schematic representation of a system with an uncertainty quantizer Q is shown, which is used to provide additional information missing in prior art systems as mentioned above. For example, it can be seen that... Figure 2As seen in the diagram, the uncertainty quantizer Q has an input interface II for receiving input data from a different set of sources or multiple different sources, including image acquisition sources and non-image sources. Therefore, the input data can include imaging data I, such as images like chest X-ray images, CXR or computed tomography (CT) images, or images from any other type of image acquisition. Furthermore, and as explained previously, further non-imaging data is provided. This non-imaging data is referred to as signals S. Examples include biomarker signals S1, clinical report signals S2, laboratory signal sets S3, and physiological measurement sets S4 such as temperature, heart rate, etc. Thus, the input interface II connects the uncertainty quantizer Q to input dataset sources (not shown in the diagram) such as temperature sensors, heart rate sensors, laboratory systems, etc. Alternatively, a modeling or accumulating database, such as an electronic health record (EHR), can also be used as an input data source DB. The processor P is configured to implement and execute the image fusion algorithm. The result r of the image fusion algorithm is provided on the output interface OI. The result r is a prediction of the (quantified) uncertainty of the evaluation calculated by the medical algorithm based on the received input data. The result r can be provided on the user interface UI. The user interface (UI) can also serve as a human-machine interface, receiving configuration data provided by the user, such as determining which input data source is currently unavailable or only available at high cost. This configuration data will be processed by information fusion algorithms and / or selection algorithms.

[0041] exist Figure 2 In the example provided, a CXR image is used as input. However, it should be noted that this is merely one example in the set of examples, and the invention is certainly not limited to this type of image modality. Therefore, MRI images, ultrasound images, and images from other acquisition modalities can also be used as input sources.

[0042] This invention provides a learning system for quantifying predictive random uncertainty, such as ambiguity prediction at the instance level. At this level, there are two levels of ambiguity that can be quantified: 1. Ambiguity between captured anomalous classes: This is the primary type of ambiguity that needs to be addressed. For example, for nearly 15% of positive cases of nodules / consolidation in CXR, decisions about the class cannot be made based on imaging information. This high degree of class overlap is not unique to the mentioned disease; for example, it can be found between pleural effusion and consolidation, or consolidation and atelectasis. This type of ambiguity can be modeled using various methods for uncertainty quantification, including fuzzy prediction, evidence learning, subjective logic, etc. This leads to systems capable of accurately identifying these 15% of cases by generating multiple labels for the same anomalous condition in the images. As in Figure 6As can be seen, the highlighted areas in the form of bounding boxes b can indicate two potential abnormalities: consolidation caused by infection or a lung mass. The results r are provided with a dataset that includes indications such as "60% consolidation, 30% mass, and 10% other".

[0043] 2. Training-Out-of-Domain Ambiguity: For completeness, the second type of ambiguity is derived from whether the anomalous instance is fully captured in the training distribution. This means the chance that the instance exists as part of an anomalous type not modeled in training and therefore cannot be predicted by the system. For example, in Figure 1 As can be seen, the system recognizes the non-zero chance that a bounding box can refer to something that is not part of a class modeled during device training.

[0044] According to a preferred embodiment, by using additional information from a non-imaging source S to simulate the behavior of an expert radiologist, greater clarity is achieved when evaluating such ambiguous cases. In the context of outpatient chest X-ray evaluation, there exists a range of factors (based on non-imaging information) that can guide radiologists' decisions on how to evaluate a case. Figure 6 In specific examples, this would mean changing the decision, for example, further increasing the likelihood of consolidation. In practice, these factors could refer to “patient age,” “fever indication,” “acute symptoms,” or “pain indication.” This information could be provided with the examination order or can be seen in the patient’s file. For example, if a patient presents with a fever (assuming it is caused by an infection, interpreting consolidation as part of the infection process), the expert could increase the confidence in consolidation. Furthermore, if the patient is younger, changes in the lung mass are further reduced—assuming the prevalence of lung cancer is very low in young people. This non-imaging information can be modeled and applied in, for example… Figure 6 The confidence level of the guidance system is described in the context. Prior knowledge can be used (e.g., lung masses are more likely in older patients than in younger patients). In the example above, using auxiliary information and knowing that the patient is 28 years old and has a fever, the chance of consolidation can increase to 90%. Such a change can have a significant impact on clinical decisions and patient triage by nature (e.g., avoiding unnecessary CT scans).

[0045] While other types of information from electronic health records or general patient history are not typically used for chest radiographic evaluation, they can be invaluable for differential diagnosis on chest CT. For example, conditions like eosinophilic pneumonia may present with fever and cough, similar to COVID-19. On CT, eosinophilic pneumonia may appear as ground-glass opacities and consolidation, similar to COVID-19, with or without the cobblestone road sign. This makes it difficult to distinguish eosinophilic pneumonia from the biomarker COVID-19 using CT alone. Therefore, this invention proposes the use of additional information from source sets S1, S2, S3…Sn, DB.

[0046] To better distinguish it from COVID-19 in the examples above, it is helpful to consider the following additional information: Clinical manifestations of slow onset of symptoms. - Associated with asthma; - Increased eosinophil count in bronchoalveolar lavage fluid and blood samples; - Distribution in the upper lung region.

[0047] By integrating the EHR system into the radiology workflow, relevant non-imaging and imaging information that can be used to provide outcome datasets for differential diagnosis becomes available to radiologists.

[0048] Generally, it is assumed that during training / inference, in addition to the image I, other relevant signals S are provided, as mentioned earlier. These signals S are encoded as x1, x2, ... x N , where N indicates the number of sources used for non-imaging signals. For any signal x k The following properties hold true: - For a given instance / sample, the signal may or may not be present. - There are inherent uncertainties in signals / measurements, which are quantified according to the evaluation task. (Heteroscedasticity, random uncertainty). - x k With any other x j In contrast, different distributions are possible (we allow categorical variables, continuous variables, and complex high-dimensional signals, etc.).

[0049] The robust static information fusion using deep learning models is explained in more detail below.

[0050] In this scenario, assume the signals are non-imaging signals x1, x2, ... x N All sources are available (note that the signal may still be missing for any number of reasons). Multiple techniques can be used for information fusion, including but not limited to deep fusion models and graph neural networks.

[0051] Deep fusion model: Interactive information , Where Y represents the system prediction; thus, the goal is to use all input information and leverage redundancy. Assume that each signal is surrounded by noise, which is quantified as uncertainty u(I); u(x) k One can use deep robust information fusion methods [6], while modeling the propagation of uncertainty through deep models [7], for example, using Bayesian deep learning [8]. Signal coding architectures (e.g., variational autoencoders) can be used to compress heterogeneous high-dimensional inputs and simplify the learning process.

[0052] Graph Neural Models: Just as deep fusion models help maximize mutual information in selecting non-imaging signals, graph neural networks can maximize mutual information in selecting signals x1, x2, ... x N The correlations between these connections promote the maximization of entropy. Non-imaging signals are connected via complex hidden underlying structures that are not always traceable. In such cases, graph neural networks can not only learn the hidden structures but also perform prediction tasks when the structures are unavailable [10-11]. By evaluating graph entropy (e.g., von Neumann entropy, Shannon entropy), we can identify and preserve important correlations without getting lost in the complexity of these hidden structures.

[0053] Von Neumann entropy: Assuming all non-imaging signals can be represented in the same latent space, let Indicates having a vertex set The graph of the combination of edge set E and weight matrix W. The Laplacian matrix of the combined graph G is defined as: , Where S is a diagonal matrix and its diagonal elements The density matrix of graph G is defined as , Where tr is the trace of the matrix.

[0054] Therefore, the entropy of graph G is given by the following equation. .

[0055] Figure 3 and Figure 4 Two graphs constructed using a greedy algorithm with minimum and maximum entropy are shown to explore their properties

[12] . Under constraints using the same number of edges or associations, Figure 3 The entropy of the graph in the middle is less than Figure 4 The entropy of the graph in the image. Because... Figure 3The graph in the least entropy construction is such that almost half the connections from the first layer to the bottom layer are blocked, and several vertices are deactivated. Conversely, graphs with higher regularity and “balance” tend to have greater entropy.

[0056] The following describes the information distillation process for the optimal selection of input sources or (additional) information provided by these sources using deep reinforcement learning.

[0057] In the second scenario (where the non-imaging signal is entirely unavailable; it is partially available, or completely unavailable), a subset of the non-imaging signal sources is hidden. Without loss of generality, assume that only x1, x2, ... x are hidden when K < N. K Available. In addition to the noise / uncertainty associated with each signal, we will also consider the cost of acquisition or measurement. k is associated. One can imagine this cost increasing during the construction phase of the training database (in the sense of the cost of collecting data from clinical sites), or during inference as a response to a user's request, for example, "Based on the current information, the prediction is Y with high uncertainty if variable x..." K+1 If it becomes available, then that uncertainty could be significantly reduced (of course, there is a cost for each measurement / clinical test)."

[0058] One can formulate the problem as follows: Given a subset of S additional information sources (from set x) that minimizes acquisition costs while optimally reducing uncertainty in prediction. k+1 … x N What is it? Without considering the cost element, a potential solution is to use a feature selection strategy, which is equivalent to the selection of the signal source, to minimize the uncertainty around the prediction Y[9].

[0059] This problem can be formulated within the context of reinforcement learning. Assume the decision process (DP) is (non-)Markovian. , Where S denotes the state space, A denotes the action space, T denotes the random transition process, and R denotes the reward function, and Indicate the discount factor. The state is defined by observable information (initially I, x1, x2, ... x). K Actions are allowed from (x) k+1 , ... x NIn this context, additional sources are selected. DP is non-Markovian in the sense that an action cannot be performed twice. The reward function R can be designed to minimize either the cost or the prediction uncertainty around Y. Furthermore, joint optimization is possible, i.e., minimizing prediction uncertainty while not exceeding a threshold of the total cost used for selection. Powering reinforcement learning models with deep architectures will allow for efficient modeling of complex and diverse input signals. As mentioned above, similar strategies can be used in the chapters related to robust static information fusion using DNNs to design learning architectures (i.e., Bayesian models, uncertainty propagation models, etc.). Q-learning or actor-critic strategies can be applied.

[0060] Unavailable information sources: Using the actor critic architecture, one can also model situations where a subset of actions is unenforceable. In other words, during inference, the user can indicate whether / whether a certain subset (x) of the source can / will not be provided. k+1 … x N In this case, the optimization model will avoid these actions.

[0061] Figure 5 This is a flowchart of a method for providing uncertainty predictions based on machine learning results. After the method begins, in step 1, input data is received from both imaging and non-imaging sources. In step 2, an information fusion algorithm is provided in the computer's storage MEM, and the received input data is forwarded to this algorithm, which is executed in step 3. After execution, in step 4, uncertainty predictions of the results of the machine learning model M are provided to the result r. Optionally, the method may branch back to step 1 to request more input data and / or branch back to step 2. The latter is likely if updates to the image fusion algorithm and / or model can be provided, and updates need to be applied to and performed on the data. This optional process step is described in... Figure 5 The middle section is depicted via dashed lines. Another optional step 5 is to apply or execute a selection algorithm to select a subset of the provided input data, which minimizes the cost function and / or reduces uncertainty by using a reinforcement learning model. This improves performance because relevant input data can be indicated and selected to be provided to the information fusion algorithm.

[0062] Typically, a single unit or device can perform several functions listed in the claims. The fact that certain measures are listed in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

[0063] Any reference symbols in the claims should not be construed as limiting the scope.

[0064] Unless otherwise specified, individual embodiments or aspects and features described in connection with the accompanying drawings may be combined or interchanged with each other without limiting or broadening the scope of the invention, provided that such combination or interchange is meaningful and in the sense of the invention. Advantages described with respect to specific embodiments of the invention or with respect to specific figures are also advantages of other embodiments of the invention wherever applicable.

[0065] List of references: .

Claims

1. A computer-implemented method for providing uncertainty predictions for medical assessment of imaging data (I) provided by a machine learning system (M), comprising the following method steps: - Receive (1) an input dataset, which includes imaging data (I) and non-imaging data that have been provided to the machine learning system (M), each non-imaging data being represented as a signal (S) with a certain degree of noise, the noise being quantized as uncertainty, wherein the non-imaging data includes medical or healthcare data in digital format or representation, which does not include image data acquired from the imaging modality; - Provides (2) an information fusion algorithm, wherein the information fusion algorithm is an algorithm for combining different datasets provided in different formats, the different datasets including imaging data and non-imaging data, and wherein, The information fusion algorithm uses information fusion models and / or graph neural networks, which are optimized to maximize the entropy in non-imaging data. - The received input dataset is applied (3) to the provided information fusion algorithm, and the propagation of uncertainty is modeled by the information fusion algorithm based on the provided input dataset to predict the uncertainty of the medical assessment provided by the machine learning system (M) as the result (r).

2. The method according to claim 1, wherein the uncertainty is accidental uncertainty.

3. The method according to claim 2, wherein, Greedy algorithms are used to optimize the entropy of information fusion models and / or graph neural networks.

4. The method of claim 3, wherein the entropy is the von Neumann entropy.

5. The method according to any one of claims 1-4, wherein, The method further includes: - Apply the (5) selection algorithm to select a subset of the provided input data, which minimizes the cost function and / or reduces uncertainty by using a reinforcement learning model.

6. The method according to any one of claims 1-4, wherein, The input data in the set of input data sources (S, DB) may or may not exist, and the method provides a guided decision on which of the non-existent input data sources (S, DB) will reduce uncertainty and / or minimize the cost function.

7. The method according to any one of claims 1-4, wherein the input data providing the set of input data sources (S, DB) includes data measured and / or acquired from imaging modalities and / or from medical databases (DB).

8. The method according to claim 5, wherein, Reinforcement learning models are based on non-Markov decision processes. ， Where S denotes the state space, A denotes the action space, T denotes the random transition process, and R denotes the reward function, and Indicate the discount factor, where the action indicates that an additional input data source (S, DB) is provided.

9. The method according to claim 8, wherein, The reward function is defined as minimizing cost and / or minimizing prediction uncertainty.

10. The method according to any one of claims 1-4, wherein, The non-imaging data includes biomarkers, clinical records, image annotations, medical report dictations, measurements, laboratory values, diagnostic codes, data from the EHR database, and / or patient memory data.

11. The method according to claim 5, wherein, Uncertainty propagation models, including Bayesian deep models and / or Q-learning and / or actor-critic learning, are used for the reinforcement learning.

12. The method according to any one of claims 1-4, wherein, The uncertainty propagation model is used in the information fusion model.

13. The method according to any one of claims 1-4, wherein, The information fusion model can handle situations where a subset of the input data sources (S, DB) is unavailable or only available at a specific cost.

14. The method according to any one of claims 1-4, wherein, The uncertainty of the prediction is patient-specific, and / or imaging data-specific, and / or signal-specific.

15. The method according to any one of claims 8, 9, and 11, wherein, On the user interface (UI), a set of interactive buttons is provided so that the user can indicate that the input data source (S, DB) is unavailable during inference, or indicate that the action space of the non-Markov decision process is limited to the input data source (S, DB). The set of interactive buttons can be used by the user to select the type of optimization.

16. An uncertainty quantizer for medical evaluation of imaging data provided by a machine learning system (M), adapted to perform the method according to any one of claims 1-15, said uncertainty quantizer comprising: - Input interface (II) for connecting to input dataset source (S, DB) to receive input dataset, which includes imaging data and non-imaging data that have been provided to the machine learning system, each non-imaging data being represented as a noisy signal, the noise being quantized as uncertainty; -Memory (MEM) is used to store information fusion algorithms; - A processing unit (P) is configured to apply the received input dataset to a provided information fusion algorithm, and simultaneously model the propagation of uncertainty through the information fusion algorithm based on the provided input dataset to predict the uncertainty of the medical assessment provided by the machine learning system (M); - Output interface (OI) is used to provide the uncertainty of the prediction as a result (r).

17. The uncertainty quantizer of claim 16, wherein the uncertainty is accidental uncertainty.

18. A medical system for medical evaluation of imaging data, the medical system being provided by a machine learning system (M) having a set of medical input data sources (S, DB) and an uncertainty quantizer according to any one of claims 16-17.

19. A computer program product comprising program elements, wherein when said program elements are loaded into or executed on a computer's memory, said program elements cause said computer to perform the steps of a computer-implemented method for providing uncertainty prediction for medical evaluation of imaging data provided by a machine learning system, according to any one of claims 1-15.

Images