Methods and systems for providing treatment response predictions based on full slide images

Abstract

The present disclosure relates to methods and systems for providing treatment response predictions based on full slide images. Embodiments disclosed herein relate to a computer-implemented method and corresponding system for providing treatment response prediction (TRP) for a patient suffering from a cancerous disease. Methods and systems are based on obtaining (S10) a full slide image (WSI) of a patient showing a tissue sample associated with a cancerous disease; providing (S20) a prediction function (PF) configured to derive a treatment response prediction (TRP) for one or more treatment options from the full slide image (WSI); and applying (S30) the prediction function (PF) to the full slide image (WSI) in order to provide a treatment response prediction (TRP).

Description

Technical Field

[0001] Various examples of the present disclosure are directed to providing treatment response predictions based on whole slide images. Specifically, various examples of the present disclosure are directed to methods and systems configured to process whole slide images to derive treatment response predictions. In other words, examples of the present disclosure are directed to automated processing of image data to provide clinically actionable results / diagnoses. Background Art

[0002] Radiation therapy is commonly applied to cancerous tumors because of its ability to control cell growth. Ionizing radiation works by directly and indirectly damaging the DNA of cancer cells, leading to cell death. In order not to damage normal tissues, such as skin and organs, through which the radiation must pass to treat the tumor, shaped radiation beams are aimed from several exposure angles to intersect at the tumor, providing a much larger absorbed dose there than in surrounding healthy tissue.

[0003] Radiation therapy works in conjunction with chemotherapy, immunotherapy, and other cancer therapies, and has been used before, during, and after other treatments for susceptible cancers.

[0004] The question of whether to treat with radiation therapy (ie, whether the tumor is susceptible to radiation therapy) and the coordination of radiation therapy treatment plans with other treatment options are difficult issues that require considerable experience on the part of the radiation oncologist and seriously affect the patient's health.

[0005] Furthermore, in clinical routine, physicists have to base their decisions on a particular treatment on a large number of additional factors. Moreover, physicists do not only consider one treatment option, but must weigh multiple options. This results in a large amount of data and information that physicists have to manage. Furthermore, one problem with current decision assistance systems is that it is often not obvious to the user how reliable a certain result of such an automated process is. Summary of the invention

[0006] Therefore, an object of embodiments of the present disclosure is to provide a method and system for automatically predicting a cancer patient's treatment response to one or more treatment options under discussion. In particular, an object of embodiments of the present disclosure is to achieve reliable treatment response prediction by effectively using the patient's available data.

[0007] This object is solved by a method for providing a prediction of a treatment response, a corresponding system, a corresponding computer program product and a computer-readable storage medium according to the main claim.Alternative and / or preferred embodiments are object of the dependent claims.

[0008] In the following, the technical solution according to the present invention is described with respect to the claimed device and with respect to the claimed method. The features, advantages or alternative embodiments described herein can also be assigned to other claimed objects, and vice versa. In other words, the claims for solving the method of the present invention can be improved by describing or claiming features with respect to the device. In this case, for example, the functional features of the method are implemented by the target unit or element of the device.

[0009] The technical solution will be described with respect to a method and system for providing information and also with respect to a method and system for providing a prediction function. Features and alternative forms of implementation of data structures and / or functions of a method and system for providing a prediction can be transferred to similar data structures and / or functions of a method and system for providing a prediction function. Similar data structures can be identified in particular by using the prefix "training". In addition, the prediction function used in the method and system for providing information can be adjusted and / or trained and / or provided in particular by a method and system for adjusting the prediction function.

[0010] According to one aspect, a computer-implemented method for providing a treatment response prediction for a patient with a cancerous disease is provided. The method includes a plurality of steps. One step involves obtaining a whole slide image of the patient, the whole slide image showing a tissue sample associated with the cancerous disease. Another step involves providing a prediction function configured to derive a treatment response prediction for one or more treatment options from the whole slide image. Another step involves applying the prediction function to the whole slide image to generate a treatment response prediction. Another step involves providing a treatment response prediction.

[0011] Cancerous diseases may involve neoplasms (also referred to as "tumors"), particularly benign neoplasms, neoplasms in situ, malignant neoplasms, and / or neoplasms of uncertain / unknown behavior associated with the patient's body.

[0012] In particular, cancerous diseases may include tumors of internal organs of the patient, such as lung tumors, prostate cancer, lymph node cancer, and the like.

[0013] The whole slide image to be processed by the prediction function may be a two-dimensional digital image having a plurality of pixels. The size of the whole slide image may be at least 4.000×4.000 pixels, or at least 10.000×10.000 pixels, or at least 1E6×1E6 pixels.

[0014] A full slide image can image a tissue section or slide of a patient. Preparing a tissue section from a tissue sample can include preparing a portion (e.g., using a punch tool) from the tissue sample, wherein the portion is cut into micron-thick slices, i.e., a tissue section. Another word for a portion is a block or punch biopsy. Under microscopic observation, a tissue section can illustrate the fine tissue structure of a tissue sample, and particularly the cells or cell structures contained in the tissue sample. When observed on a larger length scale, a full slide image can illustrate an overview of tissue structure and tissue density. The tissue may be taken from a tumor suffered by the patient. In particular, the tissue can illustrate the manifestation of a patient's cancerous disease, such as the cells of a tumor.

[0015] The preparation of tissue sections can also include staining the tissue sections using histopathological staining. In this case, staining can be used to highlight different structures in the tissue sections, such as, for example, cell walls or nuclei, or to test medical indications, such as, for example, cell proliferation levels. In such cases, different histopathological stains are used for different purposes.

[0016] In order to create a full slide image, the stained tissue section is digitized or scanned. For this reason, a suitable digitizing station such as a full slide scanner is utilized to scan the tissue section, which preferably scans the entire tissue section mounted on the object carrier and converts it into a pixel image. In order to preserve the color effect from histopathological staining, the pixel image is preferably a color pixel image. Since in prediction, the overall impression of the organization and the finely resolved cell structure may be of great significance, a full slide image usually has a very high pixel resolution. The data size of a single image can reach several gigabytes usually.

[0017] A treatment response prediction can be thought of as a reading of how promising a certain treatment option or therapy is for a patient given a whole slide image. It can include a number proportional to the expected success rate of at least one treatment option. According to some examples, the number can include a percentage. The percentage can represent the match, suitability, or success rate of a potential treatment option given the whole slide image.

[0018] The treatment response prediction may include an indication of one or more treatment options. In particular, the treatment response prediction may include a reading for each of a plurality of treatment options.

[0019] A treatment option may be a discussed therapy or a portion of a therapy for treating a patient's cancerous disease. A therapy may be predefined. Different treatment options may differ in providing different kinds of therapy or in providing the same kind of therapy but with different parameters, such as different dosage regimens for a particular drug.

[0020] According to some examples, one or more of the treatment options may relate to treatments administered to the patient prior to acquiring the tissue sample that forms the basis of the whole slide image.

[0021] The predictive function may generally be thought of as an image processing function that is configured to process image data included in a whole slide image.The predictive function may be configured to classify the whole slide image according to one or more of the treatment options.

[0022] By basing treatment predictions on whole slide images, resources available for most cancer cases can be utilized. Furthermore, depicting cancerous growths at a cellular level, whole slide images give insight into the expected behavior of tumor cells and, thereby, the expected behavior of the tumor with respect to different treatment options. Furthermore, automated processing using the prediction function can provide an objective readout and relieve the user from the task of manually selecting and evaluating different options.

[0023] According to some examples, the prediction function is configured to extract one or more features from the whole slide image and generate a treatment response prediction based on the extracted features.

[0024] The extracted features may give clues about how the underlying cancerous disease will respond to one or more of the treatment options. Feature categories (or types of features) may be predetermined and in particular learned. One or more features may be integrated in a so-called feature vector, or in this case in a histopathology feature vector. One or more features may also be designated as digital biomarkers characterizing the whole slide image according to one or more treatment options.

[0025] One or more features may describe or may be based on the morphology of the tissue (and in particular the cancer cells depicted in the whole slide image). In other words, one or more of the features may include morphological information, such as the shape, size, color, composition, or tumor infiltration of tumor cells; specific markers, such as for information about proliferation state or immune checkpoint inhibition; information about the response to existing therapies, such as the percentage of necrotic tissue (e.g., no necrotic tissue means no response and indicates that a certain treatment is ineffective). One or more of the features may be tangible to a human user, that is, may be directly interpretable to a human user. In addition, one or more of the features may be abstract and have no obvious meaning to a human user. According to some examples, one or more features may be semantic features.

[0026] According to some examples, generating a treatment response prediction can include comparing one or more features to one or more of the treatment options and / or selecting at least one treatment option from the plurality of treatment options based on the one or more features. The comparison can include comparing the one or more features to corresponding characteristics associated with the respective treatment options. In other words, the one or more features can be classified according to the one or more treatment options to generate a treatment response prediction.

[0027] According to some examples, the prediction function can be configured to map features to higher-level features or meta-features that characterize the whole slide image according to one or more categories related to selecting a treatment option, such as sensitivity to the treatment option, probability of recurrence of the treatment option, survival rate, or other more abstract meta-features. The process of mapping can also be referred to as classification. Based on these meta-features, a treatment response prediction can be provided. According to some examples, this can involve selecting one or more treatment options based on the meta-features. According to some examples, one or more of the meta-features can be provided as part of the treatment response prediction.

[0028] According to some examples, the method further comprises defining a plurality of tiles, respectively, in the whole slide image, wherein the prediction function is configured to extract one or more features for each of at least a portion of the tiles, and to generate a treatment response prediction based on the extracted features, in particular by calculating an average feature from the extracted features.

[0029] Tiles can be thought of as sub-regions or patches of the corresponding whole slide image. Tiles also include image data in the form of pixels. Tiles are typically much smaller than the underlying whole slide image. Tiles can be defined by placing a regular grid on the underlying whole slide image. Each tile can be of the same size. Using tile-by-tile processing, a large whole slide image can be split into smaller parts that are easier to handle from a computational perspective.

[0030] According to some examples, the prediction function can be a trained function.

[0031] Typically, the trained function mimics the cognitive functions of humans associated with other human minds. In particular, through training based on training data, the trained function can adapt to new environments and can detect and extrapolate patterns. Other terms for the trained function can be a machine-learned function or machine learning function (abbreviated as ML function), a trained machine learning model, a trained mapping specification, a mapping specification with trained parameters, a function with trained parameters, an artificial intelligence-based algorithm or a machine learning algorithm (abbreviated as ML algorithm).

[0032] Typically, the parameters of the trained function can be adjusted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of the trained function can be iteratively adjusted through several steps of training.

[0033] In particular, the trained function may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, a genetic algorithm and / or an association rule. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0034] As a general rule, a neural network comprises a number of layers. The input to the first layer is the input image (in this case, a whole slide image). Each layer may apply one or more mathematical operations to the input values, e.g., convolution, nonlinear excitation, pooling operations, to give just a few examples. The input to a layer may be formed by the output of the previous layer (feedforward). Feedback values ​​or skip-connection skipping layers are possible.

[0035] A neural network for digital pathology (i.e., as may be included in the prediction function) may be configured to infer at least one feature from a whole slide image. In particular, a convolutional neural network may be employed as a trained function in the prediction function. For example, ResNet-18 may be used, see Ayyachamy, Swarnambiga, et al., "Medical image retrieval using Resnet-18." Medical Imaging 2019: Imaging Informatics for Healthcare, Research, and Applications. Vol. 10954. International Society for Optics and Photonics, 2019. In addition, VGG-16 or VGG-19 CNN can be used, see: Mateen, Muhammad et al., "Fundus image classification using VGG-19 architecture with PCA and SVD." Symmetry 11.1 (2019): 1; or Kaur, Taranjit and Tapan Kumar Gandhi. "Automated brain image classification based on VGG-16 and transfer learning." 2019 International Conference on Information Technology (ICIT). IEEE, 2019.

[0036] According to some examples, the prediction function may include a convolutional neural network. A convolutional neural network is a neural network that uses a convolution operation instead of a general matrix multiplication in at least one of its layers (the so-called "convolutional layer"). Specifically, the convolutional layer performs a dot product of one or more convolution kernels with the input data / image of the convolutional layer, wherein the entries of the one or more convolution kernels are parameters or weights adjusted by training. In particular, Frobenius inner products and ReLU activation functions may be used. The convolutional neural network may include additional layers, such as pooling layers, fully connected layers, and normalization layers.

[0037] According to some examples, the prediction function may include a ResNet architecture. ResNet stands for Residual Neural Network. ResNet is a deep learning model in which a weight layer learns a residual function with reference to the layer input. A residual network is a network with skip connections that perform identity mapping, merged with the layer output by addition. This enables ResNets with tens or hundreds of layers to be easily trained and approach better accuracy.

[0038] ResNet may include multiple convolutional layers for processing. According to some examples, the prediction function may include a so-called ResNet-50. ResNet-50 is a 50-layer convolutional neural network that includes 48 convolutional layers, one maximum pooling layer, and one average pooling layer.

[0039] The training data for training the prediction function can include a data set of multiple patients (>1000) with a specific cancerous disease. The data set for each patient can include at least one whole slide image and information about the treatment options followed and the corresponding results. The results can be long-term survival, prognosis, disease progression, etc. that can be used for treatment response prediction. The results and thus the ground truth used for training can be annotated by human experts.

[0040] Patient data can be clustered according to the treatment options that were followed respectively.For each treatment option, a prediction function can then be learned to predict the treatment response based on the corresponding whole slide image and optionally any additional data.

[0041] According to some examples, providing the treatment response prediction may include outputting the treatment response prediction to a user via a user interface.

[0042] According to one aspect, the one or more treatment options include at least one of: radiotherapy treatment, immunotherapy treatment, chemotherapy treatment, and / or treatment by surgical intervention.

[0043] Each treatment option may specify the parameters of the corresponding treatment. In the case of a radiotherapy treatment, this may include radiotherapy settings, such as dose, beam geometry, dose rate, etc., which may be input into the radiotherapy treatment modality. In the case of a chemotherapy treatment, this may include the type and dose of medication to be administered to the patient. In the case of an immunotherapy treatment, this may include the type of immunomodulator to be used.

[0044] According to some examples, the prediction functionality is configured to derive treatment response predictions for a plurality of predefined treatment options.

[0045] This has the advantage that the prediction function can be tuned more specifically, which can lead to better prediction results.

[0046] Predefined treatment options may relate to standard treatment options discussed for a particular cancerous disease. Treatment options may be predefined based on medical guidelines, previous treatments of other patients with the same cancerous disease, insurance policies, etc.

[0047] According to some examples, the prediction function may be configured to select at least one treatment option from a plurality of predefined treatment options based on the whole slide image (or any supplemental information), and provide the selected at least one treatment option as a treatment response prediction.

[0048] According to some examples, treatment options may include a combination of the foregoing options, for example, a combination treatment including a radiotherapy treatment and a chemotherapy treatment.

[0049] According to some examples, the treatment options may include a treatment plan involving different treatment steps over the course of a predetermined period of time. The treatment steps may involve radiotherapy treatment, immunotherapy treatment, chemotherapy treatment, treatment by surgical intervention, and / or any combination thereof.

[0050] According to other examples, multiple treatment options may be combined based on treatment response predictions to form a treatment plan, and the treatment plan may be provided / output to a user.

[0051] According to some examples, treatment options can be combined based on treatment response predictions of the treatment plan (e.g., including an estimated success rate for each treatment option) and clinical compatibility. Combining different treatment options makes it possible to select the most promising form of therapy for the patient, which may be advantageous if there is no clear preference based on treatment response predictions.

[0052] According to some examples, the treatment response prediction includes: predicted sensitivity of the cancerous disease to one or more treatment options, probability of disease recurrence based on one or more treatment options, and / or predicted survival rate of a patient utilizing one or more treatment options.

[0053] Sensitivity and / or probability of recurrence may be provided as a number, such as a percentage, or in the form of more abstract categories, such as high, medium, or low.

[0054] Survival rates may be provided as the probability of a patient surviving within a given time frame, which may be measured in years.

[0055] According to some examples, calculating the predicted survival rate can be based on comparison of the patient to a cohort of patients with the same cancerous disease who have undergone treatment (eg, based on the whole slide image, features derived therefrom, or any supplementary information).

[0056] The inventors have recognized that the morphology of cells shown in a whole slide image can often provide an indication of how the cells will respond to a certain therapy (e.g., to the ionizing radiation involved in radiation therapy). Predicting the sensitivity (another word might be vulnerability) of a tumor to treatment options based on whole slide images can therefore indicate how promising a certain therapy is and serve as a valuable metric for prediction of treatment response.

[0057] Regarding the probability of recurrence, the inventors have realized that this metric depends on both the morphology of the cancerous disease visible in the whole slide image and the treatment options. This metric can provide additional information for the user to decide on the best treatment option(s) for the patient.

[0058] The advantage of survival probability is that it takes into account the probability of disease recurrence and the side effects of treatment options.

[0059] According to one aspect, the method further includes: providing a certainty calculation module, the certainty calculation module being configured to output a certainty metric for a corresponding treatment response prediction, the certainty metric measuring the confidence of the corresponding treatment response prediction; applying the certainty calculation module to obtain the certainty metric for the treatment response prediction; and providing the certainty metric.

[0060] The certainty metric (or confidence metric, or confidence level, or certainty score, or trust level) can indicate how confident the prediction function is in predicting treatment response for the underlying whole slide image.

[0061] According to some examples, the certainty metric may include a value or a set of values ​​(e.g., a vector) that indicates the degree of certainty or uncertainty that a treatment response prediction or a portion of a treatment response prediction is affected according to a model or algorithm and / or generally indicates the quality of a treatment response prediction. According to some examples, obtaining the certainty metric may include assigning a value / certainty metric to each element of the treatment response prediction.

[0062] According to some examples, a certainty measure may be provided for the treatment response prediction as a whole. According to other examples, a certainty measure may be provided for each element of the treatment response prediction, in particular for each treatment option included in the treatment response prediction.

[0063] The deterministic computational module may be applied to whole slide images and / or treatment response predictions and / or any intermediate processing results of the prediction function.

[0064] By introducing a measure of certainty, the quality of treatment response predictions can be assessed and quantified. This allows users to make more informed decisions about available treatment options.

[0065] According to some examples, the deterministic calculation module may be a trained function that may have been trained to derive at least one deterministic metric for treatment response prediction based on whole slide images and / or behavior of a prediction function.

[0066] According to one aspect, the deterministic calculation module can be integrated into the prediction function.

[0067] Specifically, the certainty measure can be based on the output layer of the prediction function. For example, a possible confidence measure can be the output of the logit function, which can be implemented in the final layer of the artificial neural network of the corresponding prediction function. More complex confidence measures can be calculated based on a probabilistic interpretation of the output of the logit function.

[0068] Alternatively, a separate module may be used to obtain the certainty metric. Specifically, such a module may include a softmax unit configured to determine at least one certainty metric based on a softmax action selection, and / or such a module includes a Bayesian unit configured to determine at least one certainty metric based on a Bayesian inference.

[0069] In softmax action selection, statistical methods are employed. For example, one can assume that the logit output by the logit function follows a Boltzmann distribution, and then associate a confidence measure with the likelihood found using logistic regression. Another possibility is to use Bayesian inference. There are different Bayesian-based methods available that are said to produce very strong confidence levels associated with the predictions of artificial neural networks.

[0070] According to some examples, the deterministic computation module includes a Bayesian model.

[0071] Such Bayesian models are based on artificial Bayesian networks. A Bayesian network is a probabilistic graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph, with variables as vertices or nodes and dependencies as edges. Given a random variable at a parent node, each node of the network is assigned to the conditional probability distribution of the random variable it represents. The distribution can be arbitrary. However, discrete or normal distributions are often used. The parents of a vertex are those vertices with edges leading to that vertex. A Bayesian network is used to represent the common probability distribution of all variables involved as compactly as possible using conditional dependencies. The conditional dependencies (independence) of a subset of variables are combined with prior knowledge.

[0072] The Bayesian model is trained to identify uncertain prediction results, and thereby to identify corresponding uncertain data bases. Uncertain data are, for example, whole slide images that the prediction function cannot classify with sufficient reliability according to one or more treatment options. Uncertainty can be determined by extracting feature data at different layers of the Bayesian model and determining a statistical distribution based on these data. Uncertainty can then be calculated as the standard deviation of the mentioned statistical distribution.

[0073] Bayesian models enable the modeling of causal relationships between random variables using conditional probabilities. They therefore help understand how a single inaccuracy or failure propagates through the layers of a trained function and how it can significantly affect the outcome. Bayesian models are therefore able to reliably predict the certainty or confidence of the output provided by a complex system such as a prediction function.

[0074] According to some examples, the deterministic computing module includes:

[0075] -Feature pyramid pooling algorithm,

[0076] - Monte Carlo dropout algorithm,

[0077] - Monte Carlo depth algorithm, and / or

[0078] -Deep Ensemble algorithm.

[0079] The pyramid pooling method is described in "Pyramid attention network for semantic segmentation" by Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang, arXiv-preprint arXiv:1805.10180. Feature pyramid pooling is used to perform a spatial pyramid attention structure on high-level outputs, combined with global pooling to learn better feature representations and a global attention upsampling module on each decoder layer to provide global context as a guide for low-level features to select category localization details. The feature pyramid pooling algorithm is used with features as input data and output data Figure 1 The input data and output data mentioned above that are appropriately prepared as labeled data can also be used as training data for training the feature pyramid pooling algorithm.

[0080] Monte Carlo dropout and / or Monte Carlo deep algorithms and / or deep ensemble learning methods are used to determine the uncertain data. The Monte Carlo dropout method involves extracting feature data in different layers of the artificial neural network. The extracted feature data is used to determine the statistical distribution of the features, also known as sample variance. Based on the statistical distribution, the uncertainty can be calculated as a standard deviation.

[0081] The Monte Carlo Dropout algorithm receives feature maps of different levels of the uncertainty model as input data and generates feature maps as output data. The mentioned input data and output data appropriately prepared as labeled data can also be used as training data for training the Monte Carlo Dropout algorithm.

[0082] The Monte Carlo deep algorithm also receives the feature map of the image as input data and output data. The mentioned input data and output data appropriately prepared as labeled data can also be used as training data for training the Monte Carlo deep algorithm.

[0083] Deep integration works with multiple, for example 15, base learners. These base learners include AI-based models working in parallel, preferably artificial neural networks. Bayesian inference is then performed on these base learners. Based on the statistics of different results of different base learners, sample variance is used as an uncertainty measure. The deep integration algorithm is performed using a set of input data including different subsets of image data or feature maps to feed the so-called base learners. In addition, the output data also includes different subsets of image data or feature maps, or alternatively, includes a map indicating the sample variance of the input data. The input data and output data mentioned, which are appropriately prepared as labeled data, can also be used as training data for training the deep integration algorithm.

[0084] The preceding examples for deterministic modules enable confidence values ​​to be reliably attributed to treatment response predictions. Therefore, different examples may bring different advantages, such as Monte Carlo dropout or Monte Carlo deep being more intertwined with the trained function. Therefore, they may require less training. Deep ensembles are more independent and require less modification of the trained function.

[0085] According to some examples, the deterministic calculation module includes a novelty detection algorithm, which is configured to detect input and / or output parameters of the prediction function that are not fully represented in the training of the prediction function, and the step of obtaining the certainty metric includes calculating the certainty metric based on the detection results of the novelty detection algorithm.

[0086] In other words, the novelty detection algorithm may be configured to detect that in particular input data, in particular input image data, does not use the predictive function in the sense that input data of that type has been largely or not seen at all during training.

[0087] According to some examples, novelty detection may be configured to detect inconsistencies in the behavior of a prediction function.

[0088] According to some examples, the novelty detection algorithm can again be based on a Bayesian model. Specifically, for example, through unsupervised learning, an endogenous and unsupervised Bayesian model representing the interrelationship between the image data and the processing of the image data by the prediction function can be trained. This Bayesian model can then be used to detect inconsistencies in the behavior of the prediction function.

[0089] According to some examples, a novelty detection algorithm comprises an autoencoder neural network algorithm comprising: an encoder branch configured to encode input data; and a decoder branch configured to decode an encoded representation of the input data output by the encoder branch, wherein anomalies are detected based on a reconstruction error between the input data and output data output by the decoder branch.

[0090] By using novelty detection algorithms, situations where the prediction function is less familiar or unfamiliar can be automatically identified. In such situations, the prediction function is likely to be unable to provide confident results. Therefore, identifying these situations leads to a reliable certainty measure.

[0091] According to one aspect, the step of providing includes outputting the confidence measure together with the treatment response prediction to a user via a user interface.

[0092] By providing a confidence measure, the user is provided with direct information about how "trusted" the results of the automated process are. Thus, the user may be provided with additional information for deciding the best treatment for the patient.

[0093] According to one aspect, the method further comprises determining whether the treatment response prediction is conclusive based on the confidence measure, and if the treatment response prediction is not conclusive: determining a piece of information about the patient suitable for making the treatment response prediction conclusive.

[0094] The information segment may be information that is not included in the whole slide image. In particular, the information segment may relate to non-whole slide image data. In particular, the information segment may relate to supplementary information about the patient. The information segment may relate to information from the patient's electronic health record that may potentially help increase the confidence of the prediction of treatment response. The information segment may relate to hypothetical information that may or may not yet be available.

[0095] According to some examples, the information segment may include at least one of the following: a medical report of the patient; an additional whole slide image different from the whole slide image, wherein, optionally, the additional whole slide image may be stained with a different histopathological stain than the whole slide image; radiological image data; demographic information of the patient such as age, gender, comorbidities, risk factors, etc.; laboratory data and / or omics data.

[0096] By identifying pieces of information, not only why a treatment response prediction was not conclusive, but also what could theoretically make it more conclusive, this helps users decide on treatment options and decide on next steps.

[0097] According to other examples, the information segment may relate to anamnestic findings of the patient that the user may be aware of. Thus, the method may include formulating an anamnestic question for the information segment, providing the anamnestic question to the user via a user interface, (optionally) receiving an answer from the user via the user interface, (optionally) modifying the treatment response prediction (e.g., by entering the answer in a prediction function), (optionally) modifying the certainty metric (e.g., by applying a certainty calculation module), and (optionally) providing the modified treatment response prediction and / or certainty metric (e.g., to the user via the user interface).

[0098] According to one aspect, the method further comprises providing an indication of the information piece to the user via the user interface. This not only makes the process more transparent, but also provides a further actionable result with which the user can initiate the necessary steps for obtaining the information piece.

[0099] According to some examples, determining whether the treatment response prediction is conclusive may include comparing a confidence metric to a predetermined threshold, and determining whether the treatment response prediction is conclusive based on the comparison. Specifically, if the confidence metric is found to be below the predetermined threshold in the comparison step, the treatment response prediction may be determined to be inconclusive.

[0100] According to some examples, the uncertainty calculation module may be configured to output the information segment, and the information segment is determined based on applying the uncertainty calculation module.

[0101] Specifically, as explained above, the uncertainty calculation module can be configured to determine the uncertain data basis in the processing of the prediction function. In turn, this can provide information about data elements that will make the prediction more certain. For example, if the prediction function has been trained to base the prediction on a certain data element that is not available for the current situation, the uncertainty calculation module can determine that the data element does not exist and what consequences this has on the certainty of the prediction. On this basis, it can then be determined whether the missing data element needs to be marked.

[0102] According to some examples, the prediction functionality is configured to provide a treatment response prediction in the form of a probability set, wherein each probability in the probability set reflects a probability that the cancerous disease is responsive to one of a plurality of predefined treatment options, and the information segments are selected based on the probability set, and in particular based on one or more statistical properties of a probability distribution of the probability set.

[0103] The statistical characteristics may include at least one of entropy, variance, skewness, kurtosis, etc. of the probability distribution.

[0104] According to some examples, the information segments are selected based on their predictive impact on the probability. In particular, this may involve determining hypothetical changes to one or more statistical properties if the information segments would be available.

[0105] According to some examples, the predictive impact is defined as the predictive information gain associated with each information segment, where the predictive information gain of the information segment is specifically given by:

[0106] MAX{P(Res1)[H(PD|WSI)-–H(PD|WSI,PI=Res1)],P(Res2)[H(PD|WSI)

[0107] -H(PD|TPD,PI=Res2)],…,P(Res n )[H(PD|WSI)-–H(PD|WSI,PI=Res n )}

[0108] in:

[0109] H(PD|WSI) is the initial entropy H of the probability distribution PD of the calculated probabilities of the treatment options based on the whole slide image;

[0110] H(PD|WSI, PI=Res1) is the entropy H of the probability distribution PD assuming that the information fragment PI indicates the first result Res1;

[0111] H(PD|WSI, PI=Res2) is the entropy H of the probability distribution PD assuming that the information fragment PI indicates the second result Res2;

[0112] H(PD|WSI,PI=Res n ) is assumed that the information segment PI indicates the nth result Res n The entropy H of the probability distribution PD;

[0113] Question Q; and

[0114] P(Res1),P(Res2),…,P(Resn ) are the probabilities that the information fragments indicate the first result to the nth result, respectively.

[0115] According to the above embodiments, entropy is used as a statistical property of a probability distribution and a MAX function is used as a measure for weighting possible outcomes indicated by pieces of information. By this, a reliable and reproducible way of identifying relevant pieces of information can be defined.

[0116] According to one aspect, the method further includes accessing a healthcare information system including an electronic medical record of the patient, retrieving the information segments from the healthcare information system, processing the information segments to provide an updated prediction of treatment response and optionally an updated confidence metric, and providing the updated prediction of treatment response and optionally an updated confidence metric.

[0117] Processing the information segments may include inputting the information segments into a prediction function and / or a deterministic computation module.

[0118] By automatically acquiring and processing pieces of information, the user can be further reassured that he is being provided with an improved prediction of treatment response without intervention.

[0119] According to some examples, the piece of information includes radiological image data depicting manifestations of a cancerous disease within a patient's body.

[0120] According to some examples, the step of processing the information fragments includes: providing an image analysis module configured to extract radiological observable objects from the radiological image data, applying the image analysis module to the radiological image data to obtain the radiological observable objects, and additionally generating an updated treatment response prediction and optionally an updated confidence metric based on the radiological observable objects. Thus, the radiological observable objects can have a form as described herein.

[0121] According to one aspect, the method further comprises accessing a healthcare information system comprising healthcare data (electronic medical record) of the patient, determining whether the information segment is available in the healthcare information system, and if the information segment is not available: generating a corresponding notification to a user.

[0122] With this notification, the user can be automatically pointed to information that will help reach a more certain prediction of treatment response. This enables the user to initiate the necessary steps to obtain the information piece.

[0123] According to one aspect, the piece of information relates to a medical examination of the patient, and the method further comprises generating instructions for performing the medical examination, and providing the instructions.

[0124] The instructions may include human-readable and / or machine-readable instructions.For example, the instructions may include steps and / or parameters for performing a medical examination.

[0125] The medical examination may include an examination at the patient's body to obtain one or more measurements. The measurements may be used as information fragments. The examination may include a radiological examination, a laboratory examination of a patient sample, a genomic examination of a patient sample, a (further) histopathological examination, or a general examination. The medical examination may be performed with a corresponding modality configured to perform the medical examination, such as a radiographic imaging modality, a laboratory testing modality, a genomic testing modality, etc. The modality may be connected to a healthcare information system.

[0126] According to some examples, the instructions include human-readable instructions, and the instructions are provided to a user via a user interface.

[0127] According to some examples, the instructions include machine-readable instructions that are configured to control a modality configured to perform a medical examination to perform the medical examination, and the method also includes: entering / forwarding the instructions in / to the modality (so as to control the modality to perform the medical examination), and / or providing the instructions to a user via a user interface (for review and further use).

[0128] According to some examples, a medical examination may include generating an additional whole slide image that is different from the whole slide image.

[0129] The additional whole slide image may differ from the whole slide image in that it has been stained with a different histopathology stain than that used for the whole slide image.

[0130] By basing the prediction on differently stained whole slide images, complementary image information is provided that can improve the prediction. Since each histopathology stain highlights specific structures in a tissue section, the histopathology stain of the additional whole slide image can be used to specifically highlight structures that help rule out or confirm certain treatment options. In particular, the histopathology stain of the additional whole slide image can be a stain that is less commonly used in clinical routine than the histopathology stain of the whole slide image and can only be obtained on request if the measure of certainty is too low.

[0131] According to some examples, the histopathological stain of the whole slide image is an H&E stain and / or the histopathological stain of the additional whole slide image is an immunohistochemical stain, in particular a stain comprising a keratin targeted biomarker.

[0132] In this regard, H&E stands for hematoxylin and eosin. Hematoxylin stains the cell nucleus, and eosin stains the extracellular matrix and cytoplasm. H&E is the most widely used stain in digital pathology, which also makes the initial prediction widely applicable.

[0133] Immunohistochemical stains, or IHC stains for short, involve the process of selectively identifying antigens (proteins) in cells of a tissue section by utilizing the principle that antibodies specifically bind to antigens in biological tissues. Through this, structures that cannot be reached using other stains such as H&E can be highlighted. Therefore, additional readouts can be provided to further elaborate on the prediction of treatment response.

[0134] In particular, IHC stains can include biomarkers (e.g., in the form of antibodies) configured to target (cytoskeleton) keratin. Keratin forms a part for cytoskeleton and limits the mechanical properties of cells. Therefore, the abundance of keratin constitutes a good tumor marker, and can characterize the "invasiveness" of tumor cells, because keratin expression levels often change in tumor cells. Specifically, IHC stains can include keratin biomarkers of targeting different keratin forms (e.g., CK-5, CK-8, CK-14, CK-18 (wherein, "CK" represents "cytoskeleton keratin")). In itself, IHC stains can include different or additional biomarkers, such as p63 and AMACR biomarkers.

[0135] According to some examples, the additional whole slide images may be from the same tissue sample, in particular the same tissue portion, of the patient. For example, the whole slide image and the additional whole slide image may relate to consecutive proximal slices of the tissue sample. Furthermore, it is conceivable that the additional whole slide images are prepared from different tissue samples, in particular from different locations in the patient's body. For example, the whole slide image and the additional whole slide image may relate to different biopsies.

[0136] Furthermore, according to some examples, generating an additional whole slide image may include: providing an image processing function configured to simulate, based on image data of the whole slide image, image data depicting a tissue section stained with a histopathology stain different from the histopathology stain of the whole slide image; and generating the additional whole slide image by applying the image processing function to the whole slide image.

[0137] The image processing function can be a machine-learned function that has been trained according to the above-described task of simulating additional whole slide images. In particular, the image processing function can be an image-to-image neural network, or more specifically, a convolutional image-to-image neural network. According to some examples, the image processing function can be a generative adversarial network (GAN). According to some examples, the image processing function can be trained based on whole slide images and "real" additional whole slide images that can be provided as described herein.

[0138] Providing additional whole slide images through the image processing function allows the automatic generation of additional whole slide images without the need for manual processing steps.

[0139] According to some examples, a medical examination may include generating radiological image data.

[0140] Thus, the instructions may include an imaging protocol for generating radiological image data. The imaging protocol may include an indication of the patient's body region to be imaged with the imaging modality, the type of imaging modality, and / or settings and parameters for controlling the imaging modality and for preparing the patient (e.g., in terms of placement of the patient or administration of contrast media, etc.). Furthermore, the instructions may include scheduling a medical examination in a healthcare information system, for example, by assigning a medical imaging modality and, if necessary, by assigning a technician to perform the examination.

[0141] Radiological image data may be processed as described herein. In particular, one or more radiologically observable objects may be obtained as described herein.

[0142] According to some examples, the medical examination may include a laboratory test, in particular at least one of a blood test, a flow cytometry test, and / or a gene sequencing test.

[0143] The mentioned laboratory tests may provide additional insights into the aggressiveness of the cancerous disease and / or the patient's constitution, which may help further validate treatment options and improve uncertainty measures.

[0144] Blood tests can include identifying inflammatory markers in a patient's blood sample, such as C-reactive protein (CRP) values ​​or alkaline phosphatase (AP) values, the amount of red blood cells, or tumor markers. Thus, inflammatory markers can indicate the presence of inflammation associated with a cancerous disease and give insight into the progression of the disease. The same applies to AP values. Likewise, too many or too few red blood cells may suggest specific problems associated with a cancerous disease, particularly bone marrow problems. Similarly, tumor markers can further accurately determine the cancerous disease and, thereby, the available treatment options.

[0145] With flow cytometry, physical and chemical properties of cell populations of samples taken from patients can be measured. In particular, so-called FACS tests (FACS = fluorescence activated cell sorting) can be used. Flow cytometry can be used to diagnose health diseases, in particular blood cancers, and is therefore effective in further characterizing the health status of patients. In turn, flow cytometric measurements can be used to validate treatment options and improve definitive measurements.

[0146] Using, for example, gene sequencing or (genome sequencing or cancer genome sequencing) of a patient's tumor tissue sample, specific and unique changes that the patient has experienced to develop a cancerous disease can be identified. Based on these changes, a more personalized treatment strategy can be defined, and treatment options can be further specified.

[0147] According to one aspect, the method further comprises obtaining supplemental information associated with the patient from a healthcare information system, wherein the predictive function is further configured to derive a treatment response prediction additionally based on the supplemental information, and the step of applying comprises applying the predictive function additionally to the supplemental information.

[0148] According to some examples, the supplemental information may be information not included in the whole slide image. For example, the supplemental information may include another whole slide image as described herein. In particular, the information fragment may relate to non-whole slide image data. For example, the supplemental information may include radiology image data as described herein. According to another example, the supplemental information may be non-image data of the patient. According to some examples, the supplemental information may include data from a laboratory test as described herein. According to some examples, the supplemental information includes structured and / or unstructured natural language text.

[0149] According to some examples, the supplemental information includes one or more of the following elements:

[0150] - Additional whole slide images of the same or different tissue samples / biopsies from the patient, and / or

[0151] - Radiological image data, and / or

[0152] - Data from laboratory tests (laboratory data), and / or

[0153] - The patient's previous medical reports, and / or

[0154] - prior knowledge of at least one type of cancerous disease, and / or

[0155] - Medical guidelines applicable to the patient, and / or

[0156] -The patient's electronic health record.

[0157] According to some examples, the electronic health record may include a patient's medical history, including any pre-existing conditions, comorbidities, risk factors, referral letters, demographic information such as age or gender, and the like.

[0158] Obtaining the supplemental information may include querying a healthcare information system such as a HIS (hospital information system), a LIS (laboratory information system), an EMR system (electronic medical record system), etc. to obtain the patient's supplemental information. For example, such supplemental information may be obtained in the form of one or more EMR files (electronic medical record files). In addition, querying the healthcare information system may be based on a patient identifier that electronically identifies the patient in the system, such as an ID or a patient name.

[0159] By taking into account the supplementary information, improved predictions of treatment responses can be achieved, as the supplementary information can provide additional insights into the cancer the patient has. This, in turn, can help further narrow down possible treatment options. This reduces user workload, as the user does not need to manually take into account the supplementary information, and can allow for more targeted treatment protocols that are better suited to specific patients and cancer types.

[0160] According to some examples, the method further includes extracting a supplemental feature vector from the supplemental information, wherein the prediction function is further configured to generate a histopathology feature vector based on the whole slide image, and determine a treatment response prediction based on a combined feature vector of the supplemental feature vector and the histopathology feature vector.

[0161] According to some examples, the method also includes providing multiple different data processing modules, each of which is configured to extract corresponding observable objects from specific data elements of the supplementary information, selecting at least one data processing module from the multiple different data processing modules based on the supplementary information (or specific data elements included in the supplementary information), applying the selected data analysis module to the supplementary information (specific data elements) to obtain corresponding observable objects, wherein the prediction function is also configured to derive a treatment response prediction additionally based on the corresponding (one or more) observable objects, and the applying step includes additionally applying the prediction function to the corresponding observable objects.

[0162] According to some examples, corresponding (one or more) observable objects may respectively include corresponding feature vectors (which may be supplementary features or part of supplementary feature vectors), wherein the prediction function is further configured to generate a histopathology feature vector based on the whole slide image, and determine a treatment response prediction based on the corresponding (one or more) feature vectors and a combined feature vector of the histopathology feature vector.

[0163] Specific data elements may include laboratory data, radiology image data, structured or unstructured text such as medical reports, etc. The data processing modules may include dedicated data processing modules, such as image or text processing modules. For example, the data processing modules may include a semantic data extraction module and / or a radiology image analysis module as described herein. By providing different modules, the method is able to process multimodal data, which can make treatment response prediction more accurate.

[0164] When processing information fragments, substantially the same steps can be applied. In particular, the selection can be performed based on a specific information fragment.

[0165] According to one aspect, the method further comprises providing a (semantic) data extraction module configured to search the electronic medical records of the patient to obtain the supplementary information (or information fragments), the (semantic) data extraction module preferably comprising a large language model, wherein the step of obtaining comprises accessing the electronic medical records of the patient in a healthcare information system, and applying the data extraction module to the electronic medical records in order to obtain the supplementary information (information fragments).

[0166] The (semantic) data extraction module may generally be configured to identify relevant information for providing a prediction of a treatment response in a corpus of structured or unstructured text data associated with a patient. The (semantic) data extraction module may be rule-based, for example, by having been programmed to search for a set of predefined keywords. Furthermore, the (semantic) data extraction module may include a trained function that has been trained to acquire knowledge of the inherent syntax and semantics of human languages.

[0167] A large language model (LLM) is a language model characterized by its large size. In particular, a large language model can be based on a transformer architecture. According to some examples, a large language model can include or can be based on models available on the filing date, such as GPT models (e.g., GPT-3.5 and GPT-4), PaLM, LLaMa, LLaMa 2, Falcon, Whisper, etc.

[0168] With semantic data extraction models, text data can be automatically searched for relevant information. This also enables the utilization of unstructured text data, which would otherwise be difficult to process. The advantage of large language models is that they can effectively handle synonyms, complex semantic relations, and long-range dependencies in the input data. In addition, transformer networks are able to process data in parallel, which saves computational resources in inference.

[0169] According to one aspect, the supplemental information includes radiological image data depicting manifestations of the cancerous disease in the patient's body.

[0170] The radiology image data may include one or more medical image data sets. The medical image data set may be a two-dimensional image. Furthermore, the medical image data set may be a three-dimensional image. Furthermore, the medical image may be a four-dimensional image, in which there are three spatial dimensions and one dimension like time. Furthermore, the medical image data set may include a plurality of individual medical images.

[0171] Medical image data sets include image data, for example, in the form of a two-dimensional or three-dimensional array of pixels or voxels. Such an array of pixels or voxels may represent color, intensity, absorption or other parameters that vary with two-dimensional or three-dimensional position and may be obtained, for example, by appropriate processing of measurement signals obtained by a medical imaging modality or image scanning facility.

[0172] A medical image dataset may be a radiological image dataset depicting a body part of a patient. Thus, it may contain two-dimensional or three-dimensional image data of a body part of a patient. The medical image may represent an image volume or a cross-section through an image volume. A body part of a patient may be included in the image volume.

[0173] Medical imaging modalities correspond to systems for generating or producing medical image data. For example, medical imaging modalities may be computed tomography systems (CT systems), magnetic resonance systems (MR systems), angiography (or C-arm X-ray) systems, positron emission tomography systems (PET systems), etc. Specifically, computed tomography is a widely used imaging method and utilizes "hard" X-rays generated and detected by a spatially rotating instrument. The resulting attenuation data (also referred to as raw data) is processed by computer analysis software to produce detailed images of the internal structures of the patient's body parts. The resulting image set is referred to as a CT scan, which may constitute a plurality of series of continuous images to present the internal anatomical structures in a cross section perpendicular to the axis of the human body. Magnetic resonance imaging (MRI), to provide another example, is an advanced medical imaging technique that utilizes the effect of a magnetic field affecting the motion of protons. In an MRI machine, the detector is an antenna, and the signal is analyzed by a computer to create a detailed image of the internal structure in any part of the human body.

[0174] Medical images may be stored in a standard image format such as the Digital Imaging and Communications in Medicine (DICOM) format and stored in a memory or computer storage system such as a Picture Archiving and Communication System (PACS), a Radiology Information System (RIS), etc. Whenever DICOM is mentioned herein, it should be understood that DICOM refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, for example, according to the DICOM PS3.12020c standard (or any later or earlier version of the standard).

[0175] A manifestation may be an image of a body part or organ affected by a cancerous disease. Thus, radiological image data may show lesions, tumors, nodules, etc. as manifestations of a disease.

[0176] By considering radiological image data, orthogonal information about the disease can be gathered. Specifically, radiological image data will show the tumor at a coarser level but in relation to other body compartments of the patient, such as organs, compared to a full slide image. In addition, it may be possible to estimate the overall morphology of the cancerous growth and whether and how the primary tumor has spread.

[0177] According to one aspect, the method further comprises providing an image analysis module configured to extract radiologically observable objects from radiological image data, applying the image analysis module to the radiological image data to obtain the radiologically observable objects, wherein the prediction function is further configured to derive a treatment response prediction additionally based on the radiologically observable objects, and the step of applying comprises additionally applying the prediction function to the radiologically observable objects.

[0178] According to some examples, the radiological observable object includes a radiological feature vector and / or a plurality of different radiological observable objects may be integrated in the radiological feature vector, wherein the prediction function is further configured to generate a histopathological feature vector based on the whole slide image, and determine a treatment response prediction based on a combined feature vector of the radiological feature vector and the histopathological feature vector.

[0179] By deriving radiological observables from radiological image data, relevant features related to treatment response prediction can be automatically extracted. The observables can be learned or predetermined.

[0180] According to one aspect, the radiological observable includes at least one of a tumor burden of the patient in the radiological image data, a visual characteristic of a lesion depicted in the radiological image data, and / or a temporal evolution of a lesion depicted in the radiological image data.

[0181] Tumor burden indicates how severely a patient's body or body part is affected by the tumor.

[0182] For example, tumor burden can be detected by applying a suitable contrast agent in a PET-CT examination and then analyzing the image data performed by an appropriately configured image analysis module. A PET-CT examination is a combination of positron emission tomography (PET) and computed tomography (CT). PET can visualize metabolic processes in the body. To this end, a trace amount of radiolabeled substances is administered to the patient. These substances are distributed throughout the body and accumulate in certain tissues such as tumors.

[0183] For example, FDG (F-18 deoxyglucose) can be used as a marker. FDG is a glucose molecule labeled with radioactive fluorine. Since cancer cells have an increased glucose consumption compared to healthy cells, FDG accumulates more in diseased cells. With the help of a PET camera device, the different distributions in the cells of the body are visible. Even cancer lesions measuring a few millimeters can be detected in this way.

[0184] The combined PET-CT device makes it possible to perform computed tomography scans almost simultaneously. By combining the two methods, cell areas with high glucose metabolic activity can be detected and thus provide information about the tumor burden.

[0185] Visual characteristics may include the size and / or shape and / or morphology and / or contour of the lesion. For example, the morphology may include the level of calcification or coagulation of the lesion, while the contour may include the level of spiculation or lobulation.

[0186] The temporal evolution may include the growth or shrinkage of the size of the lesion over time. Furthermore, the temporal evolution may include the new formation of lesions or the disappearance of lesions.

[0187] Radiological observables involve values ​​that are difficult for users to estimate or even quantify. At the same time, these values ​​have been shown to be very valuable for determining treatment response. Therefore, the automatic extraction of such values ​​is particularly beneficial for further supporting users and making reasonable treatment response predictions.

[0188] According to one aspect, the treatment options include at least two different radiation therapy options, the at least two different radiation therapy options differing in at least one of the following:

[0189] - dose distribution,

[0190] - Dose threshold limits,

[0191] - dose rate,

[0192] - fractionation,

[0193] - use of proton / photon or electron radiation, and / or

[0194] - Use of coplanar or non-coplanar beams.

[0195] In other words, decision support is provided that allows deciding on the most promising radiotherapy option. The inventors realized that tumor characteristics that can be derived from whole slide images can give insights into how the tumor will react to different kinds of ionizing radiation. For example, if the tumor is generally sensitive to ionizing radiation, a lower dose may be sufficient, which has fewer side effects.

[0196] According to some examples, the treatment response prediction includes one or more parameters for a radiation therapy treatment of the patient, wherein the one or more parameters include at least one of: dose distribution, dose threshold limit, dose rate, fractionation, use of proton / photon or electron radiation, and / or use of coplanar or non-coplanar beams.

[0197] According to one aspect, the one or more treatment options include radiation therapy treatment at a dose rate greater than 40 Gy / second.

[0198] Dose rates greater than 40 Gy / second are often referred to as FLASH radiotherapy. FLASH is a technique that involves delivering ultra-high dose rate radiation to a target. In preclinical models, FLASH has been shown to reduce radiation-induced toxicity in healthy tissues compared to conventional radiation therapy without compromising the anti-cancer effects of the treatment. The FLASH effect specifies that normal tissues are not harmed at ultra-high dose rates (>40 Gy / second) compared to conventional dose rate (~0.1 Gy / second) irradiation while maintaining tumor control, and has the potential to increase the therapeutic rate of radiotherapy. Notably, non-cancerous cells exposed to ultra-high dose rate radiotherapy are more likely to survive than non-cancerous cells exposed to conventional dose rates. FLASH may have advantages when it comes to organ at risk (OAR) dose limitations or any type of normal tissue toxicity. FLASH may have disadvantages when it comes to fractionated schemes and multi-beam therapy.

[0199] By predicting the treatment response to ultra-high dose rates in radiotherapy, an indication is provided to the user if FLASH is problematic in being used to treat the patient. In particular, the combination of information extracted from whole slide images (which can provide a readout of the sensitivity of tumor cells to FLASH) and radiology image data (which can provide information about the size and location of the tumor) as input data for prediction has been shown to produce confident results in this regard.

[0200] According to one aspect, a system for providing a treatment response prediction for a patient with a cancerous disease is provided, the system comprising an interface unit and a computing unit. The interface unit is configured to obtain a whole slide image of the patient, the whole slide image showing a tissue sample associated with the cancerous disease. The computing unit is configured to: host a prediction function configured to derive a treatment response prediction for one or more treatment options from the whole slide image; apply the prediction function to the whole slide image to generate a treatment response prediction; and provide the treatment response prediction via the interface unit.

[0201] The computing unit can be implemented as a data processing system or a part of a data processing system. Such a data processing system can, for example, include a cloud computing system, a computer network, a computer, a tablet computer, a smart phone, etc. The computing unit can include hardware and / or software. The hardware can include, for example, one or more processors, one or more memories and combinations thereof. One or more memories can store instructions for performing method steps according to the present invention. The hardware can be configurable and / or operable by software. Typically, all units, subunits or modules can, for example, be connected via a network or a corresponding interface at least temporarily exchange data with each other. Therefore, each unit can be positioned to be separated from each other.

[0202] The interface unit may include an interface for data exchange with a local server or a central web server via an Internet connection in order to receive the whole slide image and / or the prediction function. The interface unit may be further adapted to interface with one or more users of the system, for example, by displaying the results of the processing of the computing unit to the user (e.g., in a graphical user interface) or by allowing the user to adjust parameters for image processing or visualization in order to select a whole slide image for processing and / or review treatment options.

[0203] According to other aspects, the present invention also relates to a digital pathology image analysis system, the digital pathology image analysis system comprising at least one of the above systems and a healthcare information system, the healthcare information system being configured to acquire, store and / or forward at least a whole slide image and optionally also other data such as supplementary information. Thereby, the interface unit is configured to receive the whole slide image from the digital pathology image system.

[0204] According to some examples, the healthcare information system includes one or more archive stations for storing whole slide images and / or supplemental information, which can be implemented as cloud storage devices or local or extended storage devices. In addition, the healthcare information system can include one or more imaging modalities, such as slide scanning devices or radiology imaging modalities, etc.

[0205] According to one aspect, a computer-implemented method for providing a prediction function is provided. The method includes multiple steps. A first step involves providing a machine-learnable prediction function configured to provide a treatment response prediction for a treatment option for treating a cancerous disease based on a whole slide image. Another step involves providing input training data, the input training data comprising a whole slide showing a tissue sample associated with a patient's cancerous disease. Another step involves providing training output data, the training output data comprising a patient's treatment response to at least one treatment option. Another step involves inputting the training input data into the prediction function to obtain an intermediate treatment response prediction. Another step involves comparing the intermediate treatment response prediction with the output training data. Another step involves adjusting the prediction function based on the comparing step. Another step involves providing an adjusted prediction function.

[0206] According to another aspect, a training system for providing a prediction function for providing a prediction of a treatment response is provided. The system includes an interface implemented to receive the prediction function and input and output training data. The system also includes a computing unit configured to run the prediction function. The computing unit is further configured to perform a training step according to the method for providing a prediction function described herein.

[0207] According to another aspect, the invention relates to a computer program product comprising program elements which, when loaded into a memory of a computing unit, cause a computing unit of a system to perform one or more steps according to the method aspects and examples described herein.

[0208] According to another aspect, the invention relates to a computer-readable medium having stored thereon program elements which can be read and executed by a computing unit of a system according to one or more method aspects and examples described herein when the program elements are executed by the computing unit.

[0209] Implementing the invention by means of a computer program product and / or a computer readable medium has the advantage that already existing provisioning systems can be easily adapted by software updates in order to work as proposed by the invention.

[0210] A computer program product may be, for example, a computer program or include another element next to the computer program itself. The other element may be hardware, such as a storage device on which the computer program is stored, a hardware key for using the computer program, etc.; and / or software, such as documentation or a software key for using the computer program. The computer program product may also include development materials, a runtime system and / or a database or library. The computer program product may be distributed among several computer instances.

[0211] Independent of grammatical usage of the term, individuals with either masculine or feminine identities are included within the term. BRIEF DESCRIPTION OF THE DRAWINGS

[0212] The characteristics, features and advantages of the above invention and the ways to achieve them become clearer and easier to understand according to the following description of the embodiments, which will be described in detail with reference to the accompanying drawings. The following description does not limit the invention to the embodiments included. The same parts, parts or steps may be marked with the same reference numerals in different drawings. Generally, the drawings are not drawn to scale. In the following drawings:

[0213] Figure 1 schematically depicts a system for providing treatment response prediction according to an embodiment,

[0214] Figure 2 schematically depicts a method for providing treatment response prediction according to an embodiment,

[0215] Figure 3 schematically depicts a data flow in a method for providing treatment response prediction according to an embodiment,

[0216] Figure 4 schematically depicts a method for providing prediction of treatment response according to another embodiment,

[0217] Figure 5 schematically depicts a data flow in a method for providing treatment response prediction according to another embodiment,

[0218] Figure 6 schematically depicts a method for providing prediction of treatment response according to another embodiment,

[0219] Figure 7 schematically depicts a data flow in a method for providing treatment response prediction according to another embodiment,

[0220] Figure 8 schematically depicts a prediction function for predicting a treatment response according to an embodiment,

[0221] Fig. 9 schematically depicts a method for providing a prediction function for predicting a treatment response according to an embodiment, and

[0222] Fig.10 A system for providing a prediction function for predicting a treatment response according to an embodiment is schematically depicted. DETAILED DESCRIPTION

[0223] Figure 1A system 1 for providing a treatment response prediction TRP associated with a whole slide image WSI to a user U is depicted. In this regard, the system 1 is adapted to perform a method according to one or more embodiments, for example, as described with reference to Figures 2 to 7 Further described. In addition, the system 1 can be configured to host, provide and / or apply Figure 8 A prediction function PF is shown which has been adapted to derive a treatment response prediction TRP from the whole slide image WSI and optionally additional supplementary information SI.

[0224] According to some examples, a user U of the system 1 (in the sense of an operator controlling the system 1 ) may typically relate to a healthcare professional, such as a physician, clinician, technician, pathologist, radiologist, or the like.

[0225] The system 1 includes a user interface 10 (as part of an interface unit) and a processing system 20 (as part of a computing unit). In addition, the system 1 may include a medical information system 40 or be connected to a medical information system 40. The medical information system 40 may generally be configured to acquire and / or store and / or forward whole slide images WSI, F-WSI, and supplementary information SI. For example, the medical information system 40 may include one or more archive / review stations (not shown) for whole slide images WSI. The archive / review station may be implemented by one or more databases. In particular, the archive / review station may be implemented in the form of one or more cloud storage modules. Alternatively, the archive / review station may be implemented as a local storage device or an extended storage device, for example, as a PACS (Picture Archiving and Communication System). According to some examples, the medical information system 40 may also include one or more medical imaging modalities (not shown) for generating a whole slide image WSI, such as a slide scanner, but also radiological imaging modalities, such as a computed tomography system, a magnetic resonance system, an angiography (or C-arm X-ray) system, a positron emission tomography system, a mammography system, an X-ray system, etc.

[0226] Figure 3 , Figure 5 and Figure 7 A schematic representation of a whole slide image WSI is depicted. Typically, a whole slide image WSI, F-WSI depicts a tissue section prepared from a tissue sample of a patient. The section can be prepared by cutting the tissue sample into micrometer thick sections. The whole slide image WSI depicts a tissue section under microscopic observation. Thus, the tissue section can show the fine tissue structure of the tissue sample, and in particular the cells or cell structures contained in the tissue sample.

[0227] Typically, a whole slide image will show different tissue types, such as healthy tissue with healthy cells, cancerous tissue with cancer cells, and other tissues, such as amorphous tissue or necrotic tissue. In particular, cancer cells can indicate how a tumor will respond to one or more therapy options. This is because the morphology of cells and cancerous areas provides an indication of the aggressiveness of the tumor and the sensitivity of the tumor to various treatment options. In addition, healthy cells may also be relevant to choosing the right treatment option, because it can be inferred from the morphology of healthy tissue how it will be damaged under a specific treatment option and therefore what the most likely side effects of the treatment option are.

[0228] The whole slide image WSI may depict tissue stained with a specific histopathological stain in order to highlight features in the tissue slide for examination. The most common stain is the H&E (Hematoxylin and Eosin) stain. Therefore, the whole slide image WSI will usually be based on H&E staining. Furthermore, different stains may also be considered, for example, in the form of additional whole slide images F-WSI of the patient. These additional whole slide images F-WSI may be part of the supplementary information SI, which may be retrieved to protect the treatment response prediction TRP.

[0229] The supplementary information SI (or associated data) may be any data that provides additional information related to the patient and / or the whole slide image WSI. The supplementary information SI may include image data, such as an additional whole slide image F-WSI of the patient, which is, for example, acquired at a different time point than the whole slide image WSI and / or prepared in a different manner, in particular using a different histopathological stain. In addition, the image data may include radiological image data RID acquired using a radiological imaging modality. In addition, the supplementary information SI may include non-image data or data with mixed-type content, which includes medical images and non-image content such as text. The non-image data may relate to non-image examination results, such as laboratory data, vital signs records (including, for example, ECG data, blood pressure values, ventilation parameters, oxygen saturation levels), etc. In addition, the supplementary information SI may include structured and unstructured medical text reports related to a previous examination or current examination of the patient. In addition, the non-image data may include personal information of the patient, such as gender, age, weight, insurance details, etc.

[0230] The supplementary information SI may be obtained in the form of one or more electronic medical records EMR of the patient. The supplementary information SI may be stored in the healthcare information system 40. For example, the supplementary information SI may be stored in a dedicated database of the healthcare information system 40 (e.g., a laboratory information system (LIS) or an electronic health / medical record database).

[0231] The radiology image data RID may be a three-dimensional image data set acquired, for example, using an X-ray system, a computer tomography system, or a magnetic resonance imaging system or other system. The image information may be encoded in a three-dimensional array of m times n times p voxels. The radiology image data set may include a plurality of image slices stacked in a stacking direction to span the image volume covered by the radiology image data RID.

[0232] Furthermore, the radiology image data RID may comprise two-dimensional medical image data, wherein image information is encoded in an array of m by n pixels. According to some examples, these two-dimensional medical images may have been extracted from the three-dimensional radiology image data RID.

[0233] Radiology image data RID as well as whole slide images WSI, F-WSI may be formatted according to the DICOM format. DICOM (=Digital Imaging and Communications in Medicine) is an open standard for the communication and management of medical imaging information and related data in healthcare informatics. DICOM can be used to store and transmit medical images and associated information, enabling the integration of medical imaging devices such as scanners, servers, workstations, printers, network hardware, and picture archiving and communication systems (PACS). DICOM is widely adopted by clinical syndicates, hospitals, and for smaller applications such as doctors' offices or clinics. DICOM data objects consist of a plurality of attributes, including items such as patient name, ID, etc., and also include special attributes containing image pixel data and metadata extracted from the image data.

[0234] The user interface 10 may include a display unit and an input unit. The user interface 10 may be implemented by a mobile device such as a smart phone or a tablet computer. In addition, the user interface 10 may be implemented as a workstation in the form of a desktop PC or a laptop computer. The input unit may be integrated in the display unit, for example, in the form of a touch screen. As an alternative or in addition, the input unit may include a keyboard, a mouse or a digital pen and any combination thereof. The display unit may be configured to display a representation of the whole slide image WSI, an excerpt from the supplementary information SI, one or more treatment response predictions TRP and / or deterministic metrics CM.

[0235] The user interface 10 may also include an interface computing unit configured to execute at least one software component for serving the display unit and the input unit so as to provide a graphical user interface for allowing the user U to select the case of the target patient to be reviewed and to perform various inputs. In addition, the interface computing unit may be configured to communicate with the medical information system 40 or the processing system 20 to receive the whole slide image WSI and any supplementary information SI. The user U may activate the software component via the user interface 10, and may obtain the software component, for example, by downloading the software component from an Internet application store. According to an example, the software component may also be a client-server computer program in the form of a web application running in a web browser. The interface computing unit may be a general-purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application-specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other now known devices for processing image data. The user interface 10 may also be implemented as a client.

[0236] The processing system 20 may comprise subunits 21 to 25 configured to process the whole slide image WSI, F-WSI and the supplementary information SI in order to provide a treatment response prediction TRP for the particular treatment option in question for treating the cancerous disease of the patient.

[0237] The processing system 20 may be a processor. The processor may be a general purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other currently known devices for processing image data. The processor may be a single device or multiple devices operating in series, in parallel, or individually. The processor may be the main processor of a computer such as a laptop or a desktop computer, or may be a processor for processing some tasks in a larger system such as a medical information system or a server. The processor is configured by instructions, design, hardware, and / or software to perform the steps discussed herein. The processing system 20 may be included in the user interface 10. Alternatively, the processing system 20 may include a real or virtual computer group, such as a so-called "cluster" or "cloud". Such a server system may be a central server, such as a cloud server; or a local server, such as a server located at a hospital or radiology site. In addition, the processing system 20 may include a memory such as a RAM for temporarily loading the medical image data set MIDS. According to some examples, such a memory may also be included in the user interface 10.

[0238] Subunit 21 is a data retrieval unit. Subunit 21 is configured to access and search the medical information system 40 to obtain information fragments PI and / or supplementary information SI that can make the prediction more conclusive. Specifically, subunit 21 can be configured to formulate search queries and parse them to the medical information system 40. According to some examples, the search query can be based on the electronic patient identifier of the patient in the medical information system 40.

[0239] The subunit 22 can be conceived as a data analysis unit. The subunit 22 is configured to derive observables RA-VF, HP-VF, SI-VF from the available data WSI, F-WSI, EMR, RID. Specifically, the subunit 22 can be configured to extract histopathological feature vectors HP-VF from the whole slide images WSI, F-WSI, extract radiological feature vectors RA-VF from the radiological image data RID and / or extract data feature vectors SI-VF from the patient's electronic medical record EMR. To this end, the subunit 22 can be configured to host and run multiple dedicated processing modules RA-IAM, HP-IAM, LLM of the prediction function PF.

[0240] Subunit 23 may be conceived as a prediction unit. Subunit 23 may be configured to acquire observables provided by data analysis unit 22 and classify these observables according to one or more treatment options. On this basis, the patient's response to the treatment options may be predicted and provided as a treatment response prediction TRP. Subunit 23 may be configured to host and run a prediction module PM of a prediction function PF.

[0241] The subunit 24 can be conceived as a deterministic computing unit. The subunit 24 can be configured to calculate a deterministic measure CM, which indicates how confident the treatment response prediction TRP is. The subunit 24 can also be configured to determine additional data that can make the treatment response prediction TRP more credible. This additional data is also designated as an information segment PI. The information segment PI can be part of the supplementary information SI available to the patient. In addition, the information segment PI can be hypothetical information that does not yet exist. The subunit 24 can be configured to host and run a correspondingly configured deterministic computing module CCM of the prediction function PF.

[0242] The subunit 25 may be configured as an input / output unit. The subunit 25 may be configured to provide a treatment response prediction TRP for display to the user U via the user interface 10. In addition, the subunit 25 may be configured to check whether the information segment PI as determined by the subunit 24 is available in the healthcare information system 40. In addition, the subunit 25 may be configured to generate an instruction INS, which is directed to (newly) generate the information segment PI, wherein the instruction INS may be directed to the user U as a human-interpretable output, or directed to an appropriate modality for generating the information segment PI as a machine-readable control signal. In addition, the subunit 25 may be configured to output the instruction INS to the user U or to an appropriate modality via the user interface 10.

[0243] The designation of the different subunits 21 to 25 will be explained by way of example and not limitation. Thus, the subunits 21 to 25 may be integrated to form a single unit (e.g., in the form of a "computing unit"), or may be implemented by computer code segments configured to execute corresponding method steps running on a processor or the like of the processing system 20. The same is true for the interface computing unit. Each subunit 21 to 25 and the interface computing unit may be individually connected to other subunits and / or other components of the system 1 that require data exchange to perform method steps.

[0244] The processing system 20 and (one or more) interface computing units together can constitute the computing unit of the system 1. It is worth noting that the layout of the computing unit, i.e., the physical distribution of the interface computing unit and the subunits 21 to 25, is arbitrary in principle. Specifically, the processing system 20 can also be integrated in the user interface 10. As already mentioned, the processing system 20 can alternatively be implemented as a server system, such as a cloud server, or a local server such as a server located at a hospital or a radiation site. According to such an implementation, the user interface 10 can be designated as a "front end" or "client" facing the user U, while the processing system 20 can be conceived as a "back end" or server. For example, the communication between the user interface 10 and the processing system 20 can be performed using the https protocol. The computing power of the system can be distributed between the server and the client (i.e., the user interface 10). In a "thin client" system, most of the computing power exists at the server. In a "thick client" system, more computing power and possible data exist on the client.

[0245] The various components of the system 1 can be connected to each other at least temporarily for data transmission and / or exchange. The user interface 10 communicates with the processing system 20 via the (data) interface 26 to exchange, for example, whole slide images WSI, treatment response predictions TRP or instructions INS for generating missing information PI. For example, the processing system 20 can be activated based on a request, wherein the request is sent by the user interface 10. In addition, the processing system 20 can communicate with the medical information system 40 to retrieve the case of the target patient. As an alternative or in addition, the user interface 10 can communicate directly with the medical information system 40. The medical information system 40 can also be activated based on a request, wherein the request is sent by the processing system 20 and / or the user interface 10. The data interface 26 for data exchange can be implemented as a hardware interface or a software interface, such as a PCI bus, USB or FireWire. Data transmission can be implemented using a network connection. The network can be implemented as a local area network (LAN), such as an intranet or a wide area network (WAN). The network connection is preferably wireless, for example as a wireless LAN (WLAN or Wi-Fi). In addition, the network can include a combination of different network examples. The interface 26 for data exchange together with the components for interfacing with the user U are regarded as constituting an interface unit of the system 1 .

[0246] Figure 2 A method for providing a text summary TS according to an embodiment is described. The corresponding data flow is Figure 3 The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated.

[0247] At step S10, a whole slide image WSI is provided. This may involve selecting a whole slide image WSI from a plurality of cases stored, for example, in the medical information system 40. The selection may be performed manually by the user U, for example by selecting appropriate image data in a graphical user interface running in the user interface 10. Alternatively, the whole slide image WSI may be provided to the computing unit by the user U by uploading the whole slide image WSI to the processing system 20.

[0248] A whole slide image WSI is prepared from a tissue sample of a patient's cancerous disease and may be stained with an H&E stain.

[0249] At step S20, a prediction function PF is provided. The prediction function PF is configured to process the whole slide image WSI, and optionally to process supplementary information SI such as the patient's radiological image data RID or electronic medical record EMR, so as to obtain clues about how the patient or the cancer suffered by the patient will respond to multiple different treatment options. To this end, the prediction function PF may include multiple different data processing modules RA-IAM, HP-IAM, LLM, which are respectively configured to extract characteristics from different input data WSI, F-WSI, RID, EMR. These characteristics may also be referred to as observable objects of the corresponding input data. The observable objects may include multiple features respectively. Therefore, each observable object may have the form of feature vectors HP-VF, RA-VF, SI-VF. The feature vectors HP-VF, RA-VF, SI-VF may be mapped to one or more treatment options by a classification or prediction module PM. In other words, a mapping indicating the goodness of the matching of (one or more) feature vectors with one or more treatment options is performed. Based on the mapping, a treatment response prediction TRP may be calculated.

[0250] Treatment options may be predefined and relate to different treatments that are typically administered to patients with a cancerous disease. Treatment options may include radiation therapy treatment options, chemotherapy treatment options, surgical treatment options, immunotherapy treatment options, or any combination of the foregoing. Additional different treatment options may specify different parameters for the same type of treatment, such as drug dosages and / or radiation therapy parameters.

[0251] Further details on the prediction function PF will be combined with Figure 8 The prediction function PF may be provided by keeping it available in a memory of the processing system 20, for example, as an executable computer code.

[0252] At step S30, a prediction function PF is applied to the whole slide image WSI so as to automatically process the whole slide image WSI to provide a treatment response prediction TRP as a form of patient diagnosis. Specifically, this may involve extracting a histopathology feature vector HP-VF from the whole slide image WSI and classifying the feature vector HP-VF according to at least one, preferably multiple, treatment options. The classification or mapping may indicate how well the histopathology feature vector HP-VF "matches" the various treatment options. This "match" may then quantify the likelihood that the corresponding treatment option will be successful for the patient and may form the basis of the treatment response prediction TRP.

[0253] At step S40, a treatment response prediction TRP may be provided. This may involve showing the treatment response prediction TRP to the user U in the user interface 10 (eg, in a suitable graphical user interface).

[0254] According to some examples, the treatment response prediction TRP may include a recommendation for a certain treatment option, such as a treatment option with the highest success rate. Specifically, the treatment response prediction TRP may include a recommendation for a radiotherapy treatment involving a specific dose distribution, dose threshold limit, dose rate, fractionation, use of proton / photon or electron radiation, and / or use of coplanar or non-coplanar beams. In addition, the treatment response prediction TRP may include a recommendation for a radiotherapy treatment involving FLASH.

[0255] Figure 4 A method for providing a treatment response prediction TRP according to an embodiment is depicted. The corresponding data flow is Figure 5 The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated. Specifically, according to Figure 4 and Figure 5 The method is directed to an alternative workflow involving the calculation and use of a deterministic metric CM. Figure 2 and Figure 4 Method steps denoted by the same reference numerals in the accompanying drawings correspond to the same method steps.

[0256] The certainty metric CM can indicate how confident a treatment response predicts TRP. For example, the certainty metric CM can be a real number selected from the interval between 0 and 1. Figure 8 To further illustrate, the deterministic metric CM may be provided by a dedicated deterministic computation module CCM.

[0257] At step S21, a deterministic calculation module CCM is provided. Step S21 may be an optional sub-step of step S20.

[0258] The deterministic calculation module CCM may be part of the prediction function PF. The deterministic calculation module CCM may be configured to derive a deterministic metric CM based on processing of the actual prediction module PM of the prediction function. Furthermore, the deterministic calculation module CCM may be configured to base the deterministic metric CM on the quality of the input data, in particular the whole slide image WSI. To this end, the prediction function may comprise a quality assessment module QAM configured to determine the quality of the input whole slide image WSI.

[0259] At step S31, a deterministic computation module CCM is applied. Step S31 may be an optional sub-step of step S30. This may mean that the deterministic computation module CCM is applied to the input and / or output data, i.e. the whole slide image WSI and any supplementary information SI and / or treatment response predictions, and / or any intermediate processing results such as feature vectors RA-VF, SI-VF, HP-VF, and / or the outputs of the individual layers of the image or data analysis modules RA-IAM, HP-IAM, LLM and / or the prediction module PM.

[0260] At step S41 , the certainty measure CM is provided. Step S41 may be an optional sub-step of step S40. At step S41 , the certainty measure CM may be provided to the user U via the user interface 10. Furthermore, the certainty measure CM may be provided for further automated processing steps.

[0261] One of the automated further processing steps S50 may involve determining whether the treatment response prediction TRP is conclusive, or in other words, sufficiently certain. To this end, the certainty measure CM may be compared to a predefined threshold. If the certainty measure CM is below the threshold, the treatment response prediction TRP may be considered inconclusive.

[0262] Subsequently, optionally at optional step S60 it may be determined what (hypothetical) pieces of information PI may be used to improve the deterministic measure CM. Systematically, this may be derived from the processing of the deterministic calculation module CCM at step S31. This is because the deterministic calculation module CCM is configured to observe the various processing steps of the prediction function PF and is therefore able to identify which part of the processing is performing poorly.

[0263] For example, if the whole slide image WSI is not of sufficient quality, an additional whole slide image F-WSI may be identified as potentially helpful in determining the treatment response prediction TRP. Furthermore, if the treatment response prediction TRP is undecided between two treatment options, it may be derived which non-image information may be suitable for deriving a clearer prediction. This may be a counter-indication for a certain treatment that may be derived from the patient's electronic medical record EMR or demographic information about the patient such as age, sex, co-morbidity or a form of therapy that has been administered. Furthermore, at step S60, the radiology image data RID may be considered to contribute to improving the confidence measure CM. This is because the radiology image data RID may, for example, give insight into how much the tumor has spread.

[0264] At optional step S70, it may be determined whether the information segment PI determined at step S60 is already available in the healthcare information system 40. To this end, the healthcare information system 40 may be accessed and queried to obtain the patient's electronic medical record EMR. Subsequently, the electronic medical record EMR may be searched to obtain the information segment PI. According to some examples, this may include applying a suitably configured data analysis module LLM to the electronic medical record EMR. For example, the data analysis module LLM may include a large language module as described herein.

[0265] If the information piece PI is already available, the information piece PI may be pulled from the healthcare information system 40 at step S71 .

[0266] If the information fragment PI is not available, a corresponding notification may be provided to the user U at step S72. By this, the user U may initiate the step of collecting the information fragment PI himself.

[0267] As an alternative, instructions INS for generating the piece of information PI may be automatically generated as a form of further assistance. The instructions INS may be directed to the user U, which enables the user U to obtain the piece of information PI. For example, this may relate to a protocol for preparing a further whole slide image F-WSI or an imaging protocol for a radiological examination of a patient.

[0268] Furthermore, the instructions INS may directly include machine-readable instructions that may be used directly to control devices in the healthcare information system 40. For example, such devices may include a medical imaging modality or a scheduling system for performing scans or preparing whole slide images WSI.

[0269] According to some examples, such instructions INS can be generated by a large language model.

[0270] At step S74 , the instruction INS may be provided. This may involve inputting the instruction INS in corresponding devices in the healthcare information system 40 for controlling these devices. Furthermore, this may involve outputting the instruction INS to the user U via the user interface 10 .

[0271] At optional step S80, any piece of information PI obtained may be processed by inputting it in a prediction function PF in order to obtain an updated treatment response prediction TRP and optionally an updated certainty measure CM.

[0272] Subsequently, at optional step S90, an updated treatment response prediction TRP and / or an updated certainty measure CM may be provided substantially as described in conjunction with steps S40 and S41.

[0273] Figure 6A method for providing a treatment response prediction TRP according to an embodiment is depicted. The corresponding data flow is Figure 7 The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated. Specifically, according to Figure 6 and Figure 7 The method is directed to an alternative workflow involving the use of supplementary information SI to generate treatment response predictions TRP. Figure 2 and Figure 6 The method steps indicated by the same reference numerals in the figure correspond to the same method steps. Figure 6 The method steps can be Figure 4 A combination of method steps.

[0274] In an optional sub-step of step S10 , supplementary information SI may be obtained for processing by the prediction function PF together with the whole slide image WSI. The supplementary information SI may be obtained by querying the medical information system 40 , for example based on an electronic patient identifier of the patient in the medical information system 40 .

[0275] In order to process different kinds of supplementary information SI, the prediction function PF may include dedicated processing modules LLM, RA-IAM in addition to the image analysis module HP-IAM for processing histopathology whole slide images WSI, F-WSI. The modules may be configured to extract observable objects in the form of feature vectors RA-VF, HP-VF, SI-VF, respectively, which may be cascaded and mapped to different treatment options in the prediction module PM. Different processing modules may be selected and applied based on the content of the supplementary information SI.

[0276] For example, the processing module may include a data extraction module LLM that may be provided at step S22. The data extraction module LLM may be configured to find and extract the sought information from structured or unstructured text data such as medical reports. To this end, the data extraction module LLM may be a semantic data extraction module, in particular a large language model.

[0277] Furthermore, the processing module may comprise an image analysis module RA-IAM for extracting relevant observable objects from the radiology image data RID.At step S23 a radiology image analysis module RA-IAM may be provided.

[0278] At step S32, a radiological image analysis module RA-IAM is applied to the radiological image data RID to obtain a corresponding radiological observable object RA-VF. The radiological observable object RA-VF may be in the form of a feature vector RA-VF.

[0279] Similarly, at step S33, the data extraction module LLM is applied to the patient's electronic medical record EMR to obtain the corresponding data observable object SI-VF. The data observable object SI-VF may be in the form of a feature vector SI-VF.

[0280] Subsequently, any data observables SI-VF, RA-VF thus obtained may be input in the prediction module PM of the prediction function PF together with the histopathology observables HP-VF derived from the whole slide images WSI, F-WSI.

[0281] exist Figure 8 In FIG. 4 , a schematic representation of a prediction function PF according to an embodiment is shown.

[0282] like Figure 8 As shown, the prediction function PF may include three branches: a branch for processing whole slide images WSI, F-WSI using the histopathology image analysis module HP-IAM, a branch for processing radiology image data RID using the radiology image processing module RA-IMA, and a branch for processing electronic medical records EMR (particularly, non-image data such as structured or unstructured text) using the data extraction module LLM. The "radiology branch" and the "EMR branch" are optional.

[0283] The respective processing modules RA-IAM, HP-IAM, LLM in each branch may be configured to extract observables relevant to predicting the patient's treatment response from the corresponding input data. The observables may be in the form of feature vectors RA-VF, HP-VF, SI-VF, respectively. The feature vectors RA-VF, HP-VF, SI-VF may be learned by machine learning techniques and / or may be predefined. The latter may be questioned in the following case: it is already clear that a certain information is particularly relevant, such as age or gender (in the case of electronic medical records EMR) or the size and distribution of lesions (in the case of radiological image data RID).

[0284] For the histopathology image analysis module HP-IAM, there are various implementations that can be used to extract the feature vector HP-VF from the whole slide images F-WSI, WSI. For example, CNN can be used. For example, ResNet, in particular ResNet-18 or ResNet-50 can be used to convert the image data of the whole slide images WSI, F-WSI into the feature vector HP-VF.

[0285] In order to achieve efficient processing, each whole slide image WSI, F-WSI can be divided into N tiles or patches. For each tile, an intermediate feature vector i-HP-VF can be extracted, and the intermediate feature vectors i-HP-VF can be averaged to obtain the final feature vector HP-VF of the corresponding whole slide image WSI, F-WSI.

[0286] Also with regard to the radiological image analysis module RA-IAM, there are multiple established options for deriving a feature vector RA-VF from the radiological image data RID. Preferably, the radiological image analysis module RA-IAM also includes ResNet, in particular ResNet-50. The radiological image analysis module RA-IAM can be trained specifically for the prediction function PF. Alternatively or additionally, the radiological image analysis module RA-IAM may include pre-configured algorithms that are not specific to the prediction function PF (i.e., trained outside the prediction function PF) and can generally be used for reading radiological images, such as lesion detection and quantification algorithms. The results of these pre-configured algorithms can also be fed into the radiological feature vector RA-VF.

[0287] The data extraction module LLM may include a transformer architecture. In particular, the data extraction module LLM may include a large language module LLM. The transformer network is a neural network architecture that typically includes an encoder, a decoder, or both an encoder and a decoder. In some cases, the encoder and / or decoder are composed of several corresponding encoding layers and decoding layers, respectively. There is an attention mechanism in each encoding layer and decoding layer. The attention mechanism - sometimes called self-attention - relates a data item (such as a word) in a series of data items to other data items in the series. For example, the self-attention mechanism allows the model to examine a phrase in a sentence and determine the relative importance of other phrases in the sentence to the examined word.

[0288] In particular, the encoder can be configured to convert the input (text) into a digital representation. The digital representation can include a vector for each input token (e.g., each word). The encoder can be configured to implement an attention mechanism so that each vector of a token is influenced by other tokens in the input. In particular, the encoder can be configured so that the representation parses the desired output of the transformer network.

[0289] In particular, the decoder can be configured to transform an input into a sequence of output tokens. In particular, the decoder can be configured to implement a masked self-attention mechanism so that each vector of a token is only influenced by other tokens on one side of the sequence. Furthermore, the decoder can be autoregressive, meaning that intermediate results (e.g., previously predicted token sequences) are fed back.

[0290] According to some examples, the output of the encoder is input into the decoder.In addition, the transformer network may include a classification module or unit configured to map the output of the encoder or decoder to a set of learning outputs, such as a desired feature vector SI-VF.

[0291] The training of the transformer model according to some examples can occur in two stages - a pre-training stage and a fine-tuning stage. In the pre-training stage, the transformer model can be trained on a large data corpus to learn the potential semantics of the problem. Such a pre-trained transformer model can be used for different languages. For some applications described herein, fine-tuning can include further training the transformer network with medical text and / or medical ontologies (e.g., RADLEX and / or SNOMED) with expert annotated meanings. In particular, using the latter, the transformer model according to some examples can learn typical relationships and synonyms of medical expressions.

[0292] For a review of Transformer Networks, see Vaswani et al., “Attention Is All You Need”, arXiv: 1706.03762, 12 June 2017, which is incorporated herein by reference in its entirety.

[0293] According to some examples, the data extraction module LLM is configured to derive a plurality of predefined features SI-VFs from the electronic medical record EMR, which are to be added to the feature vector(s) extracted from the whole slide image WSI, F-WSI and / or the radiology image data RID. The predefined features SI-VFs may be predetermined influencing factors that will influence the decision of the human user U regarding one or more treatment options, such as age, gender, relevant laboratory values ​​(e.g. PSA in the case of prostate cancer), comorbidities, already performed therapies and their success, metastasis, etc.

[0294] The individual feature vectors RA-VF, HP-VF, SI-VF can be cascaded to form a combined feature vector VF. The combined feature vector VF can then be input into a prediction module PM. For the prediction module PM, a regressor model can be used. The prediction module PM can output a prediction of a treatment response TRP for one or more treatment options. For example, treatment options can be pre-configured based on training data.

[0295] Optionally, the prediction function PF may comprise a module for determining a certainty measure CM of the treatment response prediction TRP. This may comprise a quality assessment module QAM (which may also be based on ResNet) configured to detect tiles and / or entire whole slide images WSI, F-WSI that are not suitable for feature vector HP-VF extraction (e.g. because they do not include tissue or because of incomplete staining). For example, the certainty calculation module CCM may use the number of rejected tiles (or, in general, the result of the quality assessment module QAM) to calculate the certainty measure CM.

[0296] In addition, the prediction function PF may include a deterministic calculation module CCM. Specifically, the deterministic calculation module CCM may be based on a Bayesian network with Monte Carlo dropout in each module of the prediction function. Taking the histopathology image analysis module HP-IAM as an example, the whole slide image WSI, F-WSI may be input to a convolutional encoder-decoder. The convolutional encoder-decoder includes a plurality of convolutional layers, a batch normalization layer, and a rectified linear activation function layer. These layers form different levels with different feature resolutions, also referred to as convolutional blocks. A new level with a lower resolution may be generated using a pooling layer, and a new level with a higher resolution may be generated using an upsampling layer. In addition, some dropout layers are each inserted between a pooling layer and a subsequent convolutional layer or between a pooling layer and an upsampling layer or between a convolutional layer and an upsampling layer. The dropout layer adds dropout after each convolutional block. These dropouts are used to determine sample variance as an uncertainty measure. In a similar manner, other data processing modules LLM, RA-IAM, and prediction module PM may also be supplemented with dropout layers.

[0297] Finally, the dropout is normalized by the Softmax layer. Then, random dropout is performed to obtain random dropout samples. The mean of the random dropout samples is used to determine the prediction result, and the variance of the random dropout samples is used to determine the uncertainty model including the uncertain data in order to obtain the certainty measure CM of each data item processed.

[0298] In other words, a Monte Carlo dropout scheme can be implemented where the deterministic computation module CCM is integrated in the module of the prediction function PF in the form of a dropout layer, a Softmax layer, and random dropout. By this, dropout regularization is utilized to produce more reliable predictions and estimate the prediction certainty measure CM.

[0299] However, there are also other possible implementations in which the deterministic calculation module CCM is more separated from the actual prediction function PF, for example in the form of an independent module that processes the output of the prediction function and / or the raw input data of the prediction function in order to estimate the deterministic measure CM. For example, such an independent module may take the form of a novelty detection algorithm as described herein.

[0300] Fig. 9 A method for providing a prediction function PF for predicting a treatment response prediction TRP based on input data is depicted. Thus, the input data comprises at least one whole slide image WSI and optionally supplementary information SI. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention.

[0301] The first step T10 aims to provide a plurality of training data sets. The training data sets respectively include a training input data set and a corresponding verified output data set. The training input data set is preferably the same type of input data to be processed by the prediction function PF that is deployed and easy to train. Therefore, the training input data sets all also include at least a whole slide image WSI. In addition, the training input data set may include supplementary information SI as described above. The verified output data set includes an indication of the (one or more) treatment options followed and the (one or more) treatment responses verified. Specifically, the training data set may include a data set of a plurality of patients (>1000) suffering from a specific cancerous disease. The data set of each patient may respectively include at least one WSI and information about the treatment options followed and the corresponding treatment responses. The treatment response may be long-term survival rate, prognosis, disease progression, etc. The training data sets may be clustered according to the treatment options followed respectively. For each treatment option, the prediction function PF may then learn to predict the treatment response based on the corresponding whole slide image WSI and optionally the supplementary information SI.

[0302] Next, at step T20, the training data set is provided to the (not easily trainable) prediction function PF.

[0303] In step T30 , based on the training input data of the training data set, the prediction function PF will determine an intermediate treatment response prediction according to the learning task.

[0304] In a subsequent step T40 , the performance of the prediction function PF (ie the quality of the intermediate treatment response prediction) is evaluated based on a comparison of the verified treatment response and the intermediate treatment response prediction.

[0305] At step T50 , this comparison is used as a loss function to adjust the weight of the prediction function PF.

[0306] At step T60, the steps of obtaining intermediate treatment response predictions (step T30) and comparing the results with the verified treatment responses (step T40) are repeated using paired sets of training input data sets and output data sets until the prediction function PF is able to generate acceptable results (i.e., until a local minimum of the loss function is reached). Once all pairs have been used, the pairs are randomly shuffled for the next round.

[0307] Fig.10 An embodiment of a system 200 for providing (training) a prediction function PF is shown. The system includes a processor 210, an interface 220, a memory 230, a storage device 240, and a database 250. The processor 210, the interface 220, the memory 230, and the storage device 240 may be embodied by a computer 290. The processor 210 controls the overall operation of the computer 200 by executing computer program instructions, which define such operations. The computer program instructions may be stored in the memory 230, or in the storage device 240, and loaded into the memory 230 when the computer program instructions need to be executed. The storage device 240 may be a local storage device as a component of the system 200, or a remote storage device accessible via a network, such as a component of a server or a cloud system. Figure 8 The illustrated method steps may be defined by computer program instructions stored in the memory 230 and / or storage device 240 and controlled by the processor 210 executing the computer program instructions.

[0308] The database 250 is a storage device, such as a cloud or local storage device, used as an archive of the training data set as described above. The database 250 can be connected to a computer 290 to receive one or more medical images. The database 250 and the computer 290 can also be implemented as a single device. The database 250 and the computer 290 can also communicate wirelessly or with a wired connection over a network. The interface 220 is configured to interact with the database 250.

[0309] As long as it makes sense, the various embodiments or their various aspects and features can be combined or exchanged with each other without limiting or expanding the scope of the invention. Where applicable, the advantages described with respect to one embodiment of the invention are also advantageous for other embodiments of the invention. Independent of the grammatical term usage, individuals with male or female identities are also included in the terms.

Claims

1. A computer-implemented method for providing a treatment response prediction (TRP) for a patient suffering from a cancerous disease, the method comprising: - obtaining (S10) a whole slide image (WSI) of said patient showing a tissue sample associated with said cancerous disease, - providing (S20) a prediction function (PF) configured to derive a treatment response prediction (TRP) for one or more treatment options from the whole slide image (WSI), - applying (S30) the prediction function (PF) to the whole slide image (WSI) so as to generate the treatment response prediction (TRP), and - providing (S40) the treatment response prediction.

2. The method according to claim 1, wherein: The one or more treatment options include at least one of the following: - Radiotherapy treatment, -Immunotherapy treatment, - Chemotherapy treatment, and / or - Treatment by surgical intervention.

3. A method according to any one of the preceding claims, wherein: The treatment response prediction (TRP) includes: - the predicted sensitivity of the cancerous disease to the one or more treatment options, - the probability of recurrence of the disease based on the one or more treatment options, and / or - A predicted survival rate for said patient using said one or more treatment options.

4. The method according to any one of the preceding claims, further comprising: - providing (S21) a certainty calculation module (CCM), the certainty calculation module (CCM) being configured to output a certainty metric (CM) for a corresponding treatment response prediction (TRP), the certainty metric (CM) measuring a confidence level of the corresponding treatment response prediction (TRP), - applying (S31) said certainty calculation module (CCM) to obtain a certainty measure (CM) for said treatment response prediction (TRP), and - providing (S41) said certainty measure (CM).

5. The method according to claim 4, wherein: The step of providing (S41) the deterministic measure (CM) comprises: - outputting the confidence measure (CM) together with the treatment response prediction (TRP) to a user (U) via a user interface (10).

6. The method according to any one of claims 4 or 5, further comprising: - determining (S50) based on said confidence measure (CM) whether said treatment response prediction (TRP) is conclusive, and - If the treatment response prediction is not conclusive: determining (S60) a piece of information (PI) related to the patient different from the whole slide image (WSI), said piece of information (PI) being suitable for making the treatment response prediction (TRP) conclusive.

7. The method according to claim 6, further comprising: - access to a healthcare information system (40) comprising the healthcare data (EMR) of said patient, - retrieving (S71) said piece of information (PI) from said healthcare information system (40), - processing (S80) said piece of information (PI) so as to provide an updated treatment response prediction (TRP) and optionally an updated confidence measure (CM), and - providing (S90) said updated treatment response prediction (TRP) and optionally said updated confidence measure (CM).

8. The method according to any one of claims 6 or 7, further comprising: - accessing (S60) a healthcare information system (40) comprising the healthcare data (EMR) of said patient, and - determining (S70) whether said piece of information (PI) is available in said healthcare information system (40), wherein, - If the information piece is not available: generating (S72) a corresponding notification to the user (U) via the user interface (10).

9. The method according to any one of claims 6 to 8, wherein: - the piece of information (PI) relates to a medical examination of the patient, and the method further comprises: - generating (S73) instructions (INS) for performing said medical examination, and - providing (S74) said instruction (INS).

10. The method according to any one of the preceding claims, further comprising: - obtaining supplemental information (SI) associated with the patient from a healthcare information system (40), wherein: - the prediction function (PF) is further configured to derive the treatment response prediction (TRP) additionally based on supplementary information (SI), and - The step of applying (S30) comprises additionally applying said prediction function (PF) to said supplementary information (SI).

11. The method according to claim 10, further comprising: - providing (S22) a data extraction module (LLM), the data extraction module (LLM) being configured to search the electronic medical record (EMR) of the patient to obtain supplementary information (SI), the data extraction module (LLM) preferably comprising a large language model, wherein the step of obtaining the supplementary information (SI) comprises: - accessing the electronic medical record (EMR) of the patient in the healthcare information system (40), and - applying (S33) said data extraction module (LLM) to said electronic medical record (EMR) in order to obtain said supplementary information (SI).

12. The method according to any one of claims 10 or 11, wherein: - said supplementary information (SI) comprises radiological image data (RID) depicting the manifestation of the cancerous disease in the body of said patient.

13. The method according to claim 12, further comprising: - providing (S23) an image analysis module (IAM) configured to extract radiologically observable objects (RA-VF) from the radiological image data (RID), and - applying (S32) said image analysis module (IAM) to said radiological image data in order to obtain said radiologically observable object (RA-VF), wherein: - the prediction function (PF) is further configured to derive the treatment response prediction (TRP) additionally based on the radiological observable object (RA-VF), and - The step of applying (S30) comprises additionally applying said prediction function (PF) to said radiologically observable object (RA-VF).

14. The method according to claim 13, wherein: The radiologically observable object (RA-VF) includes at least one of the following: - the patient's tumor burden in the radiological image data (RID), - visual characteristics of lesions depicted in said radiological image data (RID), and / or - The temporal evolution of the lesion depicted in said radiological image data (RID).

15. A method according to any one of the preceding claims, wherein: The treatment options include at least two different radiation therapy treatment options, the at least two different radiation therapy options differing in at least one of the following: - dose distribution, - Dose threshold limits, - dose rate, - Grading, - use of proton / photon or electron radiation, and / or - Use of coplanar or non-coplanar beams.

16. A method according to any one of the preceding claims, wherein: The one or more treatment options include radiation therapy treatment at a dose rate greater than 40 Gy / second.

17. A system (1) for providing a treatment response prediction (TRP) for a patient suffering from a cancerous disease, the system (1) comprising an interface unit and a computing unit, wherein: - the interface unit is configured to obtain (S10) a whole slide image (WSI) of the patient showing a tissue sample associated with the cancerous disease, and - the computing unit is configured to: - hosting (S20) a prediction function (PF) configured to derive a treatment response prediction (TRP) for one or more treatment options from the whole slide image (WSI), - applying (S30) the prediction function (PF) to the whole slide image (WSI) so as to generate the treatment response prediction (TRP), and - providing (S40) said therapy response prediction (TRP) via said interface unit.

18. A computer program product comprising program elements which, when loaded into a memory (93) of a computing unit (92) of a system (91), cause the computing unit (92) to execute the steps of the method according to any one of claims 1 to 16.

19. A computer-readable medium (93) having program elements stored thereon, which, when executed by a computing unit (92) of a system (91), can be read and executed by the computing unit (92) to perform the steps of the method according to any one of claims 1 to 16.

Images