Systems and methods for patient stratification by machine learning immune checkpoint inhibitor response prediction

Abstract

According to the present invention, a novel method for stratifying immuno-oncology (IO) responders and non-responders is provided. The method is based on deep-learned features extracted by a well-trained automated AI-based model, going beyond traditional radiomics standards and opening new perspectives for the broader adoption of machine learning solutions in both patient care and drug development. Based on the latest advances in machine learning, the method proposed herein allows for non-invasive prediction of a patient's tumor response to immuno-oncology therapy-based treatment. The method proposed herein works not only for early stage conditions with whole organ and lesion-agnostic analysis for prediction, but also for advanced metastatic stage with multi-organ analysis for disease-agnostic and stage-agnostic prediction according to response criteria defined by RECIST 1.1 evaluation method.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63,071,062, filed August 27, 2020.

[0002] The present disclosure generally relates to radiomics methods and frameworks for stratifying patients according to their immunotherapy response.

[0003] The present invention more particularly relates to an automated radiomics method for stratifying a plurality of patients according to their immunotherapy response. [Background technology]

[0004] Radiomics refers to the high-throughput, quantitative extraction and analysis of image features from a variety of medical images acquired, for example, by computed tomography (CT), positron emission tomography (PET) or magnetic resonance imaging (MRI).

[0005] In the early 2010s, AI applied to radiomics was subject to renewed interest when well-trained features for a given task were shown to be more appropriate than the standard handcrafted solutions prevalent at the time. Since then, AI has witnessed significant progress.

[0006] In particular, the field of immuno-oncology (IO) is repeatedly faced with challenges that directly impact the efficiency and performance of pharmaceutical companies in drug development. Among the most important challenges faced in the field of immuno-oncology are the development of non-invasive methods to monitor immune checkpoint inhibitor (ICI) biomarkers, as well as the recruitment of patients who respond to immunotherapy in clinical trials to improve overall response rates (ORR) and accelerate drug development, among others. As part of the effort to deliver the right medicines to the right "responding" patients, the statistical behavior and observation of various patterns of these ICI markers and patient information has highlighted the possibility of creating predictive models for each of the different population cohorts.

[0007] 1. Several non-invasive methods for determining imaging biomarkers that can be associated with clinical biomarkers and treatment response, based on a combination of classical radiomics feature extraction methods and machine learning methods; and 2. Several non-invasive methods for determining cancer prognostic biomarkers based on deep learned features exists.

[0008] Among the methods of the first bulleted type, the paper titled “A deep learning-facilitated radiomic solution for the prediction of lung lesion shrinkage in non-small cell lung cancer trials” by A. Chen et al. discloses a deep learning-based approach to predict lung lesion response based on radiomics features extracted from lung CT scans of patients in non-small cell lung cancer clinical trials. The approach starts with classifying lung lesions at various sites into primary and metastatic lesions. The next step involves automatic segmentation to extract 3D volumes of lung lesions. Radiomics features are then extracted from lesions from pre-treatment scans and first follow-up examinations to predict which lesions will shrink in diameter by at least 30% during treatment, with such shrinkage being defined as a partial response by the Response Evaluation Criteria in Solid Tumors (RECIST) guidelines. Finally, random forest (RF), gradient boosting, and multilayer perceptron classifier models are trained on such cases to be able to predict whether the diameter will shrink by 30% for new data not seen before.

[0009] Another paper, titled “Radiomic response evaluation of recurrent or metastatic head and neck squamous cell cancer (R / M HNSCC) patients receiving pembrolizumab on KEYNOTE-012 study” by K. Taylor et al. and published in the Journal of Clinical Oncology, aims to identify radiomics features associated with lesion-level response (LLR) and overall response (OR) to immunotherapy. For this purpose, they extract tumor and peritumoral radiomics features for 406 lesions of 132 patients treated with pembrolizumab and aggregate the extracted radiomics features with high Pearson correlation. Each obtained cluster is then represented by its medoids, which leads to candidate radiomics features. The authors found that some of these radiomics features correlate significantly with LLR or OR.

[0010] A further paper by S. Trebeschi et al., entitled “Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers” and published in the Annals of Oncology, aims to extract image features that can be (mostly indirectly) associated with clinical biomarkers. Image features are extracted via standard methods and an existing library, namely PyRadiomics, which provides a fixed, predefined set of extractable image features. The final prediction model is based on such a set of image features.

[0011] The models used for patient selection and differentiation have at least two different approaches: classical statistical models or machine learning (ML) models. In this study, only one indirect symptom (association) with overall survival is considered for the final conclusion. Furthermore, additional data from follow-up examinations are necessary to estimate the shrinkage level. Moreover, standard radiomics features are extracted from the entire volume of the lesion and mainly its boundary, without considering the information surrounding the lesion.

[0012] The second bulleted type of methods aims to avoid any prior knowledge of what visual features are relevant to address a given problem in computer vision. In the context of cancer prognosis, the parameters of the deep model are optimized with respect to an objective function to extract image features that directly correlate with the clinical outcome we wish to predict.

[0013] Among the methods of the type listed in the second bullet above, for example: 1. The paper by S. Wang et al., entitled “Deep learning provides a new computed tomography-based prognostic biomarker for recurrence prediction in high-grade serous ovarian cancer,” published in Radiotherapy and Oncology. 2. A paper titled “Deep learning in head & neck cancer outcome prediction” by A. Diamant et al. 3. A paper titled “Bladder Cancer Treatment Response Assessment in CT using Radiomics with Deep-Learning” by Kenny H. Cha et al. 4. The paper entitled “Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging” by Y. Xu et al., and 5. The paper entitled “The Classification of Renal Cancer in 3-Phase CT Images Using a Deep Learning Method” by S. Han et al., published in the Journal of Digital Imaging. can be cited.

[0014] Such non-invasive methods for determining biomarkers for cancer prognosis based on deep learned features have several drawbacks. More specifically, the methods from papers 1, 2, 4 and 5 as itemized above are not directly optimized for cancer prognosis because the features are learned for another task (photo classification or image compression). In the method from paper 3, the neural network is trained on a task that is better performed by radiologists. Summary of the Invention [Problem to be solved by the invention]

[0015] A general object of the present invention is to provide an automated radiomics method for stratifying a plurality of patients according to their immunotherapy response, which makes it possible to alleviate at least one of the drawbacks of the state of the art.

[0016] Another object of the present invention is to provide such a method that allows radiologists to perform tasks that cannot be easily performed.

[0017] Another object of the present invention is to provide a method that is flexible to obtain radiomics features that provide appropriate patient selection and / or can robustly distinguish patients and their treatment responses.

[0018] Another object of the invention is to provide such a method that allows to extract some image features that are not predefined and / or are independent of the organ of interest and / or that are strongly correlated with clinical outcome. [Means for solving the problem]

[0019] According to a first aspect of the present invention there is provided a radiomics method for stratifying a plurality of patients according to their immunotherapy response, comprising: For each patient among the plurality of patients, a set of radiological images of at least slices of a body part of a patient; and Clinical metadata records for patients (clinical ground truth) providing a extracting from said set of radiographic images those in which the first determined lesion is scanned, which are then collected into a batch of radiographic images related to the first determined lesion; For each radiological image of the batch, performing a 2D analysis of the radiological image by implementing a pre-trained deep neural network (DNN) model to extract at least one numerical score from the radiological image, said at least one numerical score being correlated with biomarkers known to be predictive of a patient's immunotherapy response, preferably also provided independently in said clinical metadata record for the patient; aggregating previously extracted numerical scores to determine a patient-level numerical score for a batch of radiological images; classifying the patients into one of a first class of patients whose patient-level numerical score is below a pre-defined threshold and a second class of patients whose patient-level numerical score is equal to or greater than a pre-defined threshold by implementing a pre-trained machine learning (ML) based trained classification model; correlating said previously determined classification of the patient with clinical metadata records relating to the patient to determine a score vector associated with the determined first lesion; thus achieving said stratification of said plurality of patients according to their immunotherapy response. A method is provided.

[0020] Thus, a novel method is provided to stratify immuno-oncology (IO) responders and non-responders. This unique method is based on deep learned features extracted by a well-trained automatic AI-based model, going beyond traditional radiomics standards and opening new perspectives for a broader incorporation of machine learning solutions in both patient care and drug development.

[0021] Based on the latest machine learning advances, the method proposed herein allows to non-invasively predict a patient's tumor response to a treatment based on immuno-oncology therapy. As will become clear below, the method proposed herein works not only for early stage conditions with whole organ and lesion agnostic analysis to make a prediction, but also for advanced metastatic stages with multi-organ analysis to potentially make a disease agnostic and stage agnostic prediction according to the response criteria defined by the evaluation method known under the acronym "RECIST 1.1".

[0022] According to some embodiments (examples) of the first aspect of the present invention, The method includes, for at least one patient of the plurality of patients, The method further includes generating a visual representation of the at least one patient stratification associated with the first determined lesion. The step of generating a visual representation comprises: generating a 2D heat map in response to each radiological image of the batch; generating a volumetric heat map including a 3D representation of the first determined lesion in response to the 2D heat map and the previously determined patient-level numerical score, the volumetric heat map including a 3D representation of the first determined lesion that reveals where relevant information for scoring the first determined lesion is located; and optionally visually matching and searching for patterns or correlations using at least one machine learning method in response to the volumetric heatmap and clinical metadata records for the at least one patient to generate the visual representation of a stratification of the at least one patient associated with the first determined lesion; Including, The deep neural network (DNN) model may be trained according to a set of radiological images of a training cohort of patients to determine a threshold value to be considered for classification; The machine learning (ML) based classification model is trained with respect to a cut-off value (low vs. high) as a function of the patient-level numerical score by using the classification model to establish a correlation with clinical metadata records related to the patient; 2D analysis of each radiographic image in the batch pre-processing the radiological image by windowing; generating at least one multi-channel lesion patch associated with the first determined lesion in response to the previously pre-processed radiological image; Implementing the pre-trained deep neural network (DNN) model in response to at least one multi-channel lesion patch; Includes. The method comprises, after the pretreatment step, defining a bounding box for the first determined lesion on each preprocessed radiological image; The method further includes the step of causing the generating step to be performed according to a defined bounding box. The implementation steps (of the pre-trained DNN model) include: Computing local features by implementing a pre-trained Convolutional Neural Network (CNN) model; performing average pooling of the computed local features; Scoring the pooled local features by logistic regression; Includes. As an alternative to the last bullet, the implementation steps (of the pre-trained DNN model) can be: Computing local features by implementing a pre-trained Convolutional Neural Network (CNN) model; weighting the computed local features with an attention mechanism; performing average pooling of weighted local features; Scoring the pooled weighted local features by logistic regression; Including, The correlation step may be performed by implementing a comparison function with a metric evaluation between the class into which the patient is classified and said clinical metadata record; The clinical metadata record can include a low or high immuno-oncology response of the patient; According to the last two items, the metric assessment is performed between the patient-level numerical score and the low or high value of the patient's immuno-oncology response; With two or more determined lesions scanned on the radiological image, the method may further include repeating the steps of the method for at least one other determined lesion and then aggregating the score vector determined for the first determined lesion with the score vector determined for the at least one other determined lesion.

[0023] According to some embodiments (examples) of the first aspect of the present invention, the radiological image comprises: Computed tomography (CT) images, and Magnetic resonance imaging (MRI) images obtained by MRI At least one of Optionally, the radiological image is one of a non-contrast phase, an arterial phase, a portal venous phase, and a delayed phase radiological image.

[0024] According to some embodiments (examples) of the first aspect of the present invention, the radiological image is provided as a Digital Imaging and Communications in Medicine (DICOM) image file.

[0025] According to some embodiments (examples) of the first aspect of the present invention, a clinical metadata record (clinical ground truth) is generated from clinical ground truth obtained for each patient of said plurality of patients.

[0026] According to some embodiments (examples) of the first aspect of the present invention, the method includes, prior to providing the radiographic image: The method further comprises the step of controlling the quality of said radiological images and thus retaining radiological images having a quality value superior to the determined value.

[0027] Further aspects of the present invention relate to a non-transitory computer readable medium storing instructions which, when implemented by at least one digital processing device, perform at least the steps of the method according to the first aspect of the invention, and / or a computer program product comprising instructions which, when implemented by at least one digital processing device, perform at least the steps of the method according to the first aspect of the invention.

[0028] Other objects, features, and advantages of the present invention disclosed herein, as well as various embodiments / aspects thereof, may become apparent in light of the following description of certain exemplary embodiments.

[0029] Reference may now be made in detail to embodiments of the present disclosure, non-limiting examples of which are illustrated in the accompanying drawings. Some of the figures may be in diagram form. For clarity of illustration, some of the elements in the figures may be exaggerated and other elements may be omitted.

[0030] Any text appearing on the drawings (legends, notes, reference numbers, etc.) is incorporated herein by reference. [Brief description of the drawings]

[0031] [Figure 1] 2 is a flow chart of an embodiment of a method according to the present invention; [Diagram 2] 2 is a flow chart of one embodiment of the steps of a method as shown in FIG. 1, comprising a slice-by-slice 2D analysis. [Diagram 3] FIG. 3 shows a schematic representation of a first example of the steps consisting of a slice-by-slice 2D analysis as shown in FIG. 2. [Figure 4] FIG. 3 shows a schematic representation of a second example of the steps consisting of a slice-by-slice 2D analysis as shown in FIG. 2. [Diagram 5] 1 is a flow chart of an embodiment of a method according to the present invention specifically designed for patients at the metastatic stage. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Radiomics is the process of extracting and analyzing quantitative features, called radiomics features, from diagnostic images, especially radiological images. A cluster of radiomics features is a "radiomics signature", or in other words an "image signature".

[0033] According to the present invention, the term "scan" is a procedure that provides a radiological image resulting from at least one medical imaging technique. It is particularly foreseen to use at least one of the following two medical image acquisition techniques or modalities within the framework of the present invention: 1) Computed tomography (CT) scans of patients with and without contrast injection before the scan; and 2) Magnetic resonance imaging (MRI) with and without contrast injection prior to the scan, along with magnetic resonance elastography (MRE).

[0034] A non-contrast radiographic image is generated without injecting a contrast agent into the patient before scanning. A contrast radiographic image is generated by injecting a contrast agent into the patient before scanning. The contrast agent may be designed to highlight certain areas in order to make selected at least one of the blood vessels, organs and tissues more visible during the scan and potentially at different defined times after injection. Thus, a contrast radiographic image may be aimed at highlighting the arterial phase during a first scanning step and the portal venous phase during a second scanning step, for example by using the same contrast agent at different defined times after injection or by using different contrast agents. With respect to at least one of the first and second scanning steps introduced above, a corresponding scanning step is delayed by a defined period of time, so that another phase, called a delayed phase, may be observed during the scanning according to the invention. Such a scan of the delayed phase may make it possible to highlight other specific areas in order to make other features more prominent with respect to the arterial and / or portal venous phase. The acquisition of the delayed phase during a further scanning step of the scanning according to the invention may be performed either after the acquisition of the arterial phase and / or the acquisition of the portal venous phase.

[0035] A scan according to the present invention may include a series of scanning steps performed by at least one implementation of one of the two medical imaging techniques mentioned above to provide a radiological image of interest.

[0036] The objective function indicates how much each variable in a given problem contributes to the value to be optimized.

[0037] According to the present invention, the term "machine learning" refers to computer algorithms that can perform classification automatically without explicit programming. The computer algorithms build said mathematical models from training data.

[0038] The term "clinical ground truth" or "clinical metadata" refers to data about patients obtained by physical, medical, or laboratory examination, which may include at least one of each patient's age, sex, height, weight, BMI, ethnic origin, medical condition, risk factors, nodule size, IO response (e.g., low or high), treatment type, and tumor type.

[0039] A heat map (or heat map) is a data visualization technique that displays the magnitude of a phenomenon in a two-dimensional color layout. The color variations can be by hue or intensity, giving the reader clear visual cues as to how the phenomena are clustered or vary across space.

[0040] The term "windowing" refers to a process used to modify images in a computed tomography (CT) or MRI scan. More specifically, it refers to a technique for modifying radiological images from a CT or MRI scan machine that can reveal either soft tissue, or bone structures, or skin by selecting a range of gray levels. For example, bone windows, lung windows.

[0041] The term "patch" refers to a portion of a radiological image, for example as defined by windowing of said radiological image.

[0042] The term "multi-channel lesion patch" refers to a multi-dimensional matrix (depth-height-width) that represents a set of patches, where depth refers to the number of channels and height and width refer to the size of the lesion patch.

[0043] In convolutional neural networks, pooling refers to a form of nonlinear downsampling. Average pooling divides the input image into a set of rectangles and for each such subregion, outputs the average value of the pixels.

[0044] Because image-level radiomics features show very promising results and high performance of statistical models, we hypothesize that artificial intelligence (AI) and especially deep learning (DL) based algorithms can automatically quantify radiomic features that are associated with, and therefore can serve as, non-invasive radiomics biomarkers for immunotherapy response.

[0045] The following detailed method is proposed to extract image features without any predefined constraints.

[0046] The overall pipeline of a preferred embodiment of the method proposed herein is shown in the accompanying Figures 1, 2, 3 and 4.

[0047] Figure 1 shows how a patient is scored and classified, and how these outputs can be explained to the patient. These scoring and classification, and potentially the explanation outputs, are derived by a slice-by-slice 2D analysis, as shown, for example, by Figure 2. This 2D analysis involves a deep neural network model, which can be embodied in two ways, shown for illustrative purposes in Figures 3 and 4, respectively.

[0048] FIG. 1 shows a flow chart of one embodiment of the method proposed herein.

[0049] According to an embodiment shown in FIG. 1, the present invention provides a method 100 for stratifying a plurality of patients according to their immunotherapy response.

[0050] For each patient of the plurality of patients, method 100 comprises: a radiological image of a slice of a body part of a patient, such as the abdomen, comprising an organ exhibiting a first determined pathology; A clinical metadata record about the patient (clinical ground truth); as data associated with each patient (110).

[0051] The radiological image may be selected from at least the following: Computed tomography (CT) images, and MRI images obtained using magnetic resonance imaging (MRI).

[0052] These can be, for example, radiographic images of one of the non-contrast, arterial, portal venous and delayed phases, depending on which phase is known to be necessary to enhance an ad-hoc parameter characterizing said first determined lesion. Typically, at least two radiographic images are provided per slice during a scan, said at least two radiographic images being of different phases.

[0053] The radiological images are preferably provided as Digital Imaging and Communications in Medicine (DICOM) image files.

[0054] The quality of each of said radiological images may be controlled before being provided 110. Such control allows for the retention of radiological images whose quality value is better than a determined threshold.

[0055] For each of a plurality of patients, his / her clinical metadata record may be generated from clinical ground truth obtained by physical examination, medical examination, or clinical examination.

[0056] For each patient of the plurality of patients, the method 100 further comprises a step (120) of extracting or selecting from said set of radiographic images those in which a first determined lesion was scanned, which are then collected into a batch of radiographic images associated with the first determined lesion.

[0057] Then, the method 100 further comprises a step of a 2D analysis performed on each slice containing the first determined lesion, more specifically, for each radiographic image of the batch, said step comprising a step of performing (130) said 2D analysis for extracting, from each radiographic image of the batch, as corresponding image feature, at least one numerical score that correlates with a biomarker known to be predictive of the patient's immunotherapy response, by implementing a pre-trained Deep Neural Network (DNN) model.

[0058] It should be noted here that said biomarkers known to be predictive of a patient's immunotherapy response may indeed be known from the state of the art, and the presently detailed method 100 is not limited to any particular one of known biomarkers. Similarly, said at least one numerical score correlating with such biomarkers is at least as known from the state of the art. If necessary, more than one numerical score can be extracted for each biomarker. Preferably, said at least one numerical score is correlated with a biomarker that is also provided independently in said clinical metadata record for the patient.

[0059] Thus, the method 100 can, on the one hand, take advantage of the latest research that already defines relevant biomarkers for tracking the evolution of any kind of disease that a patient may have, and, on the other hand, can be improved by taking advantage of new knowledge that future research will provide to the skilled artisan.Therefore, there are no limitations on the biomarkers envisaged, on the type of image features that can be extracted, or on the way in which such extraction is performed, other than that the biomarkers must be known as predictive of a patient's immunotherapy response, and that the extraction can be performed on each radiological image of said batch.

[0060] Regarding the Deep Neural Network (DNN) model, there are no limitations regarding the instructions defining it, which may be a home-made Deep Neural Network (DNN) model or another model already known or proposed in the future and designed to allow for the extraction of a numerical score as defined above.

[0061] The Deep Neural Network (DNN) model must be trained, which may be performed according to a set of radiological images of a training cohort of patients. This training may make it possible to determine the thresholds to be taken into account for the subsequent classification (150), as will be described in more detail below.

[0062] Thus, similar to the second type of method itemized in the background section of this application, the proposed method 100 makes it possible to avoid any prior knowledge of what visual features are relevant to address a given problem in computer vision. In other words, the proposed method 100 encompasses the extraction of image features that are not predefined and do not necessarily depend on the organ of interest, while making use of information that is not easy, or even impossible, to extract in each radiological image by a naked eye analysis performed by an experienced radiologist.

[0063] Figure 2 shows a flow chart of one embodiment of step 130 of the method 100 shown in Figure 1. Said step 130 consists of the aforementioned slice-by-slice 2D analysis.

[0064] With reference to FIG. 2, the aforementioned 2D analysis may consist of the following sub-steps: First, each radiographic image is pre-processed by applying at least one pre-defined relevant windowing (131). Then, either a manual or automatic bounding box around the first determined lesion may be generated (132). In some cases, the entire radiographic image may be selected. Then, the pre-processed radiographic image and, if applicable, the bounding box are used to generate at least one multi-channel lesion patch (133). Then, the at least one multi-channel lesion patch is processed by a deep model that outputs a score (134). It is noted here that the calculation resulting in this score may be further analyzed with the aim of generating a heat map highlighting the most informative areas of the at least one multi-channel lesion patch (135).

[0065] Returning to FIG. 1, the method 100 further comprises a step (140) consisting of aggregating the previously extracted numerical scores to determine a patient-level numerical score. This numerical score is defined for each of a plurality of patients. It can comprise a single real or complex value, or it can consist of a collection of numerical scores in the form of a matrix, set, list, or vector. Thus, the format in which the extracted (130) numerical scores are aggregated (140) may be selected from among several known, home-grown, or future formats, the only requirement being that the aggregated numerical score is comparable in a manner relevant to the aforementioned thresholds to be considered for the subsequent classification (150), representing the patient's response to the immunotherapy.

[0066] After the detailed aggregation step 140 above, the method 100 includes the aforementioned classification step 150, which consists of classifying 150 the patients into one of a first class of patients whose patient-level numerical score is below the predefined threshold and a second class of patients whose patient-level numerical score is equal to or greater than a predefined threshold by implementing a machine learning (ML) based classification model.

[0067] Thus, each of the plurality of patients is classified according to the respective patient's immunotherapy response.

[0068] Moreover, the method 100 further comprises a step (160) consisting of correlating the classification of a given patient of the plurality of patients considered with a clinical metadata record relating to said given patient in order to determine a score vector associated with the first determined lesion. More specifically, each clinical metadata record may comprise a clinically assessed value of the patient's immuno-oncological response. This value may simply be "low" or "high".

[0069] This result is one of the tasks known in the technical field concerned here as patient stratification. In practice, each of the multiple patients is stratified into low and high immunotherapy response by taking into account not only the deep-learned image features extracted from their radiological images, but also their strong correlation with clinical ground truth (GT) or at least one related information among those usually found in clinical ground truth (GT).

[0070] Potentially, even if this constitutes an important advantage achievable by the method 100 proposed herein compared to the prior art, the method may furthermore make it possible to derive a visual explanation of the achieved stratification, either for data scientists developing software solutions for radiomics, for medical practitioners in charge of the stratified patients, and for the stratified patients themselves. The optional aspect of the derivation of such a visual explanation is indicated in FIG. 1 by the fact that the relevant blocks are dashed.

[0071] More particularly, said visual representation of the achieved stratification comprises: generating (135) a 2D heat map in response to each radiographic image of the batch; and Depending on the 2D heatmap and the previously determined patient-level numerical score, a step (170) may be derived of generating a volumetric heatmap including a 3D representation of the first determined lesion that reveals where relevant information for scoring the first determined lesion is located.

[0072] The generating step 135 has already been described above. This step can be performed directly after the 2D analysis 130 as detailed above. The generating step 170 requires knowledge of the patient-level numerical scores and therefore can be performed after the aggregating step 140.

[0073] The generation of said visual explanation for the achieved stratification may further comprise a step 180 consisting of visual matching and searching for patterns or correlations according to the volumetric heatmap and clinical metadata records for said at least one patient.

[0074] The output of the 2D analysis may therefore be split in two ways: a patient-level numerical score may be derived for multiple patients, and a visual interpretation of this (these) scores may be provided via a heat map that reveals where in the scanned patient slices the relevant information for scoring is located.

[0075] FIG. 3 shows a schematic example of an example of a slice-by-slice 2D analysis as shown in FIG. 2, and FIG. 4 shows a schematic example of another example of the same steps.

[0076] Therefore, two versions of the deep neural network model used have been considered, said two versions being shown in Figures 3 and 4, respectively.

[0077] In the first version shown in FIG. 3, local features are computed by implementing a pre-trained convolutional neural network (CNN) model (1340), then pooled together by simply averaging them (1341), and then the pooled local features are scored by logistic regression (1342).

[0078] In the second version shown in Figure 4, after computing 1340 the local features, an attention mechanism is used to derive 13411 weights for said local features, and if necessary, these local features are pre-formed into vector form, in which case the local features are pooled 13412 by weighted averaging.

[0079] The first version can be trained with less data than the second version, which may however lead to higher performance, as the attention weights can provide more information for the subsequent generation of visual explanations as already mentioned above.

[0080] In view of the above, the method 100 according to the first aspect of the present invention appears to enable stratification of patients who respond to immuno-oncology (IO) and those who do not.

[0081] The method 100 proposed herein is based on deep learned features extracted by a well-trained automated AI-based model, going beyond traditional radiomics standards and opening new perspectives for a broader uptake of machine learning solutions in both patient care and drug development.

[0082] More particularly, the method 100 proposed herein allows for the non-invasive prediction of a patient's tumor response to an immuno-oncology based treatment.

[0083] As is evident throughout the above detailed description, the method 100 proposed herein works for early stage conditions through whole organ and lesion agnostic analysis to make predictions.

[0084] Nevertheless, as will become evident in the description given below of further embodiments of the method 100 proposed herein, the method 100 proposed herein can also function for advanced metastatic stages through multi-organ analysis making disease-agnostic and stage-agnostic predictions.

[0085] FIG. 5 is a flow chart of one embodiment of the method 100 proposed herein, specifically designed for metastatic stage patients.

[0086] Figure 5 shows that in the case of advanced stage (metastasis with several lesions in the same organ or multiple organs), the above process can be deployed for each of the several lesions, or for a selection thereof. The results from each of the multiple lesion analyses can then be aggregated to predict the overall patient response. Such aggregation can also be performed according to, for example, RECIST 1.1 evaluation criteria.

[0087] More specifically, the aggregation of the set of patient-level numerical scores and optionally the 2D heatmap may include the implementation of an automated AI-based ML model for predicting patient response (value), or rather a comparison of the obtained prediction with a given RECIST criterion value to output a RECIST evaluation criterion metric.

[0088] The embodiments and aspects of the invention detailed herein may be described and illustrated in conjunction with systems, devices, and methods that are intended to be exemplary and illustrative, rather than limiting in scope. Specific configurations and details may be described to provide an understanding of the invention. However, it should be apparent to one skilled in the art that the invention can be practiced without some of the specific details presented herein. In addition, some well-known steps or components may only be generally described or even omitted for clarity of description.

[0089] Some processes may be presented and described as a series of (sequential) steps, it being understood that the sequential steps are exemplary and that the steps may be performed in an order different from that presented, some steps as described may be omitted, and some additional steps may be omitted from the sequence or described elsewhere.

[0090] Reference may be made to the disclosures of prior patents, publications and applications. Portions of text and figures from these sources may be presented herein, but may be modified, edited, or commented upon to more smoothly harmonize with the disclosure of this application. Citation or identification of any reference should not be construed as an admission that such reference is available as prior art to the present disclosure.

[0091] The methods described herein can be implemented on one or more computers in any form. The components may be implemented as dedicated applications or in a client-server architecture, including a web-based architecture, and may include functional programs, codes, and code segments. Any of the computers may include a processor, a memory for storing and executing program data, a permanent storage device such as a disk drive, a communication port for handling communication with external devices, and a user interface device including a display, a keyboard, a mouse, and the like. If any software or algorithms are included, they may be stored on a computer-readable medium, such as a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, as program instructions or computer-readable code executable on the processor. Also, the computer-readable recording medium may be distributed over computer systems coupled to a network such that the computer-readable code is stored and executed in a distributed manner. The medium may be read by the computer, stored in the memory, and executed by the processor.

[0092] For the purposes of promoting an understanding of the principles of various embodiments of the invention, reference has been made to preferred embodiments illustrated in the drawings, and specific language has been used to describe the embodiments. However, no limitation of the scope of the invention is intended by this specific language, which should be construed to encompass all embodiments that would normally occur to one skilled in the art.

[0093] The invention may be described in terms of functional block components and various processing steps. Such functional blocks may be realized by any number of hardware and / or software components configured to perform the specified functions. For example, the invention may employ various integrated circuit components, e.g., memory elements, processing elements, logic elements, look-up tables, etc., that may perform various functions under the control of one or more microprocessors or other control devices. Similarly, where elements of the invention are implemented using software programming or software elements, the invention may be implemented using any programming or scripting language, such as C, C++, Java, Assembler, etc., and various algorithms are implemented using any combination of data structures, objects, processes, routines, or other programming elements. Functional aspects may be implemented in algorithms executed on one or more processors. Additionally, the invention may employ any number of conventional techniques for electronics configuration, signal processing and / or control, data processing, etc. [Explanation of symbols]

[0094] 100 ways

Claims

1. 1. A radiomics method for stratifying a plurality of patients according to their immunotherapy response (100), comprising: For each patient of the plurality of patients, Providing, a set of radiological images of at least slices of a body part of a patient; and A clinical metadata record (clinical ground truth) relating to said patient providing a Extracting (120) from the set of radiographic images those in which a first determined lesion is scanned, which are then collected into a batch of radiographic images related to the first determined lesion; for each radiological image of the batch, performing (130) a 2D analysis of the radiological image by implementing a pre-trained deep neural network (DNN) model to extract from the radiological image at least one numerical score, the at least one numerical score being correlated with biomarkers known to be predictive of the immunotherapy response in the patient, preferably also provided independently in the clinical metadata record for the patient; aggregating (140) the previously extracted numerical scores to determine a patient-level numerical score for the batch of radiological images; classifying (150) the patients into one of a first class of patients whose patient-level numerical score is below a pre-defined threshold and a second class of patients whose patient-level numerical score is equal to or greater than the pre-defined threshold by implementing a pre-trained machine learning (ML) based classification model; correlating (160) the previously determined classification of the patient with the clinical metadata record relating to the patient to determine a score vector associated with the first determined lesion; achieving said stratification of said plurality of patients according to their immunotherapy response; A method (100).

2. For at least one patient of the plurality, generating (135, 170, 180) a visual representation of the stratification of the at least one patient associated with the first determined lesion; The method (100) of claim 1, further comprising:

3. The step of generating a visual description (135, 170, 180) generating (135) a 2D heat map in response to each radiographic image of the batch; generating (170) a volumetric heat map including a 3D representation of the first determined lesion in response to the 2D heat map and the previously determined patient-level numerical score, the volumetric heat map including a 3D representation of the first determined lesion that identifies where relevant information for scoring the first determined lesion is located; The method (100) of claim 2, comprising:

4. visually matching and searching for patterns or correlations using at least one machine learning method in response to the volumetric heatmap and the clinical metadata records for the at least one patient to generate the visual representation of a stratification of the at least one patient associated with the first determined lesion (180). The method (100) of claim 3, further comprising:

5. 5. The method (100) of any one of claims 1 to 4, wherein the deep neural network (DNN) model is trained on a set of radiological images of a training cohort of patients to determine the threshold values ​​to be considered for classification (150).

6. 6. The method (100) of any one of claims 1 to 5, wherein the machine learning (ML) based classification model is trained for a cut-off value (low vs. high) depending on the patient-level numerical score by using the classification model to establish a correlation with clinical metadata records related to the patient.

7. said 2D analysis of each radiographic image of said batch comprising: pre-processing (131) the radiological image by windowing; generating (133) at least one multi-channel lesion patch associated with the first determined lesion in response to the previously pre-processed radiological image; Implementing (134) the pre-trained deep neural network (DNN) model in response to the at least one multi-channel lesion patch. The method (100) of any one of claims 1 to 6, comprising:

8. After the pre-processing step (131), and defining (132) a bounding box for the first determined lesion on each preprocessed radiological image; The method (100) of claim 7, wherein the generating step (133) is performed in response to the defined bounding box.

9. The step of implementing (134) Computing local features (1340) by implementing a pre-trained Convolutional Neural Network (CNN) model; an average pooling step (1341) of the computed local features; Scoring (1342) the local features that have been average pooled (1341) by logistic regression; 9. The method (100) of claim 7 or 8, comprising:

10. The step of implementing (134) Computing local features (1340) by implementing a pre-trained Convolutional Neural Network (CNN) model; weighting the computed local features with an attention mechanism (13411); a step of performing average pooling of weighted local features (13412); Scoring (1342) the weighted local features that have been average pooled (13412) by logistic regression; 9. The method (100) of claim 7 or 8, comprising:

11. 11. The method (100) of any one of claims 1 to 10, wherein the correlating step (160) is performed by implementing a comparison function with a metric evaluation between the class into which the patient was classified and the clinical metadata record.

12. 12. The method of claim 11 (100), wherein the clinical metadata record comprises a low or high immuno-oncological response of the patient.

13. 13. The method of claim 12 (100), wherein the metric assessment is performed between the patient-level numerical score and the low or high value of the immuno-oncological response of the patient.

14. The radiation image is Computed tomography (CT) images, and Magnetic resonance imaging (MRI) images obtained by MRI At least one of Optionally, the radiological image is a radiological image of one of a non-contrast phase, an arterial phase, a portal venous phase, and a delayed phase. The method (100) according to any one of claims 1 to 13.

15. 15. The method (100) of any one of claims 1 to 14, wherein the radiological image is provided as a Digital Imaging and Communications in Medicine (DICOM) image file.

16. 16. The method (100) of any one of claims 1 to 15, wherein two or more determined lesions are scanned on the radiological image, the steps of the method (100) are repeated for at least one other determined lesion, and then the score vector determined for the first determined lesion and the score vector determined for the at least one other determined lesion are aggregated.

17. A program comprising instructions which, when implemented by at least one digital processing device, perform the steps of the method (100) according to any one of claims 1 to 13.

Images