Method for determining the influence of a plurality of regions of a medical image on a prediction model

WO2024246351A1PCT designated stage expired Publication Date: 2024-12-05INSTITUT CURIE +1

Patent Information

Application Number
PCT/EP2024/065126
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-05-31
Publication Date
2024-12-05

Smart Images

  • Figure EP2024065126_05122024_PF_FP_ABST
    Figure EP2024065126_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method implemented by computer means for determining the influence of a plurality of regions in a medical image (I) of a patient on a prediction of a pretrained prediction model (PM) based on said image (I), comprising: (a) determining image features (F1, F2, F3) of a plurality of regions (A1, A2, A3) on said image (I), (b) transforming said features (F1, F2, F3) of each region (A1, A2, A3) in an aggregated tensor (F) inputted into said pretrained prediction model, (c) calculating, based on said features, the Shapley value of each region, said Shapley value representing the influence of said region on said prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Method for determining the influence of a plurality of regions of a medical image on a prediction modelFIELD OF THE INVENTION

[0001] [Incorporated by Reference (Rule 20.6), 22.07.2024]The present invention relates to a method implemented by computer means for determining the influence of a plurality of regions in a medical image of a patient on a prediction of a pretrained prediction model based on said image.BACKGROUND OF THE INVENTION

[0002] [Incorporated by Reference (Rule 20.6), 22.07.2024]Prediction models are increasingly being used in medical imaging to assist in the diagnosis, prognosis, and treatment planning of various diseases, including cancer. Medical imaging, such as X rays, magnetic resonance imaging (MRI), computed tomography (CT) scans, and positron emission tomography (PET) generate large amounts of digital data that can be analyzed using machine learning and other data analysis techniques to extract meaningful patterns and features.

[0003] [Incorporated by Reference (Rule 20.6), 22.07.2024]Prediction models in medical imaging are typically developed using techniques that involve training models on datasets of medical images and associated clinical data. These models learn to identify patterns and features in the images that are indicative of certain diseases, such as tumors or other abnormalities.

[0004] [Incorporated by Reference (Rule 20.6), 22.07.2024]Once a prediction model has been trained, it can be used to analyze new medical images and make predictions about the presence, severity, and progression of a disease. For example, a prediction model for lung cancer may analyze lung images and identify regions of the lung that are suspicious for cancer, allowing radiologists and other medical professionals to target biopsies or other diagnostic tests to these regions.

[0005] [Incorporated by Reference (Rule 20.6), 22.07.2024]Prediction models in medical imaging can also be used to assist in treatment planning and monitoring. For example, a prediction model may analyze images of a tumor and predict how the tumor is likely to respond to different treatment options, such as chemotherapy or radiation therapy.

[0006] [Incorporated by Reference (Rule 20.6), 22.07.2024]These models have shown promise in improving the accuracy and efficiency of medical image analysis.

[0007] [Incorporated by Reference (Rule 20.6), 22.07.2024]However, medical imaging is a highly specialized field that requires expertise in both imaging technology and clinical interpretation.

[0008] [Incorporated by Reference (Rule 20.6), 22.07.2024]In addition, medical imaging is subject to variability in image quality and interpretation, which can affect the accuracy of prediction models. The quality of medical images can be influenced by factors such as patient motion, image resolution, and imaging technique. Additionally, the interpretation of medical images can be influenced by factors such as the experience and expertise of the interpreting radiologist.

[0009] [Incorporated by Reference (Rule 20.6), 22.07.2024]Also, prediction models based on medical images are often highly complex and opaque, making it difficult for individuals without specialized training to understand how the model arrives at its predictions. These models may involve multiple layers of processing, feature extraction, and / or statistical analysis, which can be difficult to interpret without a deep understanding of the underlying algorithms and mathematical concepts.

[0010] [Incorporated by Reference (Rule 20.6), 22.07.2024]Finally, prediction models based on medical images may also be subject to biases and limitations in the training data, which can affect the accuracy and generalizability of the model.

[0011] [Incorporated by Reference (Rule 20.6), 22.07.2024]While prediction models based on medical images hold great promise for improving disease diagnosis, prognosis, and treatment outcomes, their interpretation requires specialized knowledge and expertise.

[0012] [Incorporated by Reference (Rule 20.6), 22.07.2024]One approach to understanding the influence of different regions on a prediction is to use techniques such as feature attribution or visualization. However, the results of these techniques can still be difficult to interpret, particularly for clinicians or other non-experts.

[0013] [Incorporated by Reference (Rule 20.6), 22.07.2024]The aim of this document is to propose a method that helps to provide a better understanding of a prediction model and the specific regions of an image that are most relevant to the prediction.SUMMARY OF THE INVENTION

[0014] [Incorporated by Reference (Rule 20.6), 22.07.2024]To that aim, the present document proposes a method implemented by computer means for determining the influence of a plurality of regions in a medical image of a patient on a prediction of a pretrained prediction model based on said image, comprising:(a) determining image features of a plurality of regions on said image,(b) transforming said features of each region in an aggregated tensor inputted into said pretrained prediction model,(c) calculating, based on the features, the Shapley value of each region, said Shapley value representing the influence of said region on said prediction.

[0015] [Incorporated by Reference (Rule 20.6), 22.07.2024]A medical image may be defined as a visual representation of internal body structures or functions acquired using various imaging modalities. These images are generated using hardware and software systems that capture and process data, producing two-dimensional (2D) or three-dimensional (3D) images that aid in the diagnosis, treatment planning, and monitoring of various medical conditions. Medical images typically contain information about tissue density, composition, and function and are interpreted by radiologists, physicians, or other medical professionals with specialized training in image analysis.

[0016] [Incorporated by Reference (Rule 20.6), 22.07.2024]A medical image may take many different forms, depending on the imaging modality used and the type of medical condition being investigated.

[0017] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said regions may be obtained or extracted from said image by segmentation.

[0018] [Incorporated by Reference (Rule 20.6), 22.07.2024]Segmentation may refer to the process of dividing or partitioning an input image or data into multiple segments or regions (set of pixels or voxels) based on certain criteria, such as color, texture, intensity, shape, or other features. More precisely, image segmentation is the process of assigning a label, a class or a category to every pixel or voxel in an image such that pixels with the same label share certain characteristics. The goal of segmentation may be to identify and isolate different objects or regions of interest within an image or data, so that they can be analyzed, processed, or classified separately.

[0019] [Incorporated by Reference (Rule 20.6), 22.07.2024]Several algorithms and techniques may be used for segmentation, including thresholding (for example intensity thresholding), clustering, edge detection, region growing, and deep learning-based approaches.

[0020] [Incorporated by Reference (Rule 20.6), 22.07.2024]In digital image processing and computer vision, an image feature or feature is a piece of information about the content of an image, typically about whether a certain region of the image has certain properties. Features may be specific structures in the image such as points, edges, or objects. Features may also be the result of a general neighborhood operation or feature detection applied to the image. Other examples of features are related to motion in image sequences, or to shapes defined in terms of curves or boundaries between different image regions.

[0021] [Incorporated by Reference (Rule 20.6), 22.07.2024]More broadly a feature is any piece of information which is relevant for solving the computational task related to a certain application. This is the same sense as feature in machine learning and pattern recognition, though image processing has a very sophisticated collection of features.

[0022] [Incorporated by Reference (Rule 20.6), 22.07.2024]Extracting features from an image involves identifying meaningful patterns or characteristics in the image that can be used for further analysis. Several techniques may be used for feature extraction, including:- Edge detection: This involves detecting the boundaries between objects in an image. Edges can be detected using techniques such as the Canny edge detector or the Sobel operator.- Feature detection: This involves identifying specific points or regions in an image that are likely to be important, such as corners or keypoints. This can be done using techniques such as the Harris corner detector or the Scale-Invariant Feature Transform (SIFT).- Texture analysis: This involves identifying patterns or textures in an image that are characteristic of certain objects or materials. Texture analysis can be done using techniques such as Gabor filters or Local Binary Patterns (LBP).- Color analysis: This involves extracting color information from an image, which can be used to distinguish between different objects or materials. Color analysis can be done using techniques such as color histograms or color-based segmentation.- Deep learning: This involves using deep neural networks to learn features directly from the image data, without the need for hand-crafted features. Techniques such as convolutional neural networks (CNNs) have been highly successful for image classification and recognition tasks.

[0023] [Incorporated by Reference (Rule 20.6), 22.07.2024]A tensor is a multi-dimensional array of numerical values that can be used to represent data or parameters.

[0024] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said tensor may be a vector.

[0025] [Incorporated by Reference (Rule 20.6), 22.07.2024]A vector is a special type of tensor formed by a list of numerical values called scalars that represent some attributes of a data point. The number of dimensions of said vector is the number of said values.

[0026] [Incorporated by Reference (Rule 20.6), 22.07.2024]A prediction model in computer vision is a type of machine learning model that uses image analysis techniques to make predictions or classifications based on visual data. Computer vision models can be trained to identify patterns and features within images and use this information to make predictions or classifications based on new images.

[0027] [Incorporated by Reference (Rule 20.6), 22.07.2024]When applied to oncology, for example, prediction uses image analysis techniques to predict outcomes related to cancer. These models typically analyze medical images and use advanced algorithms to identify patterns and features that are relevant to the prediction of certain outcomes.

[0028] [Incorporated by Reference (Rule 20.6), 22.07.2024]There are many different types of prediction models in computer vision in oncology, each with its own specific focus and application.

[0029] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said prediction model may belong to:- Tumor detection models: These models are designed to identify and locate tumors within medical images, allowing for earlier detection and more accurate diagnosis of cancer.- Prognostic models: These models use image analysis to predict the likelihood of disease progression and overall survival for patients with cancer.- Treatment response models: These models use image analysis to assess the effectiveness of different treatments for cancer, particularly to predict the effectiveness of different treatments for cancer, allowing for more personalized and effective treatment plans.

[0030] [Incorporated by Reference (Rule 20.6), 22.07.2024]A treatment for cancer includes but is not limited to targeted therapy, immunotherapy such as immune checkpoint therapy and immune checkpoint inhibitor therapy, use of co-stimulatory antibodies, chemotherapy, radiotherapy. As used herein, the term "antitumor chemotherapy" or " chemotherapy" has its general meaning in the art and refers to a cancer therapeutic treatment using chemical or biochemical substances, in particular using one or several antineoplastic agents or chemotherapeutic agents. As used herein, the term "immunotherapy" refers to a cancer therapeutic treatment using the immune system to reject cancer. The therapeutic treatment stimulates the patient's immune system to attack the malignant tumor cells. Suitable examples of radiation therapies include, but are not limited to external beam radiotherapy (such as superficial X-rays therapy, orthovoltage X-rays therapy, megavoltage X rays therapy, radiosurgery, stereotactic radiation therapy, Fractionated stereotactic radiation therapy, cobalt therapy, electron therapy, fast neutron therapy, neutron-capture therapy, proton therapy, intensity modulated radiation therapy (IMRT), 3-dimensional conformal radiation therapy (3D-CRT) and the like).

[0031] [Incorporated by Reference (Rule 20.6), 22.07.2024]Prediction models in oncology are a valuable tool for improving cancer diagnosis, treatment, and management, and are increasingly being used in clinical settings to support decision-making and improve patient outcomes.

[0032] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment of the invention, any prediction model according to the invention disclosed herein may be used as a diagnostic tool, or for diagnose of a cancer, for retrieving biological data related to a cancer (including the stage of cancer, the severity of cancer), for predicting the outcome of a cancer.

[0033] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said prediction may be the survival rate or time of survival of said patient.

[0034] [Incorporated by Reference (Rule 20.6), 22.07.2024]There are several prediction metrics related to the survival of a patient that may be used to assess the prognosis of a patient.

[0035] [Incorporated by Reference (Rule 20.6), 22.07.2024]These metrics may include:- Overall Survival (OS): This refers to the length of time from the start of treatment until death from any cause.- Progression-Free Survival (PFS): This refers to the length of time from the start of treatment until the disease starts to progress or spread.- Disease-Free Survival (DFS): This refers to the length of time from the end of treatment until the disease returns or progresses.- Event-Free Survival (EFS): This refers to the length of time from the start of treatment until any event that indicates disease progression, such as the need for further treatment.- Relative Survival (RS): This refers to the survival rate of patients with a particular disease compared to the survival rate of the general population.- Recurrence-Free Survival (RFS): This refers to the length of time from the end of treatment until the disease returns.

[0036] [Incorporated by Reference (Rule 20.6), 22.07.2024]These metrics can be used to evaluate the effectiveness of different treatments and to help guide treatment decisions for patients with various conditions.

[0037] [Incorporated by Reference (Rule 20.6), 22.07.2024]A pre-trained model or trained model is a machine learning model that has already been trained on a dataset and is pre-configured to perform a prediction on a specific task.

[0038] [Incorporated by Reference (Rule 20.6), 22.07.2024]During training, the model is exposed to the training data and adjusts its parameters in response to the input data, with the goal of minimizing the difference between the model's predictions and the true values in the training data.

[0039] [Incorporated by Reference (Rule 20.6), 22.07.2024]The inference phase of a machine learning model involves using the trained model to make predictions on new, unseen data. In this phase, the model takes in input data and generates an output, based on the knowledge it has gained during the training phase. The inference phase is typically faster than the training phase since the model has already learned its parameters and does not need to adjust them further.

[0040] [Incorporated by Reference (Rule 20.6), 22.07.2024]The Shapley values of image feature tensors representing multiple different regions of an image indicate the contribution of each region to the prediction made by the prediction model.

[0041] [Incorporated by Reference (Rule 20.6), 22.07.2024]The Shapley value can provide insights into how different regions of an image contribute to the output of a machine learning model and can be useful for interpreting and explaining the model's behavior.

[0042] [Incorporated by Reference (Rule 20.6), 22.07.2024]More particularly, in the context of the present document, Shapley values calculate the average marginal contribution of a feature tensor towards a model score.

[0043] [Incorporated by Reference (Rule 20.6), 22.07.2024]In the following paragraphs, it will be detailed how the Shapley values assigned to regions of interest in an image can be calculated.

[0044] [Incorporated by Reference (Rule 20.6), 22.07.2024]Exact formulation

[0045] [Incorporated by Reference (Rule 20.6), 22.07.2024]Let us consider an imageIcharacterized by an ensemble of N regions . Each region is represented by a vector ofpfeatures. For simplicity it will now be considered that .Let us now consider the value function which takes as an input a subset of regions (i.e. ) and returns a prediction .vcan be written as follows: with withgan aggregation function which combines all the regions of a subset into a single vector oflfeatures andfa trained prediction model which returns a real value prediction given a vector oflfeatures.For instance,gcould be the mean over all features that characterize regionαijof the subset: For a regression task,fcould be a linear combination of the aggregated features computed byg(in this examplel = pbut in general it is not necessarily the case): The Shapley value of region of imageI, associated with the aggregation-prediction function , is defined as: WhereS(A)is the collection of all permutations of is a permutation ofA(e.g. ), and represents the set of regions ranked lower thanαiin the ordering defined by permutationσ.

[0046] [Incorporated by Reference (Rule 20.6), 22.07.2024]Approximation methods

[0047] [Incorporated by Reference (Rule 20.6), 22.07.2024]Computing the Shapley values of an image with a large numberNof regions can be computationally intractable sinceN!it requires operations (e.g for an image with 8 regions we need 40,320 operations to compute the exact Shapley value of one of these regions). Fortunately, the above formulation gives us a straightforward Monte Carlo scheme to estimate the Shapley value: withπ⊂S(A)a uniform sample of permutations of sizen. This Monte Carlo estimator is an unbiased estimator that converges asymptotically at rate according to the Central Limit Theorem.

[0048] [Incorporated by Reference (Rule 20.6), 22.07.2024]In order to increase the precision of the estimator, we can use antithetic sampling that is a variance reduction technique for Monte Carlo methods: whereσ-kcorresponds to the reversed permutation ofσk(e.gσk= (α1, α3, α4, α2), σ-k= (α2, α4, α3, α1)withN = 4)[6].

[0049] [Incorporated by Reference (Rule 20.6), 22.07.2024]The above-mentioned computer-implemented method may be implemented using images other than medical images, for example using digital histopathological images.

[0050] [Incorporated by Reference (Rule 20.6), 22.07.2024]A digital histopathological image is a high-resolution digital image of a tissue sample that has been stained and examined under a microscope to identify abnormalities and diagnose diseases. Histopathology is the study of tissue structures and the changes that occur at the cellular and molecular levels in response to disease or injury.

[0051] [Incorporated by Reference (Rule 20.6), 22.07.2024]In histopathology, tissue samples are typically obtained from biopsies or surgical specimens and are processed to create thin sections that can be mounted on slides and stained with various dyes to highlight different cellular structures and components. The resulting histopathological images can be digitized and examined by pathologists to identify abnormalities and diagnose diseases, such as cancer.

[0052] [Incorporated by Reference (Rule 20.6), 22.07.2024]Digital histopathological images can be very detailed, with high magnification and resolution, allowing pathologists to observe subtle changes in cellular structures and identify specific types of cells and tissues. Advances in digital imaging and artificial intelligence have also led to the development of automated methods for analyzing histopathological images, which can aid in diagnosis and treatment planning.

[0053] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said method may further comprise:(d) representing the influence of each region on said image, based on the Shapley value of the corresponding zone.

[0054] [Incorporated by Reference (Rule 20.6), 22.07.2024]The influence of each region on said image may be represented through a heat map.

[0055] [Incorporated by Reference (Rule 20.6), 22.07.2024]In the context of computer vision, a heat map is a graphical representation of the activation levels of individual pixels or region (i.e., region) in an image. Heat maps are commonly used to visualize the regions of an image that are most important or relevant to a specific task or application.

[0056] [Incorporated by Reference (Rule 20.6), 22.07.2024]Heat maps are created by applying a color map to an image, where the color of each pixel or region corresponds to its activation level or importance. Typically, warmer colors such as red, orange, and yellow may be used to indicate regions of high activation, while cooler colors such as blue and green may be used to indicate regions of low activation.

[0057] [Incorporated by Reference (Rule 20.6), 22.07.2024]Heat maps can here be used to highlight the regions of an image that are most likely to have a high influence on the prediction.

[0058] [Incorporated by Reference (Rule 20.6), 22.07.2024]Such heat map may be a useful tool, allowing a user to better understand the inner workings of the prediction model and visualize the regions of an image that are most relevant to the specific prediction.

[0059] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said image features of each region may comprise at least one radiomic feature of the corresponding zone.

[0060] [Incorporated by Reference (Rule 20.6), 22.07.2024]Radiomic features may be divided into at least said groups:- First Order Features- 3D shape features- 2D shape features- Gray Level Co-occurrence Matrix (GLCM) Features- Gray Level Size Zone Matrix (GLSZM) Features- Gray Level Run Length Matrix (GLRLM) Features- Neighboring Gray Tone Difference Matrix (NGTDM) Features- Gray Level Dependence Matrix (GLDM) Features

[0061] [Incorporated by Reference (Rule 20.6), 22.07.2024]More particularly:- First order features describe the distribution of voxel intensities within the image region defined by the mask through commonly used and basic metrics. Said first order features may comprise at least one of the following features: Energy, Total Energy, Entropy, Minimum, 10th percentile, 90th percentile, Maximum, Mean, Median, Interquartile Range, Range, Mean Absolute Deviation (MAD), Robust Mean Absolute Deviation (rMAD), Root Mean Squared (RMS), Standard Deviation, Skewness, Kurtosis, Variance, Uniformity.- 3D shape features are descriptors of the three-dimensional size and shape of the ROI. These features are independent from the gray level intensity distribution in the ROI and are therefore only calculated on the non-derived image and mask. Said 3D shape features may comprise at least one of the following features: Mesh Volume, Voxel Volume, Surface Area, Surface Area to Volume ratio, Sphericity, Compactness 1, Compactness 2, Spherical Disproportion, Maximum 3D diameter, Maximum 2D diameter (Slice, Row or Column), Major Axis Length, Minor Axis Length, Least Axis Length, Elongation, Flatness.- 2D shape features are descriptors of the two-dimensional size and shape of the ROI. These features are independent from the gray level intensity distribution in the ROI and are therefore only calculated on the non-derived image and mask. Said 2D shape features may comprise at least one of the following features: Mesh Surface, Pixel Surface, Perimeter, Perimeter to Surface ratio, Sphericity, Spherical Disproportion, Maximum 2D diameter, Major Axis Length, Minor Axis Length, Elongation.

[0062] [Incorporated by Reference (Rule 20.6), 22.07.2024]These radiomics features are for example defined at the URL https: / / pyradiomics.readthedocs.io / en / latest / features.html# and may be implemented using the library pyradiomics.

[0063] [Incorporated by Reference (Rule 20.6), 22.07.2024]At least part of the radiomics features is defined by the image biomarker standardization initiative (IBSI) which provide standardized image biomarker nomenclature and definitions (https: / / ibsi.readthedocs.io / en / latest / #).

[0064] [Incorporated by Reference (Rule 20.6), 22.07.2024]The mathematical definitions of these features are independent of imaging modality and can also be found in the literature [1], [2], [3], [4], [5].

[0065] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said image may be a 2D or 3D image.

[0066] [Incorporated by Reference (Rule 20.6), 22.07.2024]A 2D image is a two-dimensional representation of an object or scene, consisting of a rectangular grid of pixels, where each pixel represents a single point in the image. Pixels are small, discrete units that are typically arranged in rows and columns, and each pixel contains information about the color and intensity of light at that particular point in the image.

[0067] [Incorporated by Reference (Rule 20.6), 22.07.2024]On the other hand, a 3D image is a three-dimensional representation of an object or scene, consisting of a rectangular grid of voxels, where each voxel represents a small volume of space in the image. Voxels are similar to pixels, but instead of representing a single point in the image, they represent a small, three-dimensional cube of space. Each voxel contains information about the properties of the material or tissue at that particular location in the image.

[0068] [Incorporated by Reference (Rule 20.6), 22.07.2024]For 2D medical images, common modalities include:- X-ray: uses electromagnetic radiation to produce images of bones and other dense structures in the body- Ultrasound: uses high-frequency sound waves to produce images of soft tissues and organs For 3D medical images, some common modalities include:- CT: can create 3D images by stacking and processing multiple cross-sectional images obtained from the scan- MRI: can create 3D images by stacking and processing multiple 2D images obtained from the scan, or by using advanced techniques such as 3D volumetric imaging- Positron Emission Tomography (PET): uses a radioactive tracer emitting positrons to create 3D images of metabolic activity in the body- Single Photon Emission Computed Tomography (SPECT): uses a radioactive tracer emitting gamma photons to create 3D images of the distribution of the tracer in the body

[0069] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said prediction model comprises an ensemble of prediction sub-models, the prediction of said model being calculated based on the prediction of at least one sub-models.

[0070] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said sub-models are also called ensemble models.

[0071] [Incorporated by Reference (Rule 20.6), 22.07.2024]Ensemble models combine the predictions of multiple individual models (sub-models) to improve overall predictive accuracy. Once the individual models have made their predictions, the next step is to aggregate these predictions into a final prediction for the ensemble model.

[0072] [Incorporated by Reference (Rule 20.6), 22.07.2024]There are several methods that can be used to aggregate the prediction values of ensemble models, including:- Averaging: This involves simply taking the average of the individual model predictions. This is a common and straightforward method of aggregation.- Weighted averaging: This involves taking a weighted average of the individual model predictions, where the weights are determined based on the performance of each model on a validation set.- Stacking: This involves training a second-level model to combine the predictions of the individual models. The second-level model takes the predictions of the individual models as input features and makes a final prediction.

[0073] [Incorporated by Reference (Rule 20.6), 22.07.2024]The choice of aggregation method will depend on the specific characteristics of the data and the individual models being used. In general, averaging is a simple and effective method, while more complex methods such as stacking may be more effective when the individual models have different strengths and weaknesses. It may be useful to evaluate the performance of the ensemble model using appropriate validation techniques, and to carefully consider the trade-offs between performance and complexity when selecting an aggregation method.

[0074] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said prediction model or at least one of said sub-models may be a Cox proportional hazards model.

[0075] [Incorporated by Reference (Rule 20.6), 22.07.2024]A proportional hazards model, also known as a Cox proportional hazards model, is a type of survival analysis model used to analyze the relationship between one or more predictor variables and the time to an event of interest. It may be used to analyze the survival of patients with a particular disease or condition.

[0076] [Incorporated by Reference (Rule 20.6), 22.07.2024]The proportional hazards model assumes that the hazard rate (i.e., the probability of an event occurring at a given time) is proportional across different levels of the predictor variables. This means that the effect of each predictor variable is constant over time, and the hazard ratio (HR) between two groups is constant over time.

[0077] [Incorporated by Reference (Rule 20.6), 22.07.2024]The model estimates the hazard function, which is a function of time that describes the probability of an event occurring at any given time, and the hazard ratio, which represents the relative hazard of an event occurring between two groups. The hazard ratio may be used to compare the survival of two groups, such as a treatment group and a control group, or to identify factors that may be associated with increased or decreased risk of an event.

[0078] [Incorporated by Reference (Rule 20.6), 22.07.2024]The proportional hazards model can be extended to include multiple predictor variables, interactions between predictor variables, and time-dependent effects of predictor variables.

[0079] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said prediction model may be another suitable regression model, for example a Support Vector Machine model, Random Forest Model, or a Deep Learning Based model (for example a convolutional neural network or CNN).

[0080] [Incorporated by Reference (Rule 20.6), 22.07.2024]A Support Vector Machine (SVM) is a type of supervised learning algorithm used for classification or for regression analysis. It works by finding the hyperplane that best separates the different classes in the data. SVMs are known for their ability to handle high-dimensional data and non-linear relationships between variables.

[0081] [Incorporated by Reference (Rule 20.6), 22.07.2024]A Random Forest Model is an ensemble learning algorithm that combines multiple decision trees to make a final prediction. It works by randomly selecting subsets of features and samples from the data to build multiple decision trees, which are then combined to make a more robust prediction. Random Forest Models are known for their ability to handle noisy and unbalanced data, and for their ability to identify important features in the data.

[0082] [Incorporated by Reference (Rule 20.6), 22.07.2024]A Convolutional Neural Network (CNN) is a type of deep learning algorithm adapted for image data. It works by using convolutional layers to extract features from the input data, followed by pooling layers to reduce the dimensionality of the data. The resulting features are then passed through fully connected layers to make a prediction. CNNs are known for their ability to automatically learn features from the data, and for their ability to handle large and complex datasets.

[0083] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said regions may comprise at least one tumoral region of said image.

[0084] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment of the invention, said regions are issued from a tumoral region of a liquid (hematological) cancer or a solid cancer. In a particular embodiment, said regions are issued from a tumoral region of a bladder cancer, bone cancer, brain cancer, breast cancer, cervical cancer, colon cancer, esophageal cancer, gastric cancer, head & neck cancers, leukemia, liver cancer, lung cancer, melanoma, mesothelioma, multiple myeloma, myelodysplastic syndrome, ovarian cancer, pancreatic cancer, prostate cancer, rectal cancer, renal cancer, sarcoma, skin cancer, testicular cancer, thyroid cancer or uterine cancer.

[0085] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said region is preferably issued from a hematologic cancer, in particular Hodgkin lymphoma; non-Hodgkin lymphoma; myeloma; leukemia, including Acute Lymphocytic Leukemia, Chronic Lymphocytic Leukemia, Acute Myeloid Leukemia, Chronic Myeloid Leukemia.

[0086] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said region may also preferably be issued from a lung cancer, in particular breast cancer corresponding to ductal carcinoma, lobular carcinoma, invasive breast cancer, inflammatory breast cancer, metastatic breast cancer, hormone receptor positive breast cancer, hormone receptor negative cancer, HER2 positive breast cancer, HER2 negative breast cancer, triple-negative breast cancer. In a more particular embodiment, said regions are issued from a triple-negative breast cancer

[0087] [Incorporated by Reference (Rule 20.6), 22.07.2024]In particular, the cancer may be metastatic.

[0088] [Incorporated by Reference (Rule 20.6), 22.07.2024]A tumoral region refers to a region of abnormal tissue growth or mass, typically caused by uncontrolled or abnormal cell proliferation. This can occur in various organs or tissues throughout the body, including the brain, breast, lung, liver, and skin, among others.

[0089] [Incorporated by Reference (Rule 20.6), 22.07.2024]The presence and location of a tumoral region can be a key factor in the diagnosis and treatment of various medical conditions, such as cancer.

[0090] [Incorporated by Reference (Rule 20.6), 22.07.2024]The characteristics of a tumoral region may vary depending on the type of tissue involved and the stage of the disease. Tumors can be benign, meaning they do not spread to other parts of the body, or malignant, meaning they have the potential to spread and cause harm to surrounding tissues or organs. In some cases, tumors may also be classified based on their location or the type of cells involved in the abnormal growth.

[0091] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said image may represent at least partly the lungs of a patient.

[0092] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said regions may comprise at least one at least one region of an organ of a patient.

[0093] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said organ may be the spleen of the patient.

[0094] [Incorporated by Reference (Rule 20.6), 22.07.2024]The present document also concerns a computer software, comprising instructions to implement at least a part of the above-mentioned method, when the software is executed by a processor.

[0095] [Incorporated by Reference (Rule 20.6), 22.07.2024]The present document also concerns a computer device comprising:- an input interface to receive at least one medical image,- a memory for storing at least instructions of a computer program according to the preceding claim,- a processor accessing to the memory for reading the aforesaid instructions and executing then the method according to any of the above-mentioned method,- an output interface to provide an indication based on the Shapley value of the features of each region.

[0096] [Incorporated by Reference (Rule 20.6), 22.07.2024]An input interface for receiving an image is a hardware or software component that allows a user or system to input an image into a computer or other digital device for processing or analysis. There are several types of input interfaces for receiving images, including:

[0097] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said input interface may comprise file upload or network protocols.

[0098] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said input interface may be an interface between a medical imaging machine and a computer. Said interface may involves a data transfer protocol that allows images to be transmitted from the imaging machine to the computer for processing and analysis.

[0099] [Incorporated by Reference (Rule 20.6), 22.07.2024]Some common software or hardware components of an input interface for medical imaging might include DICOM (Digital Imaging and Communications in Medicine) protocol, PACS (Picture Archiving and Communication System) or Network interfaces. Alternatively, images may be transferred via removable media such as USB drives or CDs.

[0100] [Incorporated by Reference (Rule 20.6), 22.07.2024]A memory for storing at least instructions of a computer program is known as a program memory or instruction memory. This type of memory may store the instructions that the processor needs to execute a program, along with any associated data.

[0101] [Incorporated by Reference (Rule 20.6), 22.07.2024]Several types of memory may be used, including Read-Only Memory (ROM), Flash Memory, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random-Access Memory (RAM), Cache Memory and / or Virtual Memory.

[0102] [Incorporated by Reference (Rule 20.6), 22.07.2024]The present document also concerns a computer-readable non-transient recording medium on which a computer software is registered to implement the above-mentioned method when the computer software is executed by a processor.

[0103] [Incorporated by Reference (Rule 20.6), 22.07.2024]The present document also concerns a computer-readable non-transient recording medium is a storage medium that is capable of storing data or information that can be read by a computer system.

[0104] [Incorporated by Reference (Rule 20.6), 22.07.2024]The term "non-transient" indicates that the data stored on the medium remains even after the medium is removed from the computer system or the power is turned off.

[0105] [Incorporated by Reference (Rule 20.6), 22.07.2024]Examples of computer-readable non-transient recording media include hard disk drives, solid-state drives, optical disks (such as CDs or DVDs), USB drives, and memory cards. These storage devices are commonly used to store software programs, data files, and other digital content that can be accessed and processed by a computer system.

[0106] [Incorporated by Reference (Rule 20.6), 22.07.2024]In contrast, a transient medium is one that only temporarily stores data and is not capable of retaining the data once it is removed from the computer system or the power is turned off. Examples of transient media include computer memory (such as RAM) and cache memory.

[0107] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said output interface may comprise a display such as a monitor, a projector, a mobile device screen or a virtual reality headset screen.

[0108] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment of the invention, the prediction model is used as a diagnostic tool or in a method of diagnostic or for the diagnosis of a cancer, for retrieving biological data related to a cancer (including the stage of cancer, the severity of cancer), or for predicting the outcome of a cancer.

[0109] [Incorporated by Reference (Rule 20.6), 22.07.2024]To this end, the method of the invention may comprise the provision of a medical image from a patient for whom a diagnostic has to be made, and / or for whom biological data related to a cancer have to be retrieved, and / or for whom prediction of the outcome of a cancer is sought. The patient may be suspected of having a cancer, in particular any cancer disclosed herein. The patient may have been diagnosed with a cancer, in particular any cancer disclosed herein. When the patient has or had cancer, the patient may be treated or may have been treated by a cancer therapy.

[0110] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment, the method of the invention includes a step of determining whether the patient has cancer.

[0111] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment, the method of the invention includes a step of determining whether the patient positively respond to a treatment against cancer.

[0112] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment, the method of the invention is a method for treating cancer in a patient in need thereof, wherein the method comprises a step of determining that the patient has a cancer. In a preferred embodiment, the method further comprises a step of administering to the patient an effective amount of a therapeutic agent or treating the patient with a cancer therapy, according to the standard of care (SOC) recommendations for treating the cancer which affects the patient.

[0113] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment, the invention is a method for diagnosing a cancer, wherein any method according to the invention disclosed herein further comprises a step of determining whether or not the patient has a cancer, in particular a cancer as listed herein.

[0114] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment, the invention relates to a method for treating a patient in need thereof, wherein the patient has been determined to have a cancer by performing the method of the invention, wherein after performing the method according to the invention, a step of treating the patient with a therapeutic agent or treating the patient with a cancer therapy, according to the standard of care (SOC) recommendations for treating the cancer which affects the patient is performed.

[0115] [Incorporated by Reference (Rule 20.6), 22.07.2024]In an embodiment, the invention is a method for predicting the outcome of a cancer, wherein any method according to the invention disclosed herein further comprises a step of predicting the outcome of a cancer in a patient. The outcome includes the Overall Survival (OS), the Progression-Free Survival (PFS), the Disease-Free Survival (DFS), Event-Free Survival (EFS), the Relative Survival (RS), and / or the Recurrence-Free Survival (RFS). An oncologist is able to determine these outcomes according to his knowledge.BRIEF DESCRIPTION OF THE DRAWINGS

[0116] [Incorporated by Reference (Rule 20.6), 22.07.2024]Other features, details and advantages will be shown in the following detailed description and on the figures, on which:- figure 1 schematically shows an example of a computer device according to the present document,- figure 2 is a diagram illustrating an example of the method of the present document- figure 3 shows the Shapley values for different regions and PET images of two different patients.The annexed drawing includes meaningful colors. Although the present application is to be published in black and white, a colored version of the annexed drawing was filed before the European Patent Office.DETAILED DESCRIPTION OF AN EMBODIMENT

[0117] [Incorporated by Reference (Rule 20.6), 22.07.2024]Figure 1 schematically shows an example of a computer device 1 according to the invention. Said computer device 1 comprises:- an input interface 2,- a memory 3 for storing at least instructions of a computer program,- a processor 4 accessing to the memory 3 for reading the aforesaid instructions and executing the method according to the present document,- an output interface 5.

[0118] [Incorporated by Reference (Rule 20.6), 22.07.2024]The annexed drawing includes meaningful colors. Although the present application is to be published in black and white, a colored version of the annexed drawing is filed before the Office.

[0119] [Incorporated by Reference (Rule 20.6), 22.07.2024]General Method

[0120] [Incorporated by Reference (Rule 20.6), 22.07.2024]As mentioned above, the present document proposes a method (see figure 2) implemented by computer means for determining the influence of a plurality of regions in a medical image I of a patient on a predictionPof a pretrained prediction modelPMbased on said imageI, comprising:(a) determining image featuresF1, F2, F3of a plurality of regionsA1, A2, A3on said imageI,(b) transforming said featuresF1, F2, F3of each regionA1, A2, A3in an aggregated tensor (for example a vector)Finputted into said pretrained prediction modelPM,(c) calculating, based on the featuresF1, F2, F3,the Shapley value of each regionA1, A2, A3,said Shapley value representing the influence of said regionA1, A2, A3on said predictionP.

[0121] [Incorporated by Reference (Rule 20.6), 22.07.2024]Prediction Model

[0122] [Incorporated by Reference (Rule 20.6), 22.07.2024]In the following paragraphs, we will discuss the process of training the prediction model.

[0123] [Incorporated by Reference (Rule 20.6), 22.07.2024]The dataset comprises positron emission tomography (PET) with 2-deoxy-2-[fluorine-18]fluoro-Dglucose (18F-FDG) images I of 100 metastatic Non-Small Cell Lung Cancer (NSCLC) patients treated with a combination of immunotherapy and chemotherapy (i.e., 100 images).

[0124] [Incorporated by Reference (Rule 20.6), 22.07.2024]All tumor fociA1, A2, A3were automatically segmented and each segmentation was manually adjusted by an experienced nuclear medicine physician.

[0125] [Incorporated by Reference (Rule 20.6), 22.07.2024]The number of tumor foci may vary.

[0126] [Incorporated by Reference (Rule 20.6), 22.07.2024]Tumor foci refer to localized areas or regions of a tumor that can be visually distinguished from the surrounding tissue. Tumor foci can be used to describe the size and extent of a tumor and can be a crucial factor in determining the severity of the disease and the appropriate treatment options.

[0127] [Incorporated by Reference (Rule 20.6), 22.07.2024]For each tumor lesion, first order and 2D and / or 3D shape radiomic featuresF1, F2, F3have been calculated (FE) using pyradiomics. For each image of the image dataset, said image featuresF1, F2, F3have been aggregated (FA) into an aggregated tensorF(for example a vector) over all tumor lesions with a simple function (e.g.,average, max, standard deviation...).

[0128] [Incorporated by Reference (Rule 20.6), 22.07.2024]The prediction modelPMcomprises an ensemble of several Cox proportional hazards sub-models. The outputs of the sub-models are standardized and averaged to obtain a final risk score as a prediction valueP.

[0129] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said prediction model is trained based on the 100 aggregated tensors calculated on the basis of the 100 images of the dataset.

[0130] [Incorporated by Reference (Rule 20.6), 22.07.2024]We assessed the performance of this ensembling method by estimating the concordance index with a cross-validation scheme. The aggregated radiomic model succeeded in capturing prognostic information for overall survival in the images (C-index cross-validation: 0.668).

[0131] [Incorporated by Reference (Rule 20.6), 22.07.2024]As mentioned above, in machine learning, ensembling is a technique where multiple models are trained on the same data and their outputs are combined to make a final prediction. The purpose of ensembling is to improve the accuracy and robustness of the final prediction.

[0132] [Incorporated by Reference (Rule 20.6), 22.07.2024]The performance of an ensembling method can be evaluated using various metrics, such as accuracy, precision, recall, and the concordance index (also known as the C-index). The concordance index is a metric particularly relevant in survival analysis, where it measures the ability of a model to correctly rank the survival times of individuals.

[0133] [Incorporated by Reference (Rule 20.6), 22.07.2024]To estimate the concordance index of an ensembling method, the data is split into several subsets or folds, using a cross-validation scheme. In each fold, a different subset of the data is used for testing and the rest is used for training the ensembling method. This process is repeated several times, with different subsets of the data used for testing in each iteration.

[0134] [Incorporated by Reference (Rule 20.6), 22.07.2024]The concordance index is then calculated for each iteration, using the predictions of the ensembling method on the test data. The average of these concordance indices across all the iterations is taken as the final estimate of the performance of the ensembling method.

[0135] [Incorporated by Reference (Rule 20.6), 22.07.2024]By using a cross-validation scheme, we can get a more accurate estimate of the performance of the ensembling method, as it is tested on data that it has not seen during training. This helps to avoid overfitting, which is a widespread problem in machine learning where the model performs well on the training data but poorly on new, unseen data.

[0136] [Incorporated by Reference (Rule 20.6), 22.07.2024]Assessing the performance of an ensembling method using the concordance index with a crossvalidation scheme is a rigorous and reliable way to evaluate the effectiveness of the method.

[0137] [Incorporated by Reference (Rule 20.6), 22.07.2024]Finally, we applied our interpretation tool to patients with extreme predicted risk, concordant with their survival (i.e., short survival and high risk, long survival and low risk).

[0138] [Incorporated by Reference (Rule 20.6), 22.07.2024]Interpretation of the prediction model

[0139] [Incorporated by Reference (Rule 20.6), 22.07.2024]An interpretation process is then performed to effectively complement the feature importanceanalysis by providing insight into the specific behavior of the model for each patient.

[0140] [Incorporated by Reference (Rule 20.6), 22.07.2024]To perform such interpretation of the trained prediction model, the Shapley value of the features of each region is calculated (or approximation of such Shapley values), each Shapley value representing the influence of the corresponding region on said prediction.

[0141] [Incorporated by Reference (Rule 20.6), 22.07.2024]Representation of the Shapley values

[0142] [Incorporated by Reference (Rule 20.6), 22.07.2024]Then the influence of each region may be represented on each medical image, based on the Shapley value of the corresponding zone.

[0143] [Incorporated by Reference (Rule 20.6), 22.07.2024]Said representation may be represented through a heat map.

[0144] [Incorporated by Reference (Rule 20.6), 22.07.2024]Figure 3 shows the Shapley values in form of heat maps, for 6 NSCLC patients with low predicted risk and long survival (top row) and 5 patients with high predicted risk and short survival (bottom row). In this figure, the Maximum Intensity Projection (MIP) PET images are used to visualize all the Shapley values together. Blues lesions correspond to those that contribute to lower the risk prediction while red ones correspond to those that contribute to increase it.

[0145] [Incorporated by Reference (Rule 20.6), 22.07.2024]As shown in figure 3, when using the representation of the Shapley values, it has appeared that for some patients with long survival, the low-risk predictions made by the prediction model were significantly influenced by distant bone metastases while for some patients with short survival, their high-risk predictions were influenced by high volume lesions but also by specific metastatic sites like supraclavicular or cervical lymph nodes.

[0146] [Incorporated by Reference (Rule 20.6), 22.07.2024]Such representation makes it easy to identify the lesions that contributed most to each patient's prognosis.

[0147] [Incorporated by Reference (Rule 20.6), 22.07.2024]The following paragraphs describe an experimental implementation of the method according to the present document. Said implementation is a non-limiting example of said method.

[0148] [Incorporated by Reference (Rule 20.6), 22.07.2024]Data

[0149] [Incorporated by Reference (Rule 20.6), 22.07.2024]In this example, baseline 18F-FDG-PET images of 100 metastatic non-small-cell-lung cancer patients treated with first-line immune checkpoint inhibitors combined with platinum-based chemotherapy are collected. All tumor foci are automatically segmented and manually adjusted by an experienced nuclear medicine physician. The patients' response to treatment are monitored using Overall Survival (OS), Progression-Free Survival (PFS) and the best overall RECIST response observed during follow-up. Clinical data (e.g., sex, age, BMI, ...) are available for all the patients.

[0150] [Incorporated by Reference (Rule 20.6), 22.07.2024]Extraction of radiomic features

[0151] [Incorporated by Reference (Rule 20.6), 22.07.2024]Before extracting image (aka radiomic) features, PET images are all resampled to a voxel size of 2mm3using a cubic B-spline interpolator. All the segmented tumor regions are post-processed with an absolute threshold of 2.5 standardized uptake value (SUV) units, meaning that all voxel values less than 2.5 are excluded from the segmented tumor regions.

[0152] [Incorporated by Reference (Rule 20.6), 22.07.2024]For each segmented tumor, ten shape and first order radiomic features are extracted using the pyradiomics (v3.0.1) [7] python package.

[0153] [Incorporated by Reference (Rule 20.6), 22.07.2024] Table 1: Summary of the ten radiomic features extracted with pyradiomics for each segmented lesion of each patient. The quartile dispersion feature is computed using the corresponding IBSI formula [5].

[0154] [Incorporated by Reference (Rule 20.6), 22.07.2024]Aggregation of radiomic features at the patient level

[0155] [Incorporated by Reference (Rule 20.6), 22.07.2024]Six aggregation strategies are used to combine the information coming from the different lesions ofa patient into a single vector of features:- The mean value, the maximum value, and the standard deviation of each radiomic feature across all the lesions of the patient.- The Total Metabolic Tumor Volume (TMTV) for each patient (i.e., the sum of the volumes of all the lesions).- The maximum Euclidean distance between the 2 lesions that were the farthest apart for each patient (Dmax) [8].- The number of organs including a lesion for each patient.

[0156] [Incorporated by Reference (Rule 20.6), 22.07.2024] Table 2: Clinical features collected for each patient at baseline.

[0157] [Incorporated by Reference (Rule 20.6), 22.07.2024]Survival prediction

[0158] [Incorporated by Reference (Rule 20.6), 22.07.2024]In this example focus is made on the prediction of the Overall Survival. An ensemble of penalized Cox models is trained to predict a risk score for each patient. The CoxnetSurvivalAnalysis function of the python package scikit-survival is used with hyperparameters l1_ratio = 0.1 and alphas = [0.1]. Each Cox model is preceded by a standardization step (pre-processing).Three different Cox models are combined:1. A Cox model with the aggregated mean value, max value, and standard deviation of each radiomic feature (30 input features in total).2. A Cox model with TMTV, Dmax, the mean intensity of a healthy region of the liver, the mean intensity of a healthy region of the spleen and the number of organs including a tumor.3. A Cox model with the clinical features listed in table 2.The final risk score predicted by the ensemble model is computed as the average of the risk scores predicted by the three models. These risk scores are standardized beforehand using the mean and the standard deviation of the predictions made by each model on the training data set.Some patient-level features like clinical variables or the mean intensity of healthy regions in the liver and the spleen are used as input of the predictive model. These features are transformed into lesionlevel features by considering that their values are the same for every lesion of the same patient (e.g., if the patient is 64 years old the age variable will have the value 64 for every lesion of this patient). Creating these lesion-level features allows to comply with the framework described above to compute Shapley values.

[0159] [Incorporated by Reference (Rule 20.6), 22.07.2024]Statistical analysis

[0160] [Incorporated by Reference (Rule 20.6), 22.07.2024]In order to estimate the generalization performance of the proposed ensemble model a 5-fold crossvalidation scheme is used. To avoid any data leakage, all feature scaling operations are performed on the training set of each fold. Uno's concordance index is computed on the test set of each fold using the function concordance_index_ipcw of the python package scikit-survival. The five concordance indices are then averaged to obtain a final estimate of the performance of the proposed model.The ensemble model is then re-trained on the full data set. Finally, the Shapley values of the lesions of the patients are computed with the lowest predicted risks as well as those with the highest predicted risks.

[0161] [Incorporated by Reference (Rule 20.6), 22.07.2024]References

[0162] [Incorporated by Reference (Rule 20.6), 22.07.2024][1] Galloway, Mary M (1975). "Texture analysis using gray level run lengths". Computer Graphics and Image Processing. 4 (2): 172 179. Doi:10.1016 / S0146-664X(75)80008-6.[2] Pentland AP (June 1984). "Fractal-based description of natural scenes". IEEE Transactions on Pattern Analysis and Machine Intelligence. 6 (6): 661 74. Doi:10.1109 / TPAMI.1984.4767591. PMID 22499648. S2CID 17415943.[3] Amadasun M, King R (1989). "Textural features corresponding to textural properties". IEEE Transactions on Systems, Man, and Cybernetics. 19 (5): 1264 1274. Doi:10.1109 / 21.44046.[4] Thibault G, Angulo J, Meyer F (March 2014). "Advanced statistical matrices for texture characterization: application to cell classification". IEEE Transactions on Bio-Medical Engineering. 61(3): 630 7. Doi:10.1109 / TBME.2013.2284600. PMID 24108747. S2CID 11319154.[5] Zwanenburg A, Vallières M, Abdalah MA, Aerts HJWL, Andrearczyk V, Apte A, Ashrafinia S, Bakas S, Beukinga RJ, Boellaard R, Bogowicz M, Boldrini L, Buvat I, Cook GJR, Davatzikos C, Depeursinge A, Desseroit MC, Dinapoli N, Dinh CV, Echegaray S, El Naqa I, Fedorov AY, Gatta R, Gillies RJ, Goh V, Götz M, Guckenberger M, Ha SM, Hatt M, Isensee F, Lambin P, Leger S, Leijenaar RTH, Lenkowicz J, Lippert F, Losnegård A, Maier-Hein KH, Morin O, Müller H, Napel S, Nioche C, Orlhac F, Pati S, Pfaehler EAG, Rahmim A, Rao AUK, Scherer J, Siddique MM, Sijtsema NM, Socarras Fernandez J, Spezi E, Steenbakkers RJHM, Tanadini-Lang S, Thorwarth D, Troost EGC, Upadhaya T, Valentini V, van Dijk LV, van Griethuysen J, van Velden FHP, Whybra P, Richter C, Lock S (2020). "The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput[6] Mitchell R, Cooper J, Frank E, Holmes G. Sampling permutations for Shapley value estimation. J Mach Learn Res. 2022 Jan;23:1-46.[7] van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, Beets-Tan RGH, Fillion-Robin JC, Pieper S, Aerts HJWL. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017 Nov 1;77(21):e104-e107.[8] Cottereau AS, Nioche C, Dirand AS, Clerc J, Morschhauser F, Casasnovas O, Meignan M, Buvat I.18F-FDG PET Dissemination Features in Diffuse Large B-Cell Lymphoma Are Predictive of Outcome. J Nucl Med. 2020 Jan;61(1):40-45.

Claims

[Incorporated by Reference (Rule 20.6), 22.07.2024]A method implemented by computer means for determining the influence of a plurality of regions in a medical image of a patient on a prediction of a pretrained prediction model based on said image, comprising:(a) determining image features of a plurality of regions on said image,(b) transforming said features of each region in an aggregated tensor inputted into said pretrained prediction model,(c) calculating, based on said features, the Shapley value of each region, said Shapley value representing the influence of said region on said prediction.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to claim 1, further comprising:(d) representing the influence of each region on said image, based on the Shapley value of the corresponding zone.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to one of the preceding claims, wherein said image features of each region comprise at least one radiomic feature of the corresponding zone.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to any of the preceding claims, wherein said image is a 2D or 3D image.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to any of the preceding claims, wherein said prediction model comprises an ensemble of prediction sub-models, the prediction of said model being calculated based on the prediction of at least one sub-models.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to any of the preceding claims, wherein said prediction model or at least one of said sub-model is a Cox proportional hazards model.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to any of the preceding claims, wherein said regions comprise at least one tumoral region of said image.[Incorporated by Reference (Rule 20.6), 22.07.2024]The method according to any of the preceding claims, wherein said regions comprise at least one region of an organ of a patient.[Incorporated by Reference (Rule 20.6), 22.07.2024]A computer software, comprising instructions to implement at least a part of the method according to any of the preceding claims, when the software is executed by a processor.[Incorporated by Reference (Rule 20.6), 22.07.2024]A computer device comprising:- an input interface to receive at least one medical image,- a memory for storing at least instructions of a computer program according to the preceding claim,- a processor accessing to the memory for reading the aforesaid instructions and executing then the method according to any of the method according to claim 1 to 8,- an output interface to provide an indication based on the Shapley value of the features of each region.[Incorporated by Reference (Rule 20.6), 22.07.2024]A computer-readable non-transient recording medium on which a computer software is registered to implement the method according to any of the claims 1 to 8, when the computer software is executed by a processor.

Citation Information

Patent Citations

  • Assessment of endothelial cells and corneas at risk from ophthalmological images

    US20210052157A1

  • Disease characterization and response estimation through spatially-invoked radiomics and deep learning fusion

    US20210097682A1

Cited By

  • Medical image texture analysis method, system and equipment based on artificial intelligence and medium

    CN121505002A