Method for classifying medical images of a patient's tissue

Abstract

The present invention relates to a method implemented by computer means for classifying at least one medical image of at least one tissue of a patient, and to a method implemented by computer means, the method comprising the following iterative steps, namely, (a) segmenting at least one region of interest of the image, the region representing at least one region of the tissue; (b) extracting at least one feature from the region of interest; (c) classifying the region of interest based on the feature; (d) outputting the classification of the region of interest and including an inference phase.

Description

Technical Field

[0001] The present invention relates to a method implemented by computer means for classifying at least one medical image of at least one tissue of a patient.

Background Art

[0002] Pancreatic cancer is characterized by a low 5-year survival rate (5-10%) due to late diagnosis. In fact, the early stages of the disease are asymptomatic or dominated by non-specific symptoms. Furthermore, at these stages, potential tumors are small (<2 cm) or isodense, which makes it difficult for radiologists to detect them. Therefore, the disease remains undiagnosed, the cancer grows, and by the time the cancer has progressed considerably, such intervention becomes impossible even though surgical intervention is the only treatment option. It has been found that only 10% of patients are eligible for surgery.

[0003] In such a situation, it is important to diagnose the disease at an early stage in order to improve eligibility for surgery and thus overall survival. Recently, many studies have shown that there are predictive signs of pancreatic cancer that can be seen on CT (computed tomography) images up to 15 months before cancer diagnosis. One of the most important of these signs is the dilation of the main pancreatic duct, which should alert the radiologist to the potential presence of a tumor and encourage the radiologist to prescribe complementary tests for the patient. However, since the main pancreatic duct is a small structure within the pancreas, it is difficult to detect this sign and it can easily be missed, especially when examining the entire CT abdominal scan.

[0004] Currently, the only way to detect the dilation of the main bronchus is to carefully examine the CT scan, manually measure the diameter of the bronchus on the image, and compare it with the standard value. However, this is a tedious task, and not all radiologists are aware of the importance of looking for such signs. Furthermore, since bronchial cancer remains a rare disease and radiologists are pressured by a large number of examinations to be reviewed in a short time, radiologists will not necessarily examine this important sign.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

[0006] Since there is currently no automatic tool for detecting the dilation of the portal CT scanner based on its analysis, the present invention proposes a pipeline for automatically analyzing this type of image and detecting the dilation of the main bronchus.

[0007] Such a pipeline may also be useful for detecting other diseases based on other tissues of the patient.

Means for Solving the Problem

[0008] Therefore, this document proposes a method implemented by computer means for classifying at least one medical image of at least one tissue of a patient, said method comprising the following iterative steps, namely: (a) segmenting at least one region of interest of said image, said region representing at least one region of said tissue; (b) extracting at least one feature from said region of interest; (c) classifying said region of interest based on said feature; (d) outputting said classification of said region of interest including an inference phase.

[0009] In digital image processing and computer vision, image segmentation is a method aimed at dividing a digital image into at least one segment or a set of pixels. More precisely, image segmentation is a process of assigning labels to every pixel or voxel in an image, as a result of which pixels having the same label share a certain feature.

[0010] In the context of this document, the result of step (a) is a set of pixels or voxels belonging to the corresponding region of said tissue.

[0011] Furthermore, in digital image processing and computer vision, a feature is a piece of information about the content of an image, typically about whether a certain region of the image has a certain property. A feature may be a specific structure in the image such as a point, an edge, or an object. A feature may also be the result of a general neighborhood operation or feature detection applied to the image. Other examples of features relate to motion in an image sequence or to a shape defined with respect to a curve or a boundary line between different image regions.

[0012] More generally, a feature is a piece of information relevant to solving a computer task related to an application. This is generally the same meaning as a feature in machine learning and pattern recognition, but image processing has a very high set of features.

[0013] In the context of the present invention, classification becomes the problem of identifying to which category or set of classifications an observation part (here, the region of interest) belongs. The classification may include a first classification or normal classification and a second classification or abnormal classification.

[0014] Thus, the present disclosure proposes a pipeline in which, instead of directly classifying an input image, classification is performed on features extracted from segmented regions of the input image.

[0015] Such a pipeline makes it possible to better detect different classifications in regions of medical images that are usually difficult to classify.

[0016] The image may be a CT scan image, such as a portal CT scan image.

[0017] The image can be provided by other radiation modalities, such as MRI, PET / CT, or PET / MR.

[0018] The tissue may be the pancreas.

[0019] At least one region of interest may include a region representing the main pancreatic duct.

[0020] At least one region of interest may include a region representing the pancreas.

[0021] At least one region of interest may include a region representing a pancreatic lesion.

[0022] In addition to the main duct, more specific results can be obtained by segmenting the kidney and / or any kidney lesions, and when segmenting the main duct, it helps to ensure that the duct does not mix with, for example, part of the kidney or part of the lesion.

[0023] Step (a) can be implemented using a fully convolutional neural network.

[0024] The fully convolutional neural network may be nnUNet.

[0025] nnUNet automatically designs a segmentation pipeline based on the UNet architecture by relying on a heuristic problem-solving method applied to the data that enables nnUNet to estimate key parameters. To automatically perform the preprocessing steps, the dataset characteristics are estimated. This involves the automatic specification of options in the model design (number of layers, convolutional kernel size, convolutional blocks, etc.). The training procedure (data augmentation, schedule learning rate, etc.) is also implemented.

[0026] Step (b) can be implemented using radiomics, by means of size and shape-based features, descriptors of the image intensity histogram, and / or descriptors of the relationships between image voxels.

[0027] In the medical field, radiomics is a method of extracting a large number of features from medical images using data characterization algorithms. These features, called radiomics features, have the potential to discover patterns and characteristics on images that are difficult to understand with the naked eye.

[0028] This combination of parameters seems to be the most relevant for the purposes of this document.

[0029] Radiomics features can be divided into the following five groups. - Size- and shape-based features, - Descriptors of image intensity histograms, - Descriptors of relationships between image voxels (e.g., gray-level co-occurrence matrix (GLCM), run-length matrix (RLM), size-zone matrix (SZM), and neighborhood gray-tone difference matrix (NGTDM)-derived texture), - Texture extracted from filtered images, and - Fractal features

[0030] The mathematical definitions of these features are independent of the imaging modality and can be found in the literature. - Galloway, Mary M (1975). "Texture analysis using gray level run lengths". Computer Graphics and Image Processing 4 (2), 172–179, doi:10.1016 / S0146-664X(75)80008-6 - Pentland AP (June 1984). "Fractal-based description of natural scenes". IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (6), 661–74, doi:10.1109 / TPAMI.1984.4767591, PMID 22499648, S2CID 17415943 - Amadasun M, King R (1989). "Textural features corresponding to textural properties". IEEE Transactions on Systems, Man, and Cybernetics 19 (5), 1264–1274, doi:10.1109 / 21.44046 - Thibault G, Angulo J, Meyer F (March 2014), "Advanced statistical matrices for texture characterization: application to cell classification", IEEE Transactions on Bio-Medical Engineering 61 (3), 630-637, doi:10.1109 / TBME.2013.2284600, PMID 24108747, S2CID 11319154

[0031] Step (c) can be implemented using a Support Vector Machine classifier.

[0032] Step (c) can be implemented using a Support Vector Machine classifier with a Radial Basis Function kernel.

[0033] Due to the very diverse available features, feature reduction can be performed to remove redundant information. Additionally, features that are unstable and non-reproducible can be removed. This is because low-fidelity features can lead to false discoveries and non-repeatable models.

[0034] Such feature reduction or selection improves the classification performance in the next step.

[0035] For this purpose, the method can include a training phase in which the features in step (b) are selected by quantifying the distribution of the features in each of the classifications and selecting features if there are statistically significant differences in the distributions between the classifications.

[0036] The difference in distribution between the classifications can be quantified using a Student's t-test.

[0037] The Student's t-test is a statistical test for comparing the means of two groups of samples. The purpose of the test is to determine whether the means of the two groups are statistically significantly different.

[0038] The method can include a training phase in which the features of step (b) are selected by performing hierarchical clustering analysis.

[0039] In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis that requires constructing a hierarchy of clusters.

[0040] Strategies for hierarchical clustering fall into two types. - Agglomerative. This is a "bottom-up" approach. Each observation starts as its own cluster, and pairs of clusters merge as the hierarchy rises. - Divisive. This is a "top-down" approach. All observations start in one cluster, and as the hierarchy descends, recursive splitting is performed.

[0041] The results of hierarchical clustering are usually represented by a dendrogram.

[0042] To determine which clusters should be combined (in the agglomerative approach) or where the clusters should be split (in the divisive approach), a measure of dissimilarity between pairs of observations is required. Such a measure can be achieved by using an appropriate measure method (measurement of the distance between pairs of observations) and a linkage criterion that defines the dissimilarity of the pairs as a function of the pairwise distances of the observations in the pairs.

[0043] The choice of an appropriate measure method will affect the shape of the clusters because some elements may be relatively closer to each other under one measure method than another.

[0044] Some known measure methods for hierarchical clustering are as follows. - Euclidean distance - Squared Euclidean distance - Manhattan (or city block) distance - Maximum distance (or Chebyshev distance) - Mahalanobis distance

[0045] Hierarchical clustering analysis can be performed using single-link clustering.

[0046] In statistics, single-link clustering is one of several methods of hierarchical clustering. Single-link clustering is based on grouping clusters in a bottom-up fashion (agglomerative clustering), and at each step, it combines two clusters that contain the closest pair of elements that do not yet belong to the same cluster as each other.

[0047] The clusters may be formed using a correlation distance.

[0048] The distance is given by the following formula.

[0049]

Number

[0050] In the above formula, u and v are vectors that contain all the elements belonging to clusters u and v,

[0051]

Number

[0052] and

[0053]

Number

[0054] is the average of the values that make up vectors u and v, . is the dot product between u and v.

[0055] By using the correlation distance instead of other available measurement methods, it is possible to evaluate the redundancy between features and thus exclude highly correlated features that do not bring any further value to the model.

[0056] This document also proposes computer software that includes instructions for implementing at least part of the method according to any of the previous claims when the software is executed by a processor.

[0057] This document - an input interface for receiving medical images, and - a memory for storing at least the instructions of the computer program according to the previous claims, and - a processor that reads the above instructions and then accesses the memory to execute the method according to this document, and - an output interface for providing the classification of the area of interest also proposes a computer device comprising the same.

[0058] This document also proposes a computer-readable non-transitory recording medium on which computer software is recorded for implementing the method according to this document when the computer software is executed by a processor.

[0059] Other features, details, and advantages will become apparent in the following detailed description and the drawings.

Brief Description of the Drawings

[0060]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0061] The accompanying drawings include meaningful colors. Although this application will be published in black and white, a color version of the accompanying drawings was filed before the EPO.

[0062] Figure 1 schematically shows an example of a computer device 1 according to the present invention. The computer device 1 includes - an input interface 2, and - a memory 3 for storing at least instructions of a computer program, and - a processor 4 that reads the above instructions and accesses the memory 3 to execute the method according to the present invention, and - an output interface 5 and comprises.

[0063] Figure 2 schematically shows the steps of the training phase and the testing phase of the method according to this document, executed by a computer device.

[0064] Training of the Segmentation Model The training phase includes a first step of training a segmentation model.

[0065] In this step, a database of 660 portal CT scan images is used as the training set. For each image in the training set, the liver, liver lesions, and the main bile duct were annotated by a radiologist. In other words, the corresponding pixels of each image in the training set were annotated by a radiologist, and at least one label, here a liver label, a liver lesion label, or a main bile duct label, is assigned to each pixel, called the ground truth label.

[0066] To improve performance, the images are cropped around the liver to help the model focus on this tissue.

[0067] The segmentation model is nnUNet (「nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation」 Iseense et al., September 28, 2018) implemented according to the code provided at the link https: / / github.com / MIC-DKFZ / nnUNet.

[0068] The model is a fully convolutional neural network and is trained on the training dataset. The trained model is capable of segmenting each of the three structures (liver, lesion, main bile duct).

[0069] After training, the trained segmentation model is saved in memory with its hyperparameters.

[0070] Segmenting these three structures instead of just the main bile duct helps to obtain more specific results and ensures that when segmenting the main bile duct, the duct does not mix with, for example, part of the liver or part of a lesion.

[0071] Figure 4 shows, on the left, an example of an image from the training set and, on the right, the result of the segmentation step using the trained segmentation model.

[0072] Pixels belonging to the pancreas are shown in red, pixels belonging to the main pancreatic duct are shown in blue, and pixels belonging to the lesion are shown in green.

[0073] Using such a trained model, the main pancreatic duct is segmented for each image in the training set.

[0074] Feature Selection For each segmented main pancreatic duct, features can be extracted within the regions separated by the segmentation (the blue regions in Figure 4).

[0075] The features are calculated using radiomics and belong to three different groups of features, namely, size and shape-based features, descriptors of the image intensity histogram, and / or descriptors of the relationships between the image voxels.

[0076] The features are calculated using the pyradiomics library (version 3.0.1, dated October 11, 2020, https: / / github.com / AIM-Harvard / pyradiomics).

[0077] The feature selection step aims to select the most relevant features from the 108 available features in these three groups. Such selection improves the performance of the classification model.

[0078] Each image in the training set is also related to the ground truth label given by a radiologist as to whether the main pancreatic duct is dilated (if abnormal) or not dilated (if normal).

[0079] To proceed with the feature selection, for each of the 108 features, the distributions for the extended group and the non-extended group are calculated, and a Student's T-test is run. One well-known measure of this test is the p-value, which indicates the degree of significance. If the p-value is low, this means that the two groups have statistically different distributions for the corresponding feature.

[0080] Accordingly, the corresponding feature is pre-selected if the p-value is below a threshold such as 0.01, and otherwise, the feature can be removed.

[0081] Next, as seen in Figure 3, during the feature selection process, hierarchical clustering analysis is performed and related features are grouped. Such analysis is performed for all pre-selected features, and if a group of features has a correlation above a threshold such as 0.95, only one feature from this group is finally selected and the other features from the same group are removed.

[0082] The criterion for selecting features within a group of correlated features is based on the range of values. The feature with the largest range of values is preserved and the others are excluded.

[0083] Here, only 44 features are selected from the 108 features described above.

[0084] The 44 selected features of the library (split into groups) are as follows. Size or Shape-based - 3D Features: Area-Volume Ratio, Sphericity, Major Axis Length, Maximum 2D Diameter Column, Minor Axis Length, Maximum 2D Diameter Slice, Minimum Axis Length, Elongation, Maximum 2D Diameter Row, Flatness Intensity Histogram - First-order Statistics: 10th Percentile, Minimum Value, Variance, Homogeneity, Range, 90th Percentile, Maximum Value, Energy, Total Energy, Skewness Relationships between Image Voxels - Gray-level Co-occurrence Matrix (GLCM): Contrast, Difference Mean, Cluster Prominence, Cluster Shade, Correlation Relationships between Image Voxels - Gray Level Run Length Matrix (GLRLM): Long Run High Gray Level Emphasis, Run Variance Relationships between Image Voxels - Gray Level Size Zone (GLSZM): Large Area Emphasis, Large Area Low Gray Level Emphasis, Large Area High Gray Level Emphasis, Gray Level Non - uniformity, Size Zone Non - uniformity Relationships between Image Voxels - Neighborhood Gray Tone Difference Matrix (NGTDM): Complexity, Usage, Contrast, Roughness Relationships between Image Voxels - Gray Level Dependence Matrix (GLDM): High Dependence Low Gray Level Emphasis, High Dependence High Gray Level Emphasis, Low Dependence Emphasis, Dependence Non - uniformity Normalization, Low Dependence High Gray Level Emphasis, Dependence Variance, High Dependence Emphasis, Dependence Non - uniformity

[0085] Training of Classification Model Selected features of each segmented liver in the training set are calculated. To train an SVM classifier with an RBF kernel, said features and corresponding extended or non - extended labels are used.

[0086] The use of an SVM classifier with an RBF kernel is disclosed, for example, in the article "SVM Classification with Linear and RBF kernels", Vasileios Apostolidis - Afentoulis et al., July 2015.

[0087] More specifically, said n features (where, for example, n is equal to 44) are input into the SVM classifier as an n - dimensional tensor. The classification predicted by the SVM classifier is the above - mentioned extended classification (in case of abnormality) or non - extended classification (in case of normality).

[0088] The Support Vector Machine algorithm is effective in balanced classification but does not work well in an unbalanced dataset.

[0089] The SVM algorithm finds a hyperplane decision boundary that best separates examples into two classifications. The separation is done flexibly through the use of a margin that allows for misclassification of a few points. By default, this margin favors the majority classification on an unbalanced dataset, but it can be updated to dramatically improve the performance of the algorithm on datasets with skewed classification distributions, taking into account the importance of each classification.

[0090] This modification of SVM, which weights the margin penalty in proportion to the importance of the classification, is often called weighted SVM or cost-sensitive SVM.

[0091] Thus, the imbalance between the expanded and non-expanded cases in the training set in the case of an unbalanced training dataset can be compensated during the training period to avoid classification bias.

[0092] Such compensation can be easily implemented using available libraries such as the scikit-learn library.

[0093] After training, the classification SVM model trained with those hyperparameters is saved in memory.

[0094] Testing phase After the training phase, a test set of portal CT scan images is used in the testing phase. The test set is composed of 281 images. The images in the test set are independent of the images in the training set.

[0095] The following steps are performed for each image in the test set. - Segmentation of the main bronchus based on the previously trained segmentation model - Extraction of n selected features of the segmented main bronchus (where n is equal to 44) To predict whether the main duct is dilated or not, a classification model is used to classify the corresponding n-dimensional tensor representing the n features.

[0096] Figures 6 and 7 show the results obtained respectively through drawings showing the ROC (Receiver Operating Characteristic) curve and the confusion matrix. The validation phase confirms that the obtained pipeline can predict whether the main duct has dilation or no dilation with good accuracy.

[0097] In fact, Figure 6 shows a high area under the curve (AUC) of 94%.

[0098] Furthermore, based on the confusion matrix in Figure 7, the proposed pipeline has a sensitivity of 87%, a specificity of 80%, and an accuracy of 87% respectively. This high sensitivity emphasizes the pipeline's ability to systematically detect the dilation of the main duct, which is important for the management of early-stage patients.

[0099] The validation phase confirms that the obtained pipeline can predict whether there is dilation of the main duct or not with good accuracy. In other words, the trained model can generalize to new images that these models have never seen during the training phase or the validation phase.

[0100] Inference phase The steps performed during the inference phase are the same as those performed during the validation phase. The difference in the case of the inference phase is that the input data is created from a new image of the patient. Therefore, this phase refers to the deployment of the pipeline and its application example of scoring in the actual production situation based on on-site data.

[0101] Thus, to predict the dilation of the main duct of a patient, the following steps are performed for one or more portal CT scan images of the patient. - Segmentation of the main duct of each image based on a previously trained segmentation model - Extraction of n selected features of the segmented main bronchus (where n is equal to 44) - Using a classification model to classify the corresponding n-dimensional tensor representing the n features to predict whether the main bronchus is dilated or not - Output the predicted classification.

[0102] Here, based on the predicted classification, a radiologist can interpret it to determine that a tumor potentially exists and that complementary tests need to be prescribed for the patient.

Description of Signs

[0103] 1 Computer device 2 Input interface 3 Memory 4 Processor 5 Output interface

Claims

1. A method, performed by computer means, for classifying at least one medical image of at least the pancreas of a patient, comprising the following iterative steps, namely: (a) A step of segmenting at least one region of the subject of the image, wherein the at least one region includes a region representing the main pancreatic duct of the pancreas, (b) The step of extracting at least one feature from the area of ​​interest, (c) A step of classifying the main pipes based on the characteristics in order to determine whether the main pipes are expanded or not, (d) A step of outputting the classification of the target area A method that includes an inference phase.

2. The method according to claim 1, wherein the image is a CT scan image.

3. The method according to claim 1 or 2, wherein the at least one region of the target includes a region representing the pancreas.

4. The method according to claim 1 or 2, wherein the at least one region of the target includes a region representing a pancreatic lesion.

5. The method according to claim 1 or 2, wherein step (a) is performed using a fully convolutional neural network.

6. The method according to claim 5, wherein the fully convolutional neural network is nnUNet.

7. The method according to claim 1 or 2, wherein step (b) is carried out by extracting features from the main tube only, the features comprising size and shape-based features using radiomics, image intensity histogram descriptors, and / or descriptors of relationships between image voxels.

8. The method according to claim 1 or 2, wherein step (c) is performed using a Support Vector Machine classifier.

9. The method according to claim 8, wherein step (c) is performed using a Support Vector Machine classifier having a Radial Basis Function kernel.

10. The method according to claim 1 or 2, comprising a training phase in which the features of step (b) are selected by quantifying the distribution of the features in each of the classifications and selecting the features if there is a statistically significant difference in the distribution of the features between the classifications.

11. The method according to claim 10, wherein the differences in the distribution between the classifications are quantified using Student's t-test.

12. The method according to claim 1 or 2, comprising a training phase in which the features of step (b) are selected by performing hierarchical clustering analysis.

13. The method according to claim 12, wherein the hierarchical clustering analysis is performed using single-link clustering.

Images