Image feature classification

Abstract

Methods and systems for image feature classification use NN-based learning algorithms to make decisions about features in medical images or image portions. In particular, embodiments can leverage phases of a multi-phase image to improve classification accuracy. For example, embodiments can combine different phases of a multi-phase image as training data.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of image feature classification, and in particular to classifying image features of multi-phase medical images. BACKGROUND

[0002] Quick and reliable detection and diagnosis of medical image features, such as lesions or tumors, can enable subjects, i.e. patients, to start treatment earlier and achieve better outcomes. With the increasing availability and quality of medical imaging, the need for invasive diagnostic biopsies is decreasing, driving imaging-based diagnosis to play a more important role, especially in primary liver cancer cancer, which has a unique position. However, radiological diagnosis of potentially malignant liver lesions remains a challenging task. For example, the enhancement washout or capsular features of medical images are very important, but since they are usually determined manually (e.g. by a physician) due to their presence in medical images, the determination process can be inefficient and / or can lead to different determination results between physicians with different expertise or experience.

[0003] Therefore, standardized image analysis and reporting frameworks have been developed to improve radiological diagnosis by reducing variability in imaging interpretation, improving communication with referring physicians, and facilitating quality assurance and research. One prominent example of such a reporting framework is the Liver Imaging Reporting and Data System (LI-RADS) framework for standardizing reporting and data collection for CT and MR imaging of hepatocellular carcinoma (HCC).

[0004] However, the increasing complexity of such reporting frameworks makes implementation in high-volume practices less feasible, leaving the clinical need for computational decision support tools for improving workflow efficiency unmet. Therefore, there is a need for a classification assistance system that can be integrated into the clinical workflow to provide physicians with information that will enable them to classify features of multi-phase medical images more efficiently and with improved confidence. SUMMARY

[0005] The invention is defined by the claims.

[0006] According to an example of an aspect of the present invention, there is provided a computer-implemented method for classifying an image feature of a multi-phase medical image. The method comprises: obtaining a medical image and a phase identifier associated with the medical image, the phase identifier representing a phase of the image; combining the medical image and the phase identifier to generate modified image data; providing the modified image data as an input to a layer of a neural network, NN, based learning algorithm configured to classify the image feature; and obtaining, from the NN based learning algorithm, an image feature classification result for the modified image data.

[0007] The proposed embodiments provide for image feature classification using a NN-based learning algorithm to make determinations regarding features in medical images or image portions (e.g., including lesions). In particular, embodiments can leverage phases of multi-phase images to improve classification accuracy. For example, embodiments can combine different phases of multi-phase images as training data.

[0008] The NN-based learning algorithm can be trained using a training algorithm configured to receive an array of training inputs and known outputs, where the training inputs include medical images, medical image segments, or medical image features, and where the known outputs include image feature classifications. In this way, the NN-based learning algorithm can be trained to output image feature classifications when provided with medical images and phase identifiers associated with the medical images. That is, image data and phase data can be combined, and in this way, image data for different phases can be utilized together to improve training and classification.

[0009] The medical images may, for example, include images of at least a portion of a liver. The NN-based learning algorithm can then include a Liver Imaging Reporting and Data System (LI-RADS) feature classifier trained to make determinations regarding classification of image features.

[0010] In some embodiments, combining the medical image and the phase identifier can include processing the image with an image feature generation algorithm to generate a tensor representing one or more features of the image, and combining the phase identifier with the tensor to generate a modified tensor as modified image data. Providing the modified image data to a layer of the NN-based learning algorithm can then include providing the modified tensor to a hidden layer trained to make determinations regarding classification of image features.

[0011] For example, the hidden layer of the NN can include one convolutional layer or one fully connected layer.

[0012] In one embodiment, combining the medical image and the phase identifier can include modifying data of the image based on the phase identifier to generate modified image data. For example, modifying data of the image can include concatenating the phase identifier to the image as an extracted channel. In another example, the phase identifier can include a numerical value, and modifying data of the image can then include at least one of: adding the numerical value of the phase identifier to pixel values of the image; and multiplying pixel values of the image by the numerical value of the phase identifier. Embodiments can thus employ simple mathematical operations, thereby minimizing cost and / or complexity of implementation.

[0013] In some embodiments, obtaining, for the modified image data, an image feature classification result from the NN-based learning algorithm can comprise obtaining a plurality of image feature classification results from the NN-based learning algorithm; and switching between the plurality of image feature classification results based on the phase identifier to select the image feature classification result.

[0014] The medical image can comprise a CT, MR or ultrasound image.

[0015] Exemplary embodiments can comprise the following main elements:

[0016] - a first NN layer configured to accept as input a multi-phase image (i.e. a multi-phase image) and a corresponding phase identifier.

[0017] - a second NN layer combining the image feature and the phase identifier into a vector; and

[0018] - a fully constant model receiving the combined vector and generating a LiRADS feature.

[0019] According to an example of an aspect of the present invention, there is provided a computer program comprising code means for implementing the method of any preceding claim when said program is run on a processing system.

[0020] According to another aspect of the present invention, there is provided a processing system for classifying image features of a multi-phase medical image. The system comprises: an interface configured to obtain a medical image and a phase identifier associated with the medical image, the phase identifier representing a phase of the image; and a processing unit configured to: combine the medical image and the phase identifier to generate modified image data; provide the modified image data as input to a layer of a neural network, NN, based learning algorithm configured to classify image features; and obtain, for the modified image data, an image feature classification result from the NN-based learning.

[0021] In some embodiments, the NN-based learning algorithm is trained using a training algorithm configured to receive an array of training inputs and known outputs, wherein the training inputs comprise medical images, medical image segments or medical image features, and wherein the known outputs comprise image feature classifications.

[0022] Also proposed is a lesion classification system comprising the above-mentioned system and a user interface configured to receive and display the image feature classification result from the processing system.

[0023] These and other aspects of the present invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0024] For a better understanding of the present application, and to show more clearly how it can be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which:

[0025] Figure 1 is a simplified block diagram of a system for classifying image features of a multi-phase medical image according to an example embodiment;

[0026] Figure 2 is a simplified block diagram of a system for classifying image features of a multi-phase medical image according to another example embodiment;

[0027] Figure 3 is a diagram depicting concatenation of a phase identifier "i" to an original medical image as an extraction pass;

[0028] Figure 4A is a diagram depicting adding a phase identifier "i" to an original image value pixel by pixel, where c is a constant used to balance the value range of the original image data and the phase number;

[0029] Figure 4B is a diagram depicting multiplying a phase number "i" (plus a constant c) by an original image value pixel by pixel; and

[0030] Figure 5 depicts a system according to another example embodiment;

[0031] Figure 6 is a flowchart of a computer-implemented method for classifying image features of a multi-phase medical image according to an embodiment; and

[0032] Figure 7 depicts a modification to the embodiment of Figure 6 DETAILED DESCRIPTION

[0033] The present application will be described with reference to the accompanying drawings.

[0034] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the present application. These and other features, aspects, and advantages of the systems and methods of the present application will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0035] ​According to the concept of the present invention, a method and a system for classifying image features of multi-phase medical images is proposed. The medical images and the phase identifiers associated with the medical images are combined to generate modified image data. The modified image data is provided as input to a layer of a neural network (NN) based learning algorithm, which is configured to classify the image features. For the modified image data, an image feature classification result from the NN based learning algorithm is obtained.

[0036] The embodiments are at least partly based on the realization that image data and phase data can be combined (i.e. mixed together) to provide an improved training and classification. In this way, an improved deep NN based LiRADS feature classifier can be realized.

[0037] The exemplary embodiments can be used, for example, as a clinical decision support system in a digital medical imaging platform. In particular, the proposed embodiments can be applied to an automatic cancer detection system, in particular a LiRADS automatic reporting system.

[0038] Figure 1 A system 100 for classifying image features of multi-phase medical images according to an exemplary embodiment is illustrated. The system 100 comprises an interface 105 configured to obtain multi-phase medical images 110 and a phase identifier 120 associated with each medical image. Here, the phase identifier is a phase identification number (e.g. “0” = arterial phase, “1” = portal venous phase, “2” = delayed phase) representing the phase of the associated image.

[0039] The system further comprises a processing unit 130 configured to combine the medical images and the associated phase identifiers to generate modified image data.

[0040] The modified image data is provided as input to a layer of a neural network (NN) based learning algorithm 140 of the system. Thus, one or more layers of the NN are adapted for receiving multi-phase images (i.e. multi-phase images) as input.

[0041] The structure of an artificial neural network (or simply, neural network NN) is inspired by the human brain. A neural network consists of multiple layers, each layer comprising multiple neurons. Each neuron comprises a mathematical operation. In particular, each neuron can comprise a different weighted combination of a single type of transformation (e.g. the same type of transformation, a sigmoid transformation, etc., but with different weights). In the process of processing input data, the mathematical operation of each neuron is performed on the input data to produce a digital output, and the output of each layer in the neural network is fed sequentially to the next layer. The last layer provides the output.

[0042] Several types of neural networks exist, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Embodiments of this invention employ a CNN-based learning algorithm because CNNs have proven particularly successful in image analysis and are capable of classifying images with a much lower error rate than other types of neural networks.

[0043] CNNs typically consist of several layers, including convolutional layers, pooling layers, fully connected layers, and softmax layers. Convolutional layers consist of a set of learnable filters that extract features from the input. Pooling layers are a form of non-linear downsampling that reduces data size by combining the outputs of multiple neurons in one layer into a single neuron in the next layer. Fully connected layers connect each neuron in one layer to all neurons in the next layer. Softmax layers determine the probability distribution for each output.

[0044] Methods for training machine learning algorithms are well-known. Typically, such methods involve obtaining a training dataset that includes training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and its corresponding training output data entry is used to modify the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is often referred to as a supervised learning technique.

[0045] For example, the weights of the mathematical operations for each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation, and other algorithms.

[0046] The training input data entries of the NN-based learning algorithm 140 used in system 100 correspond to example medical images, medical image fragments, or medical image features. The training output data entries correspond to image feature classifications. Therefore, the NN-based learning algorithm 140 is configured to classify image features based on its training.

[0047] Images used to train the neural network-based learning algorithm 140 can be labeled with three binary numbers to indicate the presence of enhancement, erosion, or capsule features. That is, the input training images each include a lesion and are provided after a lesion segmentation process. The presence of enhancement, erosion, or capsule features is determined by analyzing the output of the lesion segmentation process. In this way, the neural network-based learning algorithm 140 is trained to determine the presence of enhancement, erosion, and capsule features in images containing detected liver lesions.

[0048] Several pre-processing methods can be employed to improve the training samples. For example, images from the same digital imaging system can be normalized in display characteristics such as gamma correction. The images can be segmented using Otsu's method. The regions can then be modified to a standard size.

[0049] The plurality of NN-based learning algorithms can be produced by modifying existing CNN-based learning algorithms such as VGG, Inception, and ResNet.

[0050] According to the NN-based learning algorithm 140, an image feature classification result 150 is obtained for the modified image data. For example, the algorithm 140 predicts whether an enhancement feature, an erosion feature, or an encapsulation feature is present within an input medical image.

[0051] Reference is now made to Figure 2 , which depicts a system 200 for classifying image features of a multiphase medical image, according to another embodiment. Figure 2 The embodiment of Figure 1 differs from the embodiment of Figure 2 in that, in the execution example, the phase identifier of the image is combined with the image after the convolution, pooling, and down-sampling layers, but before the fully connected layers.

[0052] The input to the system 200 is a medical image 110 and its associated phase identifier 120.

[0053] The image 100 can be an arterial phase image, a portal venous phase, or an image delayed phase image generated by CT or MR. The image can be a raw slice or just a region of interest (ROI) containing the lesion.

[0054] The corresponding phase identifier is "0", "1", or "2" according to the phase the image belongs to, arterial phase, portal venous phase, and delayed phase, respectively.

[0055] Thus, in this example of Figure 2 , the input image size is WxH, and the phase number is a scalar. The input image 110 is first fed into an image feature generation module 205. The image feature generation module is composed of several NN layers including convolution, pooling, down-sampling, batch normalization, activation, etc. It can be a module of any deep learning classifier backbone without fully connected layers, for example, VGG, ResNet, Inception, etc.

[0056] The image feature generation module 205 generates a vector of size Nxl. The phase identifier is then combined with the image feature vector (by a processor 210) to generate a new vector of size (N+l)xl.

[0057] The new vector is then provided to the fully connected module 215. The fully connected module 215 generates the prediction(s) 150 of whether there is an enhancement feature / erosion feature / encapsulation feature.

[0058] In Figure 2 exemplary embodiments, the images are labeled as having clinical relevance, e.g., arterial enhancement in the arterial phase, and erosion / encapsulation in the portal / late phase only. Different phases of the images can be mixed together for training, and different phases of the images can be utilized with each other to train the image feature generation module 205.

[0059] As Figure 1 illustrated, an alternative way of combining (e.g., mixing) the phase identifier and the medical image data for the classification model is to combine them at the input side of the NN-based algorithm.

[0060] In view of this procedure of combining the medical image and the phase identifier at the input side, it is noted that various different approaches can be taken.

[0061] For example, if the phase identifier is a scalar, it can be concatenated to the image as an extracted channel. Such an approach is depicted in Figure 3 .

[0062] Figure 3 is an illustration depicting the concatenation of the phase identifier “i” to the original medical image as an extracted channel. The phase identifier “i” is a scalar. Specifically, “i” is “0”, “1”, or “2” for the arterial phase, the portal phase, and the late phase, respectively, depending on the phase the image belongs to. Furthermore, “c” is a constant used to balance the value range of the original image data and the phase identifier “i”.

[0063] According to another example, the phase identifier can be combined with the medical image to which it is associated by adding to the original image values. Such an approach is depicted in Figure 4A .

[0064] Figure 4A is an illustration depicting the addition of the phase identifier “i” to the original image values pixel-wise, where c is a constant used to balance the value range of the original image data and the phase number.

[0065] In another example, the phase identifier can be combined with the medical image to which it is associated by multiplication with the original image values. Such an approach is described in Figure 4B . Figure 4B is an illustration depicting the multiplication of the phase number “i” plus a constant “c” with the original image values pixel-wise. Here, c is a constant used to balance the value range of the original image data and the phase number.

[0066] Figure 5A system 500 according to another exemplary embodiment is depicted. In particular, Figure 5 The system 500 is Figure 1 a modified version of the embodiment of the phase information is used to implement three switches. In this way, the phase information: (i) is added to the image to generate modified image data; and (ii) is added as switch information to generate the final result.

[0067] The system comprises a deep-NN based classifier 520 with three binary outputs, i.e., o1, o2, and o3 (for the enhancement (APHE) feature, the washout feature, and the capsule feature, respectively).

[0068] The input to the NN 520 is a grayscale CT or MR image 110 containing a liver lesion and a phase identifier 120 associated with the image 110. The image 110 can be from an arterial phase, a portal venous phase, or a delayed phase. The phase of the image 110 is indicated by the associated phase identifier 120 in the form of binary phase information. The binary phase information is also used to implement three switches 530 to control the output of the NN 520. For the phase variable, the value “0” identifies that the input grayscale image is from an arterial phase, and the value “1” identifies that the input grayscale image is from a portal venous phase or a delayed phase.

[0069] The phase identifier is inverted and multiplied with o1 of the NN 520 to provide a final binary output to indicate whether an enhancement (APHE) feature is present. Here, the inversion means subtracting the phase variable from ‘1’.

[0070] The identified phase is also multiplied with o2 and o3 of the NN 520 to provide two binary outputs to indicate whether a washout feature and a capsule feature are present, respectively.

[0071] During training of the NN 520, the phase information can be selected to be propagated to the relevant training error of the previous layer, and the irrelevant training error of the backpropagation is switched off (e.g., the enhancement feature error for a portal venous phase image). That is, using the phase information to implement the switches provides a technical effect of controlling the error propagation.

[0072] According to the proposed concepts and / or embodiments described, one or more of the following advantages can be provided:

[0073] (i) Different images from different phases can share the same NN, enabling improved learning with different images; (ii) The phase information can be considered in the NN-based algorithm, improving the prediction; and

[0074] (iii) The phase information can be used to control the error propagation in the NN-based algorithm.

[0075] (iii) During training, any difficulties typically associated with labeled datasets can be reduced. For example, although erosion and capsule features are only meaningful for portal vein and delayed phase images, for arterial phase images, erosion and capsule labels have no effect on the model (because the branches from o2 and o3 to their corresponding outputs are shut off, and gradients cannot be further backpropagated to other layers).

[0076] As another example of the proposed concept, Figure 6 This is a flowchart of a computer-implemented method 600 for classifying image features of multiphase medical images according to one embodiment. In this example, the medical image includes an image of at least a portion of a diseased liver (e.g., detected by a separate lesion detection process).

[0077] The method begins with step 610, which involves obtaining a medical image and a phase identifier associated with the medical image.

[0078] Next, in step 620, the medical image and phase identifier are combined to generate modified image data. In this exemplary method, combining the medical image and phase identifier 620 includes: processing the image using an image feature generation algorithm to generate a tensor representing one or more features of the image; and combining the phase identifier with the tensor to generate a modified tensor as modified image data.

[0079] Step 630 then includes providing modified image data (i.e., modified tensors) as input to the hidden layer of a neural network-based learning algorithm configured to classify image features. Here, the neural network-based learning algorithm includes the Liver Imaging Reporting and Data System (LI-RADS) feature classifier, which is trained to make decisions related to the classification of image features.

[0080] Then, in step 640, the image feature classification result is obtained from the NN-based learning algorithm.

[0081] Understandable, acceptable Figure 6 The described method can be modified. For example, see reference... Figure 5 As described in the embodiments, the embodiments may include switching between classification results based on phase information. As an example, Figure 7 A modified embodiment is described, wherein the method is similar to... Figure 6 The method includes an additional step 750 that switches between classification results based on phase information to generate the final result.

[0082] It will also be understood that the disclosed methods are computer-implemented methods. Therefore, the concept of a computer program is also proposed, which includes code means for implementing any of the methods when the program is run on a processing system.

[0083] Those skilled in the art will be able to readily develop processors for performing any of the methods described herein. Therefore, each step of the flowchart can represent a different action performed by the processor and can be executed by the corresponding module processing the processor.

[0084] As described above, the proposed system can utilize a processor to perform data processing. The processor can be implemented in various ways using software and / or hardware to perform a variety of required functions. The processor typically employs one or more microprocessors, which can be programmed using software (e.g., microcode) to perform the desired functions. The processor can be implemented as dedicated hardware performing some functions and as a combination of one or more programmable microprocessors and associated circuitry performing other functions.

[0085] Examples of circuits that may be used in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0086] In various implementations, the processor can be associated with one or more storage media, such as volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. The storage media can be encoded with one or more programs, which, when executed on one or more processors and / or controllers, perform the required functions. The various storage media can be fixed within the processor or controller, or they can be transportable, allowing one or more programs stored thereon to be loaded into the processor.

[0087] By studying the accompanying drawings, this disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plural. A single processor or other unit can perform the functions of several items recited in the claims. The fact that certain measures are listed only in mutually different dependent claims does not mean that combinations of these measures cannot be used for advantageous purposes. Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. If the term "adapted for" is used in the claims or description, it should be noted that the term "adapted for" is intended to be equivalent to the term "configured as." No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A computer-implemented method (600) for classifying image features of a multi-phase medical image, the computer-implemented method comprising: - obtaining (610) a medical image (110) and a phase identifier (120) associated with the medical image, the phase identifier representing a phase of the image; - combining (620) the medical image and the phase identifier to generate modified image data; - providing (630) the modified image data as an input to a layer of a neural network, NN, based learning algorithm (140) configured to classify image features; and - obtaining (640) an image feature classification result from the NN based learning algorithm for the modified image data, wherein combining the medical image (110) and the phase identifier (120) comprises: - processing the image with an image feature generation algorithm to generate a tensor representing one or more features of the image; and - combining the phase identifier with the tensor to generate a modified tensor as the modified image data, and wherein providing the modified image data to the layer of the NN based learning algorithm comprises: - providing the modified tensor to a hidden layer trained to make a decision related to classification of image features.

2. The computer-implemented method of claim 1, wherein the medical image (110) comprises an image of at least a portion of a liver having a lesion, and wherein the NN based learning algorithm (140) comprises a liver imaging reporting and data system, LI-RADS, feature classifier trained to make a decision related to classification of image features.

3. The computer-implemented method of claim 1 or 2, wherein the NN based learning algorithm (140) is trained using a training algorithm configured to receive an array of training inputs and known outputs, wherein the training inputs comprise medical images, medical image segments, or medical image features, and wherein the known outputs comprise image feature classifications.

4. The computer-implemented method of claim 1, wherein the hidden layer of the NN comprises one convolutional layer or one fully connected layer.

5. The computer-implemented method of any of claims 1 to 3, wherein combining (620) the medical image (110) and the phase identifier (120) comprises: - modifying data of the image based on the phase identifier to generate modified image data.

6. The computer-implemented method of claim 5, wherein modifying the data of the image comprises concatenating the phase identifier to the image as a channel of extraction.

7. The computer-implemented method of claim 5, wherein the phase identifier comprises a numerical value, and wherein modifying the data of the image comprises at least one of: - adding the numerical value of the phase identifier to pixel values of the image; and - multiplying pixel values of the image by the numerical value of the phase identifier.

8. The computer-implemented method of any one of claims 1 to 6, wherein, Obtaining (640), from the NN-based learning algorithm, image feature classification results for the modified image data comprises: - obtaining a plurality of image feature classification results from the NN-based learning algorithm; and - switching (750) between the plurality of image feature classification results based on the phase identifier to select an image feature classification result.

9. A computer program comprising code means for implementing the method according to any one of the preceding claims when said program is run on a processing system.

10. A system (100) for classifying image features of a multi-phase medical image, the system comprising: - an interface (105) configured to obtain a medical image and a phase identifier associated with the medical image, the phase identifier representing the phase of the image; and - a processing unit (130) configured to: combine the medical image and the phase identifier to generate modified image data; provide the modified image data as input to a layer of a neural network, NN, based learning algorithm (140) configured to classify image features; and obtain, from the NN-based learning, image feature classification results for the modified image data, wherein combining the medical image (110) and the phase identifier (120) comprises: - processing the image with an image feature generation algorithm to generate a tensor representing one or more features of the image; and - combining the phase identifier with the tensor to generate a modified tensor as the modified image data, - and wherein providing the modified image data to the layer of the NN-based learning algorithm comprises: - providing the modified tensor to a hidden layer trained to make a decision related to classification of image features.

11. The system of claim 10, wherein the NN-based learning algorithm (140) is trained using a training algorithm configured to receive an array of training inputs and known outputs, wherein the training inputs comprise medical images, medical image segments, or medical image features, and wherein the known outputs comprise image feature classifications.

12. The system of any one of claims 10 to 11, wherein combining the medical image (110) and the phase identifier (120) comprises: - modifying data of the image based on the phase identifier to generate modified image data.

13. A lesion classification system comprising: - the system of any one of claims 10 to 12; and - a user interface configured to receive and display the image feature classification results from the processing unit. ​ ​

Images