Method and system for machine learning classification based on structure or material segmentation in images - Patents.com

Abstract

To provide a system and method for classifying a structure or material in an image of a subject.SOLUTION: In a system 10, a classification controller includes: a segmenter; a classifier that implements a trained classification machine learning model configured to assign to classifications respective scores indicative of a likelihood that a structure or material, or a subject falls into the respective classifications; and a result output for outputting a result indicative of the classifications and scores. The segmenter forms one or more segmentations of the structure or material in an image and generates, from the segmentations, one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories. The segmenter and the trained classification machine learning model generate, based on the segmentations maps, one or more classifications.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to computer vision systems and methods using machine learning, particularly deep neural networks, for classifying (and monitoring changes in) images (e.g. medical images) based on structure or material segmentation. Potential medical imaging applications include computed tomography (CT), magnetic resonance (MR), ultrasound, HRpQCT (High-Resolution peripheral Quantitative Computed Tomography), and lesion scanner imaging.

[0002] This application is based on and claims priority to U.S. patent application Ser. No. 16 / 448,474, filed June 21, 2019, the entire contents of which are incorporated by reference as filed. [Background technology]

[0003] Computer vision and image processing techniques have been applied to medical image analysis. Some computer-aided systems achieve the analysis with two steps: segmentation and quantitative calculation. Segmentation is the process of segmenting (or distinguishing) structures or objects in an image, such as a medical image, by distinguishing pixels (in 2D images) or voxels (in 3D images) in the image, respectively. Based on the segmentation, quantitative features such as volume, shape, and density are calculated. For example, the size and shape of the lesion can be calculated after the lesion is segmented, such as in a brain CT or MRI scan; bone mineral density can be calculated after the femoral neck is segmented in a lumbar DXA (dual-energy x-ray absorptiometry) scan. After comparing such calculations with healthy reference data, the physician can make a diagnosis or treatment decision.

[0004] For example, the T-score is a standard score given to a patient's bone mineral density when compared to a young normal reference mean. The World Health Organization (WHO) defines osteoporosis as a T-score of -2.5 or less, i.e., a bone mineral density 2.5 standard deviations below the mean for a healthy 30-year-old man / woman.

[0005] Segmentation can be accomplished manually, semi-manually, or automatically. As an example of manual segmentation, a user operates a computer to move a rectangular box over a lumbar DXA scan, thereby selecting the region of the femoral neck.

[0006] Semi-manual segmentation can be performed by an image processing program that utilizes user initialization or input. For example, a user can operate a computer to draw approximate bone boundaries on a wrist CT scan, and the program then refines the approximate boundaries into a contour that segments the bone from the surrounding tissue.

[0007] Automated segmentation can be performed by exploiting features of the object of interest (e.g., signal intensity values, edges, shapes, etc.). In one existing example, a voxel-based thresholding technique can be used to segment bone from surrounding tissues in a CT scan. Some other programs use machine learning algorithms to train a classifier to segment abnormal tissues in medical images. For example, a feature-based machine learning algorithm (e.g., Support Vector Machine (SVM) or decision tree) can be used as a classifier by using tumor and normal images as training data. The trained classifier scans through the new image in "windows" to segment any image regions that are tumorous.

[0008] Machine learning algorithms have shown promising accuracy and efficiency in this field. However, both collecting sufficient training data and annotating the training data can be quite challenging. The training images must be annotated by experts, which is a tedious and time-consuming process. It should also be noted that in some applications, even experts may find it extremely difficult or almost impossible to accurately annotate the training images. For example, in bone quality assessment, there is a transition zone in every sample that is composed of both cortical and trabecular bone. The transition zone is composed of the inner cortex adjacent to the medullary canal and the trabecular bone that is attached to the cortex and adjacent to the inner cortical surface. There is active bone remodeling in the transition zone. Identifying and segmenting this region is crucial for bone microarchitecture assessment, but due to image resolution limitations, it is practically a tall order for even experts to annotate this region accurately and consistently. Without annotated images as training data, a segmentation model cannot be trained.

[0009] In the past few years, machine learning or deep neural networks have surpassed human performance in many visual recognition tasks, such as natural image classification. In an exemplary CNN (Convolutional Neural Network) embodiment, the network includes an input layer, a hidden layer, and an output layer. Images are fed into the network through the input layer. Images are sampled and convolution operations are applied to generate the hidden layer. The output of each layer is used as the input for the next layer in the network. The output layer is fully terminated and will output a classification result. The training data are images with classification labels. The training process obtains the parameters of the neural network. After training is completed, new images are processed by the neural network with the obtained parameters to generate a classification result. For example, a deep learning neural network algorithm can be used to train a model for determining the state of diabetic retinopathy (e.g., none, mild, moderate, severe, etc.) from OCT (Optical Coherence Tomography) images.

[0010] However, this end-to-end solution brings two problems for clinical practice. First, the end-to-end solution is a black box: the input is a medical image and the output is a disease or condition classification. It is difficult to interpret the process by which the neural network arrives at a decision, which makes it difficult for users to evaluate the reliability of the classification results. Second, this solution requires a significant amount of training data. As mentioned above, in medical applications, annotating or labeling training data is a laborious and time-consuming task. Therefore, collecting sufficient training data for each type of classification result will be a considerable challenge. Summary of the Invention

[0011] According to a first aspect of the present invention there is provided a system for classifying structures or materials in an image of a subject, the system comprising: a segmenter that forms one or more segmentations of structures or materials in an image (including, for example, a medical image) and generates from the segmentations one or more segmentation maps for the image, the segmentation maps including a categorization of the pixels or voxels of the segmentation map assigned from a set of one or more respective predefined categories; a classifier implementing the trained classification machine learning model configured to generate one or more classifications based on the segmentation map and to assign respective scores to the classifications indicative of the likelihood that the structure or material or subject falls into each classification; and an output unit for outputting a result indicative of the classification and score.

[0012] In an embodiment, the classifier generates one or more classifications based on the segmentation map and non-image data for the subject, the system.

[0013] The system can be configured to train a classification machine learning model.

[0014] In an embodiment, the segmenter: i) a structural segmenter configured to generate a structural segmentation map including a categorization for pixels or voxels assigned from a predefined set of structural categories; ii) a material segmenter configured to generate a material segmentation map comprising a categorization for pixels or voxels assigned from a predefined set of material categories; and / or iii) An anomaly segmenter configured to generate an anomaly segmentation map including categorization for pixels or voxels assigned from a predefined set of abnormal / normal categories.

[0015] By way of example, the structure segmenter is configured to generate the structure segmentation map using a structure segmentation machine learning model, the material segmenter is configured to generate the material segmentation map using a material segmentation machine learning model, and the anomaly segmenter is configured to generate the anomaly segmentation map using the anomaly segmentation model. The structure segmenter can be configured to train the structure segmentation machine learning model, the material segmenter can be configured to train the material segmentation machine learning model, and the anomaly segmenter can be configured to train the anomaly segmentation machine learning model.

[0016] In an embodiment, the system further comprises a segmentation map processor configured to process the segmentation map before it is input by the classifier. By way of example, the segmentation map processor is configured to downsample the segmentation map.

[0017] In embodiments, the classification machine learning model includes a neural network, a support vector machine (SVM), a decision tree, or a combination thereof. For example, the classification machine learning model may include a neural network including a convolutional neural network and a fully connected neural network layer.

[0018] In an embodiment, the images are medical images and the classifications correspond to the structure or material or the probability that the subject will suffer from a particular condition or symptom within each time period. By way of example, the time periods include a short-term time period, a long-term time period, and at least one intermediate time period interposed between the short-term and long-term time periods. In another example, the condition or symptom is a bone fracture.

[0019] In an embodiment, the images are medical images and the classifications correspond to structures or materials or probabilities that the subject will suffer from a respective condition or condition, for example, the condition or condition is a bone-related condition.

[0020] In an embodiment, the images are medical images and the classifications correspond to probabilities for respective rates of progression of a disease or condition, for example, the classifications may include classifications corresponding to any one or more of stable, mild deterioration, and accelerated deterioration.

[0021] In an embodiment, the images are medical images and the classifications correspond to probabilities for the effectiveness of each treatment option, for example, the treatment options may include antiresorptive and / or anabolic therapy.

[0022] In an embodiment, the images are medical images and the classifications correspond to respective medical conditions, for example the medical conditions may include any one or more of osteomalacia, tumor, osteonecrosis and infection.

[0023] In an embodiment, the classification machine learning model is a model trained with image data and non-image data associated with training subjects, and generates respective scores based on image data and non-image data (typically including one or more images) associated with the subjects.

[0024] According to a second aspect of the present invention there is provided a computer implemented method for classifying structures or materials in an image of a subject, the method comprising: forming one or more segmentations of structures or materials in the image; generating from the segmentation one or more segmentation maps for the image, the segmentation maps including a categorization of the pixels or voxels of the segmentation map assigned from a set of respective predefined structure / material categories; using the trained classification machine learning model to generate one or more classifications based on the segmentation map and assigning respective scores to the classifications that indicate the likelihood that the structure or material or subject falls into each classification; and outputting the results indicative of the classification and score.

[0025] In an embodiment, a classification machine learning model is used to generate one or more classifications based on the segmentation map and non-image data for the subject.

[0026] The method may include training a classification machine learning model.

[0027] In an embodiment, the step of forming the one or more segmentations comprises: i) generating a structural segmentation map including categorization for pixels or voxels assigned from a predefined set of structural categories; ii) generating a material segmentation map comprising a categorization for the pixels or voxels assigned from a predefined set of material categories; and / or iii) generating an abnormality segmentation map that includes categorization for pixels or voxels assigned from a predefined abnormal / normal category set.

[0028] For example, the method may include generating a structural segmentation map using a structural segmentation machine learning model, generating a material segmentation map using a material segmentation machine learning model, and generating an anomaly segmentation map using an anomaly segmentation model. In particular, the method may include training a structural segmentation machine learning model, a material segmentation machine learning model, and / or an anomaly segmentation machine learning model.

[0029] According to a third aspect of the present invention there is provided a computer implemented diagnostic method, the method including the method of the second aspect.

[0030] According to a fourth aspect of the present invention there is provided a computer implemented method for training a classification machine learning model for classifying structures or materials in an image of a subject, the method comprising: Partitioning the annotated segmentation map and the annotated non-image data into a training set and a test set, such that the training set and the test set each include some annotated segmentation map and some non-image data, where the annotated segmentation map was obtained by segmenting one or more images; (a) implementing a classification machine learning model, the step including initializing parameters of the classification machine learning model; (b) Training the learning algorithm set updating parameters of the classification machine learning model by running (c) Testing set testing a classification machine learning model on the (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) to (d) if performance is insufficient in step (d); outputting the classification machine learning model as a trained classification machine learning model for deployment or flagging the classification machine learning model as a trained classification machine learning model.

[0031] This aspect may be used in conjunction or in combination with (or as part of) the second aspect, for example to train a classification machine learning model of the second aspect.

[0032] The method may include segmenting one or more images (e.g., in generating an annotated segmentation map).

[0033] In an embodiment, the method includes receiving and using more image data and non-image data to train the classification machine learning model if performance in step (d) is deemed insufficient.

[0034] The classification model can be trained using a variety of machine learning algorithms and can therefore include, for example, neural networks, support vector machines (SVMs), decision trees, or combinations thereof.

[0035] Thus, in one embodiment, the classification machine learning model comprises a neural network having a number of layers comprising artificial neurons, and the parameters include a number of layers, a number of neurons, neuron weights, and neuron function parameters. Also, testing the classification machine learning model includes testing the classification machine learning model. set This includes testing at

[0036] In an embodiment, updating the parameters includes determining the slope of the loss function.

[0037] In an embodiment, the images are medical images and the non-image data includes medical records.

[0038] In an embodiment, the method includes dividing the annotated segmentation map and the annotated image data into a training set, a development set and a test set, and performing a development analysis to examine the learning procedure and tune the parameters. set (and, if the classification machine learning model comprises a neural network, tuning the layers may also be included).

[0039] According to this aspect, there is provided a system for training a classification machine learning model for classifying structures or materials in an image of a subject, the system comprising a processor, the processor comprising: splitting the annotated segmentation map and the annotated non-image data into a training set and a test set, where the annotated segmentation map is obtained by segmenting one or more images; (a) implementing a classification machine learning model, the step including initializing parameters of the classification machine learning model; (b) Training the learning algorithm set updating parameters of the classification machine learning model by running (c) Testing set testing a classification machine learning model on the (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) to (d) if performance is insufficient in step (d); outputting the classification machine learning model as a trained classification machine learning model for deployment or flagging the classification machine learning model as a trained classification machine learning model.

[0040] According to a fifth aspect, the present invention provides a computer program comprising program code arranged to perform any one or more of the second to fourth aspects when executed by one or more computing devices. In this aspect, there is also provided a computer readable medium comprising such a computer program.

[0041] Any of the various individual features of each of the above-mentioned aspects of the invention, as well as any of the various individual features of the embodiments described herein, including the claims, may be combined in any suitable and desired manner. [Brief description of the drawings]

[0042] In order that the invention may be more clearly defined, embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0043] [Figure 1] FIG. 1 is a schematic diagram of a classification system according to an embodiment of the present invention. [Diagram 2] 2 is a high level schematic diagram illustrating the operation of the classification system of FIG. 1 in calculating a subject's fracture risk from medical image scans. [Diagram 3] FIG. 2 is a schematic diagram illustrating the operation of a segmenter of the classification system of FIG. 1. [Figure 4] FIG. 2 is a schematic diagram illustrating the operation of a classifier of the classification system of FIG. [Diagram 5] 2 is a flow diagram illustrating training of the classification neural network of the classification system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] FIG. 1 is a schematic diagram of a classification system 10 for classifying structures or materials in medical images (based on structure and material segmentation), according to an embodiment of the present invention.

[0045] Referring to FIG. 1, the system 10 includes a classification controller 12 and a user interface 14 (including a GUI 16). The user interface 14 is provided for presenting information to a user and for receiving input (including feedback) from the user; typically includes one or more displays (which may display one or more GUIs 16), a web browser, a keyboard and mouse, and optionally a printer. The classification controller 12 includes at least one processor 18 and a memory 20. The system 10 may be implemented, for example, as a combination of software and hardware on a computer (e.g., a personal computer or a mobile computing device) or as a dedicated image segmentation system. The system 10 may optionally be distributed; for example, all or some components of the memory 20 may be located remotely from the processor 18; and the user interface 14 may be located remotely from the memory 20 and / or the processor 18. For example, the system 10 may be implemented in a service-oriented architecture, and its components may communicate with each other via a communication network, such as a LAN, a WAN, or the Internet. The system 10 can be deployed in the cloud and its use can be shared by users in different locations.

[0046] In certain other embodiments, system 10 may be implemented as a stand-alone system (of software and hardware) or as stand-alone computer executable software and may be deployed at a location, for example, system 10 may be deployed at a hospital, medical clinic, or other clinical location.

[0047] Memory 20 is in data communication with processor 18 and typically comprises both volatile and non-volatile memory (and may include one or more of each memory type), including RAM (random access memory), ROM, and one or more mass storage devices.

[0048] As described in more detail below, processor 18 includes a segmentor 22 (which includes a structure segmentor 24a and an anomaly segmentor such as an abnormal material segmentor 24c), a segmentation map processor 26, and a non-image data processor 28. Processor 18 also includes an I / O interface 32 and a result output 34.

[0049] The memory 20 includes program code 36, image data 38, non-image data 40, segmentation models 42 (which in this example include a structural segmentation model 44, a material segmentation model 46, and an anomaly segmentation model as an abnormal material segmentation model), and a segmentation map 50 (which in this example include a structural segmentation map 52, a material segmentation map 54, and an anomaly segmentation map as an abnormal material segmentation map 56). The structural segmenter 24a, the material segmenter 24b, and the abnormal material segmenter 24c train their respective segmentation models 44, 46, 48 and perform segmentation on the incoming image using the segmentation models 44, 46, 48 to generate the structural segmentation map 52, the material segmentation map 54, and the abnormal material segmentation map 56, respectively.

[0050] The memory 20 also includes a classification machine learning model as a classification neural network 58 that is trained and used by the classifier 30 to perform classification using the segmentation map 50 and the non-image data 40. The classification controller 12 is implemented at least in part (and in some embodiments, entirely) by the processor 18 executing program code 36 from the memory 20.

[0051] It should be noted that since this embodiment is directed to classification of structures and / or materials in medical images, the abnormal material segmenter 24c may also be referred to as an abnormal tissue segmenter, and the abnormal material segmentation map 56 may also be referred to as an abnormal tissue segmentation map.

[0052] Generally speaking, the I / O interface 32 is configured to read or receive image data and non-image data relating to a subject and store the data as image data 38 and non-image data 40 in the memory 20 for processing. In this embodiment, the image data 38 is typically in the form of a medical image, e.g., relating to a region of the subject's body. The non-image data 40 typically includes subject or patient information from a variety of structured and unstructured data sources, collected during the subject's consultation, treatment, and follow-up visits. Subject structured data may include: subject demographics, e.g., gender, age, weight, height, etc.; clinical test results, e.g., blood test results, DNA test results, etc.; medical treatment data, e.g., drug type and dosage; and medical history data, e.g., smoking history, alcohol history, fracture history. Subject unstructured data may include text documents of clinical test results, doctor's notes, radiology reports, etc. The non-image data 40 may come in a variety of formats, e.g., numerical data, text, audio, video, etc.

[0053] The segmenter 22 processes image data 38 (comprising one or more medical images) to generate a structure segmentation map 52, a material segmentation map 54, and an abnormal material segmentation map 56 from the image data 38 using the structure segmentation model 44, the material segmentation model 46, and the abnormal material segmentation model 48, respectively, which characterize the image data 38 in different ways. The classifier 30 then inputs the resulting segmentation map 50 and the non-image data 40 and generates therefrom a result as a classification output. In this embodiment, the classification output is presented to a user or used for further analysis, via the I / O interface 32, at the result output 34 and / or the user interface 14.

[0054] The classification output of the classifier 30 (generated in this embodiment with the classification neural network 58) comprises a respective condition score for each of the one or more classifications (and preferably for each of a plurality of possible classifications). Each score represents the predicted likelihood that the subject will fall into the corresponding classification. In this embodiment for bone fragility assessment, the classifications are: "negligible fracture risk", "imminent fracture risk", "intermediate term fracture risk", and "long term fracture risk". The classification output is described in more detail below.

[0055] In an alternative embodiment, the classifier outputs a respective disease progression score for each of one or more condition progression stages, each score representing the likelihood that the current condition is expected to progress to another condition. For example, in bone fragility assessment, disease progression degrees may include "stable", "mild deterioration", and "accelerated deterioration".

[0056] In yet another embodiment, the classifier outputs a respective treatment score for each of a plurality of treatment options, each score representing the likelihood that the treatment is expected to be most effective for the patient. For example, in a bone fragility assessment, the treatment options may include "antiresorptive therapy", "anabolic therapy", and "antiresorptive plus anabolic therapy".

[0057] In further embodiments, the classification output includes a score for each of one or more possible classifications that correspond to known medical conditions or pathologies. For example, in bone fragility assessment, these classifications may include: "osteomalacia", "tumor", and "infection". In such embodiments, the resulting score represents the degree to which the subject's sample (e.g., bone) matches the classification / condition. If only one classification has a significant score, or if one classification has a score significantly greater than all others, then that classification may be considered as a diagnosis or suggested diagnosis for the corresponding condition or pathology.

[0058] In certain embodiments, the classification output includes two or more sets of such scores (selected from the examples above or other examples).

[0059] Returning to FIG. 1 , and as will be appreciated by those skilled in the art, image data 38 comprising one or more medical image images includes data generated by one or more of a variety of medical imaging modalities (e.g., HRpQCT, or High-Resolution Peripheral Quantitative Computed Tomography, etc.), implemented by one or more medical imaging devices (e.g., HRpQCT scanners). Each of these devices scans a sample (whether in vivo or in vitro) to generally create a visual representation of an interior portion of the subject. The medical images may represent, for example, a portion of a human body or an entire subject (e.g., the brain, lower back, or wrist). The medical images may be obtained by scanning the same sample or body part with different imaging modalities, which may reveal different characteristics of the same sample or body part. Medical images may be obtained using the same imaging modality for different body parts, and different body parts of the same patient may provide different perspectives to approach a better diagnosis of a disease or condition. For example, in a bone fragility assessment, both the patient's wrist and leg can be scanned with an HRpQCT scanner (although different imaging modalities can also be used to scan different samples or body parts) to provide information to assess the subject's bone quality.

[0060] The image data 38 may include 2D (two-dimensional) images, which may be represented as a two-dimensional array of pixels, or 3D (three-dimensional) images, which may be represented as a three-dimensional array of voxels. For convenience, the medical images described below are 3D images, which may be represented as a 3D array of voxels.

[0061] As described above, one or more received medical images stored in image data 38 are segmented by segmenter 22 into respective segmentation maps using the trained segmentation model. Each segmentation map 52, 54, 56 characterizes each medical image differently. The structural segmentation map 52 represents the medical image as one or more different anatomical structures from a predefined set of structures. For example, a wrist CT scan can be segmented into dense cortex, transition zone, and trabecular bone regions. Material Segmentation Map 54 represents the medical image with multiple different materials from a predefined material set. For example, a wrist CT scan may be segmented into mineralized material, fully mineralized material, red bone marrow in trabecular bone regions, and yellow bone marrow in trabecular bone regions. The abnormal material segmentation map 56 represents the medical image with normal and abnormal materials (in this example, with normal and abnormal tissues). For example, from a wrist CT scan, a tumor or a fracture may be segmented and represented as "abnormal" in the abnormal material segmentation map 56.

[0062] The segmentation map 50, in combination with the non-image data 40, is input to the classifier 30. The classifier 30 generates one or more classification outputs based on the segmentation map 50 and the non-image data 40. Because the input data for the classifier 30 is typically multi-dimensional, the classifier 30 is implemented using a machine learning algorithm, such as a neural network, a support vector machine, a decision tree, or a combination thereof.

[0063] In this embodiment, the classifier 30 is implemented using or as a classification neural network 58, which in this example includes convolutional and fully connected neural network layers (other machine learning algorithms are possible in other embodiments). The classification neural network 58 is trained with training data, as described below.

[0064] As mentioned above, the final classification output is output by the system 10 to a user via the results output 34 or user interface 14. The classification output may optionally include a visual representation of one or more corresponding segmentation maps 50. The segmentation maps 50 may be presented where useful for a user to interpret the classification output, such as to assess the reliability of the results.

[0065] FIG. 2 is a high level schematic diagram illustrating the operation of system 10 with respect to calculating a subject's risk of fracture from a medical imaging scan, in this example a wrist HRpQCT scan 62 (a negative image 62' of which is also shown). 60 As shown in Figure 2, the system 10 receives a wrist HRpQCT scan 62 that includes multiple slices. (It should be appreciated that an HRpQCT scan may have 100 or more slices, as will be appreciated by those skilled in the art, but only four slices are shown for simplicity.)

[0066] The segmenter 22 segments the scan 62 into a structure segmentation map 52, where the scan is segmented into dense cortex, outer transition zone, inner transition zone, and trabecular bone regions. The segmenter 22 segments the scan 62 into a material segmentation map 54, where the scan 62 is segmented into surrounding muscle, surrounding fat, yellow marrow hyperfatty areas, and red marrow hyperfatty areas. Data 64, including the segmentation maps 52, 54, the abnormal material segmentation map 56, and non-image data 40 (e.g., clinical factors including gender and age), is processed by the trained classifier 30 to generate a classification output. The classification output includes the segmentation maps 52, 54 and a table or report 66. The table or report 66 includes, in numerical and / or graphical form, the probability of fracture in each of the following categories of fracture risk: imminent fracture risk 68a (fracture within 2 years: t<2y), intermediate fracture risk 68b (fracture within 2 to 5 years: 2≦t≦5y), long-term fracture risk 68c (fracture within 5 to 10 years: 5≦t≦10y), and negligible fracture risk 68d. In the illustrated example, the following probabilities are seen: 95.6% risk of wrist fracture within 2 years, 2.4% risk of wrist fracture in 2-5 years, 1.6% risk of wrist fracture in 5-10 years, and 0.3% risk of negligible wrist fracture. In other words, there is only a 4.4% chance that the subject will not experience a wrist fracture in the next 5 years (combining negligible and long-term fracture risks). Although the table or report 66 does not include a diagnosis (that the subject has osteomalacia), it should be understood that these probabilities are of great value and may prompt a person to seek a diagnosis, for example by undergoing a medical examination or consultation.

[0067] 3 is a schematic diagram 70 of the operation of the segmenter 22. The segmenter 22 is configured to receive an input including one or more medical images (from the image data 38) and process the images to generate a segmentation map 50. The medical images may be obtained by scanning different body parts of a patient using the same imaging modality. For example, in some applications for evaluating bone quality, both the wrist and leg of a patient may be scanned for evaluation with an HRpQCT scanner. The medical images may be obtained by scanning the same or different body parts of a patient using different imaging modalities. For example, in some applications for evaluating bone quality, both a wrist HRpQCT scan and a lower back DXA scan are obtained for evaluation (again, recall that medical images may be obtained by scanning different samples or body parts using other imaging modalities).

[0068] Referring to FIG. 3, the segmenter 22 processes medical images 1 to n (38 1 ,...,38 n One or more processing branches 1 to n (labeled 72 in the figure) corresponding to 1 ,...,72 n When there are multiple processing branches, the Medical Image 38 1 ,...,38 n For example, different imaging modalities (74 1 ,...,74 n The segmentation branches 72 may be generated by different body parts, as in the illustrated example, different scans of a single body part, or a combination of two or more of these. 1 ,...,72 n is configured to receive an image and segment the image according to image type (eg, with different program codes 36) to generate a segmentation output (forming a segmentation map 50 of the input image).

[0069] For processing each input medical image, the segmenter 22 is configured to first select a processing branch from processing branches 1 to n according to the type of the input image. The segmenter 22 determines the image type according to the following: the sample (e.g. the scanned body part) and the imaging modality, information that can be determined from each image and that can be derived from metadata stored in the header file of the medical image and / or the file type. For example, information about the scanned body part and the imaging modality is accessible from the metadata.

[0070] Each input medical image 1-n is processed by one or more of three segmentation programs (i.e., structure segmenter 24a, material segmenter 24b, and abnormal material segmenter 24c) to produce a corresponding segmentation map 52, 54, 56. Thus, segmenter 22 processes segmenters 24a, 24b, and 24c (respectively, 24a 1 ,...,24a n , 24b 1 ,...,24b n , and 24c 1 ,...,24c n Although up to n instances of each segmenter 24a, 24b, 24c (labeled as n) are used in parallel or sequential manner, the number of instances for each segmenter 24a, 24b, 24c (ranging from 0 to n in each case) may be different.

[0071] The structure segmenter 24a, the material segmenter 24b, and the abnormal material segmenter 24c each perform processing in a branch 72. 1 ,...,72 n 3, structure segmenter 24a, material segmenter 24b, and abnormal material segmenter 24c generate respective segmentation maps corresponding to medical imaging modalities 1-n, and the resulting structure segmentation map, material segmentation map, and abnormal tissue segmentation map are correspondingly generated as structure segmentation map 52. 1 ,...,52 n, Material Segmentation Map 54 1 ,...,54 n , and abnormal tissue segmentation map 56 1 ,...,56 n However, it should be noted that in some applications it may not be possible or desirable to generate all three segmentation maps, which may be due to, for example, limitations of the image, the imaging modality, or the segmenters 24a, 24b, 24c (e.g., due to constraints on the segmenter training data).

[0072] The structure segmenter 24a, material segmenter 24b, and abnormal material segmenter 24c assign to each voxel in these segmentation maps 50 one or more "types" (or "categories") from a respective set of predefined types (or categories). Thus, in this embodiment, the structure segmenter 24a assigns a respective structure type (from a predefined set of structure types) to each voxel in the scan. For example, a wrist HRpQCT scan is segmented into a structure segmentation map 52, where each voxel in the scan is assigned a structure type (or category) from the set consisting of: "periphery tissue", "dense cortex", "transition zone", and "trabecular bone region".

[0073] The material segmenter 24b assigns a respective material type (from a set of predefined material types) to each voxel. For example, in this embodiment, the material segmenter 24b segments a wrist HRpQCT scan into a material segmentation map 54, where each voxel in the scan is assigned a material type from the set: "calcified material", "fully calcified material", "red marrow hyperlipidated material", and "yellow marrow hyperlipidated material".

[0074] The abnormal material segmenter 24c assigns each voxel a respective abnormal / normal type (from a predefined set of abnormal / normal types, such as the set including "normal" and "abnormal"). For example, in this embodiment, the abnormal material segmenter 24c segments a wrist HRpQCT scan into an abnormal tissue segmentation map 56 into a set of predefined abnormal / normal types, with each voxel in the scan being assigned a "normal" or "abnormal" status. Optionally, in certain other embodiments, the abnormal material segmenter 24c can distinguish between different types of abnormalities, and the set of predefined abnormal / normal types can include (in addition to "normal") one or more specific abnormalities specific to the sample type under consideration; if the sample is bone, these can include, for example, "fracture crack" and "bone tumor." In such embodiments, "abnormal" can optionally be included in the set for cases in which the abnormal material segmenter 24c cannot determine a specific anomaly type.

[0075] In some embodiments, the structure segmenter 24a, material segmenter 24b, and abnormal material segmenter 24c assign each type, along with a confidence limit or probability, to each voxel in the medical image. In some other embodiments, the structure segmenter 24a, material segmenter 24b, and abnormal material segmenter 24c can assign multiple types, optionally each with a confidence limit or probability, to each voxel in the medical image. For example, the structure segmenter 24a, in segmenting a wrist HRpQCT scan, can assign an ambiguous voxel both a "transition zone" and a "trabecular region" type, along with their (typically different) confidence limits or probabilities.

[0076] The segmenter 22 uses trained segmentation models 42, which may include a structural segmentation model, a material segmentation model, and an abnormal material segmentation model, to generate the segmentation map 50. The segmentation models are trained using machine learning algorithms, which may include, for example, neural networks, support vector machines (SVMs), decision trees, or combinations thereof. In these embodiments, the segmentation models are trained using a deep learning neural network having multiple layers, including: a convolutional neural network layer, a fully connected neural network layer, a normalization layer, and a multiplication layer.

[0077] Segmenter 22 may additionally (or alternatively) use non-machine learning based techniques for segmentation, such as methods based on the location of edges, corners, and transition slopes, or methods based on global features such as image histograms or brightness values. For example, U.S. Patent Publication No. 2012 / 0232375 ("Method and System for Image Analysis of Selected Tissue Structures") discloses a method for segmenting the transition zone between dense cortical and trabecular regions in a wrist CT scan based on local and global features of the bone, and in many applications it may be appropriate for segmenter 22 to use this technique.

[0078] 4 is a schematic diagram of the operation 80 of classifier 30. Classifier 30 is configured to receive input including segmentation map 50 (generated by segmenter 22) and non-image subject data 40, and process the input to generate one or more classification results.

[0079] However, both the segmentation map 50 and the non-image subject data 40 are processed by the segmentation map processor 26 and non-image data processor 28, respectively, before being passed to the classifier 30. For example, in some embodiments it may be advantageous to downsample the segmentation map 50 to a lower resolution map to enable faster image processing in the classifier 30, and if such processing is desired or required, it is performed by the segmentation map processor 26. In some embodiments, the segmentation map processor 26 sets the type of any voxels (within a particular segmentation map) that have been assigned more than one type (typically with different confidence limits or probabilities), for example by assigning the type with the higher or highest probability to the voxel.

[0080] The non-image data may include structured and unstructured data. The non-image data processor 28 is configured to employ various techniques to process any structured data by extracting features therefrom, in each case according to the structure and format of the data. For example, typically, structured data is stored and maintained in structured data stores such as database tables, .json files, .xml files, .csv files, etc. The non-image data processor 28 extracts features from the structured data by querying the required parameters and attributes from each source of data.

[0081] The non-image data processor 28 processes the unstructured data in two steps: first, convert it into structured data, and then extract features from the converted data. The conversion method used by the non-image data processor 28 is specific to each source. For example, to convert doctor's notes into structured data, the non-image data processor 28 can convert the notes into computer-recognizable text by using a trained optical character reading (OCR). The non-image data processor 28 then parses the converted text with keywords, such as "fracture," "pain," and "fall," in this example. Once the unstructured data has been converted into structured data, the non-image data processor 28 then extracts features from the now structured data.

[0082] The processed non-image data 40 and the segmentation map 22 are passed to a classifier 30, which uses these inputs to generate a classification output having a classification score (e.g., a disease status score 82, a disease progression score 84, and / or a treatment score 86).

[0083] 5 shows a flow diagram 90 for training the classification neural network 58 of the classifier 30 with deep neural networks, in accordance with an embodiment of the present invention. As shown, at S92, data including medical image data and non-image data is collected or input for training and testing. At S94, the collected images are segmented by the segmenter 22 to generate a segmentation map (described above).

[0084] In S96, the image data 38 and non-image data 40 are annotated with labels by qualified experts with domain knowledge. In medical applications, the training classification output may be determined based on the subject's clinical records. For example, if the classification output is to include a fracture probability score, the training output is the timing (after the scan) of the fracture, determined from the subject's medical history, and if no fracture is evident, recorded as "negligible risk" (or some equivalent designation). If the classification output is to include scores for categories corresponding to known medical conditions, the training output is the subject's actual medical condition, also determined from the subject's clinical records.

[0085] At S98, the data (including segmentation map 50 and non-image data 40) is divided into a training set, a development set (Dev set for short, optional in some embodiments), and a test set, each serving a different purpose: the training set is the data on which to run a learning algorithm; the Dev set is the data used for parameter tuning; and the test set is the data to be used to evaluate a trained model.

[0086] S100 implements layers of a deep learning neural network. Each layer is made up of artificial neurons, which are mathematical functions that receive one or more inputs and sum them to produce an output. Typically, each input is weighted separately and the sum is passed through a nonlinear function. As the neural network learns, the model's weights are adjusted in response to the resulting error (the difference between the network output and the annotation) until the error can no longer be reduced.

[0087] At S102, parameters of the classification neural network 58 are initialized, including the number of layers, the number of neurons, neuron weights, and neuron function parameters. At S104, a learning algorithm is run on the training data set to update the parameters of the classification neural network 58. For example, the parameters may be updated by determining the slope of a loss function. The loss function is calculated by the labeled classification output and the output generated by the classification neural network 58. Development data may be used to explore the learning procedure and tune the layers and parameters.

[0088] At S106, the classifier 30 provided with the classification neural network 58 is tested on the test data. At S108, an evaluation is made as to whether the performance of the classifier 30 is satisfactory. If the performance is unsatisfactory, the process returns to step 92 and more training data is collected.

[0089] If performance is deemed sufficient at S108, processing continues to S110, where the trained classifier 30 is output, flagged for deployment, or released for use, and processing ends.

[0090] Those skilled in the art will appreciate that many modifications may be made without departing from the scope of the invention, and in particular, it will be noted that certain features of the embodiments of the invention may be used to yield further embodiments.

[0091] It should be noted that even if a reference to prior art is made herein, such reference does not constitute an admission that the prior art is part of the public knowledge in any country.

[0092] In the appended claims and the foregoing detailed description, unless the context requires otherwise, either expressly or necessarily implied, the terms "comprises," "comprises," "comprises," and the like, are used in their inclusive sense (i.e., specifying the presence of the declared features) and do not exclude the presence or inclusion of additional features in various embodiments of the invention. [Explanation of symbols]

[0093] 12 Classification Controller 14 User Interface 16 GUI 18 Processors 20 Memory 30 classifier

Claims

1. 1. A computer-implemented method for training a classification machine learning model for classifying structures or materials in an image of a subject, the method comprising: partitioning annotated segmentation map and annotated non-image data into a training set and a test set, the annotated segmentation map being obtained by segmenting one or more images of one or more subjects, and the annotated non-image data being associated with each of the one or more subjects; (a) implementing a classification machine learning model, the step including initializing parameters of the classification machine learning model; (b) updating parameters of the classification machine learning model by running a learning algorithm on the training set; (c) testing the classification machine learning model on the test set; (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) through (d) if performance is insufficient in step (d), or outputting the classification machine learning model for deployment if performance is sufficient in step (d).

2. 10. The method of claim 1, further comprising collecting more image and non-image data to train the classification machine learning model if performance is found to be insufficient.

3. 3. The method of claim 1 or 2, wherein the classification machine learning model comprises a neural network having multiple layers comprising artificial neurons; The parameters include the number of layers, the number of neurons, neuron weights, and neuron function parameters; The method, wherein testing the classification machine learning model includes testing the classification machine learning model on the test set.

4. 4. The method of claim 1, further comprising splitting the annotated segmentation map and the annotated non-image data into a training set, a development set and a test set, and using the development set for testing a learning procedure and for tuning the parameters.

5. 1. A system for training a classification machine learning model for classifying structures or materials in an image of a subject, the system comprising a processor, the processor comprising: partitioning annotated segmentation map and annotated non-image data into a training set and a test set, the annotated segmentation map being obtained by segmenting one or more images of one or more subjects, and the annotated non-image data being associated with each of the one or more subjects; (a) implementing a classification machine learning model, the step including initializing parameters of the classification machine learning model; (b) updating parameters of the classification machine learning model by running a learning algorithm on the training set; (c) testing the classification machine learning model on the test set; (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) through (d) if performance is insufficient in step (d), or outputting the classification machine learning model for deployment if performance is sufficient in step (d).

6. The system of claim 5, wherein the system is configured to perform a step of collecting more image and non-image data to train the classification machine learning model if performance is determined to be insufficient.

7. The system of claim 5, wherein the classification machine learning model comprises a neural network having multiple layers comprising artificial neurons; The parameters include the number of layers, the number of neurons, neuron weights, and neuron function parameters; The system, wherein testing the classification machine learning model includes testing the classification machine learning model on the test set.

8. A system as described in any one of claims 5 to 7, configured to divide the annotated segmentation map and the annotated non-image data into a training set, a development set and a test set, and to perform the step of using the development set to examine a learning procedure and to tune the parameters.

9. A computer readable medium comprising a computer program configured to perform the method of any one of claims 1 to 4 when executed by one or more computing devices.

Images