Image analysis method and system

Abstract

To provide a computer-implemented image analysis method.SOLUTION: A method of the present invention includes: a step for quantifying one or more features segmented and identified from a medical image of a subject; a step for extracting a clinically related feature from non-image data on the subject; a step for evaluating the quantified feature segmented from the medical image and the feature extracted from the non-image data with a trained machine learning model; and a step for outputting one or more results on the evaluation of the features.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method and system for image analysis, particularly but not exclusively for use in medical imaging applications such as diagnosing and monitoring diseases and conditions in (e.g. musculoskeletal) imaging. Regions of interest may be 2D regions or 3D volumes within such medical images. Possible medical imaging applications include computed tomography (CT), magnetic resonance (MR), ultrasound, lesion scanner imaging, etc.

[0002] This application is based on and claims priority to U.S. patent application Ser. No. 16 / 448,460, filed June 21, 2019, the entire contents of which are incorporated by reference as filed. [Background technology]

[0003] Morbidity, mortality and costs associated with the development of musculoskeletal disorders are on the rise, in part because longer life spans are increasing the proportion of the elderly population. Early detection of musculoskeletal disorders may facilitate interventions that may ameliorate disease progression or minimize adverse outcomes (e.g., fractures).

[0004] In the field of musculoskeletal imaging, various techniques are used to detect musculoskeletal disorders, for example bone biopsy is used to detect osteosarcoma or other bone disorders, and tissue properties can be accurately measured, although bone biopsy is invasive and can result in pain and scarring.

[0005] Other approaches assess musculoskeletal disorders by analyzing musculoskeletal images to identify or quantify abnormalities, where the images are acquired using a variety of imaging modalities, including: Dual-energy X-ray Absorptiometry (DXA), Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and X-ray scanners. Each modality has specific advantages for screening and evaluating musculoskeletal disorders (e.g., bone fragility, osteoarthritis, rheumatoid arthritis, osteomalacia, and bone deformities).

[0006] For example, High-Resolution Peripheral Quantitative Computed Tomography (HRpQCT) is a high-resolution, low-dose, non-invasive technique that can be used to assess the three major determinants of bone quality: microarchitecture, mineralization, and biomechanical properties.

[0007] There are manual, semi-manual, and automated methods for assessing musculoskeletal disorders, which may involve a variety of imaging modalities. For example, the Bone Marrow Burden (BMB) score is an MRI scoring method for manually assessing bone marrow involvement in Gaucher disease. Radiologists measure the BMB score on MRI images of the lumbar spine and femur, using signal intensity and distribution according to a ranking criteria. For example, according to one known scoring system, a score is given for the lumbar spine ranging from 0 to 8 and for the femur ranging from 0 to 8, with a total score ranging from 0 to 16. A higher total BMB score indicates a more severe bone marrow involvement.

[0008] When measuring bone mineral density (BMD) using DXA images, a semi-manual method is used. The spine or lumbar region is scanned by DXA. A radiation technician or doctor selects the region of interest (ROI) (for example, different vertebral sections of a spine scan or the femoral neck of a lumbar scan). The bone mineral density of the selected region is determined based on a predefined density comparison formula. The measured density value is converted to a T-score, and this is done by comparing it with measurements from a group of same-sex young adults with peak bone mass. A T-score of ≥1 is considered normal; a T-score of -2.5 < T < -1 is classified as osteopenia; and a T-score of ≤ -2.5 is defined as osteoporosis. The T-score is considered by clinicians when examining whether to evaluate the risk of fracture occurrence and recommend treatment.

[0009] However, a major concern regarding manual or semi-manual musculoskeletal imaging methods is their labor intensiveness and reproducibility. Due to the subjectivity associated with measurement and its evaluation, accuracy and reproducibility are not guaranteed.

[0010] One automated method for evaluating musculoskeletal images is disclosed in Patent Document 1 titled "Method and System for Image Analysis of Selected Tissue Structures". This method automatically analyzes and evaluates musculoskeletal images such as wrist CT scans. The method can be automatically used to extract the radius from a wrist CT scan and segment the radius into fine structures (for example, cortical bone, transition zone, and trabecular region), and can quantify cortical porosity and trabecular density.

[0011] Known methods have focused individually on each measurement value, but this does not provide useful insights into whether specific musculoskeletal attributes can enable discrimination between diseased and non-diseased individuals. For example, the BMB score only evaluates bone marrow involvement in relation to Gaucher's disease, the BMD score only evaluates bone mineral density involvement in relation to fracture risk, and cortical porosity provides measurements for important bone attributes but not complete information about bone microstructure.

Prior Art Documents

Patent Documents

[0012] [Patent Document 1] U.S. Patent No. 9,064,320 Summary of the Invention

[0013] According to a first aspect of the present invention there is provided a computer implemented image analysis method comprising the steps of: Quantifying one or more features (e.g., including structure or material) that have been segmented and identified from the medical image of the subject; extracting non-image data relating to the subject from one or more non-image data sources; Extracting clinically relevant features from non-image data relating to the subject; evaluating the quantified features segmented from the medical image and the features extracted from the non-image data with a trained machine learning model; Outputting one or more results of the evaluation of the features.

[0014] Thus, by combining multiple features extracted from both image data and non-image data using machine learning algorithms (e.g., deep learning algorithms), the present invention enables the diagnosis and monitoring of diseases and conditions in medical images, such as musculoskeletal images.

[0015] In an embodiment, a method includes receiving an image, segmenting one or more features from the image, and identifying the segmented features from the image.

[0016] In an embodiment, a method includes receiving an image along with a feature segmented therefrom, and identifying the segmented feature from the image.

[0017] In an embodiment, the segmenting and identifying is implemented by a segmentation and identification model trained with a machine learning algorithm, the model configured to segment and identify features from the image. By way of example, the trained segmentation and identification model comprises a deep convolutional neural network trained model.

[0018] In an embodiment, the trained machine learning model comprises a disease classification model.

[0019] In an embodiment, the trained machine learning model comprises a model trained using features extracted from patient data and labels or annotations indicating disease or non-disease.

[0020] In an embodiment, the trained machine learning model comprises a deep learning neural network or other machine learning algorithm (such as a support vector machine (SVM), decision tree, or AdaBoost).

[0021] In an embodiment, the trained machine learning model is a model that has been trained to diagnose and / or monitor one or more (e.g., musculoskeletal) disorders.

[0022] In an embodiment, the method further includes (i) training the trained machine learning model, and / or (ii) updating (e.g., continuously) the trained machine learning model with new or newly analyzed additional labeled data derived from the subject data.

[0023] In an embodiment, the result comprises one or more disease classifications and / or probabilities.

[0024] In an embodiment, the method further comprises generating a report based at least on the results. Illustratively, the method comprises generating a report additionally based on information from a domain knowledge database.

[0025] In an embodiment, the assessment comprises a bone fragility assessment.

[0026] In an embodiment, the results include one or more fracture risk scores.

[0027] According to a second aspect of the present invention there is provided an image analysis system comprising: a feature quantification unit configured to quantify one or more features (structures or materials) that have been segmented and identified from the medical image of the subject; a non-image data processor configured to extract non-image data relating to the subject from one or more non-image data sources and to extract clinically relevant features from the non-image data relating to the subject; a feature evaluation unit configured to evaluate features segmented from the medical image and features extracted from the non-image data with a trained machine learning model; An output configured to output one or more results of the evaluation of the features.

[0028] In an embodiment, the system further comprises a segmenter and classifier configured to receive an image, segment one or more features from the image, and identify the segmented features from the image. In one example, the segmenter and classifier comprises a segmentation and classification model (i.e., a segmentation and classification model trained using a machine learning algorithm) configured to segment and identify features from the image. The trained segmentation and classification model may comprise a deep convolutional neural network trained model.

[0029] In an embodiment, the trained machine learning model comprises a disease classification model.

[0030] In an embodiment, the trained machine learning model comprises a model trained using features extracted from patient data and labels or annotations indicating disease or non-disease.

[0031] In an embodiment, the trained machine learning model comprises a deep learning neural network or other machine learning algorithm.

[0032] In an embodiment, the trained machine learning model is a model that has been trained to diagnose and / or monitor one or more (e.g., musculoskeletal) disorders.

[0033] In an embodiment, the system further comprises a machine learning model trainer configured to update (e.g., continuously) the trained machine learning model with additional labeled data derived from new or newly analyzed subject data.

[0034] In an embodiment, the result comprises one or more disease classifications and / or probabilities.

[0035] In an embodiment, the system further comprises a report generator configured to generate a report based at least on the results, Illustratively, the report generator is configured to generate a report additionally based on information from a domain knowledge database (e.g., of the system).

[0036] In an embodiment, the feature assessor is configured to assess bone fragility.

[0037] In an embodiment, the results include one or more fracture risk scores.

[0038] According to a second aspect of the present invention there is provided computer program code comprising instructions configured to, when executed by one or more processors, perform the image analysis method of the first aspect. This aspect may also provide a (possibly non-volatile) computer readable medium comprising computer program code as aforesaid.

[0039] Any of the various individual features of each of the above-mentioned aspects of the invention, as well as any of the various individual features of the embodiments described herein, including the claims, may be combined in any suitable and desired manner. [Brief description of the drawings]

[0040] In order that the invention may be more clearly defined, embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0041] [Figure 1] 1 is a schematic diagram of a medical image analysis system according to an embodiment of the present invention; [Diagram 2] 2A and 2B are schematic diagrams of exemplary image data stored in a memory of the medical image analysis system of FIG. [Diagram 3] Figure 3A is a schematic diagram of an example unpopulated, non-image data store of the medical image analysis system of Figure 1. Figure 3B is a schematic diagram of an example populated, non-image data store of the medical image analysis system of Figure 1. [Figure 4] FIG. 2 is a schematic diagram of the operation of the image data processor of the medical image analysis system of FIG. 1 in a bone fragility application. [Diagram 5] 2 is a schematic diagram of the operation of the image data processor of the medical image analysis system of FIG. 1 on exemplary non-image data including structured and unstructured data. [Figure 6] FIG. 2 is a schematic diagram of offline training of the machine learning model of the feature evaluation unit of the medical image analysis system of FIG. 1. [Figure 7] FIG. 2 is a schematic diagram of a report generator of the medical image analysis system of FIG. 1. [Figure 8] FIG. 2 illustrates an exemplary operation of the medical image analysis system of FIG. 1 in a bone fragility application, where the image data includes a wrist HRpQCT scan and the non-image data includes patient information. [Figure 9] 2 is a schematic diagram of an exemplary report generated by the report generator of the medical image analysis system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] FIG. 1 is a schematic diagram of an image analysis system, referred to as a medical image analysis system 10, in accordance with an embodiment of the present invention.

[0043] Referring to FIG. 1, system 10 includes image analysis controller 12 and user interface 14 (including GUI 16). User interface 14 typically includes one or more displays (one or more of which can display GUI 16), a keyboard and a mouse, and optionally a printer. Image analysis controller 12 includes at least one processor 18 and memory 20. System 10 can be implemented, for example, as a combination of software and hardware on a computer (e.g., a personal computer or a mobile computing device) or as a dedicated image segmentation system. System 10 can optionally be distributed; for example, all or some components of memory 20 can be located remotely from processor 18; user interface 14 can be located remotely from memory 20 and / or processor 18, and indeed it can include a web browser and a mobile device application.

[0044] Memory 20 is in data communication with processor 18 and typically comprises both volatile and non-volatile memory (and may include one or more of each memory type), including RAM (random access memory), ROM, and one or more mass storage devices.

[0045] As described in more detail below, the processor 18 includes an image data processor 22 (including a segmenter and classifier 24 and a feature quantifier 25), a non-image data processor 26, a feature evaluator 28 (including a precedent comparator 29), a machine learning model trainer 30, a report generator 32, an I / O interface 34, and a results section, which is a results output 36. The memory 20 includes program code 38, image data 40, non-image data 42, training data 44, a trained machine learning model 46, and domain knowledge 48. The image analysis controller 12 is implemented, at least in part, by the processor 18 executing the program code 38 from the memory 20.

[0046] Generally speaking, the I / O interface 34 is configured to read or receive image data and non-image data (e.g., in DICOM format) relating to a subject or patient into the image data 40 and non-image data 42 of the memory 20. The image data processor 22 segments (i.e., extracts) and quantifies features from the image data using the segmenter and classifier 24 and feature quantifier 25, the non-image data processor 26 extracts features from the non-image data, the feature evaluator 28 processes the features with one or more machine learning models 46, and the I / O interface 34 outputs the results of the analysis, for example to the results output 36 and / or the GUI 16.

[0047] System 10 is particularly useful in musculoskeletal image analysis, using artificial intelligence (e.g., deep neural networks) and other machine learning and computer vision algorithms to accurately and reproducibly automate feature extraction and image feature quantification, and the results may then be suitable for use in localizing musculoskeletal disease, diagnosing musculoskeletal disease, and monitoring disease progression.

[0048] 1, system 10 is configured to receive two types of subject or patient data: image data (e.g., medical images acquired at one or more site locations using one or more imaging modalities) and non-image data (structured patient data, such as medical history data and interview data, and unstructured patient data, such as physician notes and audio recordings). System 10 stores this data in image data 40 and non-image data 42, respectively. The various data formats and structures of the non-image data dictate the behavior of non-image data processor 26.

[0049] The image data processor 22 comprises two components: a segmenter and classifier 24 configured to perform segmentation and classification, and a feature quantifier 25 configured to perform feature quantification. Image data 40 is processed by the image data processor 22 to automatically segment and classify clinically relevant structures, features or materials, in this embodiment from medical images of a subject or patient, using the segmenter and classifier 24, and the feature quantifier 25 to quantify the segmented and identified clinically relevant structures, features, materials, or combinations thereof. It should be noted, however, that in some embodiments, the system 10 may be configured to receive images that have already been segmented (and, optionally, segmented features may already have been identified), thereby eliminating or eliminating the use of the segmenter and classifier 24.

[0050] In this embodiment, segmentation and identification are implemented by traditional image processing methods (i.e., non-machine learning type), such as thresholding, contour detection, blob detection, and the methods described in U.S. Pat. No. 9,064,320 (titled "Method and System for Image Analysis for Selected Tissue Structures").

[0051] However, in some other embodiments, the segmenter and classifier 24 may comprise a segmentation and classification model trained with a machine learning algorithm and configured to segment and classify structures or features of interest from images. For example, in a bone fragility application, such a model trained with a deep convolutional neural network may be used to segment and classify the radius from a wrist HRpQCT scan. The training data may be annotated wrist HRpQCT scans that distinguish voxels of the radius, ulna, and surrounding tissues.

[0052] The non-image data is processed by a non-image data processor 26 to extract clinically relevant features from the non-image data.

[0053] The quantitative features obtained from both image data and non-image data are input by the feature evaluator 28 into the machine learning model 46. The machine learning model 46 is pre-trained with labeled training data, i.e., annotations that are the machine learning ground truth. For example, to train a bone segmentation model, the training data typically comprises source images and corresponding ground truth images, i.e., images with bones annotated. To train a disease classification model, the training data typically comprises features extracted from patient data and corresponding ground truth images, which are diseased or non-disease labels / annotations. (The terms "label" and "annotation" are essentially interchangeable, although "label" is generally used in reference to diseases and conditions and "annotation" is generally used in reference to image segmentation / discrimination machine learning models.)

[0054] In this embodiment, the processor 18 includes a machine learning model trainer 30 that is configured to train the machine learning model 46 (and retrain or update train the machine learning model 46, as described below) using training data 44 (which may include new subject data), although in other embodiments the machine learning trainer 30 may be configured or used solely for retraining or update training the machine learning model 46.

[0055] A variety of machine learning algorithms (using features from both image data and non-image data) can be used to train the machine learning model 46 used in this embodiment to diagnose and monitor musculoskeletal disorders and the like, and may involve, for example, deep learning neural networks (which are preferred) or support vector machines (SVMs), decision trees, AdaBoost, and other machine learning algorithms.

[0056] In one embodiment, the pre-trained machine learning model 46 is continually updated with additional labeled data derived from newly analyzed patient data.

[0057] Thus, the feature evaluator 28 evaluates and assesses the features using the machine learning model 46, and also outputs the results (e.g. in the form of classifications and probabilities) to the report generator 32. The report generator 32 generates a report based on these results. The report generator 32 can also obtain additional information from the domain knowledge 48 and combine this information with the results, and both can be presented in the report.

[0058] For example, in a bone fragility assessment application embodiment, the feature evaluator 28 may use the machine learning model 46 to output results including a fracture risk score. The report generator 32 may be configured to obtain information represented by this score from domain knowledge 48, which may aid in interpreting the score, for example. The report generator 32 may then present both the score and the information in a report.

[0059] The final report is output via the results output 36 and / or the user interface 14 .

[0060] 2A and 2B are schematic diagrams of exemplary image data stored within image data 40. The image data includes clinically relevant medical images of a subject or patient, typically acquired at one or more regional locations using one or more imaging modalities.

[0061] For example, in a bone fragility example, as shown in Figure 2A, the input image data may involve a wrist HRpQCT scan 50. In another example, it may be desired to analyze both the peripheral and central skeleton, and as shown in Figure 2B, the input image data may include a wrist HRpQCT scan 50, a leg HRpQCT scan 52, a spine DXA scan 54, and a lumbar DXA scan 56 (and, if bone marrow information is optionally integrated, a lumbar MRI scan 58).

[0062] 3A and 3B are schematic diagrams of examples of unpopulated and populated non-image data, respectively. The non-image data received by the system 10 and stored in the non-image data 42 may contain a significant amount of useful data that may be indicative of a musculoskeletal disorder. As shown in FIG. 3A, the non-image data may include patient information from a variety of structured and unstructured data sources, collected, for example, during examination and treatment of a subject or patient. Structured data may include, for example, subject demographics such as gender, age, weight, height, etc., laboratory test results (e.g., blood test results, DNA test results, etc.), medical treatment data (e.g., drug type and dosage, etc.), medical history data (e.g., smoking history, alcohol history, fracture history, etc.), etc. Unstructured data may include text documents of laboratory test results, doctor's notes, radiology reports, etc.

[0063] As shown in Figure 3A, the non-image data in non-image data 42 can have a variety of formats, including structured video 62, text 64, numeric values ​​66, and / or audio 68, and unstructured video 72, text 74, numeric values ​​76, and / or audio 78. Figure 3B illustrates an example aspect, where the non-image data includes structured text 64 in the form of medical history or interview data 80, structured numeric values ​​66 in the form of patient measurements 82, unstructured video 72 in the form of patient gait video 84, unstructured text 74 in the form of physician notes, and unstructured audio 78 in the form of patient records 88.

[0064] The non-image data processor 26 is configured to have and use different data processing and feature extraction techniques in processing the non-image data, in each case consistent with the structure and format of each portion of data.

[0065] 4 is a schematic representation 90 of the operation of the image data processor 22 for bone fragility. In this example, a pre-trained deep neural network model is used to identify and segment 92 the radius from a wrist HRpQCT scan. The radius is then segmented and identified by the segmenter and classifier 24 into dense cortex 94, transition zone 96, and trabecular bone region 98. The attributes of the different structures and combinations of structures are then quantified by feature quantifier 25 into features such as: vBMD (volumetric bone mineral density of the whole radius) 102, cortical porosity (percentage of pores in the bone volume) 104, transition zone vBMD (volumetric bone mineral density of the transition zone) 106, trabecular zone vBMD (volumetric bone mineral density of the trabecular zone) 108, matrix mineralization level (average percentage of mineralized material) 110, medullary fat status (density of medullary fat in the trabecular zone) 112, cortical thickness (average thickness of dense cortex) 114, and trabecular tissue separation (average separation distance of trabecular tissue) 116. These features were chosen because they have been shown to be indicative of bone fragility and fracture risk. In other applications, features related to the condition of interest would be selected similarly.

[0066] 5 is a schematic representation 120 of the operation of the non-image data processor 26 on exemplary non-image data 42, including structured and unstructured data. The non-image data processor 26 extracts features from the non-image data, with various tools adapted for extracting different types of non-image data with different structures and formats.

[0067] Typically, the structured data 122 is stored and maintained in a structured data store such as a database table, a .json file, an .xml file, a .csv file, etc. The non-image data processor 26 extracts features from the structured data by querying 124 the required parameters and attributes from the data sources, thereby extracting information of interest from the structured data sources. In some cases, the structured data may be complete and of interest itself, and no querying may be required.

[0068] The unstructured data 126 may include physician notes, audio recordings, graphical reports, etc., and therefore, prior to extraction, the non-image data processor 26 may typically convert the unstructured data 126 into structured data. The conversion technique used by the non-image data processor 26 is specific to each source or type of unstructured data. For example, to convert physician notes into structured data, the non-image data processor 26 may include or utilize trained optical character reading (OCR) to convert 128 the notes into text that is recognizable by the system 10. The non-image data processor 26 then performs syntactic parsing on the text (using keywords such as "fracture," "pain," "fall," etc.).

[0069] As another example, the non-image data processor 26 can include or use a trained speech recognition model to convert 128 an audio recording of an interview into text that can be recognized by the system 10. The non-image data processor 26 then segments and organizes the converted text into structured data by reference to the original interview questions.

[0070] In another exemplary embodiment, the non-image data may include video, for example because studies have shown that gait-related patterns are associated with fracture risk, and the non-image data processor 26 processes the video to extract gait features therefrom.

[0071] Once converted 128 into structured data, the non-image data processor 26 extracts features from the structured data (previously unstructured data) by querying 124 the necessary parameters and attributes from the data source, although as noted above, no querying is necessary if the now structured data is self-contained and of interest. In the above example of walking video, after converting the video into gait features (which constitute structured data), no such querying 124 is necessary.

[0072] Ultimately, the non-image data processor 26 extracts features 130 such as gender 132 , age 134 , smoking history 136 , fracture history 138 , medical history 140 and gait 142 .

[0073] 6 is a schematic representation 144 of (offline) training of a model in machine learning model 46. First, in S146, training data is accessed or selected for processing from a database of training data 44. The training data may include subject image data and / or non-subject image data. Data having various attributes is selected for training to represent the target population. Using the techniques described above, the training data is: extracted features in S148; quantified the extracted features in S150. The training data then has a set of quantitative features.

[0074] At S152, the training data is annotated (by a human operator) with a correct diagnosis. At S154, the training data, represented by the extracted features, is combined with the diagnostic labels and fed into one or more machine learning algorithms of the machine learning model trainer 30 to train one or more models. It should be noted that various machine learning algorithms are suitable for training the models. They may be black-box algorithms (e.g., neural networks), where the decision process is beyond the scope of human knowledge. Alternatively, they may be white-box algorithms (e.g., decision trees, support vector machines, or linear discriminant analysis), where the model's decision process can be interpreted by a human. They may also be hybrid algorithms that combine white-box and black-box algorithms.

[0075] In S156, the now trained model is deployed for use and will typically be stored in machine learning model 46.

[0076] In use, features extracted from new patient data are fed into the now-trained machine learning model 46. As described above, the machine learning model 46 evaluates the features and outputs one or more results. The results may be a binary classification, a score, or a probability. In some embodiments, the new patient data can be added to the training data 44 (optional step S158) and used to retrain or update the model 46.

[0077] The training of the models and their usefulness are domain specific, as are each of the resulting machine learning models 46. Each model 46 depends on the data and the target condition (e.g., disease such as bone fragility or osteoporosis, or infection such as a particular bacterial infection). In one embodiment, one of the models 46 is trained and used to diagnose one or more diseases or conditions by generating one or more disease probability scores. In another embodiment, multiple models 46 can be trained and used to diagnose respective diseases. Similarly, in another embodiment, one of the models 46 can be trained to generate a prediction of the likelihood of development of one or more conditions (e.g., fractures), or multiple models 46 can be trained and used to generate respective predictions of the likelihood of development of respective conditions.

[0078] FIG. 7 is a schematic diagram of the report generator 32, which generates a report based at least on the results 162 obtained from the machine learning model 46. The report generator 32 can use information stored in the domain knowledge 48 in generating the report. This information can include information that can help a physician in interpreting the results output by the feature evaluation unit 28 (e.g., diagnostic and treatment guidelines). For example, in the application of the system 10 for fracture assessment, treatment guidelines for different levels of fracture risk can be presented to the physician. The report can then include both the results generated by the feature evaluation unit 28 (characterizing the subject's or patient's bones, such as a fragility score, a porosity score, a trabecular bone score, etc.) and information that can help interpret the scores and select a treatment.

[0079] This information may optionally include diagnoses of a cohort of prior subjects or patients with comparable scores, and / or treatments or therapies prescribed for that cohort. Indeed, in some embodiments, feature evaluator 28 includes an optional antecedent comparator 29 that is configured to compare the results (particularly numerical results such as scores) generated by machine learning model 46 with such information from domain knowledge 48, and to automatically form a diagnosis (by assigning a likelihood or respective probability that the subject has the disease) and treatment / therapy recommendations based on that comparison, and output these to report generator 32 for inclusion in the report.

[0080] Reports may also include results of disease progression monitoring and treatment efficacy. For example, treatment may be deemed successful if a subject's or patient's fracture risk is found to be reduced upon follow-up examination.

[0081] The report may also display similar or opposite cases. With regard to the health care professional's prognostic evaluation and treatment decisions, showing similar or opposite cases may be useful. Presenting similar or opposite cases may help train novice users or help users validate results generated from the model 46. Studying similar or opposite cases may also provide insight into how past subjects or patients responded to different treatment options, allowing the health care professional to evaluate the effectiveness of the treatment (or the estimated effectiveness of the proposed treatment) for the current patient.

[0082] All of these results and information are included in a report 166 by the report generator 32, which delivers the report 166 (via the results output 36) to the user interface 14 (e.g., a web browser, a PC application, or a mobile device app) for user viewing.

[0083] An exemplary embodiment of the system 10 in a bone fragility application is shown in Figure 8. The image data includes a wrist HRpQCT scan 172 and the non-image data includes patient demographics 174. As shown in Figure 8, the segmenter and classifier 24 (using the trained segmentation and classification model 46) identifies and extracts 176 the radius bone and segments 178 it into structures (dense cortex, transition zone, and trabecular regions). Based on this structural segmentation, the feature quantifier 25 determines the cortical porosity 180 and trabecular density 182.

[0084] The non-image data processor 26 uses the query to extract gender 186 and age 188 from the non-image data basic patient information 174. The feature evaluator 28 receives the four features cortical porosity 180, trabecular density 182, gender 186, and age 188. The feature evaluator 28 generates a structural fragility score 190 by evaluating the four features 180, 182, 186, and 188 using another trained model 46'' (in this example, a bone collapse model). Alternatively, the bone collapse model 46'' can be trained using an algorithm such as that described by Zebaze et al. ("Increased Cortical Porosity and Reduced Trabecular Density Are Not Necessarily Synonymous With Bone Loss and Microstructural Deterioration" (JBMR Plus (2018)). Here, the training data 44 represents the four features 180, 182, 186, and 188 mentioned above and has been annotated with the correct bone fragility diagnosis.

[0085] The vulnerability scores 190 and other information are used by the report generator 32 to generate a report 166, which is ultimately output to the user interface 14 (in this example, a web browser).

[0086] 9 illustrates an exemplary report 200 generated by report generator 32 of system 10. Report 200 includes several sections: subject / patient details section 202, bone volume fraction map section 204, score section 206, and score graph section 208. Subject / patient details section 202 typically includes bibliographic information (e.g., name, date of birth, sex, and age), physician name (or names of physicians), date of data acquisition, and date of data processing (which also serves as the date of report 200).

[0087] The bone volume fraction map section 204 includes an example false-color 3D reconstruction 210 of the patient's radius bone volume, a corresponding example false-color 3D reconstruction 212 of a (typically healthy) radius bone volume as a reference, and a false-color legend 214. (Note that the colors in FIG. 9 are rendered in grayscale.) The false color is used to represent "bone volume fraction," which is the volume of mineralized bone per unit volume of the sample. By presenting the example reconstructions 210, 212 side-by-side, the report 200 allows medical personnel to easily assess the extent and distribution (of bone loss in this example).

[0088] The score column 206 contains the scores generated by the feature evaluator 28. In this example, these scores are a fragility score 216, a porosity score 218, and a trabecular bone score 220, which are expressed as percentages. The fragility score 216 indicates the degree of collapse of the coexisting cortical and trabecular bone. The porosity score 218 indicates the degree of porosity of the cortical bone. The trabecular bone score 220 is a score that indicates the density of the trabecular bone region.

[0089] Optionally, these scores 216, 218, 220 may be represented graphically on each bar 222, 224, 226, for example the scores may be represented as sliders 228, 230, 232. In this example, each bar is labeled with three values: a lower value and an upper value indicating the scores at either end of the bar, and a third value indicating the watershed between the ranges of decreased and increased risk (or normal and abnormal). For example, in bar 222 for fragility score 216 in the illustrated example, the values ​​are 20%, 90%, and 70%, respectively; in bar 224 for porosity score 218 in the illustrated example, the values ​​are 20%, 60%, and 46%, respectively; and in bar 226 for trabecular bone score 220 in the illustrated example, the values ​​are 0%, 10%, and 2%, respectively.

[0090] These three values ​​(the lower value, the upper value, and the cutoff value between decreasing and increasing risk) are determined from precedent data contained, for example, in training data 44 or domain knowledge 48. Alternatively, system 10 may access one or more external databases to obtain all or some of the precedent data.

[0091] Optionally, the bars 222, 224, 226 can be colored or shaded to indicate areas of decreased 234 and increased 236 risk, providing an immediate and visual indication of whether the respective sliders 228, 230, 232 (and therefore the scores 216, 218, 220) correspond to antecedent data associated with high or low risk subjects. In FIG. 9, darker shading is used to indicate areas of higher risk.

[0092] Thus, in the score field 206, the report 200 results in the following scores: 75% for the fragility score 216 (indicating increased risk); 37% for the porosity score (indicating decreased risk); and 1% for the trabecular bone score (indicating increased risk). These scores, alone or in combination, do not constitute a diagnosis, as they do not identify a specific disease or affliction, although they do inform the user about useful indications of symptoms and how pronounced those symptoms are.

[0093] The score graph section 208 presents plots 240, 242, 244 for the fragility score (FS((%))), porosity score (PS((%))), and trabecular bone score (TS((%)), each plotted over time T. In the illustrated example, the time divisions are typically 1-2 days, but in general, each division represents the time between scans (i.e., divisions within a single graph do not necessarily represent consistent time differences). In FIG. 9, the plots 240, 242, 244 first show the scores in the score section 206, followed by the scores obtained at a series of follow-up visits for each case. The plots 240, 242, 244 thereby allow a medical professional to quickly assess changes occurring in the subject, including spontaneous changes, changes in response to treatment, and other changes.

[0094] Those skilled in the art will appreciate that many modifications may be made without departing from the scope of the invention, and in particular, it will be noted that certain features of the embodiments of the invention may be used to yield further embodiments.

[0095] It should be noted that even if a reference to prior art is made herein, such reference does not constitute an admission that the prior art is part of the public knowledge in any country.

[0096] In the appended claims and the foregoing detailed description, unless the context requires otherwise, either expressly or necessarily implied, the terms "comprises," "comprises," "comprises," and the like, are used in their inclusive sense (i.e., specifying the presence of the declared features) and do not exclude the presence or inclusion of further features in various embodiments of the invention. [Explanation of symbols]

[0097] 12 Image Analysis Controller 14 User Interface 16 GUI 18 Processors 20 Memory 40 Image data storage unit 44 Training Data 48 Domain knowledge 200 reports

Claims

1. 1. A method of performing computer-implemented image analysis on a computer, the method comprising: Quantifying the segmented and identified one or more features from the medical image of the subject; extracting non-image data relating to the subject from one or more non-image data sources; extracting clinically relevant features from the non-image data relating to the subject; evaluating the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data with a trained machine learning model; and outputting one or more results of an evaluation of the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data; The method, wherein the trained machine learning model includes a model trained using features extracted from patient data and corresponding ground truths including labels or annotations indicating disease or non-disease, and the patient data includes image data and / or non-image data.

2. 10. The method of claim 1, further comprising: a) receiving the medical image and segmenting one or more features from the medical image; or b) receiving the medical image along with one or more features segmented therefrom; and The method includes performing the step of identifying the one or more features segmented from the medical image.

3. 3. The method of claim 1 or 2, wherein the segmenting and identifying is implemented by a segmentation and identification model trained with a machine learning algorithm, the identification model configured to segment and identify the one or more features from the medical image.

4. 4. The method of claim 3, wherein the trained segmentation and discrimination model comprises a deep convolutional neural network trained model.

5. The method according to any one of claims 1 to 4, wherein the trained machine learning model is The method comprises a disease classification model and / or a deep learning neural network or other machine learning algorithm.

6. 6. The method of claim 1, wherein the trained machine learning model is a model trained to diagnose and / or monitor one or more musculoskeletal disorders.

7. 7. The method of claim 1, further comprising executing, with the processor, the steps of: (i) training the trained machine learning model; and / or (ii) updating the trained machine learning model with new or newly analyzed additional labeled data derived from subject data.

8. 8. The method of claim 1, wherein the results include: (i) one or more disease classifications; (ii) one or more disease probabilities; and / or (iii) one or more fracture risk scores.

9. 9. The method of any one of claims 1 to 8, wherein the assessment comprises a bone fragility assessment.

10. 1. An image analysis system, comprising: a feature quantifier configured to quantify one or more features segmented and identified from the medical image of the subject; a non-image data processor configured to extract non-image data relating to the subject from one or more non-image data sources and to extract clinically relevant features from the non-image data relating to the subject; a feature evaluator configured to evaluate the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data with a trained machine learning model; an output configured to output one or more results of an evaluation of the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data; The system, wherein the trained machine learning model includes a model trained using features extracted from patient data and corresponding ground truths including labels or annotations indicating disease or non-disease, and the patient data includes image data and / or non-image data.

11. 11. The system of claim 10, further comprising a segmenter and classifier configured to receive the medical image, segment one or more features from the medical image, and identify the one or more features segmented from the medical image.

12. 12. The system of claim 11, wherein the segmenter and classifier comprises a segmentation and identification model trained with a machine learning algorithm, the identification model configured to segment and identify the one or more features from the medical image.

13. 13. The system of claim 12, wherein the trained segmentation and discrimination model comprises a deep convolutional neural network trained model.

14. 14. The system of claim 10, wherein the trained machine learning model comprises a disease classification model and / or a deep learning neural network or other machine learning algorithm.

15. 15. The system of claim 10, wherein the trained machine learning model is a model trained to diagnose and / or monitor one or more musculoskeletal disorders.

16. 16. The system of claim 10, further comprising a machine learning model trainer configured to update the trained machine learning model with additional labeled data derived from new or newly analyzed subject data.

17. 17. The system of any one of claims 10 to 16, wherein the results include: (i) one or more disease classifications; (ii) one or more disease probabilities; and / or (iii) one or more fracture risk scores.

18. 18. The system of claim 10, wherein the feature assessor is configured to assess bone fragility.

19. Computer program code comprising instructions configured, when executed by one or more computing devices, to perform the image analysis method of any one of claims 1 to 9.

20. 20. A computer readable medium comprising computer program code according to claim 19.

Images