Automated identification and classification of musculoskeletal abnormalities from medical images

Abstract

A method of training an image classification model to identify and / or classify musculoskeletal abnormalities. The method includes providing two or more deep learning models pre-trained on an initial image dataset and updating the pre-trained deep learning models on a secondary image dataset of medical imaging images of a same image type as images of a target dataset. The pre-trained deep learning models are then fine-tuned on the target dataset of medical imaging images associated with a task of identifying a musculoskeletal abnormality. Features are then extracted from the deep learning models and combined into a fused feature set for the task. The machine learning classifiers are then trained on the fused feature set of the extracted features.

Description

Automated Identification and Classification of Musculoskeletal Abnormalities From Medical ImagesTECHNICAL FIELD[1] The present invention relates to the automated identification and / or classification of musculoskeletal abnormalities from medical imaging images and methods for training an image classification model to identify and / or classify said musculoskeletal abnormalities. In particular, the present invention relates to, but is not limited to, automated identification and detection of shoulder abnormalities (abnormalities in the bones, muscles and joints of the shoulder) and, more particularly, the identification and detection of fatty infiltration of the rotator cuff and supraspinatus tears in medical imaging images.BACKGROUND[2] Any references to methods, apparatus or documents of the prior art are not to be taken as constituting any evidence or admission that they formed, or form part of the common general knowledge.[3] Medical abnormalities in the musculoskeletal system (e.g., muscles, bones and joints), and in particular shoulder abnormalities such as supraspinatus tears and fatty infiltration or degeneration of the rotator cuff muscles is commonly associated with different factors, such as, age-related degenerative changes, chronic muscle imbalances, and various pathologies such as rotator cuff tears and muscle atrophy.[4] The quantification and classification of the abnormalities plays a pivotal role in assessing the severity of muscle degeneration and predicting treatment outcomes.[5] Some classification systems, such as Goutallier classification system used to categorise fatty infiltration into five grades, use visual assessment of Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) images to classify abnormalities. These classification systems provide a valuable clinical tool for surgeons and radiologists to quantify and communicate the extent of abnormalities. However, some systems lack a good Interobserver and Interobserver reliability, thereby reducing their usefulness. To overcome some limitations and increase reliability of classification systems, different simplified classification systems have been proposed. While these classifications have effectively improved the reliability of muscle assessment, they still require manual human intervention, which is timeconsuming and prone to error and thus greatly limits the clinical applicability of these approaches.[6] The emergence and rapid advance of Artificial Intelligence (Al) and Machine Learning (ML) provide promising avenues to further increase reliability of detection and classification of musculoskeletal abnormalities, and specifically those associated with the shoulder. However, the lack of large volumes of specialised data associated with medical imaging presents a barrier to applying existing deep learning models to reliable classification of medical images.[7] Furthermore, inherent biases or weaknesses in individual model inhibit reliable generalization and robustness of the classifier.[8] It is an object of embodiments of the present invention to provide a solution to the problem of identifying musculoskeletal abnormalities from medical imaging images.SUMMARY OF INVENTION[9] In an aspect, the invention provides a method of identifying and / or classifying musculoskeletal abnormalities comprising: receiving medical imaging images of a musculoskeletal region of interest; and applying the medical imaging images to a pre-trained musculoskeletal abnormality machine learning classifier to thereby produce identifications and / or classifications of musculoskeletal abnormalities in the images, wherein the pretrained musculoskeletal abnormality machine learning classifier is: trained on a fused feature set of features extracted from two or more deep learning models that are pre-trained on an initial image dataset; updated on a secondary image dataset of medical imaging images of a same image type as images of a target dataset; and fine-tuned on the target dataset of images associated with a task of identifying musculoskeletal abnormalities.

[0010] In an aspect, the invention provides a method of identifying and / or classifying shoulder abnormalities comprising: receiving medical imaging images of a shoulder; and applying the medical imaging images to a pre-trained shoulder abnormality machine learning classifier to thereby produce identifications and / or classifications of shoulder abnormalities in the images, wherein the pre-trained shoulder abnormality machine learning classifier is: trained on a fused feature set of features extracted from two or more deep learning models that are pre-trained on an initial image dataset; updated on a secondary image dataset of medical imaging images of a same image type as images of a target dataset; andfine-tuned on the target dataset of images associated with a task of identifying shoulder abnormalities.

[0011] In another aspect, the invention provides a method of training an image classification model to identify and / or classify musculoskeletal abnormalities, the method comprising: providing two or more deep learning models pre-trained on an initial image dataset; updating the pre-trained deep learning models on a secondary image dataset of medical imaging images of a same image type as images of a target dataset; fine-tuning the two or more pre-trained deep learning models on the target dataset of medical imaging images associated with a task of identifying a musculoskeletal abnormality; extracting one or more features from each of the two or more deep learning models; combining the extracted features from each of the two or more deep learning models to generate a fused feature set for the task; and training one or more machine learning classifiers on the fused feature set of the extracted features.

[0012] In another aspect, the invention provides a method of training an image classification model to identify and / or classify shoulder abnormalities, the method comprising: providing two or more deep learning models pre-trained on an initial image dataset; updating the pre-trained deep learning models on a secondary image dataset of medical imaging images of a same image type as images of a target dataset;fine-tuning the two or more pre-trained deep learning models on the target dataset of medical imaging images associated with a task of identifying a shoulder abnormality; extracting one or more features from each of the two or more deep learning models; combining the extracted features from each of the two or more deep learning models to generate a fused feature set for the task; and training one or more machine learning classifiers on the fused feature set of the extracted features.

[0013] Preferably, the method includes pre-training the two or more deep learning models on an initial image dataset.

[0014] Preferably, an image type of the medical imaging images of the secondary image dataset is the same as an image type of the medical imaging images of the target dataset. The image type may be one or more of: Computed Tomography, X- rays, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET) and single-photon emission computed tomography (SPECT). For example, the image type of the medical imaging images of the secondary image dataset and the medical imaging images of the target dataset is x-ray images.

[0015] Preferably, the method includes processing the medical imaging images to generate one or more views. In some embodiments, the method includes processing the medical imaging images of the shoulder to generate one or more views of the shoulder. Preferably, the method includes automatic processing of the medical imaging images to generate the one or more views.

[0016] Preferably, the shoulder abnormality includes a supraspinatus tear and / or fatty infiltration of a rotator cuff.

[0017] Preferably, fine-tuning the two or more pre-trained deep learning models includes freezing initial layers of the pre-trained models and any remaining layers are trained on the images associated with the task to adapt the two or more pre-trained deep learning models to the task. Preferably, freezing the initial layers includes maintaining weights and parameters of the two or more pre-trained deep learning models.

[0018] Preferably, the one or more machine learning classifiers include K-Nearest Neighbour (KNN), Support Vector Machines (SVM) and Naive Bayes.

[0019] Preferably, the two or more pre-trained deep learning models include two or more convolutional neural networks (CNNs). More preferably, the two or more pretrained deep learning models include at least two of the Xception model, the lnceptionV3 model and the MobileNetV2 model. Preferably, the two or more pretrained deep learning models are pre-trained on the ImageNet dataset.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Preferred features, embodiments and variations of the invention may be discerned from the following Detailed Description which provides sufficient information for those skilled in the art to perform the invention. The Detailed Description is not to be regarded as limiting the scope of the preceding Summary of the Invention in any way. The Detailed Description makes reference to a number of drawings as follows:Figure 1 is a flowchart for a method of training a classifier to identify and classify musculoskeletal abnormalities from medical imaging images;Figure 2 is a flowchart for a method of identifying and classifying musculoskeletal abnormalities from medical imaging images using a trained classifier;Figure 3 is a diagrammatic workflow of the method shown in Figure 1;Figure 4 illustrates a confusion matrix of individual models used in the method;Figure 5 is a comparison of expert analysis and deep learning model decision making;Figure 6 illustrates two samples with three different deep learning models with visualization on where to look to make a decision;Figure 7 illustrates a Confusion Matrix of Feature Fusion and Machine learning classifiers;Figure 8 shows a Scapular Y view on an oblique sagittal MRI where the scapula and the spine form a Y-shape;Figure 9 shows steps in cross-localizing the scapular notch and 3cm cuts. A. T1 Sagittal and coronal sequences were loaded to an image processing software. B. The scapular notch was determined on the coronal view then 3cm was measured medial to it. C, D, E. Indicators were placed on these 2 points and then crosslocalized with the sagittal view.DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0021] The Inventors have conceived of a method training an image classifier to provide automatic identification of musculoskeletal abnormalities from medical imaging images of a musculoskeletal region of interest using an image classifier, andin particular, identification of shoulder abnormalities in the form of supraspinatus tears and / or fatty infiltration of a rotator cuff.

[0022] The method involves training a model, such as convolutional neural network (CNN), by exposure to in-domain datasets to recognise medical abnormalities in the musculoskeletal system from medical imaging images of a region of interest of the musculoskeletal system.

[0023] The method utilises a combination of in-domain transfer learning, feature fusion, and machine learning classifiers to provide an improved image classifier.

[0024] A preferred method of training an image classifier to identify musculoskeletal abnormalities is shown in Figures 1 to 3. The following examples and embodiments relate to the training of an image classifier with respect to identifying and / or classifying a supraspinatus tear and fatty infiltration of rotator cuff muscles. However, the method described herein can be generally applied to training an image classifier for identifying and / classifying musculoskeletal abnormalities within the human body (or body of a living being) using medical imaging images obtained from any number of medical imaging devices configured to perform medical imaging, such as Computed Tomography, X-rays, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET) and single-photon emission computed tomography (SPECT), for example.

[0025] The method includes providing two or more deep learning models pre-trained on an initial image dataset at box 110. The Inventors have found that a minimum of two deep learning models are required. However, a greater number of deep learning models can be utilised.

[0026] The illustrated embodiment includes three deep learning models pre-trained on an initial image dataset. The three pre-trained deep learning models used in this embodiment are: Xception, Inception V3 and ModelNetV2.

[0027] In some embodiments, the image dataset used for pre-training may be the ImageNet dataset that is commonly used for pre-training deep learning models. The ImageNet dataset includes a diverse collection of natural colour images belonging to 1000 classes.

[0028] However, the Inventors observed that due to the inherent differences between the ImageNet and the target dataset used in this embodiment (i.e., magnetic resonance imaging images of supraspinatus tears), direct transfer learning from ImageNet was required to effectively capture relevant features for the specific task of interest, namely detecting supraspinatus tears on MRI scans.

[0029] The method includes a further step of updating the pre-trained deep learning models on a secondary image dataset of medical imaging images of a same image type as images of a target dataset at box 120.

[0030] This step includes updating the learning features of the pre-trained models by fine-tuning them on images similar to the target dataset and associated with the task. For example, the Inventors found it is possible to align learned features of the pretrained deep learning models with the characteristics specific to the task by further training (i.e., fine-tuning) the deep learning models on MRI scans or images related to the supraspinatus region.

[0031] In embodiments of the invention, an image type of the medical imaging images of the secondary image dataset is the same as an image type of the medical imaging images of the target dataset. The image type may be one or more of: Computed Tomography, X-rays, magnetic resonance imaging (MRI), ultrasound,positron emission tomography (PET) and single-photon emission computed tomography (SPECT). For example, the image type of the medical imaging images of the secondary image dataset and the medical imaging images of the target dataset is x-ray images.

[0032] Next, the method includes fine-tuning the two or more pre-trained deep learning models on the target dataset of medical imaging images associated with a task of identifying a musculoskeletal abnormality at box 130. This can be broadly understood as the in-domain transfer learning step.

[0033] During the in-domain transfer learning process, the initial layers of the pretrained models further trained on the secondary image dataset are frozen. Freezing these layers means their weights and parameters remain unchanged, allowing them to retain the general knowledge gained from the pre-training. In one embodiment, this allows the remaining layers of the deep learning models to be further trained using the medical imaging images associated with the task by focusing on adapting the models to the specific task of supraspinatus tear detection. By adopting this approach, the Inventors addressed two significant challenges. Firstly, the scarcity of training data specific to supraspinatus tears was mitigated by leveraging the information captured by the pre-trained models on the ImageNet dataset. This helped to enhance the performance of the models in a data-limited scenario. Secondly, in-domain transfer learning facilitated better generalization and reduced the risk of overfitting. By fine-tuning the models on the medical imaging images associated with the task, the deep learning models were able to learn task-specific features while retaining the beneficial representations acquired during their pretraining on ImageNet. This type of transfer learning also helps with the next feature extraction step.

[0034] The method then includes extracting one or more features from each of the deep learning models at box 140.

[0035] The method utilises the outputs from three different models to extract multiple features from MRI scans of the supraspinatus region. Subsequently, the extracted features from each of the deep learning models are combined at box 150. By combining these various features, the method generates a comprehensive fused feature set of the extracted features encompassing a wide range of relevant characteristics.

[0036] The Inventors observed that each of the deep learning models provided unique insights and extracted distinct features from the MRI scans. Combining these individual feature sets allows for the complementary information in each model's outputs to be utilised. This fusion process allows for a holistic and nuanced understanding of the supraspinatus region, thereby enhancing the accuracy and robustness of the tear detection system. By integrating multiple features, the method leverages the strengths of each deep learning model and overcomes potential limitations associated with relying on a single feature set. The fusion of these features provides of the capture of a broader range of information, including subtle and prominent characteristics of supraspinatus tears. This comprehensive representation provides for more accurate and reliable predictions, facilitating the early and precise identification of tears on MRI scans.

[0037] In the next step, the method includes training one or more machine learning classifiers on the fused feature set of the extracted features at box 160. In some embodiments, the machine learning classifiers include K-Nearest Neighbour (KNN), Support Vector Machines (SVM) and Naive Bayes. However, other classifiers may be used.

[0038] These one or more machine learning classifiers are trained on the fused features extracted from the medical imaging images (e.g., the supraspinatus MRI scans) input to the deep learning models.

[0039] These machine-learning classifiers are useful for developing are automated identification of supraspinatus tears by leveraging the patterns and relationships in the data. Through extensive training on the fused features, these classifiers learn to discern the subtle differences and indicative patterns associated with each class. The classifiers develop the ability to discriminate between healthy supraspinatus regions and regions exhibiting signs of tears or, in other embodiments, healthy tissue, bones and / or joints and unhealthy (i.e., abnormal) tissue, bones and / or joints.

[0040] By analysing the patterns and relationships within the fused features, the classifiers acquire proficiency in accurately identifying potential tears on MRI scans. The machine classifiers can generalize from the learned features to previously unseen data, allowing them to make reliable predictions regarding the presence or absence of tears (or other abnormalities) in new scans. The combination of the fused features and the machine learning classifiers contributes to the overall effectiveness of the method described. The classifiers learn to make informed decisions based on the complex relationships between the features and the target classes. This enables the method to achieve high accuracy in detecting supraspinatus tears, ultimately aiding healthcare professionals in making timely and accurate diagnoses.

[0041] In use, there is a method for automatic identification of a musculoskeletal abnormality from medical imaging images. This method may be part of method 10 described above, or form its own separate method 20 shown in Figure 2.

[0042] The method includes receiving medical imaging images of a region of interest in the body shown at box 170. These medical imaging images may be of a shoulder or another region of interest in the body.

[0043] In some embodiments, the method includes processing the medical imaging images to generate one or more views at box 175. For example, the method may include processing the MRI images of the shoulder to generate one or more views of the shoulder.

[0044] In some embodiments, the processing to generate one or more views includes automatic processing of the medical imaging images to generate the one or more views.

[0045] In some embodiments, the medical imaging images may be 2D images. In some other embodiments, the medical imaging images may be 3D images, wherein the 3D medical imaging images are processed to generate one or more views based on a 3D volume of the musculoskeletal region of interest.

[0046] The method then includes applying the medical imaging images to a pretrained musculoskeletal abnormality machine learning classifier to thereby produce identifications and / or classifications of musculoskeletal abnormalities in the images at box 180.

[0047] As describe above, the pre-trained musculoskeletal abnormality machine learning classifier is trained on a fused feature set of features extracted from two or more deep learning models that are pre-trained on an initial image dataset and finetuned on images associated with a task of identifying musculoskeletal abnormalities.

[0048] The identification of musculoskeletal abnormalities may include generating an output, or output data, indicating the presence of absence of a musculoskeletal abnormality, such as fatty infiltration of the rotator cuff or a tear in the supraspinatus.

[0049] The classification of musculoskeletal abnormalities may include generating an output, or output data, that grades or categorises the musculoskeletal abnormality according to a classification system applied by the musculoskeletal abnormality machine learning classifier.

[0050] In a particular embodiment, a method of training the image classifier includes the following steps:1. Preprocessing: Obtain and / or otherwise provide scans or images of the musculoskeletal region of interest in the body as input data.2. Deep Learning Models: Utilize pre-trained deep learning models, such as Xception, lnceptionV3, and MobileNetV2, which have been trained on large- scale image datasets, to capture high-level features.3. End-to-End Training: Further training the pre-trained deep learning models on a secondary image dataset having medical imaging images or scans in a domain related to the task of identifying musculoskeletal abnormalities. This includes updating the weights of the layers of the deep learning models with the same features of the target dataset.4. Transfer Learning: Fine-tune the pre-trained deep learning models on a target dataset of medical imaging images (e.g., MRI scans or images) specific to the task and region of interest (e.g., supraspinatus region). This process adapts the models to recognize and extract relevant features associated with abnormalities.5. Feature Extraction: Extract multiple features from the outputs of each deep learning model. These features represent various characteristics of the region of interest.6. Feature Fusion: Combine the extracted features from multiple deep learning models to create a comprehensive representation of the region of interest. This fusion process combines the unique and complementary information captured by each model, enhancing the overall accuracy and robustness of abnormality detection.7. Machine Learning Classifiers: Train machine learning classifiers, such as K- Nearest Neighbour (KNN), Support Vector Machines (SVM), or Naive Bayes, on the fused features. These classifiers analyse the patterns and relationships within the fused features to differentiate between healthy regions and regions exhibiting signs of abnormalities.8. Prediction: Apply the trained classifiers to new medical imaging images to make predictions about the presence or absence of musculoskeletal abnormalities in the region of interest shown in the medical imaging images and / or classify the extent, type, category and / or grade of the abnormality.

[0051] The above-described methods provide a number of improvements and advantages over existing methodologies. Firstly, embodiments of the methods provide flexibility in incorporating additional deep learning (DL) models into the system. This flexibility allows for the expansion and enhancement of the model ensemble, enabling the integration of new DL models as they become available or as the dataset grows. By adding more DL models, the method can further improve the overall performance and accuracy of the identification.

[0052] Secondly, embodiments of the methods provide enhanced feature representation by leveraging the fusion of features extracted from multiple DL models. This feature fusion process combines the unique and complementaryinformation captured by each DL model, resulting in a more comprehensive and discriminative representation of the image.

[0053] Another advantage of embodiments of the described methods is that the need to train the models from scratch when incorporating additional data is eliminated. Training DL models from scratch can be computationally expensive and timeconsuming. However, the described methods can efficiently incorporate new data by leveraging the pre-trained DL models and fine-tuning them on the new dataset. This significantly reduces the required training time and computational resources, enabling a more efficient and scalable system.Experimental Results

[0054] The evaluation of the invention involves the utilization of various metrics to assess its performance. These metrics include accuracy, specificity, recall, precision, and F1-score. These evaluation metrics were computed based on the TN, TP, FN, and FP values. The TN and TP implied precisely categorized negative and positive instances, whereas FN and FP denoted misclassified positive and negative cases, respectively. Every evaluation metric equation is presented as follows:

[0055] By employing these measures, the effectiveness and capabilities of embodiments of the invention in detecting supraspinatus tears on MRI scans can be evaluated. To conduct a comprehensive evaluation, the performance of the individual models used in the described methodologies are assessed first. This evaluation allows the strengths and weaknesses of each model in isolation to be identified. By analysing their performance, insights into their abilities to extract relevant features and make accurate predictions can be gained. After evaluating the individual models, the impact of feature fusion can be evaluated. This step involves assessing the performance of the fused features with ML classifiers. These classifiers are trained on the fused features and are responsible for making the final predictions. By assessing their performance, the accuracy and reliability of the tear detection system as a whole can then be gauged.Individual Models Evaluation

[0056] Evaluation of the individual models is conducted. The confusion matrix for each model is initially calculated, as illustrated in Figure 4. The confusion matrix provides a detailed breakdown of correct and incorrect predictions for each class within every model. This matrix allows for examination of the performance of the models across different classes and provides valuable insights into the accuracy and reliability of their predictions. By analysing the confusion matrix, it can be understood how well each model performs in classifying the test samples for each class. Among the three models (MobileNetV2, Xception, and Inception), MobileNetV2 exhibited the lowest missed test samples. The other two models, Xception and Inception, had similar missed test samples. This observation highlights the relatively higher accuracy and performance of MobileNetV2 in correctly classifying a more significantproportion of the test samples. The fewer missed samples suggest that MobileNetV2 demonstrates better predictive capabilities and a more vital ability to accurately classify instances from the test dataset.

[0057] Based on the confusion matrix values, several performance metrics summarized in Table 1 have been computed. MobileNetV2 demonstrated the highest overall performance among the DL models, achieving an accuracy of 89.5%, a specificity of 94.7%, a recall of 89.5%, a precision of 90.5%, and an F1 -score of 90.0%. These metrics indicate MobileNetV2's superior ability to classify the test samples accurately.

[0058] On the other hand, Xception and lnceptionV3 obtained comparable results. Xception achieved an accuracy of 86.9%, a specificity of 93.4%, a recall of 86.8%, a precision of 89.0%, and an F1-score of 87.9%. Similarly, lnceptionV3 achieved an accuracy of 86.6%, a specificity of 93.2%, a recall of 86.5%, a precision of 88.2%, and an F1 -score of 87.4%.

[0059] These performance measurements provide insights into the models' abilities to correctly classify the test samples, with MobileNetV2 showcasing the highest level of accuracy and precision. The specificity and recall values also highlight the models' ability to identify true negatives and true positives, respectively, correctly. Overall, MobileNetV2 outperformed Xception and lnceptionV3 regarding the accuracy and other key metrics, indicating its superior performance in the automated detection of supraspinatus tears on MRI.Table 1 : Evaluation results of Individual Models.

[0060] To better understand the decision-making process of the DL models, expert assistance in analysing the regions of interest that influenced the models' decisions was obtained. This involved requesting the expert to identify and encircle the relevant regions within the image where the decision-making process was focused. Figure 5 illustrates these circled regions, highlighted in red. Furthermore, how different DL models made decisions for the same set of images was visualized and compared. To achieve this, one sample to be visualised was selected from each DL model and examined the decision-making process. By comparing these visualizations, it was observed that the DL models employed similar approaches in analysing the images and reaching their respective decisions. This similarity in the decision-making process across different DL models provides further evidence of the robustness and effectiveness of the proposed solution.Feature fusion and Machine learning classifiers

[0061] Each model exhibited distinct patterns of missed cases from the test set, indicating that they had different strengths and weaknesses in their classification capabilities. By combining the predictions of the three models, the reliability and confidence in the final decision was increased. To achieve this, the features extracted by the three models were fused and utilized for training several machine learning (ML) classifiers. Ultimately, for accurate decision-making, the model should consider the entire muscle region in the image. By combining the features extracted from multiple models, embodiments of the invention enable a more comprehensiveanalysis of the image, incorporating different perspectives and capturing a broader range of relevant information. This comprehensive view enhances the model's ability to make informed decisions based on the overall muscle characteristics, as depicted in Figure 6.

[0062] From the ensemble of classifiers, the top three classifiers were selected. These classifiers were chosen based on their performance, accuracy, and ability to distinguish between the classes of interest effectively. The confusion matrix depicted in Figure 7 provides insights into the performance of different classifiers. Based on the matrix, it is evident that the KNN classifier has missed fewer cases compared to the other classifiers.

[0063] On the other hand, Naive Bayes and SVM have similar numbers of missed cases. The lower number of missed cases for KNN suggests that it demonstrates better accuracy and precision in correctly classifying instances from the test dataset. In contrast, Naive Bayes and SVM exhibit relatively comparable performance regarding missed cases.

[0064] The results of the feature fusion approach combined with the top machine learning classifiers are presented in Table 2. Among the classifiers, KNN achieved the highest performance, with an accuracy of 91.1%, a specificity of 95.5%, a recall of 91.1%, a precision of 93.1%, and an F1-score of 92.1%. On the other hand, Naive Bayes and SVM demonstrated almost similar results, both achieving an accuracy of 90.8%, a specificity of 95.4%, a recall of 90.8%, a precision of 92.3%, and an F1- score of 91.5% and an accuracy of 90.5%, a specificity of 95.2%, a recall of 90.4%, a precision of 92.1%, and an F1-score of 91.2%, respectively. These metrics provide a comprehensive assessment of the performance of each classifier in terms of accuracy, specificity, recall, precision, and F1 score. The higher values obtained byKNN indicate its effectiveness in correctly classifying instances, while Naive Bayes and SVM showcase comparable performance, highlighting their ability to make accurate predictions. These results demonstrate the potential of the feature fusion in combination with different machine learning classifiers to achieve reliable and robust classification outcomes.Table 2: Evaluation results of Feature fusion and Machine learning classifiers.

[0065] In another embodiment, the above methods (method 10 and method 20) can be applied to the identification of fatty infiltration of the rotator cuff. In such an embodiment, the following methodology is used.

[0066] The MRI images used to further train (i.e., fine-tune) the machine learning models were taken from T1 oblique sagittal view of 1149 de-identified MRI images. The MRI scans were performed according to a standardized protocol which covers the entire scapula to be able to fully segment the entire rotator cuff muscles. All the MRI images were 1.5T with 2mm slice thickness. Mimics 24.0 software (Materialise, Leuven, Belgium) was used to process the Digital Imaging and Communications in Medicine (DICOM) files. T1 sagittal and coronal sequences were loaded into the software. Three specific cuts were selected: the Y view, the scapular notch, and 3cm medial from the notch; Medial scapular body view (MSB). The scapular Y-view refers to the Y shape formed by the scapula and the spine on an oblique sagittal MRI of theshoulder as shown in Figure 8. The notch and the 3cm mark were cross-localized using the software by initially loading the coronal view and determining the location of the scapular notch. This was marked and 3cm was measured medially where another mark was placed. These indicators were then used to determine the specific cuts on the sagittal view, shown in Figure 9.

[0067] Upon determining the three main cuts, images of each cut were saved and used to identify each of the rotator cuff muscles: supraspinatus, infraspinatus, subscapularis, and teres minor. Each muscle was grossly segmented. The main objective is to delineate the area of each rotator cuff muscle and attribute a class for that specific muscle to be used in the Deep learning model.

[0068] Initially, each of the muscle were classified according to the Goutallier Classification by a senior shoulder orthopaedic surgeon (AG). Then, a simplified classification was proposed based on the initial Goutallier classification.

[0069] The original Goutallier Classification categorizes the degree of fatty infiltration into five stages (from 0 to 4) based on visual assessment of the images: Stage 0 corresponds to a completely normal muscle, Stage 1 contains some fatty streaks in the muscle, Stage 2 still has more muscle than fat, Stage 3 has equal amount of muscle and fat, and Stage 4 where more fat than muscle is present.

[0070] From the de-identified MRIs, 1149 images were processed and classified. 569 of these images were classified as Stage 0, 329 were Stage 1, 100 were Stage 2, 105 were Stage 3, and 46 were Stage 4, as shown in Table 3.Table 3. Distribution of images processed and classified according to the GoutallierClassification.

[0071] The 5 stages were grouped into normal, mild, and severe fatty degeneration with all Stage 0 falling under normal, Stages 1 and 2 under mild, and Stages 3 and 4 under severe. Other cuts that showed Stage 3 and 4 degenerations from the same MRI sequences not included in the first 3 cuts were also processed to alleviate data imbalance between classes. In total, 1150 images were processed and classified in Table 4.Table 4. Distribution of images classified as normal, mild, and severe fatty degeneration.

[0072] The original de-identified MRIs, the deineated rotator cuff muscles and its correspondand fatty infiltration classes are used for training and testing the methods described herein as described in the next section.RESULTS

[0073] The evaluation of embodiments of the invention (specifically, quantifying fatty infiltration of the rotator cuff) involves the utilization of various metrics to assess itsperformance. These metrics include accuracy, specificity, recall, precision, and F1- score.

[0074] By employing the described measures, the effectiveness and capabilities of the herein described embodiments in quantifying the fatty infiltration of different RC muscles from delineated MRI images can be evaluated. To conduct a comprehensive evaluation, the performance of the individual models used in the described embodiment is initially assessed. This evaluation allows for the understanding and identification of the strengths and weaknesses of each model in isolation. By analysing their performance, insights into their abilities to extract relevant features and make accurate predictions can be gained.

[0075] Among the three models (MobileNetV2, Xception, and Inception), MobileNetV2 exhibited the lowest missed test samples.

[0076] To better understand the decision-making process of the DL models, similar to the supraspinatus tear embodiment described above, expert assistance in analysing the regions of interest that influenced the models' decisions was sought. This involved requesting the expert to identify and encircle the relevant regions within the image where the decision-making process was focused. Figure 5 illustrates these circled regions, highlighted in red. Furthermore, the way different DL models made decisions for the same set of images was visualized and compared. One sample from each DL model was selected to achieve this and the decisionmaking process was examined. By comparing these visualizations, it was observed that the DL models employed similar approaches in analysing the images and reaching their respective decisions. This similarity in the decision-making process across different DL models provides further evidence of the robustness and effectiveness of the proposed solution.

[0077] Each model exhibited distinct patterns of missed cases from the test set, indicating that they had different strengths and weaknesses in their classification capabilities. By combining the predictions of the three models, the reliability and confidence in the final decision was increased. To achieve this, the features extracted by the three models were fused and utilized them for training several machine learning (ML) classifiers. Ultimately, for accurate decision-making, the model should consider the entire muscle region in the image. By combining the features extracted from multiple models, embodiments of the invention enable a more comprehensive analysis of the image, incorporating different perspectives and capturing a broader range of relevant information. This comprehensive view enhances the model's ability to make informed decisions based on the overall muscle characteristics, as depicted in Figure 5.

[0078] From the ensemble of classifiers, the top three classifiers to report in this study. These classifiers were chosen based on their performance, accuracy, and ability to distinguish between the classes of interest effectively.

[0079] The results of the feature fusion combined with the top machine learning classifiers are presented in Table 5. Among the classifiers, KNN achieved the highest performance, with an accuracy of 91.1%, a specificity of 95.5%, a recall of 91.1%, a precision of 93.1%, and an F1 -score of 92.1%. On the other hand, Naive Bayes and SVM demonstrated almost similar results, both achieving an accuracy of 90.8%, a specificity of 95.4%, a recall of 90.8%, a precision of 92.3%, and an F1- score of 91.5% and an accuracy of 90.5%, a specificity of 95.2%, a recall of 90.4%, a precision of 92.1%, and an F1-score of 91.2%, respectively. These metrics provide a comprehensive assessment of the performance of each classifier in terms of accuracy, specificity, recall, precision, and F1 score. The higher values obtained byKNN indicate its effectiveness in correctly classifying instances, while Naive Bayes and SVM showcase comparable performance, highlighting their ability to make accurate predictions. These results demonstrate the potential of the feature fusion in combination with different machine learning classifiers to achieve reliable and robust classification outcomes.Table 5: Evaluation results of Feature fusion and Machine learning classifiers.

[0080] Embodiments of the invention provide a combination of in-domain transfer learning, feature fusion, and machine learning classifiers to improve musculoskeletal abnormality identification and classification. The experimental results provided herein demonstrate that combining these components provides reliable, robust, and enhanced classification outcomes.

[0081] Extent of fatty infiltration and atrophy are some of the main characteristics which prognosticate repairability of the cuff. What may be repairable largely depends upon experience, training, access to equipment and biology. In addition, classification of muscle quality and fatty architecture is prone to bias. Embodiments of the invention disclosed herein remove or diminish that bias and minimise differences in outcome influenced by experience, training and equipment differences. Advantageously, the methods described herein allow for clinicians to make better decisions to improve patient outcomes.T1

[0082] The results provided herein demonstrate that some embodiments of the invention have an accuracy of 91.1%, a specificity of 95.5%, a recall of 91.1%, a precision of 93.1%, and an F1 -score of 92.1%

[0083] Although the specific examples provided herein relate to the use of method for identification of fatty infiltration of the Rotator cuff muscles and supraspinatus tears, the methodology is readily applicable to different muscle assessment tasks such that embodiments of the invention can be used to identify and / or classify a broad range of musculoskeletal abnormalities.

[0084] Embodiments of the invention disclosed herein provide several advantages, including faster training of models compared to training from scratch, preventing overfitting, reducing data requirements, improving generalisation, providing effective feature extraction and facilitating domain adaptation. Further advantages include flexibility in incorporating additional deep learning models into the system which improves overall performance and accuracy of identification and classification, improved robustness and decreased bias caused by a single model.

[0085] In compliance with the statute, the invention has been described in language more or less specific to structural or methodical features. The term “comprises" and its variations, such as “comprising” and “comprised of’ is used throughout in an inclusive sense and not to the exclusion of any additional features.

[0086] It is to be understood that the invention is not limited to specific features shown or described since the means herein described comprises preferred forms of putting the invention into effect.

[0087] The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended claims appropriately interpreted by those skilled in the art.

Claims

CLAIMS1. A method of identifying and / or classifying musculoskeletal abnormalities comprising: receiving medical imaging images of a musculoskeletal region of interest; and applying the medical imaging images to a pre-trained musculoskeletal abnormality machine learning classifier to thereby produce identifications and / or classifications of musculoskeletal abnormalities in the images, wherein the pretrained musculoskeletal abnormality machine learning classifier is: trained on a fused feature set of features extracted from two or more deep learning models that are pre-trained on an initial image dataset; updated on a secondary image dataset of medical imaging images of a same image type as images of a target dataset; and fine-tuned on the target dataset of images associated with a task of identifying musculoskeletal abnormalities.

2. A method of identifying and / or classifying shoulder abnormalities comprising: receiving medical imaging images of a shoulder; and applying the medical imaging images to a pre-trained shoulder abnormality machine learning classifier to thereby produce identifications and / or classifications of shoulder abnormalities in the images, wherein the pre-trained shoulder abnormality machine learning classifier is: trained on a fused feature set of features extracted from two or more deep learning models that are pre-trained on an initial image dataset; updated on a secondary image dataset of medical imaging images of a same image type as images of a target dataset; andfine-tuned on the target dataset of images associated with a task of identifying shoulder abnormalities.

3. A method of training an image classification model to identify and / or classify musculoskeletal abnormalities, the method comprising: providing two or more deep learning models pre-trained on an initial image dataset; updating the pre-trained deep learning models on a secondary image dataset of medical imaging images of a same image type as images of a target dataset; fine-tuning the two or more pre-trained deep learning models on the target dataset of medical imaging images associated with a task of identifying a musculoskeletal abnormality; extracting one or more features from each of the two or more deep learning models; combining the extracted features from each of the two or more deep learning models to generate a fused feature set for the task; and training one or more machine learning classifiers on the fused feature set of the extracted features.

4. A method of training an image classification model to identify and / or classify shoulder abnormalities, the method comprising: providing two or more deep learning models pre-trained on an initial image dataset; updating the pre-trained deep learning models on a secondary image dataset of medical imaging images of a same image type as images of a target dataset;fine-tuning the two or more pre-trained deep learning models on the target dataset of medical imaging images associated with a task of identifying a shoulder abnormality; extracting one or more features from each of the two or more deep learning models; combining the extracted features from each of the two or more deep learning models to generate a fused feature set for the task; and training one or more machine learning classifiers on the fused feature set of the extracted features.

5. The method according to any one of claims 1 to 4, wherein the method includes pre-training the two or more deep learning models on an initial image dataset.

6. The method according to any one of claims 1 to 5, wherein an image type of the medical imaging images of the secondary image dataset is the same as an image type of the medical imaging images of the target dataset.

7. The method according to claim 6, wherein the image type includes one or more of: Computed Tomography, X-rays, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET) and single-photon emission computed tomography (SPECT).

8. The method according to claim 1 or claim 2, wherein the method includes processing the medical imaging images to generate one or more views.

9. The method according to claim 8, wherein the method includes automatic processing of the medical imaging images to generate the one or more views.

10. The method according to claim 2, wherein the method includes processing the medical imaging images of the shoulder to generate one or more views of the shoulder.

11. The method according to claim 2, wherein the shoulder abnormality includes a supraspinatus tear and / or fatty infiltration of a rotator cuff.

12. The method according to any one of claims 1 to 10, wherein fine-tuning the two or more pre-trained deep learning models includes freezing initial layers of the pre-trained models and any remaining layers are trained on the images associated with the task to adapt the two or more pre-trained deep learning models to the task.

13. The method according to claim 12, wherein freezing the initial layers includes maintaining weights and parameters of the two or more pre-trained deep learning models.

14. The method according to any one of claims 1 to 13, wherein the one or more machine learning classifiers include K-Nearest Neighbour (KNN), Support Vector Machines (SVM) and Naive Bayes.

15. The method according to any one of claims 1 to 14, wherein the two or more pre-trained deep learning models include two or more convolutional neural networks (CNNs).

16. The method according to any one of claims 1 to 15, wherein the two or more pre-trained deep learning models include at least two of the Xception model, the lnceptionV3 model and the MobileNetV2 model.

17. The method according to any one of claims 1 to 16, wherein the two or more pre-trained deep learning models are pre-trained on the ImageNet dataset.

Images