Computer-implemented method for image analysis of an x-raymedical image by neural network supported skeletal classification and risk assessment

Abstract

Herein is detailed a computer-implemented method for detecting an indication in an X-ray image that a fracture is present in a body part containing skeletal bones comprised in said X-ray image, wherein firstly a first deep learning model is applied for providing a skeletal body part classification, which skeletal body part classification is modified using a risk factor detection threshold if a skeletal body part is identified in the X-ray image, and, secondly and subsequently, based on the provided skeletal body part classification and risk assessment, reconfiguring a global ROC-threshold value of a further deep learning model trained to detect fractures in X-ray images in light of the outcome of the first analysis, and using the reconfigured further deep learning model to perform a fracture analysis on the skeletal body part as classified by the first deep learning model.

Description

[0001] TITLE OF INVENTION

[0002] Computer-implemented method for image analysis of an X-ray medical image by neural network supported skeletal classification and risk assessment.

[0003] TECHNICAL FIELD

[0004] In the field of computer-implemented methods for image analysis of an X-ray medical image by neural networks there is herein disclosed a method of neural network supported skeletal classification and risk assessment.

[0005] BACKGROUND

[0006] It is estimated that more than 600 million X-ray examinations are conducted yearly for evaluating various human body parts (c.f. Pham et al, https: / / arxiv.org / abs / 2108.06490, accessed 05-09-2024) such as the lungs, heart size, bowel, and bones, not including the parallel use in veterinarian sciences, making X-raying the most commonly performed imaging procedure in clinical practice. The results are ubiquitously used in diagnosis and visualization of diseases and other health conditions, requiring a tremendous amount of resource and human effort to overcome the task of analyzing and evaluating the produced imaging and providing results to the clinicians for further use.

[0007] With the advent of computerized and computer supported image analysis, the lack of trained personnel has been offset in part by the advances in automation. In recent decades, many automatic medical image analysis systems, particularly deep learning-based systems, have been studied and deployed to support radiologists in interpreting X-ray scans. Numerous

[0008] 10332WO00 clinical support software products have been introduced, and in recent years a significant number thereof have been Al software products for clinical radiology (Pham et. Al) . Whether Al-supported or general analysis, such systems are often developed for analyzing specific anatomies (e.g., lung, abdominal, spine, etc.) and often require the identification of the human body skeletal part contained in the X-ray image prior to analysis.

[0009] For X-ray imaging and analysis, Digital Imaging and Communications in Medicine (DICOM) format is the most used format and protocol for communication in clinical practice. One of the main properties of the DICOM is the ability to define different "tags" such as age, body part, view position etc. for each DICOM object, however the reliability of these tags varies a lot from one clinical practice to another. In a previous patent application (WO 2023079156, which is herein incorporated in its entirety by reference) by the present applicant, the problem of inaccuracies in the DICOM information was discussed and solutions for Al-supported correction thereof were therein presented.

[0010] With the advent of vast, non-normalized databases of X-ray images from hospitals the need for an automated approach to classify body parts from X-ray scans is raised. In particular, being able to categorize / classif y the different medical images (e.g. X-rays) into specific body parts, view projections, ages / age groups etc. enables several critical functionalities that are of high importance for Al (artificial intelligence) products in medical imaging such as :

[0011] — Being able to analyze images for specific findings based on their body parts as some findings might be irrelevant for specific body parts or specific views,

[0012] 10332WO00 — Being able to automatically route images based on their appearance / groups to specific Al products,

[0013] — Being able to automatically route images based on their risk level related to the body part, or

[0014] — Being able to set an anatomy / body part-specific threshold or operating point.

[0015] The present invention concerns a computer-implemented method for image analysis of an X-ray medical image by neural network supported skeletal classification and risk assessment .

[0016] In the art, classification tools are generally known, c.f. e.g. US 10,902,588 or US 11,282,196, however such systems are usually arranged for classifying the patient and system information contained in the received medical images. Subsequently, the medical image must be analyzed using dedicated software for a specific body part and imaging modality and diagnosis. This approach was also suggested in Pham et al. wherein they tested the suitability of seven different deep-learning architectures for performing the body part classification correctly but left further handling of the classified DICOM- images to subsequent specialized AI- tools for each skeletal body part for obtaining a dedicated diagnosis. Also in Applicant's own patent application, WO 2023079156, a body part classifier is detailed.

[0017] It is a problem of classification tools currently known in the art that these tools are not adaptive to the results of the classification process. Rather, they in general yield (within the safety margin of the tool) a classification in accordance with the purpose of the tool. However, for the subsequent analysis it is necessary to make use of further

[0018] 10332WO00 analysis tools , which in many cases both complicates the process as well as the portability of the results between platforms . At the same time , there is an increased risk of analysis delay and analysis errors , as a single result cannot be provided to the requestor as the outcome of the classi fication process .

[0019] The invention according to the present disclosure solves this and further problems in the art by providing an adaptive approach, wherein by using a first deep learning model to provide a skeletal body part classi fication and risk assessment , provide as an outcome , information to a clinician on the skeletal body part identi fied in the X-ray image , and an associated risk assessment for severity of the clinical find . In preferred embodiments , the approach secondly and subsequently, based on the provided skeletal body part classi fication and risk assessment , reconfigures a further deep learning model in light of the outcome of the first analysis , allowing the second deep learning model to perform a fracture analysis on the skeletal body part as classi fied by the first deep learning model with increased sensitivity .

[0020] Accordingly, there is herein detailed a computer-implemented method for detecting an indication in an X-ray image that a fracture is present in a body part containing skeletal bones displayed in said X-ray image , wherein firstly a first deep learning model is applied for providing a skeletal body part classi fication, which skeletal body part classi fication is modi fied using a risk factor detection threshold i f a skeletal body part is identi fied in the X-ray image , and, secondly and subsequently, based on the provided skeletal body part classi fication and risk assessment , reconfiguring a global ROC-threshold value of a further deep learning model trained to detect fractures in X-ray images in light of the

[0021] 10332WO00 outcome of the first analysis , and using the reconfigured further deep learning model to perform a fracture analysis on the skeletal body part as classi fied by the first deep learning model .

[0022] DEFINITIONS

[0023] Table 1 - Overview of Terminology

[0024] 10332WO00

[0025]

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 : Flowchart of Al-supported fracture analysis method

[0028] 10332WO00 Figure 2: Actual ROC-curves obtained by Al-supported fracture analysis.

[0029] Figure 3: Flowchart for body part classifier module.

[0030] Figure 4: Flowchart for body part classifier training module .

[0031] Figure 5: Flowchart for risk-based automation strategy.

[0032] Figure 6: Flowchart for risk decision module.

[0033] Figure 7: Flowchart for report decision module.

[0034] Figure 8: Examples of ROC-threshold adjustment.

[0035] Figure 9: Examples of lipohemarthrosis findings

[0036] It is to be understood that the embodiments shown in the figures are for illustration of the present invention and cannot be construed as limiting the present invention . Unless otherwise indicated, the drawings are intended to be read (e.g., cross-hatching, arrangement of parts, proportion, degree, etc.) together with the specification and are to be considered a portion of the entire written description of this disclosure.

[0037] DETAILED DESCRIPTION

[0038] In Al-supported image analysis, the credibility of the analysis depends on a preset desired outcome, balancing sensitivity and specificity of the analysis, typically based on a desired threshold for obtaining the desired outcome.

[0039] Usually, the threshold (and thereby the operating point) is set based on a receiver operating curve (ROC curve) to balance the true positive rate and false positive rate. An operating point can be set to yield a very high sensitivity (and consequently low specificity) , potentially having a significant number of false positives, but, if the system returns a value of "no findings" it will have a high

[0040] 10332WO00 probability of in fact being true (high Negative Predictive Value ) . The opposite outcome can also occur for very high specificity ( and consequently low sensitivity) . This outcome can potentially yield a signi ficant number of false negatives , but very few false positives ( and thereby a high precision) - when the system returns a value of " finding" it is very sure .

[0041] In medical imaging performance , metrics such as sensitivity and speci ficity are used . Often, a high sensitivity is preferred to ensure that the Al-product does not miss too many cases with a certain pathology of interest ( e . g . a fracture ) .

[0042] As mentioned, in one aspect the invention ( c . f . Figure 1 ) according to the present disclosure provides an adaptive approach, wherein using a first deep learning model to provide a skeletal body part classi fication and risk assessment , which additionally provides as an outcome information to a clinician on the skeletal body part identi fied in the X-ray image , and the associated risk assessment , it is possible in a second and subsequent deep learning model to reconfigure , based on the provided skeletal body part classi fication and ris k assessment , the further deep learning model in light of the outcome of the first analysis . The suggested approach was found to improve the performance of the subsequent fracture analysis to such an extent that a lipohemarthrosis in the knee could be detected, which hitherto has been undetectable by Al-supported fracture analysis .

[0043] In the present disclosure , there is a focus on the concurrence between the outcome of the first and second analysis , exempli fied by the body part being analyzed for,

[0044] 10332WO00 in the first analysis, is a skeletal body part, such that in the second analysis, the skeletal body part identified can be analyzed for the presence of a bone fracture. In a more generalized context, the first and second analysis models must have a concurrent body part analysis. E.g., the present analysis model is useful also where the first model analyses MR-scans for the presence of a certain soft body tissue or organ, and the second model analyses soft body tissue for the presence of e.g. tumors. However, these alternatives are not herein exemplified.

[0045] In Figure 2, there is shown an example of the outcome of actual ROC-curves for fracture detections in various skeletal body parts using a recently developed an Al-product by present Applicant (RBf racture™) . The Al-tool analyses for fractures in more than 10 different skeletal body parts (skeletal anatomies) , twelve as shown in Figure 2, fifteen in the model exemplified herein below.

[0046] The Al-product has one generic deep learning model and therefore only one threshold that defines the system's operating point on a global level. However, as the differences in the visual appearance of the body parts result in different operating points when stratified into different body part subgroups, being able to search for fractures in a specific body part in an equally weighed model (c.f. full black curve in Figure 2) leads to detection improvements for several otherwise hard to analyze body parts and anatomies. The final output from the analysis by the deep learning model is a bounding box enclosing a region of interest in the medical X-ray image, wherein the model is confident that a fracture can be found. For clinical use, the x-ray image is modified by the software to display the bounding box as an overlay over the bone structure, to allow subsequent

[0047] 10332WO00 verification by trained medical staff of the fracture detected by the model.

[0048] Surprisingly, the present inventors have discovered that improvement to the above fracture detection can be achieved if an adaptive approach to the ROC-curve and the global ROC- value is taken, such that if the global ROC-value is replaced by a body part specific ROC-value then the overall sensitivity of the fracture detection can be improved.

[0049] To this purpose, the present inventors herein propose a computer-implemented method for enhancing the sensitivity of a downstream convolutional neural network (300) for image analysis (310) for bone fracture, the downstream convolutional neural network (300) trained for detection of bone fracture in a skeletal body part containing skeletal bones displayed in an X-ray image (10) provided as input to the downstream convolutional neural network (300) , the sensitivity of the detection of bone fracture initially characterized by a global ROC-value; the method comprising: i. provide the X-ray image (10) as input (110) to an upstream convolutional neural network (100) for image classification, the upstream convolutional neural network (100) trained for skeletal body part classification; ii. perform (120) on the X-ray image (10) an image analysis using the upstream convolutional neural network (100) receiving as classifier output (130) from the analysis at least a classifier output probability vector (131— 135) at least comprising a probability of a skeletal body part (131) being displayed in said X-ray image (10) ; iii. assign as classification output (160) the skeletal body part (131) to the X-ray image (10) as a classified

[0050] 10332WO00 skeletal body part (161) if the probability of a skeletal body part (131) being displayed in the X-ray image (10) exceeds a detection threshold defined by a detection filter (140) for the presence of the skeletal body part (131) being displayed; iv. set (200) an anatomy-specific ROC-value (260) for the classified skeletal body part (161) ; v. provide the anatomy-specific ROC-value (260) and the assigned (160) classified skeletal body part (161) as invariant information to the downstream convolutional neural network (CNN) for image analysis (300) of the X- ray (10) image for bone fracture.

[0051] Throughout the present disclosure, and in preferred embodiments of the invention, X-ray images (10) are DICOM- medical images. This allows to analyze the images without having to scan the X-ray image and digitalize it thus speeding up processing time.

[0052] In an embodiment of the invention, the computer-implemented method further comprises: vi . perform using said downstream convolutional neural network (300) an image analysis (310) of said X-ray (10) for bone fracture.

[0053] And in an embodiment thereof, the computer-implemented method further comprises: vii. modify (320) said X-ray image (10) , if said downstream convolutional network (300) detects a fracture, to display an indicator, preferably a bounding box, for a region of interest enclosing said detected fracture.

[0054] In the embodiments exemplified herein, both the upstream (100) and the downstream convolutional neural networks (300)

[0055] 10332WO00 were built on a ResNet50-plat f orm and trained as detailed herein and elsewhere , notably WO 2022129628 and WO2023079156 which are herein incorporated in their entirety . The present method, however, is not limited to only neural networks based on a ResNet , particularly an ResNet50 , image analysis platform . Rather, the method is useful whenever a downstream neural network for obj ect analysis makes use of a single , global ROC-value established during training of the neural network, but optimi zation of the output is desired for enhanced detection of the obj ect by the downstream neural network as long as the two neural networks , the upstream and the downstream, share a concurrent obj ect to be analyzed for .

[0056] In aspects of the present invention, the computer-implemented method is executed in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to cause the at least one processor to implement the methods of the present invention . In other embodiments of the present invention there is herein detailed a non-transitory computer- readable storage medium comprising a computer program stored thereon for executing by a data processing system as disclosed above the method and embodiments thereof of the present invention .

[0057] Body Part Classi fier

[0058] In order to implement the present invention, the present inventors have developed a body part classi fier as herein detailed .

[0059] Body part classi fiers are well-known in the art , such as e . g . known from US 10 , 902 , 588 , US 11 , 282 , 196 , or Pham et al . Also

[0060] 10332WO00 in Applicant's own patent application, WO 2023079156, a body part classifier is detailed. In the context of the present application, a body part classifier is a shorthand notation for the more complete description, namely a computer implemented method for body part classification using a neural network supported (medical) image analysis. As a classifier is a well-established technical term in the art, in the present disclosure, the term is used in accordance with the art, such that a classifier performs a neural network supported classification of different elements, part, species etc. contained in the source data. In the present case, the classifier is a body part classifier and the output from the body part classifier is a discrimination between different body parts in the input source which herein are X-ray images, in particular DICOM images.

[0061] The body part classifier (100) of the present disclosure further permits classification of anatomical view or position, laterality, anatomy (bone structure) , and the presence or absence of body-extraneous parts (called hardware, can e.g. be fixtures, screws, hip replacements etc. with X-ray absorption profiles higher than soft tissue and bones) is present in the image.

[0062] The body part classifier (100) as discussed herein in its current stage of development is a deep learning-based classification model with multiple classification heads, as shown in Figure 3. The X-ray image (10) is passed to the body part classifier (100) , which then outputs predicted probabilities (130) for each of the features the model is trained to detect. Each feature is extracted by the body part classifier from the visual contents of the image. Currently implemented features are:

[0063] 10332WO00 • Body part: The body part can be directly observed by looking at the image (different body parts most often look very different in x-ray images) .

[0064] • View: The view can be directly observed by looking at the image (e.g. the patella, kneecap, is not very visible in PA views but clearly visible in LATERAL views) .

[0065] • Laterality: As X-ray images are taken in standardized ways, the laterality can most often be determined directly from looking at the picture (without using markers, DICOM tags and medical reports) . In the classifier this is performed by identifying specific anatomical landmarks and determining how they are located in relation to each other. For example, if the fibula bone is located to the right of the tibia bone and the view is "AP", it can be determined that the picture shows the left knee (see left image below) . Further, if the knee is instead pictured from a lateral view, the laterality can be determined based on which side of the picture the knee is pointing towards (see right image below) . Assumptions about laterality of the anatomies can be hospital-specific and can be dependent on the clinical workflow.

[0066] • Skeletal part: The bones and bone types present in the image can be easily observed from the image.

[0067] • Hardware: The presence of metal inserts, such as fixtures, screws, joint replacements, etc. is visible in the X-ray images due to the high X-ray absorption of the hardware material.

[0068] The Body part classifier (100) in its current stage of development currently classifies Body parts ( 15) :

[0069] 10332WO00 Ankle, Chest, Elbow, Femur, Foot, Forearm, Hand,

[0070] Hip / Pelvis, Humerus, Knee, Shoulder, Spine, Tibia / Fibula, Wrist, Skull.

[0071] Views and Laterality (5) :

[0072] PA / AP, Lateral, Oblique, Other, Axial.

[0073] Skeletal part:

[0074] Bone Type, Bone Number.

[0075] Hardware :

[0076] Present or Absent.

[0077] Model Training

[0078] Present applicant has found that providing a convolutional deep neural network, such as ResNet50, is sufficient as foundation for the body part classifier (100) of the invention .

[0079] In the process disclosed in WO 2023079156, the trained network receives a medical image as an input and performs a neural network supported image analysis process thereon, whereby detection and classification of a body part is possible. The training of the neural network of the body part classifier in WO 2023079156 is discussed therein in detail in relation to Figure 15 of WO 2023079156. Training of the present body part classifier (100) follows the same overall procedure as discussed in WO 2023079156 (herein incorporated in full) , in particular as disclosed on pages 35 - 37, adapted by the below further elements and as detailed in Figure 4.

[0080] The presently programmed body part classifier (100) is likewise based on a Resnet50 backbone, with multiple, fully connected, 3-layer heads. The maximum input image size found necessary is 256x256 pixels, and the input images are normally downscaled to this resolution prior to

[0081] 10332WO00 classi fication for improvements to computational ef ficiency .

[0082] The input X-ray image is set to a grey-scale image with three channels ( same grey-scale image in each channel ) of any resolution where each pixel contains values between 0 and 1 . Before training is run, the module resi zes the image to a standardi zed si ze of currently 224x224 pixels and the image is normali zed to an intensity grading interval between approximately -2 and 2 . After this input processing, the body part classi fier neural network is trained in accordance with the process detailed in WO2023079156 and as reproduced in Figure 4 , wherein the dropout layers are applied only during training .

[0083] In general , the ResNet50 model is a mid-si ze deep learning model speciali zed for image analysis wherein the first 50 layers are pretrained for image recognition on a very large publicly available dataset ( ImageNet ) . This means that even before the neural network is trained on X-ray images , it has a basic understanding of images .

[0084] Consequently, in the context of the present invention, the ResNet50-model is used herein as an implementation model and serves as the equivalent of a screwdriver in a repair shop . The skilled person knows how to arrange and train the model for obtaining an image analysis result , but the final and desired result depends on the data set used for training and the speci fics of the training procedure as disclosed e . g . herein .

[0085] During the presently applied training, the first 41 layers of the ResNet50 encoder were frozen . This means that these layers were not further tuned during training on the x-ray images . The remaining layers were tuned on the X-ray images from the training set .

[0086] 1Q332WOOO In order to make the model as generic as possible, so that it can handle new unseen data in a reliable way, data augmentation was used. The data augmentation alters the appearance of the training X-ray images by randomly rotating, blurring, solarizing and erasing parts of them. Another way of ensuring that the model generalizes well is to apply dropout. During training, 30% of the values passing through the "Dropout 33%" blocks in the neural network architecture figure above are randomly set to 0, as indicated in Figure 4.

[0087] The currently applied model was trained on a total of 71.253 radiographs, covering the supported body-parts, views and hardware labels. For evaluation of the model during training, a tuning dataset of 6.145 radiographs composed of the above supported labels was utilized. A verification dataset containing 28.701 radiographs was used to assess the generalization performance of the model. Except for the training sets for chest, spine, skull, and humerus, the tuning dataset for a given anatomy was 500 X-ray images, and the training set for all anatomies, except chest, spine, and skull always exceeded 2000 X-ray images.

[0088] The current low amount of data for chest, spine, skull, and humerus makes the predictions associated with these body parts less secure, and the below detailed risk assessment has been adjusted to compensate for this by assigning higher risk group classification to these particular body parts. Since, as discussed in relation to Table 2, chest, spine, and skull fractures already are high risk classified on the basis of severity and potential for severity of injury, the outcome of the current stage of the training vs. risk evaluation is only significant for humerus fractures, which

[0089] 10332WO00 in the current model are evaluated as riskier than what could be considered clinically relevant.

[0090] The output of the currently body part classifier (100) is a probability vector of a length equal to the number of different combinations of body part, view, laterality, anatomy, and hardware that is supported by the model. E.g., if in one embodiment the model is able to detect "KNEE AP LEFT", "KNEE AP RIGHT", "KNEE LATERAL LEFT" and "KNEE LATERAL RIGHT", the length of the probability vector would be 4. Each value in the vector will contain the probability that the image contains that class. For the example with vector length 4, the output could be [0.1, 0.8, 0.05, 0.05] . This would mean that the model is 80% certain that the image contains the class "KNEE AP RIGHT". The sum of the probability vector is always 1.

[0091] In a preferred development of the present body part classifier (100) over the prior art, improvements have been achieved by rather than (c.f. Figure 3) directly providing the classifier output (130) from the received medical image (110) to the body part classifier (120) as classifier output results (131-135) , the classifier output (131-135) results are subsequently filtered (150) using a detection filter (140) before providing classification output (160) in the form of a classification output vector (161-165) .

[0092] In the embodiment shown in Figure 3, the classification output vector contains information on all elements of the set of classification elements (161-165) the model currently can analyze for, i.e. body part (161) , view (162) , laterality (163) , anatomy (164) , and hardware (165) , however in some embodiments, the number of elements selected from the set of classification elements (161-165) can be smaller than the

[0093] 10332WO00 full set of classification elements, (typically involving omission of the analysis for hardware) . Or, as the model develops, the current set of classification elements may become expanded by other elements of classification, e.g. in-view bone type or such like.

[0094] In accordance with the present disclosure, the detection filter (140) can filter the classifier output (130) based on one or more preset detection thresholds or anatomy dependent thresholds suitable for providing clinical or system improvement after passage of the detection filter (140) . In the detection filter (140) , the probabilities from the body part classifier are thresholded to make the final predictions. In some embodiments, the detection threshold is a cut-off filter (141-145) , and in embodiments thereof, the detection threshold is set for each classifier output (131— 135) individually as respective classifier output thresholds (141-145) .

[0095] Accordingly, in embodiments of the present invention the detection threshold of the detection filter (140) combines a cut-off filter (141-145) for an apparent detection of a skeletal body part with a consequence of analysis filter (147) and / or a clinical information filter (146) for the apparently detected skeletal body part such that the detection threshold may be lowered or increased in response to the combination of said cut-off filter (141-145) with said consequence of analysis filter (147) and / or a clinical information filter (146) for said apparently detected skeletal body part

[0096] E.g. In the above example of a 4-element vector of probabilities, there is an 80% chance that the correct anatomy is a knee, with only a 20% chance that the correct

[0097] 10332WO00 anatomy is something else of the (currently) 15 anatomies the model analyzes for. Accordingly, the predicted outcome can be considered very certain. However, if the model delivers two approximately equal answers (e.g. left or right hip) , human intervention would be necessary.

[0098] In the above two cases, the detection filter (140) can be a cut-off filter, such that only certain (above the cut-off) predictions are transmitted further for Al-supported fracture analysis, whereas predictions below the cut-off are forwarded to a qualified human analyst, such as e.g. a radiologist (c.f. Figures 6 and 7) .

[0099] In all cases, however, the effect of providing the filter is that the model provides an answer, which can be either a decision that the body part identified by the classifier is the correct body part, and that subsequent image processing can rely on this information; a decision that the classification is not certain and therefore requires human intervention; or a decision that no body part has been identified.

[0100] Another example of a detection filter (140) could be a filter based on clinical information (146) , such as e.g. prevalence (146a) , potentially augmented by patient specific data obtained e.g. from DICOM metadata or a medical record. E.g., if the medical image obtained is of lower than desired quality, e.g. blurry from having been taken on an unruly child in pain, and the classifier output (130) returns a result of equal probability between a knee and an elbow, then metadata information such as intended anatomy imaged, age and clinical information can be used to augment the analysis, as elbow fractures are more prevalent in children than in adults .

[0101] 10332WO00 Another suitable filter is provided based on risk assessment of clinical outcome and missed diagnosis in a consequence of analysis filter ( 147 ) .

[0102] For example , for avoiding automated analysis of any skull images using an automated Al-tool ( as skull fractures have a very high risk of further complications , which can potentially have fatal consequences i f not discovered early) , the detection threshold for the detection of " skull" in the body part category could be set to 20% based on a risk scenario assessment . I f the body part classi fier predicts the likelihood of an anatomy image showing a skull is above 20% detection threshold, the body part is then predicted as " skull" after passage of the filter ( regardless of whether another body part has been predicted with a higher than 20% probability) and flagged for review by a professional medical practitioner .

[0103] In Table 2 the current 5 risk categories are shown that have been implemented into the detection filter ( 140 ) of the present invention . The list of categories or types of fractures is only one of many possible implementations that can be used, based on the intended risk profile considered relevant or acceptable by an end-user, such as a medical practitioner, clinic or hospital , or a quali fying authority, such as FDA or EMA.

[0104] Table 2 : Risk categories for different body parts

[0105] 10332WO00

[0106] A particularly suitable detection filter (140) of the present invention convolutes cut-off filtering with clinical information (146) and consequence of analysis filtering (147) . Thereby very low and / or low clinical-risk anatomies can be transferred to automated fracture analysis without human intervention at low risk, and (using a set point determined by an end-user) anatomies presenting as one of either medium, high, or very high clinical-risk anatomies can be separated off for human intervention and analysis, c.f. Figure 6 and Figure 7.

[0107] For implementation of the detection filter (140) of the present invention, the risk classification (147) is always provided to the model as an external input by trained medical supervision, such as trained medical doctors and / or radiologists. However, the model per se is not dependent on the source of the information, only on the contents of the information .

[0108] 10332WO00 It is a further goal of the present invention to increase the level of automation in the evaluation and analysis process of medical images. The path towards full autonomy in medical imaging is often categorized into different levels of autonomy where current Al products are working as decision-support systems, wherein the clinician has the final responsibility for the accuracy of the analysis.

[0109] In some embodiments, consequently the computer-implemented method is arranged such that if no probability of a skeletal body part (131) being displayed in the X-ray image (10) exceeds the detection threshold defined by the detection filter (140) , the X-ray image (10) is flagged for human intervention. Unfortunately, it is a recurrent error in radiology that X-ray images (10) for various reasons do not show any of the intended body parts, and identifying these in the production flow saves time by early detection. Also, the classifier (100) may in some cases be very certain that the skeletal body part is e.g. a skull, but as discussed below, in that situation the threshold would still not be exceeded due to the risk evaluation weighing negatively and the analyzed X-ray image flagged for human intervention.

[0110] An approach based on risk-based automation strategy selection can help increase the level of automation by using different risks levels for a particular automation outcome. The risk could be determined based on both visual features in medical image and / or non-visual variables based on the DICOM information / metadata such as body parts present in the image, age, different types or classes of fractures etc.

[0111] In Figure 5 is shown a flowchart for the overall process of the present invention, namely providing (10) an x-ray image

[0112] 10332WO00 as input to a trained neural network supported body part classifier (100) , obtain a body part classifier output (160) , providing the body part classifier output (160) comprising the identified body part (161) to a risk assessment module (200) for setting an associated anatomy-specific ROC- threshold (260) for the identified body part (161) , and subsequently, perform a neural network supported fracture analysis based on the identified body part (161) and the associated anatomy-specific ROC- threshold (260) . In the preferred embodiments of the present invention, the subsequently performed neural network supported fracture analysis is only performed if the risk assessment is considered below the set-point set by a qualified clinician and reflecting an acceptable risk level. Referring to Table 2, typically an automated analysis will only be performed if the overall risk assessment is considered not higher than middle, low or very low, but most commonly and more preferably, only if the risk assessment is considered low or very low. When the risk assessment is higher than middle, an automated analysis can still, in some embodiments, be performed, however the output will always only be provided as a decision-support output to a trained clinician. In lower risk cases, and where the law permits, the automated analysis may substitute for the evaluation and diagnosis by a trained clinician .

[0113] The flowchart shown in Figure 6 depicts a decision tree, as an example and in an embodiment of the method of the invention, wherein consequently is disclosed a risk decision module (200) for selecting an analysis profile based on a risk category, such as a risk category as shown in Table 2 where, in the risk decision module (200) , the output (160) of the analysis from the body part classifier (100) preferably is checked (210) against the risk category for

[0114] 10332WO00 the identified body part (161) and a subsequent analysis profile (220,230,240) is chosen, either low-risk (220) , medium-risk (230) , or high-risk (240) respectively, for the analyzed X-ray image @ (110) .

[0115] Accordingly, in embodiments of the computer-implemented method a risk decision module (200) for selecting an analysis profile based on a risk category for a classified skeletal body part (161) is applied where, in the risk decision module (200) , the classification output (160) of the analysis from the body part classifier (100) is checked (210) against a risk category, preferably a tabulated risk category, for the classified skeletal body part (161) and a subsequent analysis profile (220,230,240) is chosen, either low-risk (220) , medium-risk (230) , or high-risk (240) respectively, for the analyzed X-ray image (10) based on the risk category

[0116] In the exemplified risk module decision tree (220) , the low- risk (220) profile subsequently (221) always leads to an automated fracture analysis (300) , and the high-risk (240) profile for the identified body part always leads (241) only to a referral to manual fracture analysis (250) before exiting (299) the risk decision module (200) , whereas (depending on the risk-profile chosen by an end-user or mandated by regulatory authorities) medium-risk (230) body parts profile analysis may (232) or may not (231) be carried out automatically or remain (as is currently the case) semi- automatically analyzed, i.e. using the fracture analysis AI- model as a decision support tool. Currently, no fracture analysis software available in the market is classified for automated analysis and decision, and consequently, the situation depicted in Figure 6 shows how this can be changed by implementing the strategies of the present invention.

[0117] 10332WO00 A risk-based automation strategy selection approach therefore can enable current products to evolve from level 2 (clinical decision support) to level 3 (conditional automation) , where specific images / categories are fully automated (e.g. low risk studies) . In Figures 7 is shown an example of such an implementation of a possible conditional automation system made possible using the present invention.

[0118] The flowchart shown in Figure 7 depicts an exemplary decision tree, as an example and in an embodiment of the method of the invention showing, how a report decision module (400) can usefully be added to the method of the present invention, for deciding whether an outcome (320) from the fracture analysis module (300) can be automatically reported (460) or must be referred to trained medical personnel (450) , such as a radiologist (450) , for human assessment of the X-ray image.

[0119] In relation to the present invention, an outcome (320) from the fracture analysis module (300) in general will be one of two outcomes (320a, 320b) but will normally be both. A first outcome (320a) is a probability vector of length 1 containing at least one element defining a probability of a fracture and at least one element defining a probability of not a fracture. A second outcome (320a) is a modified input X-ray image (110) containing at least one bounding box enclosing a region of interest, wherein the probability of a fracture is higher than a threshold value of no fracture. In the present invention, for the below evaluation, the bounding box information is also supplied to the evaluation module together with the outcome of the anatomy classification such that the analysis for certain / uncertain @ (410) can take this information into account.

[0120] 10332WO00 As shown in Figure 7 for the decision tree of the exemplified report decision module (400) in embodiments of the present invention, the outcome (320) from the fracture analysis module (300) is evaluated (410) for compliance with a desired fracture analysis outcome certainty threshold value, thereby providing a determination of whether the performed fracture analysis (300) can be considered certain (411) or uncertain (412) respectively with respect to a desired risk assessment thresholds. The desired risk assessment threshold will typically be preset as a factory- value but can, in embodiments of the present invention, be a value, which is externally provided by trained medical personnel.

[0121] A preferred example of adjustment to the desired risk assessment thresholds can e.g. be based on the risk level assigned to the given body part analyzed by the fracture analysis module (300) .

[0122] However, it is a benefit of the present invention that under normal circumstances, the anatomy-specific ROC-threshold set (260) as part of the execution of the risk assessment module (200) provides sufficient certainty of analysis that subsequent risk assessment thresholds need not be applied.

[0123] In other embodiments of the present invention, as shown in the decision tree in Figure 7, for the exemplified report decision module (400) , a risk level discrimination (420) is performed on the outcome (320) of the fracture analysis module (300) , whereby, based on the risk level set for the examined body part, different criteria are set up for whether a certain positive shall be referred for verification by trained medical personnel (450) or can be automatically reported (460) . In the exemplified report module (400) of Figure 7, for middle risk body parts (421) , middle risk

[0124] 10332WO00 certain negatives (432) are automatically reported (460) , whereas middle risk certain positives (431) are referred for verification by trained medical personnel (450) . Contrary to this, low (422) or very low (423) risk body parts are always, in the embodiment shown, automatically reported (460) . The decision which risk profile to apply, however, is outside the scope of the present invention, pertaining to the tasks of trained medical personnel. The report module (400) merely executes the selected risk profile whether provided as a preset factory setting or as input by trained medical personnel .

[0125] In the exemplified embodiment of the report decision module of Figure 7, the determination of certain / uncertain of the compliance evaluation (410) is performed prior to the risk level discrimination (420) , which is a preferred embodiment since the compliance evaluation (410) depends on the outcome (320) of the facture analysis module (300) and the risk level discrimination (420) does not (having been set during the execution of the risk decision module (200) ) . Nevertheless, this order of execution is not mandatory, merely easier to implement .

[0126] In preferred embodiments of the report decision module (400) , an X-ray image containing a body part, for which an automated report (460) has been generated, may be randomly selected (470) and referred for verification by trained medical personnel (450) as part of a quality control process, prior to exiting the report decision module (400) .

[0127] Sensitivity and specificity are performance variables independent on prevalence. Other parameters such as the negative predictive value and positive predictive which are related to prevalence can be used to clinically say how

[0128] 10332WO00 likely it is a patient has a specific disease (e.g. a fracture) .

[0129] Setting of body part specific ROC-thresholds

[0130] Central to the present invention is the surprising observation by the present inventors that significant improvements to the outcome of the fracture analysis (320) can be achieved when, rather than applying a single, system- wide, ROC-threshold for the fracture analysis (300) , each fracture analysis is performed by ROC-thresholds which are specific to a given body part. This is surprising, since it would normally be expected that while body parts are anatomically different, the X-rays are not in terms of colordepth, resolution, etc., and consequently should not give rise to variations in the ROC-thresholds. As such, it will normally be expected that by using a single, system-wise "global" ROC-threshold, it is possible to control the performance metrics for all the individual body parts. However, as shown herein, a classification of the input images into the different anatomies / body parts enables setting multiple anatomy-specific thresholds as illustrated below with significantly higher performance than previously achieved. The underlying reason, in the present inventors' observations, being that when the fracture analysis model is sure that it has the right part for analysis, it eliminates part of the uncertainty of the fracture analysis, since body part specific outcomes (such as lipohemarthrosis ) are no longer considered a risk factor but becomes part of the certainty evaluation.

[0131] Different strategies could be applied to set body-part specific threshold. The aim of a fracture detection system is to reduce the number of missed fractures, which means,

[0132] 10332WO00 the focus is on having a product with a very high sensitivity (with an acceptable number of false positives) . Another aim is to have a system with a very high NPV which means when the system says there are no fractures in a study, the probability of no fracture is very high.

[0133] One strategy is to optimize the product to have the same minimum sensitivity for all body parts (e.g. minimum of 0.9 in sensitivity) - another is to optimize it to have a minimum NPV to ensure good user experience.

[0134] The different operating points are summarized in the table below and can be related to NPV and PPV. The performance calibration of the Al product can be based on either the prevalence independent variables such as sensitivity and specificity or the prevalence dependent variables such as NPV and PPV.

[0135] If the system is proven to have a minimum NPV of 0.9 - then the system is right 9 out of 10 times when it says a patient does not have a condition (e.g. a fracture) .

[0136] In Table 3 below there are shown measured correlations between various body parts and unqualified sensitivity and specificity when the re-defined ROC-values are applied to Applicants commercial product RBfracture. While in general the sensitivity is good, low prevalence body part fractures, such as knee fractures have lowered sensitivity. However, when the ROC-threshold is adjusted for the prevalence of knee fractures, the negative prediction value (NPV) becomes on par or even better than higher prevalence fractures.

[0137] A particularly successful finding was the ability to detect the presence of lipohemarthrosis in the X-ray images of,

[0138] 10332WO00 particularly, in knees ( c . f . Figure 9 ) . In Figure 9 two examples of lipohemarthrosis fractures are shown inside the bounding boxes for respectively A: toe- fracture and B : kneefracture . Particularly interesting is that the fracturemodel even detected the lipohemarthrosis fracture along with the main fracture of the shinbone . Also , the model after resetting the ROC-value in some cases flagged ef fusion lesions as well as hardware dislocations .

[0139] Table 3 : Correlation of body part to ROC-threshold adjustment bodypart sitivity ificity Prevale V ankle 0.962 0.751 0.3 0.979 0.623 elbow 0.807 0.848 0.4 0.868 0.780 femur 0.979 0.888 0.3 0.990 0.789 foot 0.879 0.732 0.5 0.858 0.766

[0140] [liiiilfiiiitM forearm 0.886 0.805 0.5 0.876 0.820 hand 0.908 0.72 0.55 0.865 0.799 - . . : hip_pelvis 0.983 0.854 0.35 0.989 0.784 humerus 0.968 0.903 0.4 0.977 0.869 knee 0.675 0.966 0.1 0.964 0.688 shoulder 0.885 0.779 0.3 0.940 0.632 tibia fibula 0.849 0.888 0.3 0.932 0.765 wrist 0.955 0.819 0.5 0.948 0.841

[0141] In an embodiment , the general ( global ) operating threshold of the tuning data is set using a weighted Youden index, wherein, in a preferred embodiment , the weighted Youden index is defined as :

[0142] 1“

[0143] (1) Y = sensitivity + (r * sensitivity) — 1, where r = (— -)

[0144] It is now possible to find an optimum for a given set of (a, p ) for a given body part , e . g . by variation of the free parameters until an optimum is reached . This approach is good when the parameter r is not "too small" as defined by a

[0145] 10332WO00 combination of alpha and prevalence. When the r is close to 0 (e.g. 0.1) , the Weighted Youden curve gets very "flat" and unstable which means an incorrect threshold might be selected. In such cases, a cut-off preventing r from becoming too close to zero can be applied.

[0146] Table 4 : Performance differences

[0147] In Table 4 is shown how the threshold adaptation away from a single global value (in the example an ROC-value of 0.79) enhances the performance for the different body parts examined. With the above adjustments to the global ROC-value, the fracture-wise sensitivity showed increases of 0-5 percent points, the study-wise sensitivity increases of 0-6 percent points, the study-wise specificity decreases of 0-11 percent points, and the NPV increases of 0-3 percent points with a low increase in average number of false positives per study of 0-14. The optimized values given in Table 4 were obtained experimentally by recursive testing.

[0148] In general, using a set of tabulated values, wherein a respective tabulated ROC-value is assigned to a corresponding respective skeletal body part is simple to implement in the methods of the present invention and is consequently preferred, particularly for those skeletal body parts that

[0149] 10332WO00 are not classified as being high and / or very high risk skeletal body parts.

[0150] Another, more extreme, version could, in one embodiment, be to specify a very specific sensitivity (e.g. 0.99) to ensure that the system has a very high specificity across all body parts. For this methodology, one could change the operating point along the ROC curve on the tuning data until the required sensitivity is obtained and pick the specific threshold that yields this sensitivity for that specific body part. And, once set as manual / external input value, the fracture analysis forthgoing would be using this value rather than the original single global value.

[0151] This approach is illustrated in Figure 8, where the plot displays how the performance characteristic can be changed knowing that specific body part (here - elbow) . Since prevalence can also be different for the different body parts, one can adjust the threshold for the body part with prevalence in mind, which will have an impact on the NPV, thereby optimizing the performance.

[0152] CLOSING COMMENTS

[0153] Although the present invention has been described in detail for purpose of illustration, it is understood that such detail is solely for that purpose, and variations can be made therein by those skilled in the art in practicing the claimed subject matter, from a study of the drawings, the disclosure, and the appended claims.

[0154] The term "comprising" as used in the claims does not exclude other elements or steps. The indefinite article "a" or "an" as used in the claims does not exclude a plurality. A single

[0155] 10332WO00 processor or other unit may ful fill the functions of several means recited in the claims . A reference sign used in a claim shall not be construed as limiting the scope .

[0156] 10332WO00

Claims

35CLAIMS1. A computer-implemented method for enhancing the sensitivity of a downstream convolutional neural network (300) for image analysis (310) for bone fracture, the downstream convolutional neural network (300) trained for detection of bone fracture in a skeletal body part containing skeletal bones displayed in an X-ray image (10) provided as input to said downstream convolutional neural network, the sensitivity of the detection of bone fracture initially characterized by a global ROC-value; the method comprising : i. provide said X-ray image (10) as input (110) to an upstream convolutional neural network (100) for image classification, the upstream convolutional neural network (100) trained for skeletal body part classification; ii. perform (120) on said X-ray image (10) an image analysis using said upstream convolutional neural network (100) receiving as classifier output (130) from said analysis at least a classifier output probability vector (131-135) at least comprising a probability of a skeletal body part (131) being displayed in said X-ray image (10) ; iii. assign as classification output (160) at least said skeletal body part (131) to said X-ray image (10) as a classified skeletal body part (161) if said probability of a skeletal body part (131) being displayed in said X-ray image (10) exceeds a detection threshold defined by a detection filter (140) for the presence of said skeletal body part (131) being displayed; iv. set (200) an anatomy-specific ROC-value (260) for said classified skeletal body part (161) ;10332WO0036 v. provide said anatomy-specific ROC-value (260) and said assigned (160) classified skeletal body part (161) as invariant information to said downstream convolutional neural network (CNN) for image analysis (300) of said X-ray (10) image for bone fracture.

2. A computer-implemented method according to claim 1 further comprising: vi . perform using said downstream convolutional neural network (300) an image analysis (310) of said X-ray (10) for bone fracture.

3. A computer-implemented method according to claim 2 further comprising: vii. modify (320) said X-ray image (10) , if said downstream convolutional network (300) detects a fracture, to display an indicator, preferably a bounding box, for a region of interest enclosing said detected fracture.

4. A computer-implemented method according to any of the previous claims, wherein said X-ray image (10) is a DICOM medical image.

5. A computer-implemented method according to any of the previous claims, wherein said detection threshold of said detection filter (140) combines a cut-off filter (141— 145) for an apparent detection of a skeletal body part with a consequence of analysis filter (147) and / or a clinical information filter (146) for the apparently detected skeletal body part such that said detection threshold may be lowered or increased in response to the combination with said consequence of analysis filter (147)10332WO00and / or a clinical information filter (146) for said apparently detected skeletal body part.

6. A computer-implemented method according to any previous claim, wherein if no probability of a skeletal body part (131) being displayed in said X-ray image (10) exceeds said detection threshold defined by said detection filter (140) said X-ray image (10) is flagged for human intervention .

7. A computer-implemented method according to any previous claim, wherein a risk decision module (200) for selecting an analysis profile based on a risk category for said classified skeletal body part (161) is applied where in the risk decision module (200) , the classification output (160) of the analysis from the body part classifier (100) is checked (210) against a risk category, preferably a tabulated risk category, for said classified skeletal body part (161) and a subsequent analysis profile (220,230,240) is chosen, either low-risk (220) , medium-risk (230) , or high-risk (240) respectively, for the analyzed X-ray image (10) based on said risk category.

8. A computer-implemented method according to any of the claims 2 to 7, wherein a report decision module (400) is applied to the fracture analysis output (320) from the downstream convoluted neural network (300) , wherein the fracture analysis output (320) is evaluated (410) for compliance with a desired fracture analysis outcome certainty threshold value, thereby providing a determination of whether the performed fracture analysis (300) can be considered certain (411) or uncertain (412)10332WO00respectively with respect to a desired risk assessment threshold .9 . A data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement a computer-implemented method according to any of the claims 1 to 8 .10 . A non-transitory computer-readable storage medium comprising a computer program stored thereon for executing by a data processing system according to claim 9 a computer-implemented method according to any of the claims 1 to 8 .10332WO00

Images