System and machine learning based method of detecting risk of fracture

Abstract

Methods and devices are provided for detecting risk of fracture of a given unfractured body part of a subject in an x-ray image. The x-ray is obtained using an x-ray machine and is processed using a trained machine learning model. This includes inputting the x-ray image into an image processing neural network, the image processing neural network having weights trained with multiple at-risk x-ray images, each at-risk x-ray image being of a first unfractured body part of a pair of body parts of a subject, wherein a second body part of the pair of body parts of the subject was fractured. A risk rating is then generated based on at least output from the image processing neural network, the risk rating indicating whether the given unfractured body part in the x-ray image is at-risk of fracture.

Description

SYSTEM AND MACHINE LEARNING BASED METHOD OF DETECTING RISK OF FRACTUREFIELD

[0001] The present application relates to x-ray systems and methods and, in particular, a machine learning based method and device for detecting fracture risk in x-ray images.BACKGROUND

[0002] Hip fractures are a common condition worldwide, which occur when an upper portion of the femur fractures or breaks. Hip fractures often occur due to osteoporosis, with 70-90% of fracture cases stemming from this disease. In many cases, the first year of costs for a patient with a hip fracture in the US may be $26,000 USD.

[0003] The current gold standard for detecting if a hip is at-risk of fracture, is using Dual Energy X-ray Absorptiometry, or “DEXA”. DEXA involves using two different energies of x-rays that produces a measurement of bone density. This measurement, however, is known to have significant variation (as much as 30%) across its many manufacturers and instantiations. As well, access to these expensive machines is limited, and may have patients waiting weeks or months for their appointment.BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Reference will now be made, by way of example, to the accompanying drawings in which:

[0005] FIG. 1 shows, in block diagram form, an x-ray system with a Diagnosis Engine in accordance with an example embodiment of the present application;

[0006] FIG. 2 shows, in block diagram form, the Diagnosis Engine of FIG. 1 in greater detail;

[0007] FIG. 3 shows a diagram of an example computing device;

[0008] FIG. 4 shows a simplified organization of software components stored in a memory of the example computing device of FIG. 3;

[0009] FIG. 5 shows, in flowchart form, example methods of detecting risk of fracture in an x-ray image;

[0010] FIG. 6 is a confusion matrix for a sample run’s TP, FP, FN, TN distribution in an example case study using the system of FIG. 1 and / or the method of FIG. 5;

[0011] FIG. 7 displays the ROC curve for the case study sample; and

[0012] FIGS. 8 and 9 illustrate two batches of images taken from the test set in the example case study for a visual inspection on network output.

[0013] Like reference numerals are used in the drawings to denote like elements and features.SUMMARY

[0014] In one aspect, the present application describes a method of detecting risk of fracture of a given unfractured body part in an x-ray image, the method comprising: obtaining the x-ray image using an x-ray machine; and processing the x-ray image using a trained machine learning model, including, inputting the x-ray image into an image processing neural network, the image processing neural network having weights trained with multiple at-risk x-ray images, each at-risk x-ray image being of a first unfractured body part of a pair of body parts of a subject, wherein a second body part of the pair of body parts of the subject was fractured; and generating a risk rating based on at least output from the image processing neural network, the risk rating indicating whether the given unfractured body part in the x-ray image is at-risk of fracture.

[0015] In some implementations, the method further comprises classifying the risk rating into one of multiple diagnoses of fracture risk.

[0016] In some implementations, the image processing neural network comprises two convolutional neural networks (CNNs).

[0017] In some implementations, the second body part of each subject was fractured, at least in part, due to a disorder in the second body part.

[0018] In some implementations, the disorder is osteoporosis.

[0019] In some implementations, the pair of body parts of each subject are hips of each subject.

[0020] In some implementations, the pair of body parts of each subject are wrists of each subject.

[0021] In some implementations, generating the risk rating comprises: inputting the output from the image processing neural network into a multilayer perceptron (MLP) layer to produce a logistic result as the risk rating.

[0022] In some implementations, the method further comprises: inputting metadata associated with the x-ray image into a multilayer perceptron, the metadata comprising x-ray field parameters; and wherein generating the risk rating comprises: concatenating output from the multilayer perceptron with the output from the image processing neural network as concatenated output; and inputting the concatenated output into a multilayer perceptron (MLP) layer to produce a logistic result as the risk rating.

[0023] In some implementations, the x-ray field parameters include one or more of kPV, distance- to-detector, and exposure time.

[0024] In some implementations, the metadata associated with the x-ray image further comprises one or more of personal and historical information of a given subject associated with the given unfractured body part.

[0025] In some implementations, the personal and historical information include one or more of age, sex at birth, ethnicity, and family history of osteoporosis.

[0026] In a further aspect, the present application describes a computing device that includes a processor and memory storing processor-executable instructions that, when executed by the processor, are to cause the processor to carry out one or more of the methods described herein.

[0027] In yet another aspect, the present application describes an x-ray system that includes the computing device and an x-ray machine to obtain the x-ray images from the subject.

[0028] In a further aspect, the present application describes a method for training a machine learning model to detect risk of fracture of a given unfractured body part in an x-ray image, the method comprising: a) obtaining an x-ray image of a first unfractured body part of a pair of body parts of a subject, wherein a second body part of the pair of body parts of the subject was fractured; b) labelling the x-ray image of the first unfractured body part as an at-risk image; c) repeating steps a)-b) until an at-risk dataset comprising the at-risk images is formed; and d) training the machine learning model, using the at-risk dataset and a control dataset comprising images of healthy one of a pair of body parts (where neither body part was fractured), until an objective performance threshold is reached.

[0029] In some implementations, the method further includes identifying that the second body part of each subject was fractured, at least in part, due to a disorder in the second body part.

[0030] In some implementations, the method include identifying that the disorder in the second body part is osteoporosis.

[0031] In some implementations, the pair of body parts of each subject are hips of each subject.

[0032] According to another aspect, the present application discloses a non-transitory computer readable storage medium containing computer-executable instructions which, when executed, configure a processor to carry one or more of the methods described herein.

[0033] Other aspects and features of the present application will be understood by those of ordinary skill in the art from a review of the following description of examples in conjunction with the accompanying figures.

[0034] In the present application, the phrase “at least one of.. . or.. .” is intended to cover any one or more of the listed elements, including any one of the listed elements alone, any sub-combination, or all of the elements, without necessarily excluding any additional elements, and without necessarily requiring all of the elements.DETAILED DESCRIPTION

[0035] In accordance with one aspect of the present application, an x-ray system and method are described that eliminate the need for bone density measurements, and instead, uses a machine learning model to detect risk of bone fracture in an x-ray image based on visual patterns in at-risk x-ray images and, potentially, additional patient metadata. X-ray imaging of a body part is far simpler to perform than bone density measurements and is readily available to patients with little to no wait times. As well, the present methods and systems may also be performed or used in tandem with a patient’s hip x-ray imaging process.

[0036] Reference is made to FIG. 1, which shows, in block diagram form, an x-ray system 100 according to an example embodiment of the present disclosure. The x-ray system 100 includes an x-ray machine 102 configured to take x-ray images of body parts of a subject. The body parts may include bone structures of the subject, including the hips and wrists, among others, where the body part may be nonfractured. The x-ray machine 102 outputs the x-ray images, and may also record and output associated metadata, including imaging parameters that were used to take each x-ray image. The imaging parameters may include x-ray field parameters, such as peak kilovoltage (kPV), distance-to-detector, and exposure time, among others.

[0037] In the depicted embodiment, the x-ray machine 102 is connected, wired or wirelessly, to one or more computing devices with one or more processing units, such as a computing device 104 that includes a processor 106 and a memory 108. The computing device 104 may be configured to execute program instructions stored in the memory 108 that, when executed, cause the processor 106 to carry out the described functions. The processor 106 may be implemented by way of a suitable computing device, including a personal computer, laptop, tablet, server, mobile device, or special purpose computing device. In some cases, the processing units and associated hardware are integrated in a single device together with the x-ray machine 102. The x-ray machine 102 and / or integrated computing device 104 may be configured for wired or wireless connection and communication with other computing devices using suitable network communications protocols. The computing device 104 may further incorporate, or be connected to, a display 110.

[0038] The memory 108 stores instructions for execution by the processor 106, including an x-ray machine application 112 that may be stored in the memory 108. The x-ray machine application112 may be configured to communicate with the processor 106 so as to provide imaging instructions, obtain the x-ray image and associated metadata from the x-ray machine 102, and to output the x-ray image and associated metadata on the display 110 in a suitable graphical user interface for review by the user or clinician operating the x-ray machine 102. As noted above, in some embodiments, there may be no separate computing device 104, whereby the computing device 104 and the display 110 may be incorporated into the x-ray machine 102 to output the x- ray image and associated metadata on the display 110 of the x-ray machine 102.

[0039] The memory 108 may include an image database 114 in which the x-ray image and the associated metadata from the x-ray machine may be stored. The memory 108 may also include a subject database 116, which may be a database comprising one or more of personal, medical, and historical information relating to the subject or patient being imaged. The subject database 116 may include information such as the subject’s age, sex at birth, ethnicity, diagnosed conditions, and family history of osteoporosis, among others. The personal, medical, and historical information may form part of the metadata that is associated with the x-ray image.

[0040] The memory 108 may store further instructions for execution by the processor 106, including a diagnosis engine 200 with a trained machine learning fracture model 202. The diagnosis engine 200, when executed by the processor 106, may obtain the x-ray image, such as of an unfractured body part (and, optionally, obtain the metadata associated with the x-ray image) from the image database 114 and the subject database 116, or directly from the x-ray machine 102, and input the data into the machine learning fracture model 202. The diagnosis engine may then output a fracture risk diagnosis for the inputted x-ray image (of the unfractured body part). In that manner, the diagnosis engine 200 and the machine learning fracture model 202 may be implemented in software for execution by the processor 106.

[0041] In the depicted embodiment of FIG. 1, the diagnosis engine 200 and the machine learning fracture model 202 are shown to be stored locally in the memory 108 of the computing device 104. In alternative embodiments, the machine learning fracture model 202 or the entire diagnosis engine 200 may be stored remotely, such as on a cloud computing server. For example, it may be that the machine learning fracture model 202 software may be provided as a service and may be centrally hosted (e.g. and then accessed by users, clinicians, or medical practitioners via a secure webbrowser or other application) through the computing device 104 or the x-ray machine 102 using suitable network communications protocols (not shown). In some embodiments, elements of the diagnosis engine 200 may be implemented to operate and / or integrate with various other platforms and operating systems.

[0042] Reference is made to FIG. 2, which shows, in block diagram form, the diagnosis engine 200 of FIG. 1 in greater detail. As noted above, the diagnosis engine 200, when executed by the processor 106, may obtain the x-ray image, such as of an unfractured body part, (and, optionally, the metadata associated with the x-ray image) from the image database 114 and the subject database 116, or directly from the x-ray machine 102, and input the data into the machine learning fracture model 202. The machine learning fracture model 202 may take on one of a number of different forms or architecture. In the present embodiment, the machine learning fracture model 202 may have a mixed data neural network (MDNN) architecture with two main components, followed by a concatenating layer. To process the x-ray image, the first main component may be an image processing neural network 204, for example, comprising two convolutional neural networks (CNNs) 206. To process the metadata, the second component may be a multilayer perceptron 210 (as shown in FIG. 2). While the first and second components are shown as separate components, in alternative embodiments, they may be combined. The two main components may then be followed by a concatenating layer 208 and a multilayer perceptron (MLP) layer 212.

[0043] When operating without metadata, the CNNs 206 generally include a plurality of convolutional layers that process the x-ray image in order to generate an output, such as a predicted classification, a predicted label, or a logistic result for the x-ray image. Generally, a convolutional layer performs convolution processing, which may involve computing a dot product between the input to the convolutional layer and a convolution kernel. A convolutional kernel is typically a 2D matrix of learned parameters that is applied to the input in order to extract image features. Different convolutional kernels may be applied to extract different image information, such as shape information, color information, etc.

[0044] The output of the convolution layers may be an activation map of the x-ray image that encodes image features. In the present embodiment, subsequent convolutional layers process the activation map in order to determine a set of probabilities representing the likelihood that the x-ray image belongs to each of a defined set of possible classes, such as “at-risk” and “healthy”. To do this, the image processing neural network 204 contains learned parameters or trained weights that, when applied to the activation map, output the set of probabilities representing the likelihood that the x-ray image belongs to each of the defined set of possible classes.

[0045] The learned parameters or trained weights of the image processing neural network 204 were obtained by training the image processing neural network 204 with an at-risk dataset and a control dataset until an objective performance threshold was reached. The at-risk dataset comprised x-ray images of at-risk body parts, and the control dataset comprised x-ray images of healthy corresponding body parts, where neither of the subject’s body parts was / is fractured.

[0046] Notably, the x-ray images of the at-risk body parts in the at-risk dataset were identified and selected as follows. If a fracture, such as a hip fracture, occurred on one side of a subject’s body, then the unfractured body part on the opposite / contralateral side of the subject’s body (such as the opposite hip) was identified and marked as an at-risk body part (or at-risk hip). The unfractured at-risk body part was then imaged, labelled as an at-risk x-ray image, and added to the at-risk dataset. This process was repeated until a sufficient at-risk dataset was collected. Each x-ray image in the at-risk dataset may be obtained in that manner. In some cases, when identifying the at-risk body parts to be imaged, the identification included identifying that the fractured body part of each subject was fractured, at least in part, due to health of the fractured body part, and was not caused purely due to external factors. For example, the original fracture may have occurred, at least in part, due to a disorder or disease in the body part, such as osteoporosis. The compiled at-risk dataset and the control dataset were then combined and randomized into a training subset and a test subset, and the image processing neural network 204 was trained using the training subset and the test subset.

[0047] Such training may be particularly applicable for image processing of images of body parts of which subjects have a pair, such as hips, wrists, or knees. In that manner, each at-risk x-ray image in the at-risk dataset would be of a first unfractured body part of a pair of body parts of a subject, where a second body part of the pair of body parts (i.e. the corresponding contralateral body part) of the subject was / is (known to be) fractured. For example, if a subject’s left hip / femurwas fractured, the at-risk x-ray image in the at-risk dataset would be of the subject’s nonfractured right hip / femur.

[0048] The image processing neural network 204 may, thus, be configured, when executed by the processor 106, to identify visual patterns that indicate risk of fracture in x-ray images and output the probabilities representing the likelihood that the x-ray image belongs to each of the defined set of possible classes, such as “at-risk” and “healthy”. The MLP layer 212 may receive the output from the CNNs 206 to generate a logistic result. In some embodiments, if metadata is not used or inputted into the machine learning fracture model 202, the logistic result may be considered a risk rating for the x-ray image. The risk rating indicates whether the unfractured body part in the x-ray image is at-risk of fracture.

[0049] If metadata is used or inputted into the machine learning fracture model 202, the multilayer perceptron 210 may be trained to receive and process the metadata to capture holistic information about the metadata for incorporation into / with the output from the image processing neural network 204. As noted above, the multilayer perceptron 210 may be trained to process metadata inputs such as x-ray field parameters of the x-ray image (including peak kilovoltage (kPV), distance-to-detector, and exposure time, among others), and personal and historical information of the subject (including the subject’s age, sex at birth, ethnicity, diagnosed conditions, and family history of osteoporosis, among others). For different applications, different combinations of the metadata inputs may be used.

[0050] The outputs from the multilayer perceptron 210 and the image processing neural network 204 may then be inputted into the concatenating layer 208 and concatenated as internal representations or, equivalently, internal states. In other words, the output from the first multilayer perceptron 210 may be joined to the CNN branch’s internal state / representation. The concatenated internal representations may be fed into the subsequent MLP layer 212, which produces a logistic result or a risk rating for the x-ray image. The risk rating may be in the range [0, 1] (at-risk or healthy). The risk rating itself may indicate, or be used to determine, whether the unfractured body part in the x-ray image is considered at-risk of fracture, at moderate risk of fracture, or healthy.

[0051] The diagnosis engine 200 or the machine learning fracture model 202 may further have a classifier 214. The classifier 214, when executed by the processor 106, may be configured tocategorize or classify the risk rating into one of multiple diagnoses of fracture risk for the x-ray image, such as “at-risk”, “moderate”, and “healthy”.

[0052] FIG. 3 is a high-level diagram of an example computing device 300. The example computing device 300 includes a variety of modules. For example, the example computing device 300 may include a processor 310, a memory 320, an VO module 340, and a communications module 350. As illustrated, the foregoing example modules of the example computing device 300 are in communication over a bus 360.

[0053] The processor 310 is a hardware processor. The processor 310 may, for example, be one or more ARM, Intel x86, PowerPC processors, or the like.

[0054] The memory 320 allows data to be stored and retrieved. The memory 320 may include, for example, random access memory, read-only memory, and persistent storage. Persistent storage may be, for example, flash memory, a solid-state drive, or the like. Read-only memory and persistent storage are a computer-readable medium. A computer-readable medium may be organized using a file system such as may be administered by an operating system governing overall operation of the example computing device 300.

[0055] The VO module 340 allows the example computing device 300 to receive input signals and to transmit output signal. Input signals may, for example, correspond to input received from a user. Some output signals may, for example, allow provision of output to a user. The VO module 340 may serve to interconnect the example computing device 300 with one or more input devices. Input devices may, for example, include one or more of a touchscreen input, keyboard, trackball, or the like. The VO module 340 may serve to interconnect the example computing device 300 with one or more output devices. Output devices may include, for example, a display screen such as, for example, a liquid crystal display (LCD), a touchscreen display. Additionally, or alternatively, output devices may include devices other than screens such as, for example, a speaker, indicator lamps (such as, for example, light-emitting diodes (LEDs)), and printers.

[0056] The communications module 350 allows the example computing device 300 to communicate with other electronic devices and / or various communications networks. For example, the communications module 350 may allow the example computing device 300 to sendor receive communications signals. As an example, the communication module 350 may include a network connection, data port, or the like, for receiving scan images from an MRI scanner. Communications signals may be sent or received according to one or more protocols or according to one or more standards. For example, the communications module 350 may allow the example computing device 300 to communicate via a cellular data network, such as for example, according to one or more standards such as, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Evolution Data Optimized (EVDO), Long-term Evolution (LTE) or the like. Additionally, or alternatively, the communications module 350 may allow the example computing device 300 to communicate using near-field communication (NFC), via Wi-Fi (TM), via the Ethernet family of network protocols, using Bluetooth (TM) or via some combination of one or more networks or protocols. In some embodiments, all or a portion of the communications module 350 may be integrated into a component of the example computing device 300. In some examples, the communications module may be integrated into a communications chipset.

[0057] Software instructions are executed by the processor 310 from a computer-readable medium. For example, software may be loaded into random-access memory from persistent storage within memory 320. Additionally, or alternatively, instructions may be executed by the processor 310 directly from read-only memory of the memory 320. An example of the example computing device 300 may be the computing device 104.

[0058] FIG. 4 depicts a simplified organization of software components stored in memory 320 of the example computing device 300. As illustrated, these software components include, at least, application software 410 and an operating system 400.

[0059] The application software 410 adapts the example computing device 300, in combination with the operating system 400, to operate as a device performing a particular function. While a single application software 410 is illustrated in FIG. 4, in operation, the memory 320 may include more than one application software and different application software may perform different operations.

[0060] The operating system 400 is software. The operating system 400 allows the application software 410 to access the processor 310, the memory 320, the I / O module 340, and thecommunications module 350. The operating system 400 may, for example, be iOS™, Android™, Linux™, Microsoft Windows™, or the like.

[0061] The application software 410 may, when executed, cause the processor 310 to carry out operations to implement at least some portion of one or more of the methods described herein. In some cases, the application software 410 may implement some or all of a machine learning model to detect risk of fracture of a body part of a subject in an x-ray image from an x-ray machine. The example computing device 300 may be external to the x-ray machine and may receive the x-ray image via a wired or wireless connection in some cases. In some cases, the example computing device 300 may be implemented as part of the x-ray machine. That is, the x-ray machine may include the example computing device 300 and its constituent parts.

[0062] Reference will now be made to FIG. 5, which shows, in flowchart form, an example method 500 of detecting whether an x-ray image of an unfractured body part is at-risk of fracture. The method 500 may be implemented by way of suitably programmed processor-executable instructions stored in memory that, when executed, cause a computing device to carry out the described functions. In particular, the instructions may implement a machine learning model for receiving an x-ray image and outputting a risk diagnosis. In some cases, the computing device may be integrated within an x-ray machine. In some cases, the computing device may be external to and separate from the x-ray machine.

[0063] In operation 502, an x-ray image of a body of a subject is obtained. The body part in the x- ray image may be one of a pair of body parts of the subject that is unfractured. For example, in some applications, the unfractured body part in the x-ray image may be one of the subject’s hips (in operation 504) or one of the subject’s wrists (in operation 506). The x-ray image may be obtained directly from an x-ray machine, or may be retrieved from an image database. In some applications, the region of interest (ROI), such as the hip region, in the x-ray image may be cropped from the x-ray image before being processed by the trained machine learning model.

[0064] In operation 508, optionally, metadata associated with the x-ray image may be obtained. The metadata may include imaging parameters that were used to take the x-ray image. For example, at operation 510, the imaging parameters may include x-ray field parameters, such as peak kilovoltage (kPV), distance-to-detector, and exposure time, among others. The metadata mayalso, or alternatively, include one or more of personal, medical, and historical information relating to the subject whose body part was imaged (at operation 512). The personal, medical, and historical information may include information such as the subject’s age, sex at birth, ethnicity, diagnosed conditions, and family history of osteoporosis, among others.

[0065] In operation 514, the x-ray image is processed with a trained machine learning model, such as the machine learning fracture model 202 described above. This includes, in operation 516, inputting the x-ray image into an image processing neural network. Notably, the image processing neural network has weights trained with multiple at-risk x-ray images, each at-risk x-ray image being of a first unfractured body part of a pair of body parts of a subject, wherein a second body part of the pair of body parts (i.e. the contralateral body part) of the subject was fractured.

[0066] The trained machine learning model was trained as follows. To obtain the at-risk images, when a fracture was identified in one of a pair of body parts in a subject, the other nonfractured one of the pair of body parts (i.e. the corresponding contralateral body part) was labelled as “at- risk”, and imaged. The resulting x-ray image was labelled as an at-risk image and added to an at- risk dataset, that comprised of at-risk x-ray images. This was repeated until a sufficient at-risk dataset was compiled. In some cases, it was determined that the fracture in the one of the pair of body parts occurred, at least in part, due to health of the fractured body part, and was not caused purely due to external factors. For example, it may have been determined that the original fracture occurred, at least in part, due to a disorder or disease in the body part (at operation 518), such as osteoporosis (at operation 520). The compiled at-risk dataset and a control dataset (comprising x- ray images of corresponding healthy body parts, where neither body part of the subject was fractured) were then combined and randomized into a training subset and a test subset, and the image processing neural network was trained using the training subset and the test subset until an objective performance threshold was achieved.

[0067] Getting a labeled dataset is often the limiting factor in a supervised learning pipeline. This method of obtaining at-risk images and training the machine learning model has the advantage of being relatively simple to both identify the at-risk body parts and to obtain straightforward x-ray images of the corresponding contralateral body part thereof. The image processing neural networkmay, thus, be configured to output the probabilities representing the likelihood that the x-ray image belongs to each of a defined set of possible classes, such as “at-risk” and “healthy”.

[0068] If metadata was obtained at operation 508, at operation 522, the metadata may be inputted into a multilayer perceptron to capture holistic information about the metadata for incorporation into / with the output from the image processing neural network 204. The first multilayer perceptron may be trained to process metadata inputs such as x-ray field parameters of the x-ray image (including peak kilovoltage (kPV), distance-to-detector, and exposure time, among others), and personal and historical information of the subject (including the subject’s age, sex at birth, ethnicity, diagnosed conditions, and family history of osteoporosis, among others). For different applications, different combinations of metadata inputs may be inputted into the first multilayer perceptron.

[0069] At operation 524, a risk rating or other risk indicator is outputted. The risk rating may be outputted via an output device such as a display 110. The risk rating may, in some instances, be pushed to a remote computing system such as via an alert mechanism. The risk rating may, in some instances, be a numerical rating. The risk rating may be output to a digital file in at least some instances and this digital file may be associated with a user or patient file or account, in at least some implementations. In some instances, the risk rating may be sent to a remote computing system such as a patient’s or doctor’s computing system.

[0070] If no metadata was obtained at operation 508, the output from operation 516 may be directly inputted into an MLP layer (such as the MLP layer 212) at operation 528 to produce / output a logistic result. If no metadata was involved, this logistic result may be considered a risk rating for the x-ray image. The risk rating indicates whether the unfractured body part in the x-ray image is at-risk of fracture.

[0071] However, if metadata is involved, following operation 522, the outputs from operation 522 and operation 516 may then be concatenated at operation 526 as internal representations or, equivalently, internal states. At operation 528, the concatenated internal representation may be inputted into the subsequent MLP layer to produce a logistic result or a risk rating for the x-ray image. The risk rating may be in the range [0, 1] (at-risk or healthy). The risk rating itself mayindicate, or be used to determine, whether the unfractured body part in the x-ray image is considered at-risk of fracture, moderately at-risk of fracture or healthy.

[0072] At operation 530, the risk rating may further be classified into a risk diagnosis category of whether the unfractured body part in the x-ray image is at-risk of fracture, such as “at-risk”, “moderate”, and “healthy”.

[0073] EXAMPLE CASE STUDY

[0074] An example case study using the above-described system and method is set out below with discussion.

[0075] An at-risk dataset of 113 at-risk hip x-ray images and 360 healthy (or “control”) hip x-rays, was compiled, for a total of 473 images. The at-risk hip x-ray images were obtained by identifying a fracture in a hip of a subject. An x-ray image of the subject’s other hip was then taken and labelled as an at-risk hip x-ray image. This was repeated until the at-risk dataset was compiled. The at-risk and control x-ray images were randomized and divided into 378 training images and 95 test images (with 20 at-risk images, 75 healthy / control images).

[0076] After the mixed data neural network was trained, the x-ray test images were inputted into the above-described machine learning fracture model to obtain a risk rating for each test image. Each risk rating was in the range [0, 1] (where 0 is towards healthy hips and 1 towards at-risk hips). Values >0.5 were considered / approximated to be at-risk, and values <0.5 were considered to be control / healthy. A 3rdparty open-source software package, Ray Tune, was used to systematically optimize hyperparameters across various runs. The risk ratings were further categorized or classified according to Table 1:

[0077] Table 1:

[0078] The model evaluation was performed on 95 (20 at-risk) images unseen during training. Models were evaluated against a thorough suite of metrics. Throughout, a “Positive” result was the model outputting an estimate that is greater than a threshold (which was by default 0.5), and, in this case, indicates an at-risk hip:1. Precision: What fraction of the estimated Positives were True Positives?2. Recall: What fraction of the At-risk hips were detected as Positive?3. Fl: Harmonic mean of Precision and Recall together. Range is [0,1],4. Confusion Matrices: A snapshot of the distribution of True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN).5. ROC & AUC: The probability that a positive sample will be ranked higher than a negative sample.

[0079] In this case study, Recall was valued over Precision. These two metrics are at odds in practical contexts. High precision indicates a very selective, stringent model that tries to minimize False Positives (healthy hips reading as at-risk). However, this comes at the cost of letting at-risk hips slip through unnoticed (increased False Negatives, low recall). High recall indicates that the model is doing well at capturing at-risk hips, but this looser restriction lets more False Positives through. In medical applications, high recall is often preferred so positives are not missed, with the downside of potentially treating false positives. It is not a zero-sum game however, and better models will indeed score better at both precision and recall than a lesser model (which is neatly captured in the Fl score).

[0080] ROC and AUC are common metrics for classification estimators. The ROC curve, as will be shown, plots True Positive Rate (TPR) vs False Positive Rate (FPR) as the classification / decision threshold is swept from [0, 1], By default, the threshold is 0.5, where an estimate of 0.6 gets rounded to 1 as Positive result and vice-versa. However, an estimator may still be an effective discriminator, separating positives from negatives, but about a different threshold. For example, it estimates positives to be roughly in the range [0.3, 0.4] and negatives to be [0.1, 0.2], At a threshold of 0.5, this model performs terribly, but not at a threshold of 0.25. If an estimator perfectly separates its test dataset, the ROC curve will be a square, TPR=1 constant, AUC (Area under the curve) would be its maximum at 1. In practice, there is mixing of FPs andFNs, which reduces the curve away from a square, AUCcl. The worse this mixing, the worse the AUC, and a slope of 1 is as good as random estimation.

[0081] First, looking at the sample run’s TP, FP, FN, TN distribution (see FIG. 6), and the Precision, Recall, Fl -score (see Table 2), the model shows a bias towards the Control class, with only 6 of 75 Controls being FP. It would be better to see more of the At-risk images correctly identified, 9 of the 20 were FN, lowering the recall. The Positive precision was somewhat better with only 6 of 20 being FP. Overall, this model correctly labeled 80 of 95 images, giving an accuracy of 84%, and a “balanced” accuracy (the mean accuracy by class) of 74%.

[0082] The Precision, Recall, and Fl-score are given in Table 2. “Support” is the number of images of that type in the Test dataset. “Macro Average” is the flat average of the metric values, and “Weighted Average” weights the metrics by their Support count.

[0083] Table 2:

[0084] FIG. 7 displays the ROC curve for the case study sample. There is not much insight to be gathered form the form itself, beyond that it directs towards TPR=1 as it should, but an AUC of 0.82 is a good sign. There is evidently some degree of class separation. Overall, from these results, this network is indeed learning and identifying features present in the dataset.

[0085] FIGS. 8 and 9 illustrate example two batches of images taken from the test set for a visual inspection on network output. FIG. 8 shows test-set control images and the selected model estimations. FIG. 9 shows test-set at-risk images and the selected model estimations. For the given input images, the model output percentages are overlain in text along with the category of image: “Ctrl” or “Risk”. Most of the output samples are ideal, approaching the “Healthy” Risk Score for the control batch, and the “At-Risk” Risk Score for the at-risk batch, though exceptions can befound in either batch. For example, a hip might (in reality) be at-risk, but still be in the control set because the patient hasn’t fractured their hip yet. Conversely, a control hip could be in the at-risk group if the patient’s fracture was due to external circumstances and not hip-health per se.

[0086] The various embodiments presented above are merely examples and are in no way meant to limit the scope of this application. Variations of the innovations described herein will be apparent to persons of ordinary skill in the art, such variations being within the intended scope of the present application. In particular, features from one or more of the above-described example embodiments may be selected to create alternative example embodiments including a subcombination of features which may not be explicitly described above. In addition, features from one or more of the above-described example embodiments may be selected and combined to create alternative example embodiments including a combination of features which may not be explicitly described above. Features suitable for such combinations and sub-combinations would be readily apparent to persons skilled in the art upon review of the present application as a whole. The subject matter described herein and in the recited claims intends to cover and embrace all suitable changes in technology.

Claims

What is claimed is:

1. A method of detecting risk of fracture of a given unfractured body part in an x-ray image, the method comprising: obtaining the x-ray image using an x-ray machine; and processing the x-ray image using a trained machine learning model, including, inputting the x-ray image into an image processing neural network, the image processing neural network having weights trained with multiple at-risk x- ray images, each at-risk x-ray image being of a first unfractured body part of a pair of body parts of a subject, wherein a second body part of the pair of body parts of the subject was fractured; and generating a risk rating based on at least output from the image processing neural network, the risk rating indicating whether the given unfractured body part in the x-ray image is at-risk of fracture.

2. The method of claim 1, further comprising classifying the risk rating into one of multiple diagnoses of fracture risk.

3. The method of claim 1, wherein the image processing neural network comprises two convolutional neural networks (CNNs).

4. The method of claim 3, wherein the second body part of each subject was fractured, at least in part, due to a disorder in the second body part.

5. The method of claim 4, wherein the disorder is osteoporosis.

6. The method of claim 5, wherein the pair of body parts of each subject are hips of each subject.

7. The method of claim 5, wherein the pair of body parts of each subject are wrists of each subject.

8. The method of claim 4, wherein generating the risk rating comprises: inputting the output from the image processing neural network into a multilayer perceptron (MLP) layer to produce a logistic result as the risk rating.

9. The method of claim 4, further comprising: inputting metadata associated with the x-ray image into a multilayer perceptron, the metadata comprising x-ray field parameters; and wherein generating the risk rating comprises: concatenating output from the multilayer perceptron with the output from the image processing neural network as concatenated output; and inputting the concatenated output into a multilayer perceptron (MLP) layer to produce a logistic result as the risk rating.

10. The method of claim 9, wherein the x-ray field parameters include one or more of kPV, distance-to-detector, and exposure time.

11. The method of claim 10, wherein the metadata associated with the x-ray image further comprises one or more of personal and historical information of a given subject associated with the given unfractured body part.

12. The method of claim 11, wherein the personal information includes one or more of age, sex at birth, and ethnicity, and the historical information includes family history of osteoporosis.

13. A computing device for detecting risk of fracture of a given unfractured body part in an x-ray image, the computing device comprising: a processor; and a memory coupled to the processor, the memory storing processor-executable instructions that, when executed by the processor, configure the processor to: receive the x-ray image from an x-ray machine; andprocess the x-ray image using a trained machine learning model, including: inputting the x-ray image into an image processing neural network, the image processing neural network having weights trained with multiple at-risk x-ray images, each at-risk x-ray image being of a first unfractured body part of a pair of body parts of a subject, wherein a second body part of the pair of body parts of the subject was fractured; and generating a risk rating based on at least output from the image processing neural network, the risk rating indicating whether the given unfractured body part in the x-ray image is at-risk of fracture.

14. The computing device of claim 13, wherein the second body part of each subject was fractured, at least in part, due to a disorder in the second body part.

15. The computing device of claim 14, wherein the disorder is osteoporosis.

16. The computing device of claim 15, wherein the pair of body parts of each subject are hips of each subject.

17. The computing device of claim 13, wherein the processor-executable instructions, when executed by the processor, further configure the processor to generate the risk rating by: inputting the output from the image processing neural network into a multilayer perceptron (MLP) layer to produce a logistic result as the risk rating.

18. The computing device of claim 13, wherein the processor-executable instructions, when executed by the processor, further configure the processor to: input metadata associated with the x-ray image into a multilayer perceptron, the metadata comprising x-ray field parameters; andgenerate the risk rating by: concatenating output from the multilayer perceptron with the output from the image processing neural network as concatenated output; and inputting the concatenated output into a multilayer perceptron (MLP) layer to produce a logistic result as the risk rating.

19. The computing device of claim 18, wherein the x-ray field parameters include one or more of kPV, distance-to-detector, and exposure time.

20. The computing device of claim 19, wherein the metadata associated with the x-ray image further comprises one or more of personal and historical information of a given subject associated with the given unfractured body part, the personal information including one or more of age, sex at birth, and ethnicity, and the historical information including family history of osteoporosis.

21. An x-ray system comprising the computing device of claim 13 and an x-ray machine to obtain the x-ray image of the given unfractured body part.

Images