Method and system for determination of bone health

Abstract

A method and system for the early determination of bone health and risk or status of bone disease is disclosed. Determination is an incidental finding from x-rays used in routine examinations. The method and system aid determination of a person's bone health during the person's lifetime when routine x-rays are taken. The method and system thereby aid nearly continuous determination. Biomarkers are considered independently thereby characterising intricacies missed by a single DXA value. An example demonstrates independent assessment of cortical and trabecular bone.

Description

[0001] METHOD AND SYSTEM FOR DETERM I NATON OF BONE HEALTH

[0002] Field of the Invention

[0003] The present invention relates to a method and system for the early determination of bone health and risk or status of bone disease.

[0004] It is an advantage that the determination is an incidental finding from x-rays. It is a further advantage that the method and system have verification and correction stages to accurately predict a continuous measure of bone mineral density that correlates with standard reference protocols.

[0005] Osteoporosis (the term is used here to refer singly to all types of conditions on the low bone mass spectrum, including osteopenia) can affect most of the skeleton and / or a localized area. The measure of associated loss of and / or destruction of bone tissue or bone mass is one of the factors that help determine weakness and risk of fracture, along with e.g., the aspects assessed to calculate a Fracture Risk Assessment Tool score.

[0006] Early detection and prompt treatment is imperative. Bone mineral density (BMD) is one of the measures to determine the health of bone. The measure includes estimation of the amount of calcium hydroxyapatite (a calcium phosphate mineral) by means of a range of x-ray based, gamma-ray based and ultrasonic methods (for example, radiographic absorptiometry (RA), single photon and x-ray absorptiometry (SPA), quantitative computed tomography (CT), quantitative ultrasound, MRI and / or dual energy x-ray absorptiometry. Dual energy x-ray absorptiometry is often abbreviated by the letters DXA or DEXA.

[0007] A conventional x-ray image records the attenuation of the x-ray beam as it passes through tissue and the scatter of x-rays as they pass through an object of interest.

[0008] CT records low-dose x-ray scans from several directions around an object of interest, enabling construction of a 3D volume in which cortical bone, trabecular bone, and soft tissue are visible. Such a 3D image is often quantitative: each voxel recording the x- ray density measured in Hounsfield units. However, whilst the individual beam energies may be low, when summed together a CT scan entails higher radiation compared to DXA and x-ray.

[0009] During a dual energy x-ray absorptiometry (DXA) scan, two x-ray beams of different energy levels are passed through a patient’s bones. The difference in total absorption of the two beams is used to subtract out the absorption by soft tissue, leaving the absorption by bone, from which estimations of bone mineral density (BMD) and bone mineral content (BMC) are computed. DXA results are combined with other patient factors as part of fracture risk assessment, for example a Fracture Risk Assessment Tool score. DXA is thus the current ‘gold standard.’

[0010] BMD results are usually reported in terms of standard deviations from the mean of a young healthy population (T-score). A T-score greater than -1 is considered normal, whilst a T-score between -1 and -2.5 is indicative of osteopenia. A T-score less than - 2.5 is indicative of osteoporosis. An associated Z-score (derivation from the mean) corrects for age and sex.

[0011] Systems are known that build on dual energy x-ray absorptiometry (DXA) to indicate quality of bone. For example, trabecular bone score (TBS) quantifies local variations in grey-level of the greyscale DXA image as a surrogate for bone trabecular microarchitecture.

[0012] However, known measures of bone quality have limitations. For example, due to the poor resolution of DXA images, TBS provides an impression of trabecular micro architecture, rather than direct visualization as it does not take account of other determinants of bone quality, such as cortical microarchitecture or morphometry (used here to mean external shape and dimensions of a bone or part of bone).

[0013] Furthermore, dual energy x-ray absorptiometry (DXA) produces a single number describing the entire range of contributory factors to the state of bone health: for example, the measured bone attenuation is averaged across the area of the femoral head. Whereas osteoporosis, for example, causes marked changes in both the cortical and trabecular bone, often at different points in time.

[0014] Dual energy x-ray absorptiometry (DXA) is also a dedicated scan requiring specialised equipment and trained operators.

[0015] Therefore, while imaging can provide important information about osteoporotic fracture risk, many studies have indicated that BMD only partly explains bone strength. Quantitative assessment of macro- and microstructural features may improve the estimation, but important challenges remain: the balance between spatial resolution and sampling size, or between signal-to-noise and radiation dose or acquisition time, as well as the complexity and expense of the methods vs their availability and accessibility.

[0016] Furthermore, clinical challenges for bone imaging include balancing the benefits of simple bone densitometry vs the more complex architectural features of bone; or specific research requirements vs the broader clinical needs.

[0017] The relative merits of sophisticated imaging techniques must be weighed with respect to their applications as diagnostic procedures, requiring high accuracy or reliability, compared with their monitoring applications, requiring high precision or reproducibility.

[0018] No cost-effective, accurate means exists which is incidental to routine procedure, and seamlessly integrates into existing radiological services to provide sophisticated measure of bone health whilst correlating with a standard reference (e.g., dual energy x-ray absorptiometry DXA).

[0019] A method and system for determination of bone heath is needed to provide DXA compatible, accurate clinical measure of bone health and risk of disease as an incidental (or secondary) finding from x-rays acquired for investigation of another condition (e.g., assessment of trauma or osteoarthritis). Further, the method and system are needed to provide measures which are continuous, enabling borderline classifications to be easily identified, for example those patients who are exhibiting a high degree of osteopenia, and are thus almost osteoporotic.

[0020] Summary of the Invention

[0021] According to a first aspect of a method or system of measuring bone mineral density: at least one Region Of Interest, ROI, in x-ray images is identified, and the x-ray images and the ROIs are classified according to those with an intended anatomical body part; a first segmentation in the x-ray images is created that approximately conforms to a shape of a targeted feature of the intended anatomical body part within the ROI; the shape is quantitatively verified to resemble the targeted feature using a first metric determined from a perimeter of the first segmentation; a second segmentation in the x-ray images is created that approximately conforms to cortical bone of the targeted feature; and parameters describing the shape, structure, and composition of bone are mapped to the x-ray attenuations to derive a metric correlated to bone mineral density.

[0022] According to aspect of an x-ray image of an anatomical body part, the bone mineral density is classified according to the method or system disclosed herein.

[0023] Herein ‘or’ denotes either or all items to which the word ‘or’ refers. For example, a system or method refers to the system individually, the method individually, or the system and the method.

[0024] Preferably the x-ray image or the ROIs are classified according to those with an intended anatomical body part. The ROI’s may be classified according to a portion or portions of the intended anatomical body part which the ROI includes.

[0025] The present method or system provides a dual energy x-ray absorptiometry (DXA) compatible, accurate clinical measure of bone health and risk of disease as an incidental (or secondary) finding from x-rays acquired for investigation of another condition (e.g., assessment of trauma or osteoarthritis).

[0026] The present method or system further provides measures which are continuous, enabling borderline classifications to be easily identified, for example those patients who are exhibiting a high degree of osteopenia, and are thus almost osteoporotic.

[0027] It is a further advantage that a plurality of biomarkers may be considered independently to characterise the more complex architectural features of bone beyond bone densitometry that are missed by the single dual energy x-ray absorptiometry (DXA) value.

[0028] The method or system may include cropping the x-ray images leaving only the ROI within the x-ray images. Only the ROI from the x-ray image of a pelvis and containing a femoral head within the ROI may be left.

[0029] A generative adversarial network, GAN, may be applied to classify the ROI x-ray images. A model may be used to generate examples of x-ray images of specific anatomical body parts, a bidirectional GAN, BiGAN, may be trained to generate more images of the specific anatomical body parts, and an internal discriminator of the BiGAN may be used to discriminate between ROI images of the intended anatomical body part and ROI images of other anatomical body parts.

[0030] The x-ray images classified as having the intended anatomical body part may be provided to a Visual Geometry Group, VGG, ROI locator. Points may be placed around a femoral head to determine the ROI. The points may include a point at each of a femur low shaft, lessor trochanter, greater trochanter, and furthest extremity of the femoral head, and generating the ROI by placing a border around the points.

[0031] The method or system may be configured to detect metalwork. Within the ROI, the metalwork may be detected by applying a CNN to classify connected regions of the x- ray image which have similar textures into one or a plurality of metalwork classes and a non-metalwork class. A third segmentation may be used that is a selective search graph segmentation to find the connected regions. A synthetic metalwork may be added to the x-ray image to train a model. The CNN may be trained as a Visual Geometry Group (VGG) or II NET CNN.

[0032] In a primary metalwork detection method, a LINET is used to segment metalwork from a whole image. A binary mask is output. When using the LINET, the known mask of the metalwork location is used as the training target. The II NET is trained to segment the metalwork from the full synthetic image.

[0033] In a secondary metalwork detection method, selective search is used with graph-based region proposal, with Kullback-Leibler (KL) divergence and a VGG classifier. The primary or secondary metalwork detection method may be used.

[0034] Upon determination that the first segmentation is insufficient, contrast manipulations as a gamma correction may be applied to the x-ray image and a first repeat segmentation may be attempted. Upon determination that the first repeat segmentation is insufficient, contrast manipulations as a contrast limited adaptive histogram equalization correction may be applied to the x-ray image a second repeat segmentation is attempted. If all corrections fail we repeat attempts with an expanded ROI.

[0035] The first metric may include a shape loss score or an area loss score of the first segmentation. Upon determining the first metric is insufficient, the x-ray image may be corrected then the first segmentation may be repeated. The first metric may relate to a distance in the shape to a nearest edge in the first segmentation. To determine the first metric an active shape model preferably comprises points that define the outer contour of a femur including a lower shaft, a lesser trochanter, a greater trochanter, and a furthest extremity of a femoral head.

[0036] An estimate, preferably from a segmentation, of a proportion of cortical bone in an intertrochanteric region may be used in determining a cortical ratio. The cortical ratio may be used as a second metric indicative of bone mineral density or bone strength or bone brittleness or bone quality.

[0037] The second segmentation in the x-ray images is used. The second segmentation conforms approximately to cortical bone of the targeted feature. The second segmentation enables the amount of the cortical bone in the x-ray images, the ROI(s), the intended body part, or the targeted feature to be quantified. For example, it may be used to quantify cortical bone area in the x-ray image or ROI(s). The second segmentation may be used to quantify cortical bone volume or mass in a bone in the intended anatomical body part in the image or targeted feature. The second segmentation may be a cortical segmentation. The second segmentation may be implemented with a synthetically trained ll-Net model. The second segmentation may be deemed adequate when an area loss score and a shape loss score for both medial and lateral active shapes are within acceptable quantitative thresholds.

[0038] There may be a fourth segmentation of a lesser trochanter ROI or intertrochanteric ROI in the x-ray images.

[0039] There may be a crop of the x-ray image in the neck ROI, trochanter ROI, and intertrochanter ROI.

[0040] The present method and system may comprise an imaging biomarker platform. A plurality of imaging biomarkers are measured from x-ray images to form a multiparametric description of the shape, structure, and composition of bone, for example volume and composition of constituent bone type.

[0041] These are then mapped to x-ray attenuations, resulting in a metric that directly correlates with DXA-measured bone mineral density (BMD).

[0042] In a preferred embodiment each of the biomarkers are considered independently thereby characterising those intricacies missed by the single DXA value, for example independent assessment of cortical and trabecular bone.

[0043] A CNN or vision transformer with a single output neuron may be used to predict the bone mineral density. A ratio of cortical bone to trabecular bone in the at least one portion of bone in the region of interest is determined. The ratio may be indicative of the bone mineral density, strength, brittleness, or quality.

[0044] The intended anatomical body part in the x-ray image is classified from the bone mineral density as osteoporotic, osteopenic, non-osteoporotic, or healthy / non-healthy.

[0045] In an embodiment the present and system comprises: data pre-processing I cleaning - unusable images are detected and rejected, along with inappropriate images i.e. a hand x-ray mis-labelled as a pelvis x-ray. A suspicious region proposal model finds a suspicious region and a classifier classifies the region. region-of-interest locator - a region of interest (ROI) locator takes an x-ray and outputs a suitable crop. This region is then used when segmenting the important / target feature within the ROI. ensemble segmentation - ensemble segmentation segments the target feature and minimises the impact of common image quality issues such as over / under exposure and overlaid tissue artefacts by using an ensemble of image enhancement presegmentation. The ensemble repeatedly attempts to segment the target feature, and if it fails it tries a new corrected version of the image. Corrections include brightness alterations as well as ROI repositioning. active shape verifier - an active shape model is used to check that the segmentations are plausible and to effect rejection of images when the segmentation has failed to produce a shape that adequately resembles the target feature. Rejection is determined by shape loss and area loss scores, two metrics developed using the active shape model. cortical segmentation and verification - the high density and therefore high attenuation cortical bone drives a significant portion of the dual energy x-ray absorptiometry (DXA) score. The amount of cortical bone is thus quantified. Cortical segmentation and verification follow the preceding stage and a cortical ratio metric is calculated from this segmentation, estimating the proportion of cortical bone in the intertrochanteric region. dual energy x-ray absorptiometry (DXA) region cropping - accurate cropping of each region enables positive correlation with DXA bone mineral density (BMD) score. In a preferred embodiment, segmentation of the lesser trochanter region in an x-ray of a hip is performed to allow the bottom of the intertrochanteric region to be consistently located (e.g., specifically evaluating neck BMD as this is where most fractures occur).

[0046] The invention will now be described, by way of example only, with reference to the accompanying figures in which:

[0047] Brief Description of the Figures

[0048] Figure 1 shows a flow chart of a method and system for determination of bone health;

[0049] Figures 2A and 2B show image translation of a ‘handmade’ bone label image (2A) to a realistic image (2B);

[0050] Figure 2C shows metal work templates;

[0051] Figure 2D shows synthetic metal work templates applied to the known bone structure;

[0052] Figure 2E shows points of interest around a femoral head;

[0053] Figure 2F shows final ROI for assessment;

[0054] Figure 3A shows a flow chart to implement ensemble segmentation;

[0055] Figure 3B shows images of ensemble segmentation;

[0056] Figure 4A shows verification of segmentation using an active shape model;

[0057] Figure 4B shows calculation of shape loss;

[0058] Figure 4C shows area loss;

[0059] Figure 5 shows active shapes fitted to calculate lateral and medial cortical metrics;

[0060] Figure 6A shows isolation of the neck region, trochanter region and the intertrochanter region;

[0061] Figure 6B shows the midpoint; and

[0062] Figure 7 shows a Bland-Altman plot of the agreement between the present method and dual energy x-ray absorptiometry (DXA) across a range of bone heaths (as recorded by the t-score).

[0063] Detailed Description of the Invention

[0064] Referral is made to Figure 1 in which a flow chart of procedures that integrate into radiological services to provide a measure of bone health. The procedures are included in stages.

[0065] Stage 1. Region-of-interest locator stage

[0066] The first stage is shown as an embodiment of Figure 1 as a region of interest locator stage 1. In the embodiment the ROI locator stage 1 comprises a Visual Geometry Group VGG ROI locator 10. All x-ray images are passed to the ROI locator.

[0067] An example of an implementation of the region-of-interest (ROI) locator stage 1 is illustrated by Figures 2E and 2F. Figure 2F shows that the implementation determines a ‘crop’ from a pelvic x-ray image that contains a femoral head 263. The crop has a border that is a border of a region of interest containing the femoral head 264.

[0068] The ROI locator stage 1 is not limited to only finding an ROI from a pelvic x-ray image because the data preprocessing I cleaning stage 2 and region of interest locator stage 1 may be developed to target other body parts.

[0069] A full pelvic x-ray image is scaled (e.g., to a pixel range). Points of interest around the femoral head are output, Figure 2E, and these points of interest determine the ROI.

[0070] In an embodiment, four points of interest are output at consistent locations: the lower shaft, the lesser trochanter, the greater trochanter, and the furthest extremity of the femoral head. A final ROI is generated using these points and a border placed around them. The border may have a square or rectangular form as shown in Figure 2F.

[0071] Each of the x-ray images in which the border around the region of interest is defined is provided as an input to the second stage or third stage. Either the whole x-ray image may be provided or only the region of interest within the border. In an embodiment of the region of interest locator stage 1 as shown by Figures 2E and 2F, pelvic x-ray images including the crop that contains the femoral head 263 are provided.

[0072] Stage 2, Data pre-processing I cleaning stage

[0073] Figure 1 shows the data pre-processing I cleaning stage 2 starting with a ROI body part checker 20. In stage 2, cropped ROI images from x-rays images are input to the ROI part checker 10. The cropped ROI images are produced by stage 1.

[0074] In an embodiment the ROI body part checker 20 classifies the ROI image into left hip, right hip, or other. ‘Other’s are rejected.

[0075] The raw x-ray images 5 include usable images and unusable images. The data preprocessing I cleaning stage 2 detects which of the raw x-ray images 5 are unusable images. The unusable images are sequestered. The usable images are subjected to further processing in the stages which follow.

[0076] The raw images may include x-ray images of many different persons’ various body parts. The procedures may be intended to measure bone health based on x-ray images of specific body parts including a pelvis. In an embodiment the intended anatomical body is the pelvis or a femur joined to the pelvis.

[0077] Unusable images include, for example, mis-labelled x-ray images e.g., a hand x-ray image mis-labelled as pelvis. Another example of unusable images are x-ray images of a femur with metalwork that occludes the femur.

[0078] According to an embodiment which uses the ROI checker 20 to detect mis-labelled x- ray images: a) a Generative Adversarial Network (i.e., GAN) is applied to classify ROI images into anatomical sites, for example into ‘left hip, ‘right hip’ or ‘other’. X-ray images can be flagged or rejected based on the classification. In an exemplary embodiment, a Bidirectional GAN (i.e., BiGAN) is used. In training on synthetic x-ray data: i. an image-to-image translation model, such as Dual Contrastive Learning GAN (i.e. , DCLGAN), is used to generate examples of x-ray images of a specific anatomical site, such as the pelvis; ii. The synthetic x-ray image is cropped to an ROI around the femur; ii. the BiGAN is trained to generate more images of this anatomical site, such that its internal discriminator model can discriminate between images of the intended anatomical site and incorrect images; iii. the BiGAN discriminator and encoder is extracted and used as a stand-alone inference model to detect mis-labelled x-ray images; b) if the GAN model predicts the x-ray image subject is not as intended, i.e., mislabelled, the x-ray image can be flagged or rejected or sequestered for example as an unusable image of an incorrect body part 26.

[0079] As shown in Figure 1 , an x-ray image of a subject that is as intended is categorized as having a correct body part 24.

[0080] The x-ray images with the correct body part are provided to the third stage or in an embodiment they may be provided directly to the fourth stage.

[0081] Stage 3. Metalwork detection stage

[0082] The third stage is shown as an embodiment of Figure 1 as a metalwork detection stage 3. It is not necessary to include the metalwork detection stage. Rather, the x-ray images in which the border around the region of interest is defined according to the second stage, may be provided directly to the fourth stage.

[0083] In the embodiment the metalwork detection 30 operates on the x-ray images provided from the VGG ROI locator 20 to detect whether there is any metalwork within any body part that is imaged in the x-ray image.

[0084] The x-ray images in which no metalwork is detected are sequestered in a group of no metalwork 32. The x-ray images in which metalwork is detected only outside of the region of interest are sequestered in a group of metalwork elsewhere 34.

[0085] The x-ray images in which metalwork is detected in the region of interest are sequestered in a group of metalwork in all ROI’s 36. The x-ray images that are sequestered in this group may have metalwork inside the ROI and outside the ROI, or they may have metalwork only inside the ROI.

[0086] According to an embodiment to implement the metalwork detection 30, a LINET that was trained on DCL GAN synthetic x-ray images and added metalwork synthetics, segments metalwork.

[0087] According to an embodiment to implement the metalwork detection 30, metalwork is detected in the ROI as follows: aa) a selective search graph segmentation is used to find connected regions of the x-ray image which have similar textures; bb) a CNN is applied to classify the connected regions which have similar features. For example, they may be classified into 4 metalwork classes and 1 non-metalwork class. In an exemplary embodiment a Visual Geometry Group CNN (i.e. , VGG CNN) is used. In training on synthetic x-ray data: i. an image-to-image translation model, such as a DCLGAN model, is used to transfer a ‘hand-made’ bone label image as shown in Figure 2A to a realistic image as shown in Figure 2B. In an embodiment the ‘hand-made’ image is generated from the statistics of an active shape model: realistic femur shapes are produced and simpler randomisations on templates (e.g., of pelvic and spine shapes). Realistic images, as in Figure 2B, are selected from pelvic x-ray data or partial pelvis x-ray images. The model is saved at different stages to produce a suite of models, e.g. DCLGAN , CUT etc., that generate a larger variety of x-ray images. For example, the number of fat folds, length of exposure etc. ii. Using the known bone structure, a synthetic metalwork is added to the x-ray image as shown in Figure 2D. A set of metalwork templates as shown in Figure 2C is added in realistic positions; iii. Having identified the location and type of metalwork, the VGG CNN is trained to classify regions of images into one of a plurality of classes: hip replacement; pelvic shield; dynamic hip screw; screw and plate; and non-metalwork, In an embodiment that uses UNET, a known mask of the metalwork location is a training target, and the U NET is trained to segment the metalwork from a full synthetic image; cc) if metalwork affects all femur locations of interest i.e., if the metalwork overlaps with the ROI model prediction, as shown for example by Figure 2D, the image is rejected from the pipeline and not advanced to the next stage.

[0088] In an embodiment the regions are selected using a Kullback-Leibler (KL) divergence which relies on the metalwork segments being anomalous compared to the rest of the image, e.g., very bright, low texture etc., compared to the rest of the image. This embodiment may be used for the VGG approach.

[0089] In an embodiment a verification stage determines the quality of the ROI, including positioning.

[0090] As shown in Figure 1 after the third stage of metalwork detection 3, the x-ray images in the groups of no metalwork 32 and metalwork elsewhere 34 are advanced to a fourth stage.

[0091] 4. Ensemble segmentation stage

[0092] The fourth stage is an ensemble segmentation stage 4. In an embodiment shown in Figure 1 , femur segmentation 40 segments the proximal femur found in the ROI locator stage 2 wherein there is no metalwork or only metalwork elsewhere as determined in the third stage.

[0093] The femur segmentation 40 attempts to segment the ROI using a CNN. An embodiment of the femur segmentation 40 which utilizes ll-Net in the CNN is shown in Figure 3A. In an embodiment the ll-Net model is trained on synthetic data (augmented to improve model robustness) comprising randomized control points connected with Bezier curves to resemble bone shapes.

[0094] As shown in Figure 3A, an x-ray image plus ROI 310 is submitted to a segment original 320 routine. If segmentation fails, contrast manipulations are applied to the image, e.g., gamma correction, and a segmentation attempted as shown by segment gamma 330. If segmentation fails using gamma correction, then segmentation is attempted upon applying contrast limited adaptive histogram equalization correction, CLAHE, to the image. If all image versions fail, the ROI is expanded to include more of the image to enlarge the context as by a resize of the image ROI 350. Then all image versions are tried once more by subjecting them again to the routines of segment original 320, segment gamma 330, and segment CLAHE 340. If all attempts fail on the enlarged context, the image is rejected 360.

[0095] Rejection may be due to poor contrast, artefacts, lack of femur etc., and is determined by a shape loss and / or area loss score. In a preferred embodiment the ensemble segmentation stage 4 minimizes and corrects common image quality issues including, for example, but not limited to: over exposure, under exposure, or variations in brightness. This reduces rejection.

[0096] As shown in Figure 3A, a segmentation may be deemed adequate to pass 322, 332, 342 by any of segment original 320, segment gamma 330, and segment CLAHE 340 routines. Then the image and segmentations which have passed continue 370 on from the ensemble segmentation stage 4 to a fifth stage.

[0097] 5. Active shape verifier stage

[0098] The fifth stage is an active shape verifier stage 5. An active shape verifier 50 has an active shape model to verify the segmentations in the ensemble segmentation phase. In an embodiment the active shape model generated in stage 1 comprises points that define the outer contour of a femur 407 including the lower shaft 407, the lesser trochanter 409, the greater trochanter 408, and the furthest extremity of the femoral head 406. For example, in an embodiment shown in Figure 4A, there are fifty to seventy points. Using a fitting algorithm and the U-Net outputs from the fourth stage, the ‘best-fit’ realistic femur shape is determined. The shape loss score and the area loss score are calculated to determine how well the active shape matches the segmentation i.e., how realistic the segmentation is.

[0099] The shape loss score relates a distance in the active shape to a nearest edge in the segmentation. This is low when the shape tightly fits the border of the mask, and high where the mask cannot accurately be represented by a realistic proximal femur shape. Shape loss is calculated from the sum of the distances of each point in the active shape to the nearest edge found in the segmentation. The shape loss score is calculated as the total perpendicular distance between each of the control points of the transformed parameterised shape, which is to say the active shape, and the segmented area, normalised by the number of control points and image pixel spacing. The shape loss is the average perpendicular distance 429 from each active shape point 428, 425, 426 to the nearest edge 426 in the segmentation mask, as shown in Figure 4B, calculated in millimetres to normalize between different x-ray resolutions.

[0100] The area loss score may be computed as the ratio of the symmetric difference of the set of pixels in the transformed parameterised shape and the segmented area of the body part, to the total number of pixels in the segmented area. Area loss is the total percentage of segmentation area that is incorrectly identified by the active shape. This includes the active shape borders 433 and the area inside 434 the active shape that is not part of the segmentation mask 431 as shown in Figure 4C. The area 436 for which the segmentation mask and fitted active shape do not agree is outside the shape. The area loss score is lower for shapes that tightly match the segmentation compared to shapes that do not encompass much of the mask or those that leave large gaps inside their outline.

[0101] As per the ensemble segmentation stage 4, poor segmentations are reprocessed in the active shape verifier stage 5 until adequate or rejected.

[0102] As shown in Figure 1 , the image and segmentations which have passed are in a segmentation passed group 54 which is sequestered from the image and segmentations which have been rejected and are in a segmentation failed group 56. The x-ray images and segmentations in the failed group 56 proceed no further.

[0103] The x-ray images and segmentations in the passed group 54 that are to be subjected to cortical segmentation 60 proceed directly to a sixth stage and then to a seventh stage for active shape verification.

[0104] The x-ray images and segmentations in the passed group 54 that are to be subjected to lesser trochanter segmentation 80 proceed directly to an eighth stage and skip over the sixth and seventh stages.

[0105] In an embodiment a copy is made of the x-ray images and segmentations in the passed group 54 so that there is a primary and a duplicate of the primary of each of the x-ray images and segmentations. The primary of each of the x-ray images and segmentations proceeds to the sixth stage and the duplicate proceeds directly to the eighth stage.

[0106] 6. Cortical segmentation and verification stage

[0107] The sixth stage is a cortical segmentation stage 6. Cortical bone is segmented. Cortical segmentation is implemented for example with a synthetically trained ll-Net model.

[0108] 7. Active shape verification stage

[0109] As shown in Figure 1 , the cortical segmentation 60 is followed by an active shape verifier 70. The cortical segmentation 60 is verified using active shapes. In an embodiment , medial and lateral cortical bone is visible in anterior-posterior x-rays, and separate shapes are fitted to each.

[0110] Figure 5 shows active shapes fitted around a range of inner and outer cortical segmentations. The cortical portion includes the osseous tissue that forms the outermost layer of bone. The x-ray images in the left column of Figure 5 show accepted examples of active shapes fitted around a range of inner and outer cortical segmentations, i.e., medial, and lateral active shapes. The x-ray images in the left column of Figure 5 show rejected examples.

[0111] Figures 5 top left and right x-ray images show a region of interest (ROI) and the corresponding segmentations 516, 514 and 566, 564.

[0112] Figure 5 middle left and bottom left x-ray images both show a respective accepted example of segmentation. In the middle-left x-ray image, region 526 has a shape loss of 1.64, and region 536 has area loss of 19.98%, compared to the original image. In the bottom left ray image region 524 has a shape loss of 0.45 and region 534 an area loss of 9.51 %.

[0113] By contrast, the shape loss and area loss for a rejected example are much greater. Figure 5 middle right and bottom right x-ray images show examples that would be rejected. As seen, the segmentation in the rejected example is much less smooth than the segmentation in the accepted example of Figure 5 top left image. Figure 5 middle right and bottom right x-ray images show different segmented regions for the ROI. In Figure 5 middle right x-ray image, region 576 has a shape loss of 3.73, and region 586 has area loss of 56.58%, compared to the original image. In Figure 5 bottom right image, region 574 has a shape loss of 1.24 and region 585 an area loss of 45.8%.

[0114] The segmentation is adequate when the area loss and shape loss scores for both the medial and lateral active shapes are within acceptable thresholds. When, as shown in Figure 5, cortical segmentation is adequate, the x-ray images with adequate cortical segmentation proceed to the seventh stage.

[0115] 8. Lesser trochanter and dual energy x-ray absorptiometry (DXA) region cropping stage

[0116] The eighth stage is a lesser trochanter segmentation stage. The eighth stage outputs lesser trochanter segmentation 80 as shown in Figure 1. The lesser trochanter segmentation 80 uses the same ll-Net network for segmentation of protruding and hidden lesser trochanter. Training data is fully synthetic to be x-ray apparatus agnostic.

[0117] In some embodiments the lesser trochanter and dual energy x-ray absorptiometry (DXA) region cropping stage precedes a bone mineral density, BMD, inference stage.

[0118] 9. Lesser trochanter segmentation check

[0119] The ninth stage checks the plausibility of the lesser trochanter segmentation. This consists of discarding segmentations that have an area less than a given threshold. This threshold can be determined from a hyperparameter search on validation data.

[0120] 10. DXA region annotation

[0121] The DXA region cropping step isolates the neck region 603, trochanter region 608 and the intertrochanter region 605. Figure 6A shows an example with a pelvis 604 which receives a femoral head 606 that merges with the neck region 603. The neck region 603 merges with the greater trochanter 608 and the intertrochanter 605. The lesser trochanter 609 is visible as a bump on the introchanter 605. The femur lower shaft 607 merges with the intertrochanter 605 distal from the neck 603 and greater trochanter 608.

[0122] With reference to Figure 6B, the narrowest thickness of the neck has a midpoint 673 that is determined by iteratively eroding the segmentation until the narrowest point is breached. The femoral head axis is defined as the perpendicular bisector through the midpoint 673 of the narrowest thickness of the neck. At opposite sides of the narrowest thickness are active shape points P3 663 and P4 664.

[0123] The greater trochanter 638 merges with the neck at an active shape point P1 661. The neck region is defined using the active shape point P1 661 and intersection of bounding lines with the active shape points P2 662 ,P3 663 , and P4 664. The trochanter / intertrochanter boundary is defined using femur active shape point P5665 and the centre-point 671 bisecting neck region between the active shape points P1 661 and P2 662. The bottom 636 of the intertrochanter is defined as a distance from the lesser trochanter points P6 666 and P7 667, e.g., 1 cm below.

[0124] The cortical bone segmentation enables accurate calculation of the proportion of the area of the proximal femur and femur shaft occupied by cortical bone. The ratio of cortical bone to trabecular bone in the at least one portion of bone in the region of interest may also be determined. In an embodiment it is determined as the ratio of the area of the cortical segments and the area of the proximal femur within the intertrochanteric region. This metric is chosen as it is found that the Bone Mineral Density (BMD) of the inter-trochanteric region in Dual Energy X-Ray Absorptiometry (DXA) scans contributes most to the overall BMD.

[0125] 11. Bone Mineral Density, BMD, inference stage

[0126] In the BMD inference stage 11 , a convolutional neural network (CNN) or vision transformer (ViT) with a single output neuron predicts a continuous bone mineral density, BMD, value from an image This is known as the BMD inference model.

[0127] In one embodiment, the input image is a region of the bone determined by stage 10, the DEXA annotation step. The input region may be the intertrochanter, neck, greater trochanter, or any combination thereof, the combination of all these regions being the “total” DXA region. These regions are shown in Figure 6A. The input image is generated by “masking” the image outside of this specified region i.e. setting the image pixel values outside of this region to black or another constant value. This removes surrounding information which are not related to bone mineral density. The image is then input to the BMD inference model to make a prediction.

[0128] In another embodiment the specified regions may be extended slightly to, for example, include the 1cm around the DXA region. With the image fading to black around this region. This is to give the network some context and compensate for the effect of soft tissue. 1 cm is only an example, and other values could also be chosen.

[0129] In another embodiment. The region is instead generated directly from stage 4, the femur segmentation. The rest of the image outside of this region is masked in the same way, with the rest of the image fading to black immediately outside of the segmentation, or gradually after some small distance.

[0130] In another embodiment, the input region is taken directly from stage 2, and the cropped ROI is passed to the BMD inference model. The subsequent stages only being used to determine if the ROI was suitable. If the image passes some or all the subsequent checks (stages 3, 5, 7, 9), the image is passed to the BMD inference model to make a prediction.

[0131] In each of these scenarios, the BMD inference model predicts bone mineral density directly from the input image.

[0132] In another embodiment, the BMD inference model also takes other clinical risk factors such as age and gender as an input.

[0133] In an embodiment, the network classifies the image into any number of categories: for example, osteoporotic, osteopenic, non-osteoporotic, or healthy / non-healthy.

[0134] Embodiments may comprise transfer learning, where the network is pretrained on unseen images, for example, images that have been cleaned using the previous steps. The network can then be refined using the labelled data. For example : a) masked auto-encoding of images. Where the input is the image with some masking applied and the target is the unaltered image. b) training to predict other data such as age and / or gender or other features known to be related to bone mineral density. Here the input is the image, and the target is age or another related feature. c) knowledge distillation from another model. The other model may have been trained using features such as age, gender, cortical ratio, trabecular structure etc. Here the input is the unaltered image, and the target is the bone mineral density as predicted by the another model on these unlabeled images.

[0135] Embodiments may also comprise multitask learning, where the model is trained on and outputs multiple predictions, such as total and femoral neck BMD.

[0136] The invention has been described by way of examples only. Therefore, the foregoing is considered as illustrative only of the principles of the invention. References herein to a CNN, LINET, VGG, BiGAN and related elements of artificial intelligence are as adapted by a person skilled in the arts of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation shown and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope of the claims.

Claims

Claims:1 . A method of measuring bone mineral density wherein; at least one Region Of Interest, ROI, in x-ray images is identified, and the x-ray images and the ROI(s) are classified according to those with an intended anatomical body part; a first segmentation in the x-ray images is created that approximately conforms to a shape of a targeted feature of the intended anatomical body part within the ROI(s); the shape is quantitatively verified to resemble the targeted feature using a first metric determined from a perimeter of the first segmentation; a second segmentation in the x-ray images is created that approximately conforms to cortical bone of the targeted feature; parameters describing the shape, structure, and composition of bone are mapped to the x-ray attenuations to derive a metric correlated to bone mineral density.

2. A method according to claim 1 wherein a generative adversarial network, GAN, is applied to classify the x-ray images.

3. A method according to claim 2 wherein an image-to-image translation model is used to generate examples of x-ray images of specific anatomical body parts, a bidirectional GAN, BiGAN, is trained to generate more images of the specific anatomical body parts, and an internal discriminator of the BiGAN is used to discriminate between x-ray images of the intended anatomical body part and x-ray images of other anatomical body parts.

4. A method according to any preceding claim including cropping the x-ray images leaving only the ROI within the x-ray images.

5. A method according to claim 4 leaving only the ROI from the x-ray image of a pelvis and containing a femoral head within the ROI.

6. A method according to any preceding claim including providing the x-ray images classified as having the intended anatomical body part to a Visual Geometry Group, VGG, ROI locator.

7. A method according to claim 6 including placing points around a femoral head to determine the ROI.

8. A method according to claim 7 wherein the points include a point at each of a femur low shaft, lessor trochanter, greater trochanter, and furthest extremity of the femoral head, and generating the ROI by placing a border around the points.

9. A method according to any preceding claim configured to detect metalwork within the ROI by applying a CNN to classify connected regions of the x-ray image which have similar textures into one or a plurality of metalwork classes and a non-metalwork class.

10. A method according to claim 9 including using a third segmentation that is a selective search graph segmentation to find the connected regions.

11. A method according to claim 9 or 10 including adding a synthetic metalwork to the x-ray image.

12. A method according to claim 9, 10, or 11 including training the CNN as a Visual Geometry Group CNN.

13. A method according to any preceding configured to detect metalwork that uses LINET trained on DCLGAN synthetic x-ray images and added metalwork synthetics.

14. A method according to any preceding claim wherein upon determination that the first segmentation is insufficient, contrast manipulations as a gamma correction are applied to the x-ray image and a first repeat segmentation is attempted.

15. A method according to claim 14 wherein upon determination that the first repeat segmentation is insufficient, contrast manipulations as a contrast limited adaptive histogram equalization correction are applied to the x-ray image a second repeat segmentation is attempted.

16. A method according to any preceding claim wherein the first metric includes a shape loss score or an area loss score of the first segmentation.

17. A method according to any preceding claim whereupon determining the first metric is insufficient, then correcting the x-ray image and repeating the first segmentation.

18. A method according to any preceding claim wherein the first metric relates a distance in the shape to a nearest edge in the first segmentation.

19. A method according to claim 18 wherein to determine the first metric an active shape model comprises points that define the outer contour of a femur including a lower shaft, a lesser trochanter, a greater trochanter, and a furthest extremity of a femoral head.

20. A method according to any preceding claim including determining a cortical ratio from an estimate of a proportion of cortical bone in an intertrochanteric region.21 . A method according to any preceding claim wherein the second segmentation is a cortical segmentation that is implemented with a synthetically trained ll-Net model.

22. A method according to any preceding claim wherein the second segmentation is adequate when an area loss score and a shape loss score for both medial and lateral active shapes are within acceptable quantitative thresholds.

23. A method according to any preceding claim that includes a fourth segmentation of a lesser trochanter ROI or intertrochanter ROI in the x-ray images.

24. A method according to any preceding claim that includes a crop of the x-ray image in a neck ROI, a trochanter ROI, and an intertrochanter ROI.

25. A method according to any preceding claim wherein a CNN or vision transformer with a single output neuron predicts the bone mineral density.

26. A method wherein a ratio of cortical bone to trabecular bone in the at least one portion of bone in the region of interest is determined.

27. A method according to any preceding claim wherein the intended anatomical body part in the x-ray image is classified from the bone mineral density as osteoporotic, osteopenic, non-osteoporotic, or healthy / non-healthy.

28. An x-ray image of an anatomical body part wherein the bone mineral density is classified according the method disclosed in any of claims 1 to 27.

Images