System and method for shape extraction

Abstract

A system and method are disclosed to estimate bone health wherein a cross-sectional shape of cortical bone is segmented and parameterised and regions of internal bone extracted. Typically, two regions of internal bone structure are extracted and quantified (cortical and trabecular), or three regions of internal bone structure are extracted and quantified (cortical, endocortical and trabecular). Metrics including, but not limited to, absolute volume, relative volume, volumetric bone mineral density (BMD) and areal bone mineral density (BMD) are measured from the extracted regions. The system and method accurately estimate bone health as an incidental finding from standard imaging procedures. It is robust to several imaging acquisition techniques.

Description

[0001] SYSTEM AND METHOD FOR SHAPE EXTRACTION

[0002] Field of the Invention

[0003] The present invention relates to a system and method for improved estimation of bone health wherein a cross-sectional shape of cortical bone is segmented and parameterised and regions of internal bone extracted.

[0004] It is an advantage that the system and the method are robust to different imaging acquisition techniques.

[0005] It is a further advantage of the system and the method that the use a physics model to generate synthetic volumes to create synthetic images that simulate acquisition from multiple angles whereby a known volumetric ground truth enables accurate extraction.

[0006] Background

[0007] Osteoporosis (the term is used here to refer singly to all types of conditions on the low bone mass spectrum, including osteopenia) can affect most of the skeleton and / or a localized area. Low bone mass (bone mass being referred to here as bone mineral density (BMD)) causes the bone to be more fragile, increasing the risk of fracture and other skeletal diseases.

[0008] Worldwide, osteoporosis is estimated to affect one in three women and one in five men over the age of 50 (International Osteoporosis Foundation, 2023). With the ageing of the global population and persistent changes in lifestyle habits, the incidence of osteoporosis and related fractures is expected to increase markedly in the future.

[0009] However, in many countries, osteoporosis is under-diagnosed and under-treated, even in high-risk patients or patients who have already experienced fracture.

[0010] Improved methods for early identification and appropriate treatment are thus imperative : it can reduce both the quantity and severity of fragility fractures, related health problems and cost.

[0011] Bone health is assessed via various medical imaging modalities : quantitative ultrasound (QUS), peripheral quantitative computed tomography (pQCT), high- resolution peripheral quantitative computed tomography (HR-pQCT), magnetic resonance imaging (MRI), digital X-ray radiogrammetry (DXR), quantitative computerised tomography (QCT) and dual-energy X-ray absorptiometry (DXA). Of these quantitative computerised tomography (QCT) and dual-energy X-ray absorptiometry (DXA) are the most commonly used.

[0012] In quantitative computerised tomography (QCT), a CT scanner is calibrated using solid phantoms (e.g. of calcium hydroxyapatite) that represent various bone mineral densities, placed under the patient. Via this calibration, Hounsfield units are converted to bone mineral density values. The technique can be used for both peripheral and central bone mineral density (BMD) measurements, with lumbar spine and hip being preferred locations. Among the advantages of quantitative computerised tomography (QCT) are that it is not confounded by body size, atherosclerotic calcifications, or degenerative changes of bone, and can be used for both trabecular and cortical bone.

[0013] Generally during a dual-energy X-ray absorptiometry (DXA) scan, two x-ray beams of different energy levels are passed through a patient, usually the vertebrae or femur, or occasionally the forearm. The difference in total attenuation of the two x-ray beams is used to subtract the attenuation by soft tissue, leaving only the attenuation by bone, from which estimations of bone mineral density (BMD) is calculated in g / cm2, and then compared against two reference population giving two scores: T-score: comparison by standard deviation to a young adult population, matched for sex and ethnicity and classified by >-1.0: normal, <-1.0 to >-2.5: osteopenia, <-2.5: osteoporosis, <-2.5 plus fragility fracture: severe osteoporosis; and Z-score compared by standard deviation to an age, sex, and ethnicity population.

[0014] Some dual-energy X-ray absorptiometry (DXA) systems comprise an x-ray energy sensitive detector that can produce a high and low energy images from a polyenergetic beam.

[0015] However, such modalities are generally expensive, time consuming and entail varying, often high, radiation dose, which hinders frequent use. Furthermore, image resolution is limited, which hinders accurate assessment, for example, of the cortex. The cortical cortex (the outer, compact bone tissue) is a thin structure. Cortical bone mineral density (BMD) is affected by both the degree of mineralisation of bone and porosity of the cortical bone. The degree of mineralisation of bone is an intrinsic material property of bone tissue, while porosity is a property of the microarchitecture of bone. Cortical bone mineral density (BMD) increases with degree of mineralisation and decreases with porosity, while fracture risk decreases with bone mineral density (BMD) and increases with porosity. Hence a decrease in bone mineral density (BMD) due to either mechanism increases fracture risk but the relative contributions of each to bone mineral density (BMD) cannot be distinguished at low resolution. Instead, it is only possible to measure an average cortical bone mineral density (BMD).

[0016] Model-based techniques are known that aim to resolve errors resulting from the limited resolution and to more accurately estimate bone mineral density (BMD) (V. Wong, F. Caliva, F. Su et al, ‘Comparing bone shape models from deep learning processing of magnetic resonance imaging to computed tomography-based models’, JSES International (2023); S; Jang, P. M. Graffy, T. J. Ziemlewicz et al ‘Opportunistic osteoporosis screening at routine abdominal and thoracic CT’, Radiology 2019; 291 :368-369). For example, several such techniques use a transect of cortical and trabecular bone sampled from a medical scan, a parametrised estimate of the density and an estimation of the imaging blur, to model the cortical features using deconvolution.

[0017] Three-tier cortical bone mapping (CBM) models are known that represent the density in profile across the cortex at one or more surface locations (G. Treece, A. Gee, ‘Cortical Bone Mapping: Measurement and Statistical Analysis of Localised Skeletal Changes’ 2018, Current Osteoporosis Reports 16: 617-625). A number of local measurements are made over the surface of the bone, which are combined to provide surface maps of the measured features.

[0018] CBM allows detection of focal regional variations between populations, however, is also only measures the cortical thickness, the cortical bone mineral density (BMD) and the trabecular bone mineral density (BMD) as single aggregate values across the entire cortical or trabecular region included in the profile. A further limitation is where for example the cortical density varies from global estimates, which might increase the bias of cortical thickness for thin structures, even when corrected, for example, via a two-phase, model-based fitting technique.

[0019] Statistical shape and density modelling (SSDM) has been used to develop three- dimensional bone geometry and density distribution models for better prediction of individual fracture risk. For example, Bredbenner et al (T. L. Bredbenner, R. L. Mason, L. M. Havill et al, ‘Fracture Risk Predictions based on Statistical Shape and Density Modelling of the Proximal Femur', J Bone Miner Res. 2014 September ; 29(9): 2090- 2100) identified subtle changes in combinations of structural bone traits (e.g. geometric and bone mineral density (BMD) distribution traits) that appeared to indicate fracture risk: their SSDM-based prediction model, adjusted by age, correctly identified 55% of the fracture cases (and 94.7% of the non-cases), whereas the clinical standard bone mineral density (BMD) correctly identified 10% of the fracture cases (and 91.3% of the non-cases).

[0020] The utility and reliability of predictive models depends on the quantity and curation of the data, and ground-truth.

[0021] The potential benefits of synthetic data in healthcare to resolve the challenges of data acquisition, quantity, diversity, organization, robustness, adaptability, labelling, and privacy concerns are known (D. McDuff, T. Curran and A. Kadambi, ‘Synthetic data in healthcare’ 2023, arXiv:2304.03243v1 [cs.AI]). Synthetic data can be used as a sole data source or to supplement and augment real data, for example, for generative models and can be generated using a number of different approaches: physical models, statistical models and sim2real models.

[0022] Physical models require knowledge of how to parameterise the data generation process. Such models have the advantage of being interpretable and having known constraints on the samples they can and cannot produce. However, model mismatch can occur in complex scenarios and a physical model does not reflect all the variance that might be observed.

[0023] Statistical generative models capture a probabilistic representation of a dataset from which samples can be drawn. Often referred to as “generative Al”, these are trained to mimic the distribution of a dataset. As they are trained models, albeit generally unsupervised, they are a function of the data used to create them. If the data is representative, statistical models capture more faithfully the distribution of data observed than physical models and are less complex to create than some physical simulators.

[0024] When models perform differently in the real domain as compared to the synthetic domain a so-called ‘sim2real’ domain gap exists. In healthcare, it is critical that any form of synthetic patients generalize to real data. Frameworks to minimise domain gap include domain randomization (which perturbs the generation of simulated data, thereby shifting the distribution of the generated data), domain adaptation (where features from real world data area warped to match features from simulators) and differentiable simulation (which aims to form a simulator whose parameters can be optimized and learning is used to set the optimal parameters of the simulator). Both domain adaptation and domain randomization are a form of transfer learning, which involves training a model on data from one domain that will be tested on data from another domain.

[0025] Styles can be transferred on to images to improve segmentation models, for example, using GANS. If styles are transferred on to two images of the same bone taken from different acquisition angles, the pixel values which represent the attenuation of bone would not correspond. This is not a significant problem for image segmentation where the interest is 2D shape, however, it is important that any model which is intended to predict composition or 3D shape from two or more acquisition angles has as input images which correspond to the same underlying ground truth.

[0026] A further challenge is the significance of the endocortical region. The endocortical region is the region between the compact cortical bone and the porous trabecular bone, and as indicated by several studies (e.g. R. A. Pearson, G. M. Treece, ‘Measurement of the bone endocortical region using clinical CT’, Medical Image Analysis 2018; 28-40) measuring the endocortical region may provide early indication of deteriorating bone quality and / or an accurate measure of targeted regeneration via therapies. of the Invention

[0027] According to a first aspect of the invention there is a method of shape extraction from medical scanner images of internal bone to estimate metrics of bone health, wherein a cross-sectional shape of the internal bone in each medical scanner image is segmented to reveal regions of bone structure and parameterized, and the regions are extracted from the medical scanner images. The method of shape extraction facilitates inferring metrics of bone health from the shapes of internal bone structures.

[0028] Further features of the invention including a system to execute the method are disclosed in the first claim and in the dependent claims.

[0029] The method may further comprise training a model to extract the metrics from x-ray projections. The model may be trained using simulated X-ray data which are generated from real or synthetic volumes which have known (i.e. ground truth) metrics / shapes as labels.

[0030] The method and system accurately estimate bone health as an incidental finding from standard or routine imaging procedures. The bone may be segmented and parameterized (selectively cortical and trabecular, or cortical, endocortical and trabecular), either for a section of interest or iteratively along the length of a bone. A statistical model or representation may be used to generate synthetic volumes and a physics model to create synthetic images that simulate acquisition from multiple angles to generate a volumetric ground truth.

[0031] In an embodiment, a cross-sectional shape of de-mineralized cortical bone is segmented and parametrized and two regions of internal bone structure are extracted and quantified (cortical and trabecular), and I or three regions of internal bone structure are extracted and quantified (cortical, endocortical and trabecular).

[0032] It is an advantage of the system and method that it is robust to several imaging acquisition techniques. It is a further advantage of the system and method that it may use a statistical model or representation to generate synthetic volumes and a physics model to create synthetic images that simulate acquisition from multiple angles. As such, a known volumetric ground truth ensures accurate matching and resolves the inherent inaccuracies of transferring a style to each individually. In an embodiment, a CNN is used to predict the cross-sectional shape of the cortical bone. The model may be trained on synthetic x-ray bone slices, labelled with corresponding cross-sectional shapes. These are generated from a statistical model or representation a cross-sectional shape of the internal bone.

[0033] CT slices or image slices may be medical scanner images or derived from them. The shapes may have been collected by segmenting and parameterizing the cortical bone in CT slices. By extension, the de-mineralised cortical bone is segmented and parameterised, so that the model extracts and quantifies three regions of the bone, namely the compact cortical cortex, the de-mineralised endocortical region and the medullary bone. The relative proportions of these regions reflect the patient’s bone health and density.

[0034] Metrics including, but not limited to, absolute volume, relative volume, volumetric bone mineral density (BMD) and areal bone mineral density (BMD) are measured from the extracted shapes.

[0035] The invention will now be described, by way of example only, with reference to the accompanying figures in which:

[0036] Brief Description of the Figures

[0037] Figure 1 shows shape extraction and parameterisation of the cortex of a femur bone from a CT scan.

[0038] Figure 2 shows comparison of synthetic bone slice (left) and real bone slice (right);

[0039] Figure 3 shows shapes predicted from real X-ray slices;

[0040] Figure 4 shows each row of plots (right) show shape predictions for adjacent bone slices;

[0041] Figure 5 shows density distribution onto an input segmentation image that has been created from the statistical shape model; and

[0042] Figure 6 shows shape generated from statistical representation of parameters of each region, and a synthetic image, and distribution transfer using GAN. Detailed of the Invention

[0043] Figure 1 comprises four images including: an X-ray image 96 of a hip that is left most; a medical scanner image that is a CT scan 97 adjacent to the X-ray image 96, a cross section 98 of the femur bone 2 from the CT scan 97, and segmentation 99 of the cross section 98 of the femur bone 2.

[0044] In the X-ray image 96 of Figure 1 is shown a femur bone 2 connected to a pelvis 4. Parallel lines 6 and 8 indicate cross sections of the femur bone 2. The parallel lines are separated by wafer 10 of the femur bone 2. The wafer 10 is referred to as a bone slice 10 or image slice of the internal bone. The wafer 10, i.e. the slice 10, is transverse to the femoral-shaft axis 504 of the femur bone 2 shown in Figure 5.

[0045] In the CT scan 97 of Figure 1 is shown a cross section of the hip with a surface of the slice 10 on the cortex region 12 highlighted. In the segmentation 99 of Figure 1 , the surface of the cortex region 12 is highlighted around a surface of a medullary bone region 14 of the slice 10.

[0046] Figure 2 shows in two side by side top images comparison of synthetic bone slice 201 (left) and real bone slice 202 (right). Synthetic data is used to capture the variety of cross-sectional bone shape between patients and ensure the model is robust to all image acquisition methodologies.

[0047] To generate realistic synthetic data, prior knowledge of bone shape is acquired by extracting and parameterising the cortical cortex from volumetric medical scans. For example the volumetric scans may be CT volumes as shown in Figure 3. The volumetric scans or CT volumes or CT slices may be the medical scanner images or derived directly from the medical scanner images.

[0048] The prior knowledge may be used to confirm or to ensure or to arrange that the image slices or CT slices are transverse to the femoral-shaft axis. Figure 3 shows three side by side sketches. On the left of Figure 3 is a sketch of four CT slices 301 , 302, 303, 304. In the middle of the Figure 3 is shown a sketch that is an amalgamation 305 of the four CT slices. The amalgamation 305 has a thickness from the top CT slice 301 to the bottom CT slice 304. On the left of figure 3 is a sketch 306 of the cross section of the amalgamation 305. The segmented regions of the four slices are stacked in the amalgamation 105.

[0049] The compact cortical cortex is segmented in each axial slice by thresholding techniques, then the shape of the segmentation is parameterised as an array of radii extending from the centre of mass at incremental angles. In an embodiment, the shape of the cortical cortex can be parameterised as an array of ninety radii (sixty outer points 401 and thirty inner points 402) as shown in Figure 4. By extension, the de-mineralised endocortical region can also be parameterised as an array of one hundred and five radii (sixty outer points 601 and thirty middle points 602 and fifteen inner points 603) as shown in Figure 6.

[0050] A statistical model is created from these parameterisations which captures the mean and variance of bone shape(s).

[0051] In an embodiment, a 3D Morphable Model (3DMM) is created of the entire bone by segmenting the bone in CT volumes, converting to meshes and labelling a number of corresponding anchor points for each. The 3DMM is a statistical model of these points after Generalised Procrustes Analysis (GPA) which removes any translation, rotation, reflection and scaling variance, so that the points reflect shape only.

[0052] The 3DMM is used to generate new synthetic volumes with substructures (including cortical and endocortical thickness) which can then be projected (to mimic an x-ray) and used as training data for a variety of applications.

[0053] An example of a shape of region generated by a statistical model is shown in Figure 6 leftmost sketch. The shape of the region is an example of new cross-sectional shapes that are generated from the statistical model which are similar to the real extracted shapes.

[0054] As shown in Figure 6, the different bone regions defined by a newly generated shape are given mean tissue attenuations to create synthetic axial slices.

[0055] In an embodiment, the spatial distribution of bone density is modelled in the synthetic slice using a Generative Adversarial Network (GAN). Using either unpaired training data (for network structures like CycleGAN, BiGAN, DCLGAN) or paired training data (for networks like cGAN), a generator is trained which transfers a realistic density distribution onto an input segmentation image that has been created from the statistical shape model, as shown in Figures 5 and 6.

[0056] Additionally, features are modelled into synthetic slices to mitigate features in x-rays which are not physically related to bone shape or composition, for example, soft tissue gradients. In an embodiment the features modelled in the synthetic slices include varying of fat / muscle quantities surrounding the bone, changing body shape, and adjusting the rotation of the bone.

[0057] This process is repeated for an adjacent set of bone slices to create synthetic bone volume. Alternatively, this is extended from transferring a 2D distribution onto a segmented image, to a 3D distribution onto a segmented volume.

[0058] In an embodiment to simulate an x-ray passing through the bone, the volume is projected along the coronal axis resulting in a synthetic x-ray bone image (Figure 2), for example as the sum of pixels along the axis or via physics model simulation.

[0059] Additional augmentation is applied to synthetic images to mitigate artefacts in x-rays which are not physically related to bone shape or composition, for example, detector noise and scatter. Augmentation in the project images comprises introducing noise and artefacts which mimic real x-ray images.

[0060] In an embodiment, a VGG inspired CNN is trained on the synthetic x-ray slices labelled with their corresponding cross-sectional shapes. The model extracts a cross-sectional shape for a one-centimetre x-ray slice and is repeated for adjacent slices to estimate a total three-dimensional shape of the proximal femur.

[0061] With reference to Figure 5, general properties of bone width and cortical thickness 508, 509, 510 are captured, as bone shape is accurately measured from synthetic slices and variation of shape from real bone slices 501 , 502, 503.

[0062] Instead of inferring properties from a predicted shape, further embodiments may utilise bone synthetics to directly predict metrics including, for example, cortical thickness, bone mineral density (BMD), and volume of each region.

[0063] The synthetic modelling can be extended to the entire femur, or to any bone which can be successfully segmented from CTs or similar three-dimensional acquisitions. The invention has been described by way of examples only. Therefore, the foregoing is considered as illustrative only of the principles of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation shown and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope of the claims.

Claims

Claims:

1. A method of shape extraction from medical scanner images of internal bone to estimate metrics of bone health, wherein a cross-sectional shape of the internal bone in each medical scanner image is segmented to reveal regions of bone structure and parameterized, and the regions are extracted from the medical scanner images.

2. A method according to claim 1 wherein the internal bone is de-mineralized, and the regions extracted include a compact cortical cortex region, a de-mineralised endocortical region, and a medullary bone region.

3. A method according to claim 1 or 2 wherein the regions extracted include a cortical structure region and a trabecular structure region.

4. A method according to claim 3 wherein the regions extracted include an endocortical structure region.

5. A method according to any preceding claim wherein the regions extracted are then quantified.

6. A method according to any preceding claim wherein the regions are extracted from the medical scanner images either for a section of interest or iteratively along the length of the internal bone.

7. A method according to any preceding claim using a statistical representation and physics to generate synthetic volumes.8 A method according claim 7 creating synthetic images from the synthetic volumes that simulate acquisition from multiple angles.

9. A method according to any preceding claim determining absolute volume of each of the regions extracted.

10. A method according to any preceding claim determining volumetric bone mineral density of each of the regions extracted.11 . A method according to any preceding claim determining relative volume or proportion of each of the regions extracted.

12. A method according to any preceding claim wherein the medical scanner images or the regions extracted are as slices of the internal bone.

13. A method according to claim 12 receiving the medical scanner images as CT images from a CT scanner.

14. A method according to claim 12 or 13 wherein the slices are mutually parallel and stacked.

15. A method according to claim 12, 13, or 14 wherein the medical scanner images are of a femur bone and oriented transverse to the femoral-shaft axis of the femur bone.

16. A method according to any preceding claim creating a statistical representation of parameters of each region, wherein each region has a shape specifiable in form by the parameters by which the region is parameterized.

17. A method according to claim 16 wherein the parameters include radii of the shape of each region, the radii extending from a centre of mass of the internal bone.

18. A method according to claim 16 or 17 capturing a mean and a variance to represent the shape of each region, from of the parameters of each region.

19. A method according to preceding claim creating a synthetic three-dimensional model of the internal bone by meshing each of the regions and labelling a number of corresponding anchor points for each to link the regions.

20. A method according to claim 19 conducting a Generalised Procrustes Analysis (GPA) removing translation, rotation, reflection and scaling variance of the regions directing the anchor points to reflect shape specifically.

21. A method according to claim 19 or 20 generating synthetic volumes from the synthetic three-dimensional model of the internal bone, the synthetic volumes having synthetic substructures that mimic the regions of the internal bone, and projecting the synthetic volumes to mimic an X-ray of the internal bone from a selectable angle.

22. A method according to claim 19, 20, or 21 generating synthetic cross-sectional shapes from the synthetic three-dimensional model of the internal bone, wherein synthetic cross-sectional shapes mimic shapes of the regions extracted from themedical scanner images.

23. A method according to claim 21 or 22 giving mean tissue attenuations to the synthetic volumes and predicting a spatial distribution of bone density.

24. A method according to claim 23 training a generator in a Generative Adversarial Network (GAN) to transfer the spatial distribution of bone density distribution onto a segmentation image created from the synthetic three-dimensional model.

25. A computer program configured to implement shape extraction from medial scanner images of internal bone to estimate metrics of bone health according to the method any preceding claim, wherein the computer program is directly loadable on the memory of a computer for controlling the method when said program is run on a computer.

26. A system configured to implement shape extraction from medical scanner images of internal bone to estimate metrics of bone health to execute the method according to any of claims 1 to 24, the system comprising a memory of a computer, the system configured to receive the medical scanner images from a reserve or directly from a scanner and load the medical images into the memory, the system comprising the computer for controlling the method, and a display or printer in communication with the computer to render data and selected images retrieved or created in execution of the method.

27. A system and method substantially as described in the written summary of invention and detailed description with reference to the background and accompanying figures.

Images