An improved method of, and apparatus for, ultrasound examination to extract fetal characteristics

Abstract

There is provided a method of performing ultrasound examination of a fetus in utero, the method comprising the steps of: a) acquiring a plurality of ultrasound images of the fetus in utero using an ultrasound imaging apparatus, the ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of a fetus in utero; b) utilising, on a computing system, a plurality of sets of the ultrasound images in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight; c) providing, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

Description

[0001] An Improved Method of, and Apparatus for, Ultrasound Examination to Extract Fetal

[0002] Characteristics

[0003] The present invention relates to an improved method of, and apparatus for, ultrasound examination to extract fetal characteristics. More particularly, the present invention relates to a method for ultrasound examination in order to estimate the weight of a fetus, which may be used in practice by medical professionals for the diagnosis and prognosis of potential pathologies.

[0004] The assessment of fetal growth and development is an essential element of high-quality obstetric care. Abnormal developmental weight characteristics such as fetal growth restriction and macrosomia (excessive fetal growth) are strongly associated with the risk of poor maternal-fetal outcomes. For example, the detection of growth restriction is understood to be the main preventable factor for reducing stillbirth, and timely planning of labour is essential to ensure good perinatal outcomes.

[0005] However, in countries where a third-trimester growth scan is not routinely performed, only 19- 32% of all growth-restricted foetuses are currently detected. Although detection is reported to increase with a third-trimester routine scan, there is still considerable variance in the accuracy of scans, depending on operator experience and skills, both of which may impact measurement error.

[0006] The conventional approach to fetal biometry uses the well-known Hadlock or Shepard formulas. This requires the acquisition of a number of B-scan (2D brightness scan) ultrasound images of the fetus from a number of predetermined orientations. An ultrasound operator then performs a number of measurements of fetal parameters by positioning calipers on the relevant image. The fetal parameters include biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC) and femur length (FL). These metrics are then inputted into specific equations based on statistical regressions and estimations based on population analysis in order to estimate fetal growth and weight.

[0007] However, there is significant variability in operator performance which can be affected by multiple factors including image quality and human error when measuring the fetal parameters. To minimise the operator variance when estimating fetal weight, automated methods for fetal biometries have been evaluated against measurements performed by clinician experts and determined to be equivalent. However, automation of existing metrics misses additional information available in the images such as subcutaneous fat, texture of organs, convolutions of the fetal brain and other features that contain information not captured when relying on crude measurements of fetal biometries.

[0008] The present invention aims, in embodiments, to address these issues.

[0009] The following introduces a selection of concepts in a simplified form in order to provide a foundational understanding of some aspects of the present disclosure. The following is not an extensive overview of the disclosure and is not intended to identify key or critical elements of the disclosure or to delineate the scope of the disclosure. The following merely summarizes some of the concepts of the disclosure as a prelude to the more detailed description provided thereafter.

[0010] Several preferred aspects of the methods and systems according to the present invention are outlined below.

[0011] According to a first aspect of the present invention there is provided a computer-implemented method of estimating fetal weight, the method comprising the steps of: a) providing a plurality of ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of a fetus in utero; b) utilising the image data in a first machine learning model comprising one or more neural networks to generate one or more latent space representations of the anatomical structure and appearance of the fetus and one or more predicted measurements of the anatomical structure of the fetus; and c) utilising the one or more predicted measurements and the one or more latent space representations in a second machine learning model to generate an estimated weight of the fetus.

[0012] In one embodiment, the generated one or more predicted measurements corresponding to the anatomical structure of the fetus comprise one or more of: head circumference (HC); biparietal diameter (BPD); abdominal circumference (AC); and femur length (FL).

[0013] In one embodiment, the plurality of ultrasound images of the fetus comprise: a head image of the fetus; an abdomen image of the fetus; and a femur image. In one embodiment, the head image, the abdomen image and femur image are taken in or close to respective standard planes.

[0014] In one embodiment, the first machine learning model comprises a plurality of sub-networks, a discrete sub-network being allocated for each of the head, abdomen and femur images.

[0015] In one embodiment, each sub-network comprises a first neural network for generating a respective latent space representation of anatomical structure and appearance and a second neural network for generating, from the respective latent space representation, one or more predicted measurements of the anatomical structure of the fetus.

[0016] In one embodiment, the first neural network comprises a convolutional neural network (CNN). In one embodiment, the second neural network comprises a dense neural network (DNN).

[0017] In one embodiment, the sub-network allocated to the head image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the head of the fetus and a predicted measurement of the head circumference (HC) and / or the biparietal diameter (BPD) of the fetus; and / or wherein the sub-network allocated to the abdominal image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the abdomen of the fetus and a predicted measurement of the abdominal circumference (AC) of the fetus; and / or wherein the subnetwork allocated to the femur image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the femur of the fetus and a predicted measurement of the femur length (FL) of the fetus.

[0018] In one embodiment, the second machine learning model comprises a dense neural network (DNN). In one embodiment, the method further comprises: d) generating an uncertainty estimate for the estimated fetal weight.

[0019] In one embodiment, step d) further comprises: e) applying one or more transforms to the plurality of ultrasound images to generate a set of augmented images; f) performing steps b) and c) using the set of augmented images to generate an estimate of fetal weight based on the set of augmented images.

[0020] In one embodiment, the steps e) and f) are repeated N times to generate N estimates of fetal weight. In one embodiment, the one or more transforms are randomly selected from one or more of: rotation; shear; translation; brightness; contrast; and horizontal flip.

[0021] In one embodiment, step a) comprises: g) generating a plurality of ultrasound images of the fetus in utero using an ultrasound imaging apparatus.

[0022] In one embodiment, the or each latent space representation of the anatomical structure and appearance of the fetus comprises an embedding vector.

[0023] According to a second aspect of the present invention, there is provided an ensemble machine learning model for estimating fetal weight, the ensemble machine learning model comprising: a first machine learning model comprising one or more neural networks, the first machine learning model being configured to utilise a plurality of ultrasound images comprising image data representative of the anatomical structure of a head, an abdomen and a femur of a fetus in utero to generate one or more latent space representations of the anatomical structure and appearance of the fetus and one or more predicted measurements of the anatomical structure of the fetus; and a second machine learning model configured to utilise the one or more predicted measurements and the one or more latent space representations of the anatomical structure to generate an estimated weight of the fetus.

[0024] In one embodiment, the plurality of ultrasound images of the fetus comprise: a head image of the fetus; an abdomen image of the fetus; and a femur image, and wherein the first machine learning model comprises a plurality of sub-networks, a discrete sub-network being allocated for each of the head, abdomen and femur images.

[0025] In one embodiment, each sub-network comprises a first neural network for generating a respective latent space representation of anatomical structure and appearance and a second neural network for generating, from the respective latent space representation, one or more predicted measurements of the anatomical structure of the fetus.

[0026] In one embodiment, the first neural network comprises a convolutional neural network (CNN) and / or the second neural network comprises a dense neural network (DNN) and / or the second machine learning model comprises a dense neural network (DNN). In one embodiment, the sub-network allocated to the head image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the head of the fetus and a predicted measurement of the head circumference (HC) and / or the biparietal diameter (BPD) of the fetus; and / or wherein the sub-network allocated to the abdominal image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the abdomen of the fetus and a predicted measurement of the abdominal circumference (AC) of the fetus; and / or wherein the subnetwork allocated to the femur image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the femur of the fetus and a predicted measurement of the femur length (FL) of the fetus.

[0027] In one embodiment, the or each latent space representation of the anatomical structure and appearance of the fetus comprises an embedding vector.

[0028] According to a third aspect of the present invention, there is provided a computing system for estimating fetal weight, the computing system comprising: at least one hardware processor; and an analyser, the analyser comprising: a first machine learning model comprising one or more neural networks, the first machine learning model being configured to utilise a plurality of ultrasound images comprising image data representative of the anatomical structure of a head, an abdomen and a femur of a fetus in utero to generate one or one or more latent space representations of the anatomical structure and appearance of the fetus and one or more predicted measurements of the anatomical structure of the fetus; and a second machine learning model configured to utilise the one or more predicted measurements and the one or more embedding vectors to generate an estimated weight of the fetus.

[0029] In one embodiment, the generated one or more predicted measurements corresponding to the anatomical structure of the fetus comprise one or more of: head circumference (HC); biparietal diameter (BPD); abdominal circumference (AC); and femur length (FL).

[0030] In one embodiment, the plurality of ultrasound images of the fetus comprise: a head image of the fetus; an abdomen image of the fetus; and a femur image. In one embodiment, the first machine learning model comprises a plurality of sub-networks, a discrete sub-network being allocated for each of the head, abdomen and femur images.

[0031] In one embodiment, each sub-network comprises a first neural network for generating a respective latent space representation of anatomical structure and appearance and a second neural network for generating, from the respective latent space representation, one or more predicted measurements of the anatomical structure of the fetus.

[0032] In one embodiment, the first neural network comprises a convolutional neural network (CNN) and / or the second neural network comprises a dense neural network (DNN) and / or the second machine learning model comprises a dense neural network (DNN).

[0033] In one embodiment, the sub-network allocated to the head image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the head of the fetus and a predicted measurement of the head circumference (HC) and / or the biparietal diameter (BPD) of the fetus; and / or wherein the sub-network allocated to the abdominal image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the abdomen of the fetus and a predicted measurement of the abdominal circumference (AC) of the fetus; and / or wherein the subnetwork allocated to the femur image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the femur of the fetus and a predicted measurement of the femur length (FL) of the fetus.

[0034] In one embodiment, the analyser is further configured to generate an uncertainty estimate for the estimated fetal weight.

[0035] In one embodiment, the analyser is configured to generate an uncertainty estimate for the estimated fetal weight by: applying one or more transforms to the plurality of ultrasound images to generate a set of augmented images; utilising the set of augmented images to generate an estimate of fetal weight based on the set of augmented images; and repeating the applying and utilising N times to generate N estimates of fetal weight.

[0036] In one embodiment, the one or more transforms are randomly selected from one or more of: rotation; shear; translation; brightness; contrast; and horizontal flip. In one embodiment, the or each latent space representation of the anatomical structure and appearance of the fetus comprises an embedding vector.

[0037] According to a fourth aspect of the present invention, there is provided an ultrasound imaging apparatus comprising an ultrasound scanner configured to generate a plurality of ultrasound images of the fetus in utero and the computing system according to the third aspect. According to a fifth aspect of the present invention, there is provided a method of performing ultrasound examination of a fetus in utero, the method comprising the steps of: a) acquiring a plurality of ultrasound images of the fetus in utero using an ultrasound imaging apparatus, the ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of a fetus in utero; b) utilising, on a computing system, a plurality of sets of the ultrasound images in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight; c) providing, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

[0038] In one embodiment, step a) further comprises obtaining: at least one head image of the fetus; at least one abdomen image of the fetus; and at least one femur image of the fetus; and wherein each set of ultrasound images of the fetus comprises a head image, an abdomen image and a femur image.

[0039] In one embodiment, step b) further comprises: d) utilising the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; e) applying one or more transforms to the plurality of ultrasound images to generate an augmented set of ultrasound images; and f) utilising the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate.

[0040] In one embodiment, the steps e) and f) are repeated N times to generate N estimates of fetal weight for uncertainty determination.

[0041] In one embodiment, step c) comprises g) providing, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus regarding whether the one or more parameters indicative of the accuracy of the one or more of the obtained ultrasound images has met one or more predetermined thresholds.

[0042] In one embodiment, if, at step g) one or more predetermined thresholds have not been met for one or more ultrasound images, providing an instruction to the operator to repeat step a) to acquire further ultrasound images. In one embodiment, step b) further comprises: h) utilising a first subset of the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; i) applying one or more transforms to the first set of ultrasound images to generate an augmented set of ultrasound images; and j) utilising the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate for the first subset of the obtained ultrasound images; and k) repeating steps h) to g) for second and subsequent subsets of ultrasound images; wherein step c) comprises: I) providing a notification to the operator in relation to the uncertainty for one or more subsets of ultrasound images.

[0043] In one embodiment, step I) comprises providing a notification to the operator in relation to subset of ultrasound images having the lowest uncertainty.

[0044] In one embodiment, the method further comprises: automatically selecting the subset of ultrasound images having the lowest uncertainty. In one embodiment, each subset of ultrasound images of the fetus comprises a head image, an abdomen image and a femur image.

[0045] In one embodiment, the head image, the abdomen image and femur image are taken in or close to respective standard planes.

[0046] In one embodiment, the ensemble machine learning model comprises: a first machine learning model comprising one or more neural networks, the first machine learning model being configured to utilise a plurality of ultrasound images comprising image data representative of the anatomical structure of a head, an abdomen and a femur of a fetus in utero to generate one or more latent space representations of the anatomical structure and appearance of the fetus and one or more predicted measurements of the anatomical structure of the fetus; and a second machine learning model configured to utilise the one or more predicted measurements and the one or more latent space representations of the anatomical structure to generate an estimated weight of the fetus.

[0047] In one embodiment, the first machine learning model comprises a plurality of sub-networks, a discrete sub-network being allocated for each of the head, abdomen and femur images.

[0048] In one embodiment, each sub-network comprises a first neural network for generating a respective latent space representation of anatomical structure and appearance and a second neural network for generating, from the respective latent space representation, one or more predicted measurements of the anatomical structure of the fetus.

[0049] In one embodiment, the first neural network comprises a convolutional neural network (CNN) and / or the second neural network comprises a dense neural network (DNN) and / or the second machine learning model comprises a dense neural network (DNN).

[0050] In one embodiment, the sub-network allocated to the head image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the head of the fetus and a predicted measurement of the head circumference (HC) and / or the biparietal diameter (BPD) of the fetus; and / or wherein the sub-network allocated to the abdominal image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the abdomen of the fetus and a predicted measurement of the abdominal circumference (AC) of the fetus; and / or wherein the subnetwork allocated to the femur image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the femur of the fetus and a predicted measurement of the femur length (FL) of the fetus.

[0051] According to a sixth aspect of the present invention, there is provided an ultrasound imaging apparatus comprising: an ultrasound scanner configured to obtain a plurality of ultrasound images of a fetus in utero, the ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of the fetus; and a computing system comprising at least one hardware processor and configured to utilise a plurality of sets of the ultrasound images in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight; provide, based on the uncertainty estimate, a notification to the operator of the ultrasound in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

[0052] In one embodiment, the ultrasound scanner is configured to obtain: at least one head image of the fetus; at least one abdomen image of the fetus; and at least one femur image of the fetus; and wherein each set of ultrasound images of the fetus comprises a head image, an abdomen image and a femur image.

[0053] In one embodiment, the computing system is further configured to: utilise the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; apply one or more transforms to the plurality of ultrasound images to generate an augmented set of ultrasound images; and utilise the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate.

[0054] In one embodiment, the ultrasound imaging apparatus is further configured to repeat the applying and utilising steps N times to generate N estimates of fetal weight for uncertainty determination.

[0055] In one embodiment, the ultrasound imaging apparatus is further configured to: provide, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus regarding whether the one or more parameters indicative of the accuracy of the one or more of the obtained ultrasound images has met one or more predetermined thresholds.

[0056] In one embodiment, if one or more predetermined thresholds have not been met for one or more ultrasound images, the computer system is further configured to provide an instruction to the operator to repeat the acquisition to acquire further ultrasound images.

[0057] In one embodiment, the computer system is further configured to: utilise a first subset of the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; apply one or more transforms to the first set of ultrasound images to generate an augmented set of ultrasound images; and utilise the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate for the first subset of the obtained ultrasound images; and repeat the applying and utilising for second and subsequent subsets of ultrasound images; and wherein the computer system is further configured to: provide a notification to the operator in relation to the uncertainty for one or more subsets of ultrasound images.

[0058] In one embodiment, the computer system is further configured to provide a notification to the operator in relation to subset of ultrasound images having the lowest uncertainty. In one embodiment, the computer system is further configured to automatically select the subset of ultrasound images having the lowest uncertainty.

[0059] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which: Figure 1 shows a schematic diagram of a medical analysis computing system according to an embodiment;

[0060] Figure 2 shows a schematic diagram of an ensemble machine learning model for use with the medical analysis computing system of Figure 1 according to an embodiment;

[0061] Figure 3 shows a detailed schematic diagram of an ensemble machine learning model for use with the medical analysis computing system of Figure 1 according to an embodiment;

[0062] Figure 4 shows uncertainty data using the model of Figure 3;

[0063] Figure 5a shows a typical ultrasound scan image after caliper placement;

[0064] Figure 5b shows the same ultrasound scan image after image processing;

[0065] Figure 6 shows a flow chart of a method according to an embodiment;

[0066] Figure 7 shows ROC curves for the Hadlock method and for the method of the present invention;

[0067] Figure 8 shows saliency data for the ensemble machine learning model;

[0068] Figure 9 shows a flow chart of a method according to an embodiment; and

[0069] Figure 10 shows a flow chart of a method according to a further embodiment.

[0070] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numbers are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same.

[0071] Various examples and embodiments of the present disclosure will now be described. The following description provides specific details for a thorough understanding and enabling description of these examples. One of ordinary skill in the relevant art will understand, however, that one or more embodiments described herein may be practiced without many of these details.

[0072] Likewise, one skilled in the relevant art will also understand that one or more embodiments of the present disclosure can include other features and / or functions not described in detail herein. Additionally, some well-known structures or functions may not be shown or described in detail below, so as to avoid unnecessarily obscuring the relevant description.

[0073] The present invention relates to a medical analysis system and method operable to provide an estimate of fetal birth weight from ultrasound images. In embodiments, the medical analysis system and method are operable to provide an uncertainty estimate on the estimated fetal weight.

[0074] The medical analysis system comprises one or more machine learning algorithms to identify parameters of interest in scan data. The parameters are identified by means of a training process which enables classification and utilisation of specific features within images by means of one or more quantitative indicators or parameters indicative of fetal weight.

[0075] When appropriately trained, the computing process can provide estimates of fetal birth weight and an associated uncertainty value for the estimated fetal birth weight.

[0076] This is a significant improvement on known arrangements which require analysis of scan data to be performed by a medical professional or sonographer either during or after the scan session has been completed.

[0077] Medical imaging system configuration

[0078] The present invention relates to a medical analysis computing system 10. The computing system 10 may take any suitable form and may comprise, for example, a cloud-based computing system, a remote computer connected over a network, a local workstation or a dedicated medical imaging console.

[0079] Figure 1 shows a computing system 10 according to an embodiment. The computing system 10 comprises one or more physical processors 12, a computer-readable physical memory 14 and a non-transitory storage device 16 such as a hard disk drive or solid-state drive. The one or more physical processors 12 may comprise any suitable processor types; for example, central processing units (CPUs), graphical processing units (GPUs) or any other suitable processor such as FPGAs and stream processors.

[0080] The storage device 16 may take any suitable form and may include storage device(s) local to the computing system 10 and / or storage devices external to the computing system 10. For example, the storage device 16 may comprise cloud or other virtual networked storage.

[0081] The computing system 10 further comprises an interface 18 through which image data is received. The interface 18 may take any suitable form and may be in the form of a communications interface and / or a network connection to a suitable source of input medical image data.

[0082] A computing application 20 is run on the computing system and is operable to cause the computing system 10 to perform the method of the present invention. The computing application 20 comprises an analyser 22 and a training database 24 as will be described below.

[0083] The computing application 20 utilises one or more machine learning algorithms to analyse and process medical image data. In embodiments, in use, the computing application 20 comprises two aspects - a training stage and an operational stage.

[0084] In non-limiting embodiments, the computing application 20 is operable to communicate through the interface 18 with a picture archiving and communication system (PACS) 26. The PACS 26 comprise an industry-standard device and format for medical imaging.

[0085] This is, however, not intended to be limiting and other configurations may be used. The PACS 26 is operable to process data in Digital Imaging and Communications in Medicine (DICOM) format. However, again this is non limiting and the skilled person would readily be aware of other formats which could be used.

[0086] Further, in non-limiting embodiments, the computing application 20 is operable to communicate through the interface 18 with a medical imaging device 50 in the form of an ultrasound imaging apparatus 50. The ultrasound imaging apparatus 50 comprises an ultrasound scanner 52 and a reading station 54. The reading station 54 may be local to the ultrasound scanner 52 (e.g. a computer and monitor system) or may be located remotely therefrom. The reading station 54 can be used by a medical practitioner and / or sonographer to read, interpret and analyse image data and biometric data generated therefrom.

[0087] In embodiments, the ultrasound imaging apparatus 50 is connected to the PACS 26 to access and store data relating to a pregnant woman under observation. This may, in non-limiting embodiments, comprise image data as well as other biometric information relating to the fetus and pregnant woman such as date of conception or last menstrual period, or other health- or biometric-related data.

[0088] In use, the ultrasound scanner 52 of the ultrasound imaging apparatus 50 is operable to transmit ultrasound signals via a suitable probe and to recover reflected ultrasound signals in order to image the fetus. In embodiments, these measured signals may be B-mode signals which are used to generate B-scan (2D brightness scan) ultrasound images of the fetus from a number of predetermined orientations.

[0089] The predetermined orientations are taken in three standard planes to enable the determination of fetal parameters including biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC) and femur length (FL) from these images.

[0090] These fetal parameters are determined by the computing system 10 in accordance with the methods described herein.

[0091] In embodiments, the computing system 10 may be separate from the ultrasound imaging apparatus 50 and receive data from the ultrasound imaging apparatus 50 via PACS 26.

[0092] Alternatively, whilst shown schematically and separately therefrom, the computing system 10 may be integrated into the ultrasound imaging apparatus 50 and may form a part thereof. For example, the computing application 20 may be run on the computing system 10 forming part of the ultrasound imaging apparatus 50 and provide data to the medical practitioner and / or sonographer using the ultrasound imaging apparatus 50 to enable them to read, interpret and analyse image data and biometric data generated therefrom. In such embodiments, the computing application 20 may comprise a computing module executed on the computing apparatus 10 of the ultrasound imaging apparatus 50.

[0093] Alternatively, the computing application 20 may utilise medical image data obtained from other sources and may be run on a computing system 10 such as a post-processing workstation or cloud solution working on medical scan image data.

[0094] Neural network architecture

[0095] The computing application 20 will now be described with reference to Figures 2 and 3. In embodiments, the analyser 22 comprises one or more machine learning algorithms. Figure 2 shows a general schematic of a machine learning model 100. Figure 3 shows a detailed schematic diagram of a specific embodiment of the machine learning model 100 and machine learning algorithms.

[0096] In embodiments, the analyser 22 comprise an ensemble machine learning model 100 which is operable to estimate fetal weight. In embodiments, the ensemble machine learning model 100 may be configured to estimate head circumference (HC), biparietal diameter (BPD), abdominal circumference (AC), and femur length (FL) based on ultrasound images.

[0097] Figure 2 shows a schematic of the configuration of the model 100. The ensemble machine learning model 100 comprises first and second machine learning models 102, 104. The first machine learning model 102 comprises one or more machine learning algorithms and is operable to process one or more medical images 106 to predict a measurement of the input anatomical structure which is the subject of the input medical images and one or more latent space representations of the anatomical structure and appearance of the fetus. In embodiments, one or more latent space representations comprise an embedding vector corresponding to this anatomical structure and the appearance thereof.

[0098] The second machine learning model 104 is operable to receive the predicted measurements, and embedding vectors, together with additional, optional biometric data as inputs. These are input into the second machine learning model 104 which comprises a second machine learning algorithm to generate an output 108 of the fetal weight estimate from the inputted medical images 106.

[0099] Any suitable machine learning model structures and algorithms may be used that can achieve the above functionality. A specific embodiment of the model 100 is described below where the components of the first and second machine learning models 102, 104 are described with reference to Figure 3.

[0100] The medical images 106 input into the model 100 comprise three medical images 106-H, 106- A, 106-F comprising the three standard planes of head 106-H, abdomen 106-A and femur 106-F.

[0101] Each medical image 106-H, 106-A, 106-F is also associated with scaling data SH, SA, SF representative of the pixel spacing (i.e. spatial resolution) of the respective medical image 106-H, 106-A, 106-F. This is required to determine the relative scale of the respective medical image 106. This scaling data is, in embodiments, saved in the DICOM files exported from the ultrasound imaging apparatus 50 and / or PACS 26 and can be extracted for use in the model 100.

[0102] The first machine learning model 102 of the model comprises a sub-network for each of the three standard planes 106-H, 106-A, 106-F. Each sub-network comprises a convolutional neural network (CNN). CNNH is operable to process the head standard plane image 106-H, CNNA the abdomen standard plane image 106-A and CNNF the femur standard plane image 106-F.

[0103] A CNN comprises an input layer, an output layer a sequence of encoding layers therebetween. The encoding layers of a typical CNN may comprise repeated applications of convolution layers, non-linear activation functions and pooling layers for down sampling, followed by one or more dense (fully connected) layers.

[0104] In specific embodiments, the CNN model is produced using RegNetX 1.6 Gf which comprises a convolutional network design space for the design of CNNs. However, this is not intended to be limiting and other systems may be used; for example, ResNet, VGG, EfficientNet, MobileNet, DenseNet, AlexNet, GoogLeNet and / or Inception.

[0105] Each of CNNH, CNNA and CNNF is operable to process the respective standard plane medical images 106-H, 106-A, 106-F to generate an embedding vector E that corresponds to this anatomical structure. An embedding vector EH, EA, EF is generated from each of CNNH, CNNA and CNNF. and for each standard plane. The embedding vectors E enables the model 100 to encode additional information relating to the input images beyond the predicted measurements. Each embedding vector comprises a low-dimensional, learned vector representation of the relevant input image.

[0106] The respective scaling data SH, SA, SF representative of the pixel spacing (i.e. spatial resolution) of the respective medical image 106-H, 106-A, 106-F is also applied to the respective embedding vector EH, EA, EF. TO ensure that the relative scale of the respective image 106 is factored into the model 100.

[0107] In order to generate predicted measurements of the input anatomical structure which is the subject of the medical images 106, each embedding vector EH, EA, EF. is input into a respective Deep Neural Network DNNH, DNNA, DNNF.

[0108] Deep neural networks comprise artificial neurons or nodes. A weight is assigned to each input to the node. The sum of all the weighted inputs to a given node is then passed through a nonlinear activation function to transform the pre-activation level of the neuron to an output. The output then serves as input to a node in the next layer.

[0109] Each DNN DNNH, DNNA, DNNF comprises, in specific embodiments, a single layer having 512 features and is operable to generate predicted fetal biometric measurements. DNNH is configured to generate predicted measurement values for HC and BPD, DNNA is configured to generate predicted measurement values for SC and DNNF is configured to generate predicted measurement values for FL.

[0110] The data generated in the first machine learning model 102 of the model 100, namely the predicted measurements for HC, BPD, AC and FL and the embedding vectors EH, EA, EF., can then be provided to the second machine learning model 104 to generate an estimated fetal weight.

[0111] The second machine learning model 104 of the model 100 comprises a dense neural network DNNEFW having, in embodiments, three dense (i.e. fully connected) layers and activation units. In specific embodiments, each layer comprises 256 features.

[0112] Several activation functions are available, which differ with respect to how they map a preactivation level to an output value. Commonly used activation functions used are the rectifier function (where neurons that use it are called rectified linear unit (ReLU)), the hyperbolic tangent function, the sigmoid function and the softmax function.

[0113] In embodiments, the activation units comprise rectified linear activation units. However, other activation units may be used. For example, a non-exhaustive list of suitable activation units may comprise one or more of: leaky rectified linear activation unit (leaky ReLLI), Sigmoid / Logistic, Hyperbolic tangent (Tanh), Swish, Mish and exponential linear unit (ELU).

[0114] The predicted measurements for HC, BPD, AC and FL and the embedding vectors EH, EA, EF. are used as inputs to the dense neural network DNNEFW.

[0115] Optionally, feature vector C may also be provided to the dense neural network DNNEFW. Feature vector C may comprise additional empirical and / or biometric data representative of or relevant to maternal and fetal characteristics.

[0116] This data may be used as an optional further input to the dense neural network DNNEFW. Feature vector C may, in embodiments comprise data relating to one or more data parameters. These may include, but are not limited to:

[0117] 1) Demographic data (gestational age, maternal age, fetal sex, parity, BMI, ethnicity, previous birth of SGA / LGA fetus);

[0118] 2) Biochemical markers (PAPP-A, beta-HCG, s-FLT, PLGH, metabolomic markers, proteomic markers); and

[0119] 3) Electronic healthcare record (EHR) data representative of a patient’s medical history

[0120] In addition, data gathered from other empirical or computational analysis may be used in the model 100. For example, textural computational analysis on other organs (for example, the placenta, fetal heart and / or brain) may generate data which may be used in the model 100. This data may optionally be input to further refine the output from the dense neural network DNNEFW.

[0121] The output from the dense neural network DNNEFWC the second machine learning model 104 is the Estimated Fetal Weight (EFW).

[0122] Uncertainty Estimation Test time augmentation was used to estimate prediction uncertainty of the EFW. Test time augmentation comprises generating a number of random modifications to the medical images 106 and running each of the modified images through the model 100. The predictions of each corresponding image can then be used to determine the uncertainty. In other words, test time augmentation comprises the aggregation of predictions across transformed versions of an input medical image 106.

[0123] In embodiments, the uncertainty was estimated by augmenting the set of medical images 106 ten times and passing these images 106 through the model 100 to obtain multiple predictions for the fetal weight. The standard deviation of the predictions is then used as the initial uncertainty estimate.

[0124] The augmentation parameters may be the same as used in the training process (see below). Values obtained in this way correlate with the prediction errors. However, the values need to be scaled appropriately.

[0125] Figure 4 shows experimental data illustrating aspects of the uncertainty determination.

[0126] Figure 4a shows the error distribution for a plurality of bins, with predicted uncertainty on the X-axis and error in kg on the Y-axis. Four bin ranges (1 to 4) have been identified and marked on Figure 4a between each set of dashed lines. Figure 4b shows the distribution of errors for these marked bins.

[0127] Figure 4c shows mean absolute error (MAE) as a function of predicted uncertainty and Figure 4d shows N samples in bins as a function of predicted uncertainty.

[0128] Further, to evaluate how the error changes as a function of predicted uncertainty, the data was divided into bins with a width of 10. Figure 4e shows this data. As shown, a good fit can be observed for 98% of the data with predicted uncertainty in the range between 20 and 130.

[0129] Therefore, since the errors are normally distributed in the bins, the linear fit can be used to transform the uncertainty prediction into the standard deviation of the error. This enables computation of the confidence intervals for the predictions of the model 100.

[0130] Generation of image data for training, testing and validation Ultrasound were obtained from 17 hospitals in Denmark between 2008 and 2018. Birth weight data was obtained through the Danish Fetal Medicine Database and imaging data was collected from four central servers.

[0131] Optionally, the model may use additional data such as fetal and maternal characteristics (Body Mass Index (BMI), parity, preeclampsia, Gestational Age (GA), fetal sex) if required. However, in this example, this data was not needed.

[0132] Processing and selection of data

[0133] Dataset Images used for fetal biometry measurements often come with embedded markings (text and calipers) placed by clinicians and sonographers during the scan. An example of this is shown in Figure 4a.

[0134] Calipers (marker crosses) are placed on the picture to outline the anatomy to be measured, and the result is placed in a table in the lower-right corner. The table contains the value and the code of what is being measured: FL, AC, HC, and BPD.

[0135] In embodiments, Optical Character Recognition (OCR) was used to automatically classify the images as head, abdomen, femur, and other and extract the relevant measurements. The other classes are discarded.

[0136] Next, the images were aggregated based on the patient’s identification number and study date to obtain sets of images from the same examination. Furthermore, sets that did not have at least one image from each class were excluded.

[0137] Finally, fetal weight at scan time was extrapolated from the birth weight using the Marsal growth curve.

[0138] The test data set included 433,096 images from 94,538 examinations performed on 65,752 patients. The data was limited to singleton pregnancies only and was divided between training, validation, and test sets (85%, 5%, 10%), ensuring no patient overlap.

[0139] The training included 27% of 2nd-trimester images to increase the amount of training data. However, the test set contains only 3rd-trimester images with gestational age above 28 weeks. The mean gestational age in the train and test sets was 30±6.1 and 34±3.2, respectively, and the mean gestational age at birth was 40 ± 1 .6. The mean maternal age was 31 .5 ± 5.26, BMI was 24.4 ± 5.6, and parity was 0.8 ± 0.9. In terms of conception status, 91.8% of the fetuses were conceived spontaneously, 5.8% through in vitro fertilisation or intracytoplasmic sperm injection, and 2.2% through intrauterine insemination. The remaining cases were unspecified.

[0140] Furthermore, 88.3% of the patients were of Caucasian ethnicity, 3.2% were Asian, 1.6% were Oriental, 1.4% were Afro-Caribbean, and the remaining 5.5% were either unspecified or of another ethnicity.

[0141] Image enhancement

[0142] Although calipers and other markings provide a convenient means of generating a sizable, labelled dataset, it is well known that calipers may introduce extraneous information that can influence model output. To mitigate the issue of information leakage, which can affect model output, the images are subjected to inpainting to remove the aforementioned information. This is done in a two-step process.

[0143] First, the acquired image is thresholded in the Hue, Saturation, Value (HSV) colour-space. Secondly, the largest connected component is identified, and holes within the component are filled using a hole filling algorithm.

[0144] These procedures facilitate the removal of extraneous text and features outside of the ultrasound conical field of view. However, the calipers are still present within this view.

[0145] To address this issue, in embodiments, a second thresholding step is performed in the HSV colour-space. In embodiments, the resulting mask is then subjected to dilation using a 7x7 square structuring element to include elements that were not entirely segmented through thresholding alone.

[0146] The dilated mask can then subsequently be used to identify the pixels for inpainting. Inpainting can then be performed.

[0147] In Figure 4a, the original image from the database is displayed, with only the patient information being manually obscured. Figure 4b depicts the same image with the text and calipers removed. Model training

[0148] Once the data has been prepared, the model can be trained on the relevant datasets.

[0149] In non-limiting embodiments, the model 100 is trained using a suitable optimiser, such as, for example, the AdamW optimiser.

[0150] In specific embodiments, a learning rate of 1e-4, weight decay of 1e-6, and batch size 8 can be utilised. To reduce training time, the RegNetX parameters obtained from training on ImageNet data can be used as a starting point.

[0151] In the specific examples described herein, the training images were centre cropped and resized to 224x224 pixels. The images were also converted to grayscale, and further augmented with random rotation (±25° ), shear (±10° ); translation (0.05 of image size); brightness (0.2), contrast (0.2) and random horizontal flip (P = 0.5).

[0152] The model 100 was trained using a multi-task learning scheme to output the measurements: HC, BPD, AC, FL, and EFW. Images as well as all measurements were normalised to fit a 0 to 1 interval.

[0153] The regression loss is defined as follows:

[0154] - extrapolated scan weight - predicted scan weight - scan weight z-score

[0155] In specific embodiments, additional weighting parameters may be incorporated into the loss function used for estimating fetal weight. Specifically, relative error may be utilised as the base loss function, and two weighting parameters added to further refine the loss.

[0156] A first weighting parameter may be based on the z score and enables abnormal fetuses to be emphasised in the loss function. This approach enables a greater weight to be assigned to fetuses that were at higher risk for adverse outcomes.

[0157] A second weighting parameter may be used to put additional weight on training samples from the third trimester and those that were closer to their estimated date of birth. By doing so, this has the advantage that the uncertainty in the model that could arise from changes in the growth curve can be reduced.

[0158] In embodiments, the training dataset is organised such that each unique scan corresponds to one entry, but the scans can contain more than one image of each standard plane. Therefore, during training, a set of three images (head, abdomen and femur) is randomly sampled from each scan.

[0159] METHOD

[0160] The following method relates to the steps occurring during operation of the fetal estimation method using the medical analysis computing system 10. Figure 6 shows a flow chart of the method according to an embodiment.

[0161] Step 200: Provide ultrasound images

[0162] At step 200, ultrasound medical images 106 of a fetus for assessment are obtained by the medical analysis computing system 10.

[0163] In embodiments, the ultrasound medical images may be acquired by the ultrasound imaging apparatus 50 in this step. The ultrasound scanner 52 of the ultrasound imaging apparatus 50 transmits ultrasound signals via a suitable probe and recovers reflected ultrasound signals in order to generate image data of the fetus. In embodiments, these measured signals may be B-mode signals which are used to generate B-scan (2D brightness scan) ultrasound images of the fetus from a number of predetermined orientations.

[0164] The predetermined orientations may be taken in three standard planes to enable the determination by the model 100 of fetal parameters including biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC) and femur length (FL) from these images.

[0165] However, in embodiments, the images may comprise out of plane images or incorrectly aligned images (i.e. images taken in planes which are linearly or angularly offset or which deviate in geometry from the desired standard plane images). In contrast to known arrangements, the model 100 can still extract the desired fetal parameters from such images as described below. However, this need not be the case and medical images 106 obtained from another ultrasound source may be provided in this step without the step of acquisition of the images by the ultrasound imaging apparatus 50.

[0166] Optionally, the medical images 106 may be pre-processed and / or transformed for use in the model 100. In embodiments, this may involve one or more of: colour transform; cropping; scaling; skewing or resizing of the medical images 106 before provision to the model 100.

[0167] In specific embodiments, the medical images 106 are converted to grayscale and centre cropped. However, this is not to be taken as limiting and other transformations may be used.

[0168] In specific embodiments, the medical images 106 are resized to dimensions of 224 x 224 pixels. However, this is not to be taken as limiting and other scaling(s) may be used.

[0169] The three medical images 106-H, 106-A, 106-F are then provided to the model 100 of the computing application 20.

[0170] Scaling data for the medical images 106 is also provided. In the specific embodiment these are extracted from the DICOM file. In a non-limiting embodiment, they may be extracted from a scale bar in the image. The method proceeds to step 202.

[0171] Step 202: Input images to CNN

[0172] At step 202, the images 106-H, 106-A, 106-F are input into the respective convolutional neural network CNNH, CNNA and CNNF of the first machine learning model 102 of the model 100. CNNH is operable to process the head standard plane image 106-H, CNNA the abdomen standard plane image 106-A and CNNF the femur standard plane image 106-F.

[0173] The CNN comprises an input layer, an output layer a sequence of encoding layers therebetween. The encoding layers of a typical CNN may comprise repeated applications of convolution layers, activation layers and pooling layers for down sampling, followed by one or more dense (fully connected) layers. In specific embodiments, the CNN model is produced using RegNetX 1.6 Gf which comprises a convolutional network design space for the design of CNNs.

[0174] Each of CNNH, CNNA and CNNF processes the respective standard plane medical images 106- H, 106-A, 106-F to generate an embedding vector E that corresponds to this anatomical structure. In step 202, an embedding vector EH, EA, EF is generated from each of CNNH. CNNA and CNNF. and for each standard plane.

[0175] The scaling data is inputted into the relevant CNN at this stage. The method proceeds to step 204.

[0176] Step 204: Input embedding vectors to DNNs

[0177] At step 204, each embedding vector EH, EA, EF. is input into a respective Deep Neural Network DNNH, DNNA, DNNF in order to generate predicted measurements of the input anatomical structure which is the subject of the medical images 106,

[0178] Deep neural networks comprise artificial neurons or nodes. A weight is assigned to each input to the node. The sum of all the weighted inputs to a given node is then passed through a nonlinear activation function to transform the pre-activation level of the neuron to an output. The output then serves as input to a node in the next layer.

[0179] Each DNN DNNH, DNNA, DNNF comprises, in specific embodiments, a single layer having 512 features and is operable to generate predicted fetal biometric measurements. However, this is not to be taken as limiting and additional layers or layers having different numbers of features may be provided.

[0180] DNNH is configured to generate predicted measurement values for HC and BPD, DNNA is configured to generate predicted measurement values for SC and DNNF is configured to generate predicted measurement values for FL. The method proceeds to step 206.

[0181] Step 206: Input data to second machine learning model

[0182] In step 206, the data generated in the first machine learning model 102 of the model 100, namely the predicted measurements for HC, BPD, AC and FL and the embedding vectors EH, EA, EF., is provided to the second machine learning model 104 to generate an estimated fetal weight.

[0183] The second machine learning model 104 of the model 100 comprises a dense neural network DNNEFW having, in embodiments, three dense (i.e. fully connected) layers and activation units. In specific embodiments, each layer comprises 256 features. However, this is not to be taken as limiting and greater or fewer layers having different numbers of features may be provided. Several activation functions are available, which differ with respect to how they map a preactivation level to an output value. Commonly used activation functions used are the rectifier function (where neurons that use it are called rectified linear unit (ReLLI)), the hyperbolic tangent function, the sigmoid function and the softmax function. In embodiments, the activation units comprise rectified linear activation units.

[0184] In step 206, the predicted measurements for HC, BPD, AC and FL and the embedding vectors EH, EA, EF. are inputted into the dense neural network DNNEFW.

[0185] Optionally, feature vector C may be inputted into the dense neural network DNNEFW'^ required.

[0186] Step 208: Generate estimated fetal weight

[0187] At step 208, the estimated fetal weight is output from the dense neural network DNNEFW.

[0188] Optionally, uncertainty may be determined for the estimated fetal weight according to the following steps.

[0189] Step 210: Generate augmented medical images

[0190] At step 210, random modifications to the medical images 106 are generated to generate an augmented set of images. The random modifications may include, for example, random rotation (±25° ), shear (±10° ); translation (0.05 of image size); brightness (0.2), contrast (0.2) and random horizontal flip (P = 0.5). These are the same parameters as used in the training process.

[0191] The augmentation process may be carried out N times to generate N sets of augmented images. In embodiments, N may be 10.

[0192] Once one or more augmented images have been generated, the method proceeds to step 212 for one set of the N sets of augmented images.

[0193] Step 212: Generate EFW from augmented images

[0194] At step 212, steps 202 to 208 are run based on a set of augmented images. This step is repeated for further augmented images until N predictions have been carried out. Step 214: Derive uncertainty estimate

[0195] At step 214, the standard deviation of the predictions is then used as the initial uncertainty estimate. However, in this form the uncertainty metric is not meaningful.

[0196] Therefore, in embodiments, to evaluate how the error changes as a function of predicted uncertainty, the linear fit derived from the training data, can be used to transform the uncertainty prediction into the standard deviation of the error. This allows computation of the confidence intervals for the model predictions.

[0197] Experimental analysis

[0198] The dataset comprised 433,096 images from 94,538 examinations and was divided into training (85%), validation (5%), and test (10%) sets ensuring no patient overlap. In the event that multiple images of each anatomical region (femur, abdomen, head) were obtained during an examination, multiple observations were created by generating all feasible permutations of the available images.

[0199] In this study, the deep learning model estimate of fetal weight at scan time is compared to the clinical standard practice - the Hadlock formula. This formula is based on measurements of Abdominal Circumference (AC), Head Circumference (HC), and Femur Length (FL) performed by clinician during scan. The measurements are obtained automatically using Optical Character Recognition (OCR).

[0200] Moreover, the fetal weight at scan time is extrapolated from birth weight based on the Marsal growth curve (“Intrauterine growth curves based on ultrasonically estimated foetal weights" K Marsal et al, Acta Paediatr. 1996 Jul 85(7) 843-848). The standard deviation of the fetal weight is not fixed and varies as a function of the weight itself. Similarly as in Marsal et al., it was set to 12%. Therefore, the standard score is calculated z = (x- ) / 0.12p .

[0201] Moreover, fetuses with a fetal weight below the 10th percentile (z < -1.282) are referred to as Small for Gestational Age (SGA), while fetal weights above 90th (z > 1.282) are referred to as Large for Gestational Age (LGA). Normal weight fetuses are referred to as Appropriate for Gestational Age (AGA).

[0202] Relative error

[0203] The relative error between fetal weight and predictions obtained from the Hadlock formula and the Model is shown in Table 1 below.

[0204] Hadlock MRE [%] 9.12 ± 7.68 7.14 ± 5.38 9.57 ± 6.71

[0205] Model MRE [%] 7.31 ± 6.47 6.49 ± 5.02 7.31 ± 5.71 p — value < 0.0001 < 0.0001 < 0.0001

[0206] Effect size - Cohen’s d 0.25 0.12 0.36

[0207] Table 1

[0208] The results presented in the table reveal three significant observations. Firstly, the model surpasses the Hadlock formula in all categories. Secondly, the model displays higher consistency, as indicated by smaller standard deviations. Finally, the difference in performance between groups is less pronounced in the model than in the Hadlock formula, suggesting more consistent performance for abnormal groups (SGA, LGA).

[0209] This difference in performance between the two methods was found to be statistically significant, with a two-sample Welch’s t-test yielding p - values < 0.0001 in all groups, and the effect size ranges from trivial for AGA to moderate for LGA.

[0210] Classification

[0211] In the study, the performance of the model was evaluated in terms of its ability to classify small and large fetuses, which is of great clinical significance. The results of this evaluation are presented using Receiver Operating Characteristic (ROC) curves, as shown in Figure 7.

[0212] These curves show the ROC curve comparisons for SGA and LGA classifications. The total number of samples was 31386, with SGA 6152 and LGA 3270.

[0213] These curves offer a clear graphical representation of the effectiveness of the model in significantly reducing the accuracy of detection when compared to the conventional Hadlock method. The confidence intervals and standard errors for the ROC curves were calculated using the analysis defined in Hanley and McNeil: "A Method of Comparing the Areas under Receiver Characteristic Curves Derived from the Same Cases" (1983).

[0214] Table 2 shows the baseline for the data used in the study,

[0215] Table 2

[0216] A deep learning model was trained based on a large dataset to estimate fetal growth. The superior performance of the model 100 was demonstrated when compared with existing best practices such as the Hadlock formula for both SGA and LGA fetuses. The clinical implications of improved detection of SGA and LGA fetuses are significant.

[0217] The detection of SGA and LGA fetuses remains a challenge to obstetric management and prevention of stillbirth and adverse perinatal outcomes. Yet, errors in measurements of fetal biometries have a large impact on EFW and even small measurement errors may change the clinical management decisions significantly. Different approaches have been used to reduce the sources of variance in fetal weight estimation.

[0218] First, identifying at-risk pregnancies may detect women with pre-existing risk factors but growth issues for primiparous women. Second, even in countries with a third trimester ultrasound scan, the detection rates of LGA and SGA remain low (for example, around 60% for SGA fetuses). Third, initiatives to improve clinicians’ ultrasound skills have shown improvements in diagnostic accuracy when predicting birth weight for clinicians with poor skills and large errors in their estimates.

[0219] Finally, the use of fetal biometry automation has been described previously in several studies. Replicating existing workflows and methods - such as automated caliper placements - seem to reduce measurement variance due to operators.

[0220] However, conventional approaches may miss important features leveraged by the present invention. One of the reasons that the present model outperformed estimates from a large number of clinical exams may relate to the fact that the deep learning model relied on additional features in the ultrasound images in addition to those that are currently used for fetal biometries.

[0221] Figure 8 illustrates a selection of saliency maps. These represent gradient-based saliency visualisation. Each input image was passed through the model 100 times with additive Gaussian noise. The absolute value of the image gradients was averaged and smoothed.

[0222] A saliency map (right hand column of Figure 8) is shown for each input image 106-H, 106-A, 106-F (shown on the left of Figure 8). The brightness / intensity of features in each saliency map.

[0223] Considering the saliency maps of Figure 8, the deep learning model appears able to utilise the thickness around the fetal stomach and thigh as a potential indicator, which may serve as a predictor of fetal growth restriction as well as macrosomia.

[0224] The present invention has several benefits over the known art. In addition to providing clinically meaningful improvements in prediction of LGA and SGA, the model 100 also addresses the issue of uncertainty around estimating fetal weight. Clinical management decisions are currently made without any estimate of uncertainty around a given weight estimate, which may exacerbate the number of unnecessary tests and procedures including induction of labour and caesarean sections. This ultimately leads to inappropriate obstetric care and increased health expenditure.

[0225] T o better inform clinical management decisions, uncertainty estimates were developed around the fetal weight estimates produced by the model of the present invention, which is not possible using existing methods for EFW prediction.

[0226] METHOD OF USE EMBODIMENT

[0227] The following method relates to a further embodiment of the invention and comprises steps occurring during operation of a further method using the medical analysis computing system 10. Figure 9 shows a flow chart of the method according to the further embodiment.

[0228] The method of this embodiment may use, but is not limited to, the model 100 of the earlier embodiments. Any suitable ensemble machine learning model operable to extract uncertainty values from a plurality of ultrasound images may be used. Step 300: Acquire ultrasound images

[0229] At step 300, ultrasound medical images 106 of a fetus for assessment are obtained by an operator of the medical analysis computing system 10.

[0230] In embodiments, the operator of the medical analysis computing system 10 obtains a plurality of ultrasound images of the fetus in utero using the ultrasound imaging apparatus 50. The operator acquires ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of a fetus in utero.

[0231] In embodiments, the ultrasound scanner 52 of the ultrasound imaging apparatus 50 transmits ultrasound signals via a suitable probe and recovers reflected ultrasound signals in order to generate image data of the fetus. In embodiments, these measured signals may be B-mode signals which are used to generate B-scan (2D brightness scan) ultrasound images of the fetus from a number of predetermined orientations.

[0232] The predetermined orientations may be taken in three standard planes to enable the determination by an ensemble model of fetal parameters including biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC) and femur length (FL) from these images.

[0233] However, in embodiments, the images may comprise out of plane images or incorrectly aligned images (i.e. images taken in planes which are linearly or angularly offset or which deviate in geometry from the desired standard plane images). In contrast to known arrangements, the model can still extract the desired fetal parameters from such images as described below.

[0234] Optionally, the medical images 106 may be pre-processed and / or transformed for use in the model. In embodiments, this may involve one or more of: colour transform; cropping; scaling; skewing or resizing of the medical images 106 before provision to the model.

[0235] In specific embodiments, the medical images 106 are converted to grayscale and centre cropped. However, this is not to be taken as limiting and other transformations may be used.

[0236] In specific embodiments, the medical images 106 are resized to dimensions of 224 x 224 pixels. However, this is not to be taken as limiting and other scaling(s) may be used. The three medical images 106-H, 106-A, 106-F are then provided to the model of the computing application 20.

[0237] Scaling data for the medical images 106 is also provided. In the specific embodiment these are extracted from the DICOM file. In a non-limiting embodiment, they may be extracted from a scale bar in the image. The method proceeds to step 302.

[0238] Step 302: Estimate fetal weight and uncertainty

[0239] At step 302, a plurality of sets of the ultrasound images acquired in step 302 are used in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight.

[0240] This may be done in accordance with the embodiments above or may be done in accordance with any other suitable method.

[0241] The set of images may comprise three medical images 106-H, 106-A and 106-F obtained in step 300, or, if a plurality of medical images in each plane have been obtained, the set of images may be a subset of the images acquired in step 300.

[0242] In embodiments, the method may obtain fetal weight estimates from the or each set of images. Uncertainty estimates may be obtained in accordance with methods above using augmented images or via any other suitable method.

[0243] Step 304: Provide notification

[0244] At step 304, a notification to the operator of the ultrasound imaging apparatus 50 is provided in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

[0245] The notification may be in the form of one or more parameters of one or more images of a set or as acquired in step 300. In embodiments, the or each parameter may be indicative of a particular quantity or property of one or more medical images 106-H, 106-A and 106-F or information derived therefrom. In embodiments, a notification may be provided if one or more of the parameters exceeds a particular threshold for a particular property or quantity in one or more of the medical images 106-H, 106-A and 106-F or in information derivable therefrom.

[0246] For example, a notification may be provided to inform the operator that one or more of the three medical images 106-H, 106-A and 106-F has an associated uncertainty (derived in step 302) which does not meet an acceptable threshold for accuracy.

[0247] By way of further example, a notification may be provided to inform the operator that one or more of the three medical images 106-H, 106-A and 106-F has one or more parameters related to image quality which do not meet an acceptable threshold.

[0248] By way of further example, if at step 300 a plurality of images for each standard plane is obtained, a notification may be provided to inform the operator of the relative uncertainties and / or accuracies of each image.

[0249] By way of further example, if at step 300 a plurality of images for each standard plane is obtained, a notification may be provided to inform the operator which of the plurality of images obtained for each standard plane has the lowest uncertainty and / or has the highest accuracy.

[0250] Step 306: Automatically obtain images

[0251] Step 306 may be an optional step. If at step 300 a plurality of images for each standard plane is obtained and the notification optionally provided at step 304 to inform the operator which of the plurality of images obtained for each standard plane has the lowest uncertainty and / or has the highest accuracy, step 306 may be carried out.

[0252] At step 306, the subset of three medical images 106-H, 106-A and 106-F from the total set of medical images obtained in step 300 may be selected automatically for determination of fetal weight. On this basis, the operator may be notified of the images having the lowest uncertainty and that these images have been selected to determine fetal weight.

[0253] FURTHER METHOD OF USE EMBODIMENT

[0254] A further specific embodiment of the method of use of the present invention is described with reference to Figure 10.

[0255] Step 400: Acquire multiple ultrasound images At step 400, ultrasound medical images 106 of a fetus for assessment are obtained by an operator of the medical analysis computing system 10. In embodiments, step 400 proceeds as described above in respect of step 300 to acquire a plurality of medical images.

[0256] In embodiments, at step 400, the operator acquires ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of a fetus in utero.

[0257] The predetermined orientations may be taken in the three standard planes as described in step 300. However, in embodiments, the images may comprise out of plane images or incorrectly aligned images (i.e. images taken in planes which are linearly or angularly offset or which deviate in geometry from the desired standard plane images).

[0258] In embodiments, step 400 comprises obtaining multiple images for each of images representative of the head, abdomen and femur of the fetus in utero. These may be taken in, or close to, the standard planes as described above. However, in embodiments, this may not be the case. In other words, at step 400 multiple images are obtained for each of the standard planes (or intended to be in standard planes but may be offset or incorrectly aligned).

[0259] Pre-processing and other image processing operations may be performed on the images, and scaling data may be obtained as described in step 300 above. The method proceeds to step 402.

[0260] Step 402: Define one or more subset of images

[0261] At step 402, one or more subsets of the medical images acquired in step 400 are defined. At step 402, the obtained images 106-H, 106-A and 106-F can be grouped into three categories - head images 106-H, abdomen images 106-A and femur images 106-F.

[0262] The one or more subsets each comprise a head image 106-H, an abdomen image 106-A and a femur image 106-F. Any suitable permutation of captured images may be used to form a subset of images.

[0263] Once the subsets have been defined, the method proceeds to step 404.

[0264] Step 404: Estimate fetal weight and uncertainty for one or more subsets At step 404, the subsets of the ultrasound images defined in step 402 are used in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight.

[0265] A first subset of the obtained ultrasound images is selected for processing by the ensemble machine learning model to determine an estimated fetal weight and associated uncertainty.

[0266] This comprises applying one or more transforms to the first subset of ultrasound images to generate an augmented set of ultrasound images. The augmented set of ultrasound images are then inputted into the ensemble machine learning model to determine an uncertainty estimate for the first subset of the obtained ultrasound images.

[0267] The steps are then repeated for subsequent subsets of ultrasound images to obtain uncertainty estimates for each subset.

[0268] In embodiments where a two-part ensemble model is used, step 404 may comprise each of steps 202 to 214 described above, where steps 202 to 214 are performed for each subset of images to generate uncertainty estimates for each subset.

[0269] Step 406: Provide notification

[0270] At step 404, a notification to the operator of the ultrasound imaging apparatus 50 is provided in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

[0271] In this embodiment, the notification may comprise providing the operator with information in relation to the uncertainty and / or the accuracy of one or more subsets so identified. The notification may comprise an indication of the subset of images having the lowest uncertainty and / or highest accuracy, or the images within that subset may be highlighted to the operator for selection by the operator.

[0272] In embodiments, any metric of the accuracy of the images may be derived directly from the uncertainty measurements.

[0273] In embodiments, the operator may then select the best images based on the notification. Alternatively or additionally, optional step 408 may perform some steps automatically with the notification comprising information relating to the automatic selection of the subset of images having the lowest uncertainty and / or highest accuracy, or the images within that subset highlighted for the operator. This is described in step 408 below.

[0274] Step 408: Automatically obtain images

[0275] Step 408 may be an optional step or may be part of step 406. At step 408, the subset of three medical images 106-H, 106-A and 106-F from the total set of medical images obtained in step 400 may be selected automatically for determination of fetal weight. On this basis, the operator may be notified of the images having the lowest uncertainty and therefore the highest accuracy, and that these images have been selected to determine fetal weight.

[0276] Alternatively or additionally, the estimated fetal weight for each subset may be averaged and provide the operator with an averaged value for the EFL.

[0277] Variations will be apparent to the skilled person. For example, any suitable form of convolutional neural network (CNN) could be used with the present invention. For example, ResNet, VGG, EfficientNet or DenseNet technologies could be used.

[0278] The methods described herein may be embodied in one or more pieces of software. The software is preferably held or otherwise encoded upon a memory device such as, but not limited to, any one or more of, a hard disk drive, RAM, ROM, solid state memory or other suitable memory device or component configured to software. The methods may be realised by executing / running the software. Additionally or alternatively, the methods may be hardware encoded.

[0279] The method encoded in software or hardware is preferably executed using one or more processors. The memory and / or hardware and / or processors are preferably comprised as, at least part of, one or more servers and / or other suitable computing systems.

[0280] In this specification, unless expressly otherwise indicated, the word "or" is used in the sense of an operator that returns a true value when either or both of the stated conditions are met, as opposed to the operator "exclusive or" which requires only that one of the conditions is met. The word "comprising" is used in the sense of "including" rather than to mean "consisting of".

[0281] All prior teachings above are hereby incorporated herein by reference. No acknowledgement of any prior published document herein should be taken to be an admission or representation that the teaching thereof was common general knowledge in Australia or elsewhere at the date thereof.

[0282] Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components set forth herein may be separated into sub-components comprising software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components and vice-versa.

[0283] Software, in accordance with the present disclosure, such as program code and / or data, may be stored on one or more computer readable mediums. It is also contemplated that software identified herein may be implemented using one or more general purpose or specific purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the ordering of various steps described herein may be changed, combined into composite steps, and / or separated into sub-steps to provide features described herein.

[0284] While various operations have been described herein in terms of “modules,” “units” or “components,” it is noted that that terms are not limited to single units or functions. Moreover, functionality attributed to some of the modules or components described herein may be combined and attributed to fewer modules or components.

[0285] Further, whilst the present invention has been described with reference to specific embodiments and examples, those examples are intended to be illustrative only, and are not intended to limit the invention. It will be apparent to those of ordinary skill in the art that changes, additions or deletions may be made to the disclosed embodiments without departing from the scope of the invention.

Claims

CLAIMS1. A method of performing ultrasound examination of a fetus in utero, the method comprising the steps of: a) acquiring a plurality of ultrasound images of the fetus in utero using an ultrasound imaging apparatus, the ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of a fetus in utero, b) utilising, on a computing system, a plurality of sets of the ultrasound images in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight; c) providing, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

2. A method according to claim 1 , wherein step a) further comprises obtaining: at least one head image of the fetus; at least one abdomen image of the fetus; and at least one femur image of the fetus; and wherein each set of ultrasound images of the fetus comprises a head image, an abdomen image and a femur image.

3. A method according to claim 1 or 2, wherein step b) further comprises: d) utilising the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; e) applying one or more transforms to the plurality of ultrasound images to generate an augmented set of ultrasound images; and f) utilising the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate.

4. A computer-implemented method according to claim 3, wherein the steps e) and f) are repeated N times to generate N estimates of fetal weight for uncertainty determination.

5. A method according to claim 2, 3 or 4, wherein step c) comprises:g) providing, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus regarding whether the one or more parameters indicative of the accuracy of the one or more of the obtained ultrasound images has met one or more predetermined thresholds.

6. A method according to claim 5, wherein if, at step g) one or more predetermined thresholds have not been met for one or more ultrasound images, providing an instruction to the operator to repeat step a) to acquire further ultrasound images.

7. A method according to claim 1 or 2, wherein step b) further comprises: h) utilising a first subset of the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; i) applying one or more transforms to the first set of ultrasound images to generate an augmented set of ultrasound images; and j) utilising the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate for the first subset of the obtained ultrasound images; and k) repeating steps h) to g) for second and subsequent subsets of ultrasound images; wherein step c) comprises: l) providing a notification to the operator in relation to the uncertainty for one or more subsets of ultrasound images.

8. A method according to claim 7, wherein step I) comprises providing a notification to the operator in relation to subset of ultrasound images having the lowest uncertainty.

9. A method according to claim 8, wherein the method further comprises: a) automatically selecting the subset of ultrasound images having the lowest uncertainty.

10. A method according to claim 7, 8 or 9, wherein each subset of ultrasound images of the fetus comprises a head image, an abdomen image and a femur image.

11. A method according to claim 10, wherein the head image, the abdomen image and femur image are taken in or close to respective standard planes.

12. A method according to any one of the preceding claims, wherein the ensemble machine learning model comprises:a first machine learning model comprising one or more neural networks, the first machine learning model being configured to utilise a plurality of ultrasound images comprising image data representative of the anatomical structure of a head, an abdomen and a femur of a fetus in utero to generate one or more latent space representations of the anatomical structure and appearance of the fetus and one or more predicted measurements of the anatomical structure of the fetus; and a second machine learning model configured to utilise the one or more predicted measurements and the one or more latent space representations of the anatomical structure to generate an estimated weight of the fetus.

13. A method according to claim 12 when dependent upon claim 2, wherein the first machine learning model comprises a plurality of sub-networks, a discrete sub-network being allocated for each of the head, abdomen and femur images.

14. A method according to claim 13, wherein each sub-network comprises a first neural network for generating a respective latent space representation of anatomical structure and appearance and a second neural network for generating, from the respective latent space representation, one or more predicted measurements of the anatomical structure of the fetus.

15. A method according to claim 14, wherein the first neural network comprises a convolutional neural network (CNN) and / or the second neural network comprises a dense neural network (DNN) and / or the second machine learning model comprises a dense neural network (DNN).

16. A computer-implemented method according to any one of claims 13 to 15, wherein the sub-network allocated to the head image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the head of the fetus and a predicted measurement of the head circumference (HC) and / or the biparietal diameter (BPD) of the fetus; and / or wherein the sub-network allocated to the abdominal image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the abdomen of the fetus and a predicted measurement of the abdominal circumference (AC) of the fetus; and / or wherein the sub-network allocated to the femur image is operable to generate a latent space representation corresponding to the anatomical structure and appearance of the femur of the fetus and a predicted measurement of the femur length (FL) of the fetus.

17. An ultrasound imaging apparatus comprising: an ultrasound scanner configured to obtain a plurality of ultrasound images of a fetus in utero, the ultrasound images comprising image data representative of the anatomical structure and appearance of a head, an abdomen and a femur of the fetus; and a computing system comprising at least one hardware processor and configured to utilise a plurality of sets of the ultrasound images in an ensemble machine learning model comprising one or more neural networks to determine an estimated fetal weight and generate an associated uncertainty estimate for the estimated fetal weight; provide, based on the uncertainty estimate, a notification to the operator of the ultrasound in respect of one or more parameters indicative of the accuracy of one or more of the obtained ultrasound images.

18. An ultrasound imaging apparatus according to claim 17, wherein the ultrasound scanner is configured to obtain: at least one head image of the fetus; at least one abdomen image of the fetus; and at least one femur image of the fetus; and wherein each set of ultrasound images of the fetus comprises a head image, an abdomen image and a femur image.

19. An ultrasound imaging apparatus according to claim 17 or 18, wherein the computing system is further configured to: utilise the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; apply one or more transforms to the plurality of ultrasound images to generate an augmented set of ultrasound images; and utilise the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate.

20. An ultrasound imaging apparatus according to claim 19, further configured to repeat the applying and utilising steps N times to generate N estimates of fetal weight for uncertainty determination.21 . An ultrasound imaging apparatus according to claim 20, further configured to: provide, based on the uncertainty estimate, a notification to the operator of the ultrasound imaging apparatus regarding whether the one or more parameters indicative of the accuracy of the one or more of the obtained ultrasound images has met one or more predetermined thresholds.

22. An ultrasound imaging apparatus according to claim 21 , wherein if one or more predetermined thresholds have not been met for one or more ultrasound images, the computer system is further configured to provide an instruction to the operator to repeat the acquisition to acquire further ultrasound images.

23. An ultrasound imaging apparatus according to claim 17 or 18, wherein the computer system is further configured to: utilise a first subset of the obtained ultrasound images as a first set of ultrasound images in the ensemble machine learning model to determine an estimated fetal weight; apply one or more transforms to the first set of ultrasound images to generate an augmented set of ultrasound images; and utilise the augmented set of ultrasound images in the ensemble machine learning model to determine an uncertainty estimate for the first subset of the obtained ultrasound images; and repeat the applying and utilising for second and subsequent subsets of ultrasound images; and wherein the computer system is further configured to: provide a notification to the operator in relation to the uncertainty for one or more subsets of ultrasound images.

24. An ultrasound imaging apparatus according to claim 23, wherein the computer system is further configured to provide a notification to the operator in relation to subset of ultrasound images having the lowest uncertainty.

25. An ultrasound imaging apparatus according to claim 24, wherein the computer system is further configured to automatically select the subset of ultrasound images having the lowest uncertainty.

Images