Garbh-ini ultrasound image-based gestational age estimation system (GUAGE)

Abstract

The present invention comprises an Ultrasound Image-Based Gestational Age Estimation System and a method thereof. The invention introduces advanced deep learning models designed to estimate gestational age (GA) by utilizing ultrasound (US) images. The Garbh-Ini Ultrasound image-based Gestational Age Estimator (GAUGE) is specifically trained on ultrasound images of the fetal head. The invention also includes a mechanism to identify and discard images in the presence of data distribution shifts, thereby minimizing the risk of inaccurate predictions. Thus this method has provides clinicians with a sophisticated, accurate efficient and one of its kind tool for gestational age estimation.

Description

[0001] Garbh-Ini Ultrasound Image-Based Gestational Age Estimation System (GUAGE)

[0002] FIELD OF THE INVENTION:

[0003] The invention, in general, relates to the field of gestational age estimation systems. More particularly, the present invention relates to the use of ultrasound images of fetal head and deep learning models to develop a novel method to estimate gestational age during pregnancy.

[0004] BACKGROUND OF THE INVENTION:

[0005] The following background discussion includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention or that any publication expressly or implicitly referenced is prior art.

[0006] Gestational age (GA) estimation is the cornerstone of obstetric care, and its significance is widely known. GA not only influences clinical decision-making for monitoring fetal growth and other pregnancy -related complications, but also affects population-level estimates of pregnancy outcomes such as preterm birth, fetal growth restriction, and stillbirth.

[0007] Traditionally, GA is estimated using the first day of the last menstrual period (LMP) and applying Naegle’s rule. This method has several assumptions and is imprecise in a large proportion of women with recall bias, irregular menstrual periods, oral contraceptive use, and recent breastfeeding.

[0008] Current GA estimation models rely on fetal biometry measurements, which are susceptible to ethnic and pathological variations in fetal growth, especially in the second and third trimesters of pregnancy.

[0009] Thus, in the light of above discussion, it is imperative that there is need for an innovative method which does not use fetal biometry or LMP for gestational age estimation is necessary to overcome the limitations of above mentioned methods especially in populations of Low middle income countries like India where prevalence of abnormal growth patterns is high.

[0010] To address the above-mentioned challenges, the present invention provides a novel method for determining gestational age using the ultrasound images and deep learning models.

[0011] OBJECTIVE OF THE INVENTION:

[0012] The primary object of the present invention is to develop a novel methodology for gestational age estimation using ultrasound images and deep learning models.

[0013] Another object of the present invention is to build-in a foolproof mechanism in the method to avoid errors in the prediction.

[0014] Another object of the invention is to ensure the method developed will estimate gestational age accurately in even in fetuses with abnormal growth patterns.

[0015] Another object of the invention is to provide the method as an easy to use, clinically deployable stand-alone tool that can be utilised by healthcare practitioners in routine clinical care.

[0016] SUMMARY OF THE INVENTION:

[0017] The Invention provides a Garbh-Ini Ultrasound Image-Based Gestational Age Estimation System comprising: a processor to perform conformal prediction (CP) algorithm to detect and reject images when there is a data distribution shift, preventing erroneous predictions, and thereafter image segmentation where the images are manually annotated by Computer Vision Annotation Tool to generate the ground truth; said system comprises a multi-input model which captures multiple images and predicts gestational age to evaluate if this combination of images can improve the prediction. The system comprises encoder and decoder of a model based on a ResNet34 backbone and AdamW optimizer to optimize a combination of Binary Cross Entropy and Mean Squared Error Loss for the task of image segmentation and regression respectively.

[0018] The encoder extracts features from the input images which are used by decoder to segment the head region and by regression head for GA prediction and thereafter the losses of both tasks were combined to calculate a total loss which was used to optimize the model.

[0019] The system also works on the reference feature vector is determined by averaging the feature vectors of all the samples in the training dataset and the feature vector for each image is compared to the reference feature vector along with the Mahalanobis Distance (MD) between them and thereafter the accept / reject decision is done by comparing MD of the new image to distribution of MD’s calculated on training data.

[0020] DETAILED DESCRIPTION OF THE DRAWINGS:

[0021] To further clarify the advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting in their scope. The invention will be described and explained with additional specificity and detail with the accompanying drawings in which:

[0022] Fig. 1: Illustrate the dataset showing the number of participants enrolled in Garbh-Ini cohort and images considered for analyses. *POG: Period of gestation.

[0023] Fig. 2: Illustrate the architecture of GAUGE model of the present invention.

[0024] Fig. 3: Illustrate the Regression Activation Maps (RAM) showing the model’s dependency on finer details within the head region.

[0025] Fig. 4: Illustrate the Comparison of error distributions of GAUGE, Hadlock, and INTERGROWTH-2 1st models of the present invention.

[0026] Fig. 5: Illustrate the experiments demonstrating that the model is dependent on the finer details in the image of the present invention. Fig. 6: Illustrate the comparison of error distributions in fetuses with abnormal growth patterns i.e. small for gestational age (SGA) and appropriate for gestational age (AGA) groups of the present invention.

[0027] DETAILED DESCRIPTION:

[0028] For the purpose of promoting an understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates.

[0029] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be restrictive thereof.

[0030] In an embodiment, the present invention provides deep learning models for GA estimation using ultrasound (US) images taken at 18-35 weeks of pregnancy from 2207 participants of Garbh-Ini - a hospital-based prospective cohort of pregnant women in North India.

[0031] In an embodiment, the present invention provides a novel conformal prediction (CP) algorithm to detect and reject images when there is a data distribution shift, preventing erroneous predictions. The Garbh-Ini Ultrasound image-based Gestational age Estimator (GAUGE) is trained on US images of the fetal head.

[0032] In an embodiment of the present invention, GAUGE relies on the finer details in the image instead of the fetal biometry and that this leads to a similar performance across small for gestational age (SGA) and appropriate for gestational age (AGA) groups. The ability of GAUGE to consider image features beyond derived biometry suggests that GAUGE offers a better choice for populations with a high prevalence of fetal growth restriction. In an embodiment of the present invention, a scenario, the only way of achieving accurate GA estimation, particularly in the latter part of pregnancy, are to rely on fetal anatomies that are spared in growth restriction conditions or to have non-biometry- based information in the ultrasound (US) images to determine GA.

[0033] In an embodiment of the present invention, Deep learning approaches do not rely on derived biometry and can automatically learn image features associated with GA.

[0034] In an embodiment, as depicted in figure. 2, novel conformal prediction framework that identifies out-of-distribution samples and prevents erroneous predictions. Gestational age Estimator (GAUGE), a deep learning-based model for GA prediction. The GAUGE is more accurate than existing biometry-based methods, depends on the finer details in the image, and performs consistently in both SGA and AGA groups.

[0035] In an embodiment, the US images of the head, abdomen and femur collected during the second and third trimesters are extracted from the image repository using OCR-based software developed in-house. The images segregated by the OCR software had text annotations on them which are not suitable for building machine learning -based models as they could introduce model bias. Custom pre-processing techniques are used to remove the biometry values on the bottom right of the images.

[0036] Further, the GA derived from Hadlock formula using CRL at < 14 weeks of pregnancy is considered the GA gold standard and available for all training and test images. For the image segmentation task, the images of fetal head are manually annotated using Computer Vision Annotation Tool to generate the ground truth.

[0037] Estimating GA from a US image is treated as an image regression task and to model it, the inventors modified the ResNet 34 architecture by replacing the classification layer with a regression layer. Then three separate models with the above-mentioned architecture (Figure 2) were trained on second and third trimester US images of the head, abdomen, and femur images. For developing the GAUGE model, selected only those GARBH-Ini cohort participants who are enrolled at < 14 weeks and for whom CRL based dating using Hadlock formula was available.

[0038] GAUGE has a UNET++ segmentation architecture at its core. To predict GA using the same architecture we have modified it by adding a separate regression head which utilizes the features of the UNET++ encoder. The encoder and decoder of the model have a ResNet34 backbone and use an AdamW optimizer to optimize a combination of Binary Cross Entropy and Mean Squared Error Loss for the task of image segmentation and regression respectively.

[0039] In an embodiment, the present invention used all three images while training and internal testing. While randomly splitting we used participant id and ensured that all the images of the same participant are present in the either training set or the test set. The external validation dataset has only one image per participant. The images were normalized using the mean and standard deviation of the entire dataset and histogram equalization, and contrast enhancement augmentations were randomly applied.

[0040] Further, all the models are trained in Python 3.8.12 using the Fastai 2.3.0, Pytorch Lighting, and Pytorch 3.9.12 deep learning libraries. The models were trained on a Linux based server having a 72 core Intel Xeon processor, 256 GB of RAM and used a single Nvidia RTX 2080 Ti GPU with 11GB VRAM. For conformal prediction, first, feature vectors for all the training examples are extracted by passing them through the encoder. The vectors obtained were averaged to calculate a reference feature vector. The Mahalanobis distance was calculated between the reference feature vector and the feature vector of every sample in the training dataset, feature vector of every sample in the training dataset. All these distances were transformed / X2 using 1 to achieve a normal distribution and were referred as transformed Mahalanobis distance.

[0041] In an embodiment, the present invention provides a multi-input model which takes all three images (head, femur, and abdomen) and predicts GA to evaluate if this combination of images can improve the prediction. The images of the fetal head harbor all the information required to predict GA and are sufficient to build accurate models. Based on these observations, only fetal head images are used for further modelling.

[0042] In an embodiment of the present invention, the baseline regression model based on head images, designed a novel multi-task model (GAUGE) that segments the head along with GA estimation as depicted in Fig. 2. The samples which are rejected by our CP framework have higher error rates in both internal and external validation datasets substantiating its use while predicting on new samples as depicted in Fig. 2. The CP framework identified images from 7.7% participants from the internal test and 62% from the external test set as out-of-distribution samples and rejected them before the prediction in Fig. 2.

[0043] Further as depicted un figure. 2, the encoder extracts features from the input images which are used by decoder to segment the head region and by regression head for GA prediction. The losses of both tasks were combined to calculate a total loss which was used to optimize the model, b Working of the conformal prediction framework during inference. The reference feature vector was calculated by averaging the feature vectors of all the samples in the training dataset (part of the image shown in grey background). During inference, the feature vector for each image is compared to the reference feature vector and Mahalanobis Distance (MD) between them is calculated. Lastly, accept / reject decision is made by comparing MD of the new image to distribution of MD’s calculated on training data.

[0044] In an embodiment, the present invention as showed in figure. 3, Violin plots showing the distribution of error in days. The error of GAUGE is centred around zero and a smaller range when compared to the biometry-based methods of Hadlock and INTERGROWTH-2 1st. The GAUGE model is 44% (MAE in days5.11 vs 2.8) and 35% (MAE in days 4.41 vs 2.8) more accurate than the widely used Hadlock and INTERGROWTH -21st models - a big advantage.

[0045] In an embodiment, the present invention Heat maps are created using a regression activation map technique as depicted in Fig. 4. This showed that the finer details within the skull are more predictive of GA when compared to the outline of the skull (from which fetal biometry is calculated). To further investigate this finding, the present invention uses image blurring and resizing to compromise the finer details in the image while maintaining the outline of the skull as depicted in Fig 5. When tested on the images that were blurred using Gaussian filters of sizes 3x3, 5x5, 7x7 and 9x9 pixels, the MAE of the GAUGE model increased to 5.62, 7.96, 9.16, and 9.48 days respectively as depicted in Fig. 5. Lastly, similar increasing trend in MAE was found (4.81, 7.21 and 12.26 days) when the model was tested on images where the spatial resolution was reduced by 75%, 50% and 25% respectively in Fig. 5.

[0046] In an embodiment, the present invention as depicted in Fig 6 the errors of the GAUGE, Hadlock and INTERGROWTH-21st models were compared in small for gestational age (SGA) and appropriate for gestational age (AGA) groups. The GAUGE model performs consistently in SGA and AGA groups (2.66 vs 2.94 days) whereas both the Hadlock (4.41 vs 6.51 days) and INTERGROWTH-21st(4.34 vs 4.55 days) models underestimate the gestational age in the small for gestational age as compared to appropriate for gestation babies.

[0047] In an embodiment, the present invention addresses the limitations of existing clinical GA estimation models and is appropriate for use in populations with high prevalence of abnormal fetal growth. The architecture of the GAUGE model is designed to focus on the features relevant to the fetal head for GA estimation (as depicted in Fig. 2). Although segmentation of the fetal head is separate from the prediction of gestational age, it enables to achieve better accuracy and generalization than the simple image regression model.

[0048] Additionally, the present invention provides a visual representation of the region of the fetal head which would inspire more confidence in the model amongst users. Despite performing two tasks, GAUGE model has similar computational requirements (26.1 million parameters) when compared to a simple image regression model (21.3 million parameters.

[0049] In an embodiment, the present invention is to provide a novel image-based conformal prediction framework to prevent GAUGE from predicting on data where there is a distribution shift relative to the data used to train the model, thereby preventing erroneous predictions. Our approach is different from the domain adaptation techniques, where the goal is to adapt the model by using the out-of-distribution data.

[0050] In an embodiment, the present invention developed deep learning models for GA estimation that are more accurate than those used in current routine clinical practice. This is due to our state-of-the-art deep learning models and training and pre-processing techniques, a large dataset from a prospective cohort, and an accurate gold standard. As we are using the images directly as input to the machine learning model rather than using the measured values of the fetal biometry. This is an additional advantage and enables GAUGE to reduce time, effort, and the inter-operator variability which occur during the manual measurement of fetal biometry in US images. GAUGE can be packaged within a software tool that seamlessly integrates into the sonologists’ workflow.

[0051] The advantages of the current invention include but not limited to:

[0052] Unparalleled Accuracy: Integration of geographic parameters significantly enhances the precision of gestational age estimation compared to existing methods.

[0053] Global Adaptability: The tool's Al-driven customization ensures accurate assessments across diverse populations by factoring in regional variations that influence fetal growth.

[0054] Optimized Prenatal Care: Highly reliable gestational age data allows healthcare providers to personalize care plans, identify potential risks promptly, and intervene early for improved outcomes.

[0055] Reduced Maternal and Neonatal Risks: Precise estimations minimize risks associated with premature and post-term births, resulting in healthier outcomes for both mothers and babies.

[0056] Clinical Efficiency: The system streamlines processes while minimizing human error, leading to more efficient gestational age determinations.

[0057] Applications of the current invention include but not limited to:

[0058] Data-Driven Decision-Making: Clinicians can confidently base care plans, treatment strategies, and delivery timing on accurate gestational age information.

[0059] Global Health Impact: The tool supports enhanced pregnancy care worldwide, potentially contributing to reducing maternal and infant mortality, particularly in resource-constrained settings.

[0060] Research Advancement: Precision data improves the quality of perinatal research studies, furthering the understanding of pregnancy outcomes and health trends.

[0061] Telemedicine Enablement: Facilitates accurate remote consultations and assessments, expanding the reach of prenatal care.

[0062] Educational Value: Serves as an effective training tool for healthcare professionals, enhancing their skills in gestational age determination

Claims

We claim:

1. A Garbh-Ini Ultrasound Image-Based Gestational Age Estimation System comprising: a processor to perform conformal prediction (CP) algorithm to detect and reject images when there is a data distribution shift, preventing erroneous predictions, and thereafter image segmentation where the images are manually annotated by Computer Vision Annotation Tool to generate the ground truth.

2. The system as claimed in claim 1, evaluates finer details in the image to obtain a similar performance across small for gestational age (SGA) and appropriate for gestational age (AGA) groups.

3. The system as claimed in claim 1, wherein the Gestational age is derived from Hadlock formula based on CRL at < 14 weeks of pregnancy which is considered as the GA gold standard for training and test images.

4. The system as claimed in claim 1, comprises a UNET++ segmentation architecture at its core comprising a separate regression head which utilizes the features of the UNET++ encoder.

5. The system as claimed in claim 1, comprises encoder and decoder of a model based on a ResNet34 backbone and AdamW optimizer to optimize a combination of Binary Cross Entropy and Mean Squared Error Loss for the task of image segmentation and regression respectively.

6. The system as claimed in claim 1, wherein the encoder extracts features from the input images which are used by decoder to segment the head region and by regression head for GA prediction and thereafter the lossesof both tasks were combined to calculate a total loss which was used to optimize the model.

7. The system as claimed in claim 1, wherein the reference feature vector is determined by averaging the feature vectors of all the samples in the training dataset and the feature vector for each image is compared to the reference feature vector along with the Mahalanobis Distance (MD) between them and thereafter the accept / reject decision is done by comparing MD of the new image to distribution of MD’s calculated on training data.

Images