Preterm birth prediction

Abstract

Systems and methods for predicting future medical events are based on the processing of medical images. Examples include the prediction of preterm birth and gestational age based on ultrasound images. The new ability to estimate the probability of future medical events before they are predicted offers new avenues for the development of preventative treatments.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 041,360, filed June 19, 2020, and U.S. Non-Provisional Patent Application No. 17 / 352,290, filed June 19, 2021, the disclosures of which are incorporated herein by reference in their entireties. [Technical Field]

[0002] The present invention is in the field of predictive diagnostics with application to a wide range of medical conditions, including but not limited to pregnancy. [Background technology]

[0003] Preterm birth is when a baby is born more than three weeks before the estimated due date. In other words, preterm birth occurs before the start of the 37th week of pregnancy. Preterm babies, especially those born very prematurely, often have complex medical problems. In general, preterm babies can experience a range of complications. However, the earlier a baby is born, the higher the risk of complications.

[0004] A fetal sonogram is an imaging technique that uses sound waves to create a picture of the fetus inside the uterus. Fetal ultrasound images help healthcare providers assess the baby's growth and development and monitor the pregnancy. In some cases, a fetal sonogram can be used to evaluate problems that may be present at the time of the ultrasound or to confirm a diagnosis. The first fetal sonogram is usually performed in the first trimester to confirm the pregnancy and estimate the gestational age. Subsequent sonograms are generally performed in the second trimester, when anatomical details become visible. Summary of the Invention

[0005] Ultrasound, and optionally other medical imaging techniques, are used to generate predictions regarding the outcome of a medical condition, such as pregnancy. These predictions may include, among other things, expected birth date, preterm birth, and / or the need to induce birth. The predictions are optionally used to anticipate preterm birth and provide specific corrective treatment. The predictions provide a heretofore unavailable method of treatment to provide anticipatory treatment to prevent or otherwise mitigate undesirable outcomes. The systems and methods of the present invention include prediction, anticipatory treatment, and / or the development of treatment based on the predictions.

[0006] Various embodiments involve using medical images, e.g., ultrasound images, as input to machine learning systems configured to generate quantitative predictive outputs based on the images. These machine learning systems may use regression, classification, and / or other machine learning algorithms. For example, any regression algorithm that outputs a range, such as quantile regression, may be used. In some embodiments, multiple algorithms are uniquely combined into a single AI.

[0007] Various embodiments also include pre-processing of images and / or pre-training of machine learning systems. For example, image pre-processing has been found to be useful when medical images are of poor or variable quality or have varying magnitudes, such as when the images are ultrasound images.

[0008] Various embodiments include real-time feedback during image acquisition. This feedback may be directed to image selection, acquiring better images, and / or acquiring more useful images. Feedback may also be based on processing of acquired images. For example, in some embodiments, evaluation of initial images is used to guide acquisition of further images that have predictive and / or diagnostic value. Optionally, an image processing system is included in the image capture device or is connected to the image processing device using a communications network, such as a local area network or the Internet.

[0009] Various embodiments of the present invention comprise a medical prediction system configured to predict preterm birth, the system comprising: an image storage configured to store ultrasound images, the ultrasound images including, for example, a fetus; image analysis logic configured to provide an estimate that the fetus will be born preterm based on the ultrasound images; a user interface configured to provide a user with at least the estimate that the fetus will be born preterm; and a microprocessor configured to execute at least a portion of the image analysis logic. The image analysis logic optionally comprises first logic configured to estimate the gestational age of the fetus based on the ultrasound images, where the gestational age is current at the time of generation of the ultrasound images; second logic configured to estimate the time to birth of the fetus based on the ultrasound images; and logic configured to calculate the estimated gestational age of the fetus at the time of birth of the fetus. The number of days until delivery can be calculated directly without the gestational age. In this case, a healthcare provider can calculate the number of days that delivery will be early.

[0010] Various embodiments of the present invention include a method of generating a quantitative prediction of preterm birth, the method comprising: acquiring a set of medical images including a fetus; analyzing the medical images using a machine learning system to produce a quantitative prediction, the quantitative prediction including an estimate of time to birth of the fetus or an estimate of gestational age of the fetus at birth; and providing the quantitative prediction to a user.

[0011] Various embodiments of the present invention include a method of training a medical prediction system, the method comprising: receiving a plurality of medical images, optionally including ultrasound images of a pregnant fetus; optionally filtering the images; optionally classifying the images according to views or features contained therein; optionally pre-training a neural network to recognize features within images or types of images; training the neural network to provide a quantitative prediction regarding the birth of the fetus, the quantitative prediction including an estimate of the gestational age of the fetus at birth, or an estimate of the current gestational age of the fetus and an estimate of the time remaining until birth of the fetus; and optionally testing the trained neural network to determine the accuracy of the quantitative prediction.

[0012] Various embodiments of the present invention include a method of collecting ultrasound images for training a neural network, the method comprising: scraping a social media account; identifying ultrasound images of a fetus in the social media account; identifying birth announcements in the social media account; calculating the time between posting the ultrasound image to the social media account and posting the birth announcement to the social media account; and using the ultrasound image and the calculated time to train a neural network to generate a prediction that the ultrasound image is indicative of preterm labor, where the prediction is optionally a quantitative prediction.

[0013] Various embodiments of the present invention include a method for identifying beneficial treatments based on medical predictions, the method including: determining a quantitative prediction of a future medical condition, the quantitative prediction including a probability that the medical condition will occur within a future time range and based on an analysis of medical images of the patient; providing a candidate treatment for the medical condition to the patient; repeating the steps of determining the quantitative prediction of the future medical condition and providing the candidate treatment to a plurality of patients; determining whether the candidate treatment provided a statistically relevant benefit to a plurality of patients over each time range; and identifying the candidate treatment as a beneficial treatment based on the statistically relevant benefit. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a medical prediction system configured to predict medical outcomes, according to various embodiments of the present invention. [Figure 2] FIG. 1 illustrates a method for making quantitative predictions according to various embodiments of the present invention. [Figure 3] FIG. 1 illustrates a method for training a medical prediction system according to various embodiments of the present invention. [Figure 4] FIG. 1 illustrates a method for identifying beneficial treatments based on medical predictions, according to various embodiments of the present invention. [Figure 5] 1 illustrates a method for acquiring ultrasound images in accordance with various embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] While artificial intelligence systems, such as machine learning systems and expert systems, have been used to identify features in medical images, e.g., reading chest x-rays, limited progress has been made in the area of ​​medical prediction using medical images. In contrast to previous diagnostic applications of artificial intelligence, the systems and methods disclosed herein use artificial intelligence to demonstrate the predictive value of medical image, and optionally other data processing. As an illustrative example, the predictive value of such processing is described in the context of predicting preterm birth using ultrasound.

[0016] The systems and methods disclosed herein can provide quantitative predictions that a fetus will be born early (e.g., preterm birth), or quantitative predictions of some other medical event. The quantitative nature of these predictions contrasts sharply with the prior art. For example, quantitative predictions do not simply identify that a pregnancy is "at risk." As used herein, a "quantitative prediction" includes at least one of a probability, a classification within multiple preterm birth classes, or a time estimate. For preterm birth predictions, the time estimate may be expressed in terms of estimated days until birth and / or estimated gestational age at birth. These estimates may be expressed over two, three, or more time ranges. An advantage of quantitative predictions is that they can provide much more actionable information compared to a simple (and error-prone) "at risk" classification.

[0017] 1 illustrates a medical prediction system 100 configured to predict a medical outcome, according to various embodiments of the present invention. Prediction system 100 may include multiple devices, such as an ultrasound system and a computing device configured for image processing. Prediction system 100 is optionally configured to communicate between various components and / or external devices via a communications network, such as the Internet.

[0018] Processing the images to generate a prediction includes generating an estimate that a particular medical condition and / or event will occur in the future, e.g., during a time period beginning at a future date. The estimate is optionally expressed as an absolute probability or a relative probability. Furthermore, the estimate may include a time component. For example, the estimate may be that there is a 66% chance of preterm birth, that there is a higher chance of birth between 33 and 34 weeks of gestation than between 37 and 38 weeks of gestation, and / or that there is a 50% chance of birth by 34 weeks of gestation. In another example, the prediction may include the probability of developing lung cancer or breast cancer within a future time period.

[0019] Prediction system 100 includes an optional image generator 110 configured to generate an image. Image generator 110 may include a conventional ultrasound or other imaging system further configured to provide images to other elements of prediction system 100 for processing, e.g., via a computer network. In various embodiments, image generator 110 is a system including an image generation device and any combination of one or more elements of prediction system 100. For example, image generator 110 may be an ultrasound device including storage 120, image analysis logic 130, user interface 150, and feedback logic 170 (further discussed elsewhere herein). In various embodiments, image generator 110 includes a radiographic (e.g., X-ray) based imaging device, a magnetic resonance imaging device, a nuclear imaging device, an ultrasound imaging system, an elastography device, a photoacoustic device, a tomography device, an echocardiography device, a magnetic particle imaging system, a spectroscopy (e.g., near-infrared) device, etc.

[0020] In some embodiments, other elements of prediction system 100 are directly connected to or included within image generator 110. For example, image generator 110 may include an ultrasound machine having image analysis logic 130 configured to provide real-time feedback via feedback logic 170 to guide the collection of ultrasound data and / or images. In some embodiments, image generator 110 comprises a sound source, a sound detector, and logic configured to generate ultrasound images based on sounds detected by the sound detector. Optionally, image generator 110 is configured to adapt the generation of ultrasound images based on feedback from image analysis logic 130. For example, sound generation, focus, and processing may be adapted to better detect blood perfusion in small fetal capillaries in response to an indication from image analysis logic 130 that images including such information would provide better predictions and estimates.

[0021] The image generator 110 is optional in embodiments in which the prediction system 110 receives externally acquired images or in embodiments in which raw image data is used for the prediction. For example, the image generator 110 is optional in embodiments in which raw ultrasound (sonogram) data, rather than images, is processed to generate the medical prediction. In some embodiments, the images and / or raw data are received by the prediction system 100 over a communications network, such as the Internet. The images generated by the image generator 110 may include a sequence of images depicting fetal movement. For example, the sequence may show blood flow, capillary abundance, heart rate, etc. Such a sequence may also include Doppler information related to the direction and speed of movement.

[0022] Prediction system 100 further comprises storage 120. Storage 120 includes digital memory configured to store raw sensor data, medical images, medical data, executable code (logic), neural networks, etc. For example, storage 120 may be configured to store raw sensor data generated by photon or acoustic detectors, which can be used to generate X-ray or ultrasound images. As discussed elsewhere herein, storage 120 optionally includes memory circuitry and optionally includes data structures configured to manage and store any of the above data types. Ultrasound images stored in storage 120 are optionally 600x600 pixels, with random crops of 400x400 (or at least 300x300) optionally used for training and / or prediction as discussed herein. As discussed herein, "ultrasound image" optionally includes a three-dimensional rendering based on ultrasound data.

[0023] In some embodiments, the storage 120 specifically includes circuitry configured to store ultrasound images of a pregnant mother and / or fetus. The ultrasound images may be generated in one or more acquisition sessions. For example, a first set of ultrasound images may be acquired by a sonographer in a single session, and a second set of ultrasound images may be acquired in a second session occurring at least 1, 2, 5, 7, 15, 21, 30, 60, 90, or 180 days later, or any range therebetween. Ultrasound images of a particular mother and / or fetus may be generated over a period of time, including any of the above durations. For example, a mother with a high-risk pregnancy may undergo ultrasound examinations once a week. The ultrasound images optionally include Doppler data and / or a sequence of images (e.g., video) depicting fetal movement. For example, the ultrasound images may show the fetal heartbeat or blood flow. The ultrasound images may further include information about the density of fetal tissue, fluid, or bone. Images that have been found to be useful for preterm prediction include images showing fetal heart rate, umbilical arteries, uterus (including the lower uterine segment), cervix, cervical views taken specifically to measure cervical length, amniotic fluid index views, abdominal circumference (AC), bilateral head diameter (BPD) and all other views of the brain, femur, humerus, endometrium (e.g., thickness and vascularization), placental margin relative to the cervix, fetal kidneys, placenta, adnexa, etc. In some embodiments, images useful for estimating fetal gestational age are processed along with images useful for estimating placental gestational age. The difference between these estimates is then used to predict preterm birth.

[0024] The prediction system 100 further includes image analysis logic 130 configured to provide a quantitative prediction that the fetus will be born preterm based on the ultrasound image and, optionally, clinical data. This prediction includes an estimate that can take various forms. For example, the prediction may be based on an estimate of the fetus's current gestational age and an estimate of the prenatal (remaining) time (time to delivery or days earlier than the standard 280 days). Alternatively, the estimate may include a probability that the fetus will be born with indicators of preterm birth (e.g., low birth weight), optionally independent of the current gestational age. The prediction made by the image analysis logic 130 may also include a probability estimate that a physician will choose to induce labor even while still in the preterm stage or at full term. For example, the prediction may include a probability that a physician will choose to induce preterm labor due to a condition such as preeclampsia. Such a prediction may be made at least 1, 2, 3, or 4 weeks, or 1, 2, 3, 4, 5, 6, or 7 months before the induction occurs. In the case of preeclampsia, if the condition manifests in a life-threatening way, caregivers may need to induce immediately, regardless of whether the fetus is born preterm.

[0025] In some embodiments, the image analysis logic 130 includes first logic configured to estimate the gestational age of the fetus based on the ultrasound image and second logic configured to estimate the time until birth of the fetus based on the ultrasound image. For example, the first logic is configured to use the ultrasound image to estimate the gestational age at the time of generation of the ultrasound image, while the second logic is configured to estimate the time remaining until birth of the fetus. In these embodiments, further logic in the image analysis logic 130 is then configured to calculate the estimated gestational age of the fetus at birth by adding the estimated gestational age to the time remaining until birth. The first logic and the second logic are optionally located within the same machine learning system. For example, they may be included in the same neural network that receives the ultrasound image as an input and outputs both the gestational age and the time remaining. By using the same machine learning system to calculate the estimated gestational age and the time remaining until birth, errors in both calculations may be related such that the overall error is less than would be expected if the two errors were independent.

[0026] The image analysis logic 130 may be configured to perform the above calculations using a wide range of machine learning algorithms. For example, in some embodiments, the image analysis logic 130 is configured to use a regression algorithm (e.g., quantile regression) to provide a prediction that a fetus will be born preterm, and optionally to estimate time to birth. Quantile regression, and the like, predicts a range within which the actual answer is likely to lie, rather than simply a single-valued answer. In various embodiments, any regression system that predicts a range rather than a point value may be used by the image analysis logic 130. Using a range as an estimate prevents overfitting of the data and is useful when ultrasound images used for training may be mislabeled. The image analysis logic is generally configured to make predictions and estimates based on a set of ultrasound images, rather than analyzing a single ultrasound image.

[0027] In some embodiments, the image analysis logic 130 is configured to use a classification algorithm to provide a prediction that a fetus will be born preterm. In contrast to traditional classifications that categorize births into two classes, "term" and "preterm" (preterm births are 22 days or more early), the classification algorithm optionally includes three, four, or more classifications related to birth timing. A neural network using the classification algorithm can assign pregnancies to specific classes, such as a range of estimated birth times, thereby providing a quantitative prediction regarding the time remaining until birth. These classifications may include, for example, "preterm" if the number of days early is 29 days or more, "borderline" if the number of days early is 14-28 days, "term" if the number of days early is less than 14 days, etc. In practice, the date ranges for these classifications may vary by + / - 1-4 days in different implementations. In alternative embodiments, more and / or different classifications are used.

[0028] When using a classification algorithm, a "label smoothing" function is optionally applied. A smoothing function can be useful because some training images may be labeled incorrectly due to the undeveloped state of the ultrasound images at the time they are generated. This smoothing function can take the form described below, for example, where epsilon (ε) is optionally 0.05, 0.1, 0.3, or greater.

[0029] JPEG0007733922000001.jpg25166JPEG0007733922000002.jpg40148JPEG0007733922000003.jpg130166

[0030] In some embodiments, label smoothing is used when the loss function is cross-entropy, and the model applies a softmax function to the logit vector z in the penultimate layer to calculate its output probability p. Label smoothing is a regularization technique for classification problems that prevents models from overly confidently predicting labels during training and generalizing poorly. See, for example, https: / / leimao.github.io / blog / Label-Smoothing / .

[0031] In some embodiments, both regression and classification algorithms are used to predict preterm labor and / or time to birth. For example, image analysis logic 130 may include two separate neural networks, one configured to apply a regression algorithm (which outputs a range) and another configured to apply a classification algorithm. In this case, the classification algorithm is applied before the regression algorithm, and the regression algorithm is optionally applied separately to each class.

[0032] Alternatively, both the regression algorithm and the classification algorithm may be applied by the same neural network. In such an embodiment, the neural network is trained to produce both classification and regression-based predictions, both of which are quantitative. The regression algorithm outputs one or more values ​​for each selected percentile. For example, some embodiments use 10%, 25%, 50%, 75%, and 90% percentiles for the output (which represent percentiles of the quantitative prediction), and each of these percentiles may be associated with a probability and / or confidence measure. From an input of a set of images, the neural network of the image analysis logic 130 typically generates one or more values ​​for each selected percentile. Multiple outputs from different algorithms may be used to confirm predictions of preterm birth and / or time to birth. Because the regression algorithm and the classification algorithm should produce the same results, this scenario is optionally used to establish the reliability of the overall prediction.

[0033] The image analysis logic 130 may employ other machine learning algorithms or combinations thereof in addition to or as an alternative to regression and classification. For example, the image analysis logic 130 may be configured to apply regression that outputs a range of estimates, which is more accurate and / or useful than a single-point prediction. However, single-point predictions can be used when many neural networks are generated from different subsets of data (each trained on a different subset of images / data) and statistically analyzed to form averages and / or distributions. In some embodiments, Bayesian convolutional neural networks are used to capture epistemic uncertainty, which is uncertainty about model fit due to limited training data. Specifically, instead of learning specific weight (and bias) values ​​in a neural network, Bayesian methods learn a sample weight distribution that produces an output for a given input, thereby encoding the uncertainty in the weights. Bayesian networks can also be used in a similar manner in the predictive classification techniques discussed herein.

[0034] As described elsewhere herein, the image analysis logic 130 may be configured (using the above algorithms / machine learning techniques) to provide a prediction that the fetus will be born preterm. This prediction may include the probability of preterm birth, the current estimated gestational age, the estimated time until the fetus is born, and / or the estimated total gestational age. These predictions are based on processing of the ultrasound image and, optionally, other factors associated with the pregnancy. For example, the image analysis logic 130 may be configured to generate the prediction based on clinical data. This clinical data may include, for example, one, two, or more of the following: maternal genetics, maternal weight, maternal reproductive history, maternal blood glucose level, maternal cardiac function, maternal kidney function, maternal blood pressure, placental condition, maternal infection, maternal nutrition, maternal medication use (smoking and alcohol use), maternal age, maternal socioeconomic status, maternal home environment, maternal income, maternal race, and / or characteristics of the maternal cervix or uterus. Image analysis logic 130 is optionally configured to take any one or combination of these clinical data as inputs and make the inferences and predictions discussed herein based in part on these clinical data.

[0035] Prediction system 100 optionally further includes calculation logic 140 configured to calculate a useful output based on the estimates made by image analysis logic 130. For example, calculation logic 140 may be configured to calculate a total gestational age based on the current gestational age and the estimated time to birth. Calculation logic 140 may be configured to calculate the total gestational age based on a probability distribution (e.g., a distribution expressed in percentiles). Calculation logic 140 may be configured to calculate a probability of preterm birth based on the estimated time to birth or the estimated total gestational age. For example, calculation logic 140 may apply a distribution function to the estimates made by image analysis logic 130 to create a probability distribution therefrom. In some embodiments, image analysis logic 130 is configured to generate characteristics of this distribution function. For example, in some embodiments, an estimate of the reliability of the predicted time to birth and the reliability estimate can be used to determine the width (e.g., standard deviation) of the distribution function. Calculation logic 140 is optionally included in image analysis logic 110.

[0036] Prediction system 100 optionally further comprises a user interface 150 configured to provide a user with the estimates and / or predictions made using image analysis logic 130. User interface 150 optionally includes a graphical user interface (and associated logic) and may be displayed on an instance of image generator 110, a mobile device (in which case user interface 150 may include a mobile app), or a computing device remote from image generator 110 and / or image analysis logic 130. For example, user interface 150 may be configured to display at least one or two of the fetus's gestational age at birth, an estimate that the fetus will be born preterm, an estimated time until birth, and / or an estimated gestational age. In some embodiments, user interface 150 is configured to upload one or more ultrasound images by a remote user for processing by image analysis logic 130.

[0037] As discussed further herein, in some embodiments, the user interface 150 is configured to provide real-time feedback to the user. For example, the user interface 150 may be used to instruct the sonographer during an ultrasound session to generate images that will provide better predictions and / or estimates related to preterm birth.

[0038] Prediction system 100 optionally further includes a data input 160 configured to receive data related to the pregnancy, such as clinical data related to the fetus and / or the fetus's mother. Data input 160 is optionally configured to receive any of the clinical data discussed herein. The clinical data may be used by image analysis logic 130 to generate the estimates and / or probabilities discussed herein. For example, this data may include any of the clinical data discussed herein or input from a user of image generator 110. In some embodiments, data input 160 is configured to receive medical images, such as ultrasound images, from a remote source.

[0039] Prediction system 100 optionally further includes feedback logic 170. Feedback logic 170 is configured to guide the acquisition of ultrasound images based on the quality of the pregnancy-related estimates and / or predictions. For example, if analysis of ultrasound images acquired during an imaging session using analysis logic 130 results in insufficient accuracy, precision, and / or reliability of the predictions and / or estimates, feedback logic 170 may use user interface 150 to inform the user that additional ultrasound images are desirable.

[0040] Additionally, in some embodiments, the feedback logic is configured to prompt the user to acquire ultrasound images of specific features, such as fetal heartbeat movement, fetal heart rate, placenta, cervix, fetal blood flow, fetal bone development, fetal spine, fetal kidneys, fetal capillary blood perfusion, umbilical arteries, uterus, lower uterine segment, cervical views taken specifically to measure cervical length, amniotic fluid index (AFI) views, abdominal circumference (AC), bilateral cephalic diameter (BPD) and all other brain views, femur, humerus, endometrium, endometrial vascularization, placental margin relative to the cervix, fetal kidneys, adnexa, etc. In some embodiments, the image analysis logic 130 is configured to classify the ultrasound images according to the objects and / or objects contained within the images. For example, distinct object classes may include any of the views and / or features discussed herein. In such embodiments, the image analysis logic 130 may be configured to identify objects in the ultrasound images and determine that sufficient quality images of the objects exist for each object classification. (Object classification is not to be confused with classifying ultrasound images by classes of expected gestational age at birth.) If there are not enough images, then user interface 150 may be used to request that the operator of image generator 110 acquire additional images containing additional objects. Accordingly, feedback logic 170 may be configured to indicate the need to collect additional ultrasound images useful in estimating that the fetus will be born preterm. In a particular example, image analysis logic 130 may be configured to request at least one set of images indicative of the gestational age of the placenta, at least one set of images indicative of the gestational age of the fetus (e.g., fetal bone development and / or fetal heart movement), and optionally, one set of images indicative of the condition of the uterus. In some cases, feedback logic 270 is configured to guide the positioning of the image generator (e.g., ultrasound probe) to generate images more useful in estimating that the fetus will be born preterm. Such guidance may include positioning the ultrasound probe in a specific location or written / audio requests such as, "Get an image showing the full length of the femur."

[0041] In various embodiments, the feedback logic 170 is configured to induce or request the collection of new images that are useful for training future models to achieve greater accuracy.

[0042] Prediction system 100 optionally further includes training logic 180. Training logic 180 is configured to train image analysis logic 130, feedback logic 170, image acquisition logic 190, and / or any other machine learning systems discussed herein. Such training is generally directed to the end goal of learning to make a quantitative prediction and / or estimate related to whether a fetus will be born preterm. For example, training logic 180 may be configured to train image analysis logic 130 to make a quantitative prediction and / or estimate of the gestational age of the fetus at birth. As described elsewhere herein, this prediction can be made using both a quantile regression algorithm and a classification algorithm, either together or separately.

[0043] While training logic 180 may use any applicable neural network training algorithm known in the art, training logic 180 optionally includes various improvements disclosed herein to better train the neural network. For example, in some embodiments, training logic 180 is configured to pre-train the neural network of image analysis logic 130 to better recognize features in ultrasound images. This pre-training may include training on images with various orientations, contrasts, resolutions, viewpoints, etc., and may be targeted to recognizing anatomical features in ultrasound images. Pre-training is optionally performed using unlabeled data.

[0044] In some embodiments, training logic 180 is configured to generate additional training images when training images for a particular condition are sparse or infrequent. For example, once the images and features that are most predictive are known, training logic 180 can take a subset of images and use a GAN (generative adversarial network) to generate new training images that include features that predict extreme preterm birth, etc.

[0045] In some embodiments, training logic 180 is configured to train on multiple sets of images, optionally from different mothers. Training on multiple sets of images rather than a single image can reduce overfitting of the data. Preferably, each set is large enough to ensure that there are at least some images in the set that contain information useful for making the quantitative predictions discussed herein.

[0046] In some embodiments, training logic 180 is configured to train image analysis logic 130 to enhance images. For example, image analysis logic 130 may be pre-trained to enhance poor quality ultrasound images or to reveal features such as fetal capillary blood perfusion and / or vascularization that are not normally visible in the ultrasound image being processed. With such enhancements, handheld ultrasound images can be used to generate processed images to make quantitative predictions related to preterm birth.

[0047] The prediction system 100 optionally further includes image collection logic 190. The image collection logic 190 is configured to collect training images and birth information from unconventional sources, such as social media accounts. For example, the image collection logic 190 may be configured to scrape social media accounts (i.e., Instagram® or Facebook®) to automatically identify prenatal ultrasound images and then correlate those images with birth announcements posted within the same social media account. The time between the ultrasound image and the birth announcement can be used to approximate the time between the ultrasound collection and birth, i.e., the remaining gestational age until birth. Such social media collection information is optionally used by the training logic 180 to train the image analysis logic 130. In some embodiments, ultrasound images (or videos) retrieved from social media will include the date and time on the image or on the social media website. It is also somewhat common to write the number of weeks of pregnancy on the image.

[0048] In some embodiments, training logic 180 is configured to train the neural network in multiple stages, e.g., transfer learning. For example, the neural network may first be trained to recognize relevant fetal features, then trained to estimate gestational age, and then trained to provide a quantitative estimate of gestational age at birth or length of time to birth.

[0049] Prediction system 100 typically further includes a microprocessor 195 configured to execute some or all of the logic described herein. For example, microprocessor 195 may be configured to execute portions of image analysis logic 130, calculation logic 140, feedback logic 170, training logic 180, and / or image acquisition logic 190. Microprocessor 195 may include circuitry and / or optical components configured to perform these functions.

[0050] 2 illustrates a method for making a quantitative (optionally medical) prediction according to various embodiments of the present invention. The quantitative prediction is optionally used to anticipate preterm birth and provide specific corrective treatment. Thus, the method of FIG. 2 is optionally followed by appropriate therapy and / or treatment.

[0051] In image acquisition step 210, a set of one or more images is acquired. These images typically relate to a specific patient, e.g., a pregnant woman and her fetus. These images may be acquired from a source external to prediction system 100 or may be acquired using image generator 110. For example, in some embodiments, ultrasound images are uploaded to storage 120 over a computer network such as the Internet. Images may be received from an electronic medical record system. In other embodiments, images are generated using a medical imaging system, such as any of those discussed herein. The images are optionally stored in storage 120. The images may include any combination of views and / or features discussed herein and are optionally classified based on their respective views and features (object classification).

[0052] In an optional data receiving step 220, additional clinical data about the patient (mother or fetus) is received. Again, this data may be received from an electronic medical record system or may be provided by the patient and / or caregiver. The received clinical data may include any of the clinical data discussed herein and is optionally received via data input 160.

[0053] In image analysis step 230, the images acquired in image acquisition step 210 are analyzed using image analysis logic 130. The images are analyzed to produce one or more quantitative predictions. In the case of pregnancy, the quantitative predictions generally include quantitative estimates related to preterm birth of the fetus. For example, the quantitative predictions may include an estimate of the fetus's current gestational age (at the time of image capture), an estimate of the time until birth of the fetus, and / or an estimate of the fetus's gestational age at birth. The predictions are "quantitative predictions" as defined elsewhere herein. In addition to the images, the quantitative predictions are optionally further based on clinical data received in data receiving step 220. The analysis methods in image analysis step 230 may include any combination of algorithms and / or machine learning systems discussed elsewhere herein, including those discussed with reference to image analysis logic 130. For example, analyzing the medical images may include using a quantile regression algorithm and / or a classification algorithm to produce a quantitative prediction related to preterm birth of the fetus. In another example, analyzing the medical images includes using a regression algorithm to provide a prediction that the fetus will be born preterm and optionally estimating time to birth, the regression algorithm configured to classify the estimated gestational age of the fetus at birth into one of at least two or three time ranges.

[0054] Examples of quantitative predictions that may be generated in the image analysis step 230 include the probability that the fetus will be born preterm, the probability that the fetus will be born within one or more preterm ranges, the probability that labor will be induced, the probability that the fetus will be born within a future time (where "future time" refers to a time period with a future starting date), classification of the birth as "borderline preterm," "preterm," or "extremely preterm" (these classes have defined time periods), prediction of adverse medical conditions related to the pregnancy, prediction of post-birth health problems for the mother and / or fetus, prediction that the placenta will not be expelled intact, etc.

[0055] An optional provide feedback step 240 provides a user (e.g., a caregiver) with feedback regarding the acquisition of images. This feedback can be based, for example, on the quality of quantitative predictions and / or classification of images already acquired. In a specific example, during an ultrasound session, a caregiver may be requested to acquire additional images with different resolutions, different views, different features, etc. After the image acquisition step 210 and image analysis step 230 are optionally repeated, the provide feedback step 240 follows.

[0056] In a prediction providing step 250, the quantitative prediction generated in the image analysis step 230 is provided to a user, such as a patient or caregiver. The prediction is optionally also located in storage 120 and / or an electronic medical record (EMR) system. In various embodiments, the prediction is provided via a web interface, via an EMR system, via a mobile application, on the display of the image generator 110, on the display of a computing device, etc.

[0057] 3 illustrates methods for training a medical prediction system according to various embodiments of the present invention. The methods illustrated in FIG. 3 are optionally used to train image analysis logic 130, feedback logic 170, and / or image acquisition logic 190. These methods may be performed using training logic 180.

[0058] In an image receiving step 310, a plurality of medical images is received as a training set. The received medical images optionally include ultrasound images of a fetus during pregnancy. The received images may optionally be acquired using the method illustrated in FIG. 5 and stored in storage 120.

[0059] In an optional classification step 320, the received images are classified according to views or features contained within the image. For example, an image may be classified as showing a fetal heart or as showing a placenta. Classification step 320 is optionally performed by a neural network included in image analysis logic 130 and / or trained by training logic 180. Classification step 320 may also include classifying the images according to the gestational age (actual or estimated) of the fetus in each image and / or the known outcome of the respective pregnancy. For example, an image may be classified as having been generated at 12 weeks gestation, or as being of a fetus born several weeks after the image was generated, and / or as being of a fetus / mother that was born prematurely (with varying degrees of premature birth).

[0060] In an optional filtering step 330, the received images are filtered. Filtering may include removing images lacking features or views that are determined to have little or no predictive value. For example, a class of maternal bladder images may be determined to have little value in determining quantitative predictions, and images of this class may be removed from the training set. Images may also be filtered according to image quality or resolution, etc.

[0061] In some embodiments, the filtering step 330 includes balancing the images in various classes. For example, for training purposes, it may be desirable to have approximately equal numbers of extremely preterm, preterm, induced, and full-term births. Specifically, the balancing step may be used to adjust the amount of images based on gestational age at birth. Similarly, for training purposes, it may be desirable to balance the number of images in the training set based on the view and / or feature classifications determined in the classification step 320.

[0062] In an optional pre-training step 340, the neural network is optionally pre-trained to recognize features within an image or type of image. For example, the neural network within image analysis logic 130 may be pre-trained to recognize features of various orientations, resolutions, and / or qualities of ultrasound images.

[0063] In a training step 350, the neural network is trained to provide quantitative predictions regarding the birth of the fetus. As discussed elsewhere herein, the quantitative predictions may include an estimate of the gestational age of the fetus at birth, an estimate of the current gestational age of the fetus, and / or an estimate of the time remaining until birth of the fetus.

[0064] In an optional testing step 360, test images are used to test the predictions made by the neural network trained in the training step 350. This testing may be performed to determine the precision and / or accuracy of the quantitative predictions made by the neural network.

[0065] FIG. 4 illustrates a method for identifying beneficial treatments based on medical predictions, according to various embodiments of the present invention. The systems and methods disclosed herein, which enable quantitative predictions of medical conditions and / or outcomes, offer new opportunities for identifying beneficial treatments, such as treatments that can prevent or ameliorate undesirable medical conditions or events. The quantitative nature of the predictions disclosed herein allows quantitative changes in outcomes to be detected and used to evaluate the effectiveness of treatments. This approach may be used for any of the medical conditions discussed herein. Furthermore, the medical condition need not be present or evident at the time the quantitative prediction is determined. Treatments may include the administration of medications or physical treatments. In certain examples, the effectiveness of physical treatments, including bed rest, for pregnant women can be prescribed and / or evaluated based on ultrasound images acquired at least 1, 2, 3, 4, or 6 months prior to birth. Treatments may also include changes in maternal behavior, such as reducing physical activity or avoiding physically strenuous work. Treatments may further include follow-up ultrasound examinations and evaluations.

[0066] Quantitative predictions of future medical conditions or events are optionally used to selectively select populations for clinical trials. For example, assuming that extremely preterm birth occurs in less than 1% of pregnancies, it would be inefficient to administer a candidate treatment to the general population of pregnant women early in pregnancy in order to detect a benefit in the unidentified 1%. However, by identifying the 1% of pregnancies most likely to result in extremely preterm birth, the benefit of the candidate treatment can be investigated within this population. Such investigations are much more efficient and are more likely to reveal a benefit with better statistical relevance. The systems and methods disclosed herein may be used to identify such preferred populations for clinical studies. This approach is particularly beneficial for diseases that begin to develop (and benefit from treatment) well before overt symptoms appear, such as Alzheimer's disease or dementia.

[0067] In prediction determination step 410, a quantitative prediction of a future medical condition is determined for a particular patient. As discussed elsewhere herein, the quantitative prediction can include a probability that a medical condition will occur within a future time range, and the quantitative prediction can be based on an analysis of the patient's medical images. As used herein, a "future" time range refers to a time range beginning at a future time, rather than a time range beginning from the current time. For example, at the time of a second trimester ultrasound, the future time range may be one to two weeks or two to four weeks prior to full-term pregnancy. The quantitative prediction may be made using any of the systems and / or methods described herein. In various embodiments, the time delay from determining the quantitative prediction to the future time range may be weeks, months, or years. For example, in the case of preterm birth, the delay may be at least one, two, three, four, or five months.

[0068] In candidate providing step 420, a candidate treatment is provided to a specific patient. The treatment may include administration of medication and / or physical measures such as bed rest, a specific diet, physical exercise, physical therapy, mental exercises, dialysis, the use of support clothing, etc. In some embodiments, the candidate treatment is provided only to patients who meet the criteria of the quantitative prediction. Optionally, the candidate treatment is provided only to patients with a quantitative prediction of greater than 50%, 66%, or 75% that the medical condition will occur within a future time range. For example, a medication may be provided only to pregnant women who have a greater than 75% risk of extremely preterm birth of the fetus. The candidate treatment may be provided to a patient before the patient exhibits any outward symptoms of the medical condition.

[0069] In an iterative step 430, the steps of determining a quantitative prediction of a future medical condition and providing a candidate treatment are repeated for a number of patients, the number of patients may be selected to include a statistically sufficient population.

[0070] In the relevance determination step 440, it is determined whether the candidate treatment resulted in a statistically relevant benefit for multiple patients over each time range. The relevance determination step 440 is typically performed for each patient at some point after the prediction determination step 410 and the candidate providing step 420. The relevance determination step 440 may be performed for multiple respective time ranges for each patient and / or for different respective time ranges for different patients. The time ranges are time ranges for which future medical conditions were predicted in the prediction determination step 410. The relevance determination step 440 optionally includes a step of comparing patients who received the candidate treatment with patients who received a placebo. As used herein, a "statistically relevant benefit" refers to a benefit that leads to acceptance and / or government approval of a treatment, or motivation for further research / improvement of a treatment.

[0071] In the identification step 450, candidate treatments are identified as beneficial based on statistically associated benefit. Note that improvements statistically associated with a candidate treatment are also an indication that a population of patients is suitable for the intervention. This information is optionally used to improve the intervention (treatment).

[0072] Figure 5 illustrates methods for acquiring ultrasound images according to various embodiments of the present invention, in which posts to social media accounts are used to generate training data, which is optionally used to perform the training method illustrated in Figure 3. The method illustrated in Figure 5 may optionally be performed using image acquisition logic 190 and may be applied to images other than ultrasound images.

[0073] The scraping step 510 involves reviewing and accessing content from social media accounts, such as Facebook® or Instagram®, which may be performed using a browser or an automated system, such as a web crawler.

[0074] In an image identification step 520, ultrasound images containing a fetus are identified within the accessed social media accounts. The social account name (e.g., username and / or URL) and posting date of the image are recorded. Optionally, but not necessarily, the ultrasound images are downloaded and stored in storage 120.

[0075] In a birth identification step 530, the social media accounts recorded in the image identification step 520 are revisited to identify birth announcements, which are then assumed to be associated with the fetus seen in the ultrasound image.

[0076] A calculation step 540 calculates the time between posting the ultrasound image to the social media account and posting the birth announcement to the social media account, and the calculated time is then assumed to represent the time from the generation of the ultrasound image to the birth.

[0077] The ultrasound images and the calculated times are used to train a neural network, optionally included within image analysis logic 130, in a training step 550. The neural network is trained to generate an optional quantitative prediction that the ultrasound images are indicative of preterm birth.

[0078] The various methods shown in Figures 2-5 may be used in any combination, optionally performed using the system shown in Figure 1. For example, the methods of Figures 2 and 3, and / or 3 and 4 may be used together. The method of Figure 5 may also be used to generate training data for other methods disclosed herein.

[0079] EXAMPLES: The following are illustrative examples that may be included in any of the embodiments discussed herein.

[0080] All ultrasounds from every gestational age are used to train the neural networks discussed herein, although different models can be created for a range of gestational ages.

[0081] Note: The models below are just individual examples of how to create a working model. Many of the hyperparameters such as learning rate, batch size, number of epochs, etc. can be tuned without issue.

[0082] Import all the libraries used in your code.

number

[0083] The path below is a folder containing one subfolder for each predicted class: "Extremely Preterm," "Very Preterm," "Moderately Preterm," or "Normal." This is just one embodiment; any number of different classifications would work. For example, there could be two, three, four, or more classes.

[0084] Preterm births make up approximately 10% of the total dataset used, but training is likely more effective when using a balanced dataset, as in this case. The validation set is unbalanced in this embodiment to reflect the accuracy of real-world distributions, but may be balanced in other embodiments. In some embodiments, the neural network has an equal chance of being fed instances of one of the four categories. In various embodiments, at least 50% or 75% of the training set contains images that are balanced across the possible classes.

[0085] Arbitrarily, we set the image size to 400x400 pixels. Most of the images in this dataset are several times this size and are reduced in size to improve training and inference efficiency. Increasing image size increases accuracy but with diminishing returns. Alternative embodiments use images that are at least 224x224 pixels, 640x640 pixels, or 2048x2048 (pixels), or any range in between.

number

[0086] This creates an object that feeds data to the neural network during training and validation. 10% of the studies are placed into a validation set used to monitor training. The validation set contains the training data and the same distribution of preterm infants typically found in developed countries. While the data is imbalanced for training, the validation set is a natural distribution and is not balanced in any way. The batch size can be freely set to 28 images and adjusted as needed. The addition of the aug_transforms function randomly augments each image, thereby reducing overfitting. Examples of augmentations include, but are not limited to, adjusting brightness, adjusting contrast, or horizontally flipping the image. This embodiment uses binary cross entropy as the loss function, which trains as a multi-label classification problem, even though there is exactly one label for each image. Other embodiments can use other loss functions, such as categorical cross entropy or mean squared error if the data is viewed as a regression problem.

number

[0087] Create objects that provide high-level control over training and inference. Using a high value for weight decay is just one example; other forms of weight normalization may have a similar effect.

[0088] While we obtain a pre-trained network to use for transfer learning, it is also useful to train a neural network from an initial randomized formation. In this case, we use ResNet-152. Other types of resnets will also work, with more layers providing better accuracy. Many other neural networks will likely produce usable results. The following example shows one possible training scenario. Alternative embodiments may vary the parameters and steps.

number

[0089] This freezes the parameters of the convolutional portion of the neural network, allowing for training of only the linear layers. Other embodiments do not necessarily require the layer freezing step.

number

[0090] The scheduler trains the neural network for 10 epochs by gradually increasing and then decreasing the learning rate, with a maximum learning rate of 1e -3 Other embodiments may use alternative training schedules.

number

[0091] This allows for training of the entire neural network, including convolutional layers.

number

[0092] The scheduler further trains the neural network for 5 epochs by gradually increasing and then decreasing the learning rate, with a maximum learning rate of 1e -5 This becomes:

number

[0093] The scheduler further trains the neural network for 5 epochs by gradually increasing and then decreasing the learning rate, with a maximum learning rate of 1e -5 This becomes:

number

[0094] When the prediction threshold is set to 0.5, the best validation accuracy of this embodiment is 86%.

[0095] The network allows for the determination of the most useful specific anatomical view of the study for making this prediction. The specific anatomical view or other information provided to the network can be omitted if it is not generally informative, or various views can be weighted in calculating the final prediction.

[0096] Each prediction outputs a score of the model's confidence. This score can be used in real time on the ultrasound machine to notify the sonographer when it obtains views that improve accuracy. Alternatively, the software can automatically incorporate high-confidence views as it finds them, without requiring any effort from the technician. This technique can be used to create training sets for continually improving updated models, a feedback loop that incorporates better data to train future models.

[0097] A trained system is optionally used to provide real-time feedback to the ultrasound technician. For example, the system may notify the technician when it has acquired the most predictive image, or when it has not been able to do so. The system may request that the technician acquire images of specific anatomical structures of the mother, fetus, or both. When the system identifies an acquired image as predictive of preterm birth, it may request that the technician acquire additional images to confirm or refute the prediction of preterm birth. For example, if an image of a portion of the fetal anatomy (e.g., the heart) is predictive of preterm birth, the system may request that the technician acquire additional images of that anatomy or additional images of another portion of the fetal anatomy (e.g., the hands or face).

[0098] We determined that making predictions based on individual ultrasounds and then simply aggregating these individual predictions is useful but could be improved. The biggest challenge with this method is that within a single ultrasound recording session, some ultrasound images will confidently predict preterm birth, while ultrasound images of other anatomical locations will confidently predict a full-term outcome, making simple aggregation less effective. We therefore decided to create a neural network that would allow classification of many images in a single pass through the neural network or sequence of networks. Increasing the number of images simultaneously fed into a neural network generally increases accuracy. We therefore developed a more efficient way to pass many images through a network (or sequence of networks) (either in parallel or serially).

[0099] This does not necessarily have to be a classification problem: the value to predict can be a numerical value representing a desired target, and a neural network that performs regression is instead created.

[0100] Training data can be created by training a neural network to change images from one class to another when the data is restricted to certain classes. An example of a neural network that can convert images from one class to another is CycleGAN.

[0101] It is not necessary to use an image: the raw sound waves captured by the ultrasound machine before being converted into an image can also be used instead of, or in addition to, an image for this prediction.

[0102] [CNN-LSTM (Convolutional Neural Network - Long Short-Term Memory)] We then determined that we could create a neural network that could extract useful information from each image and aggregate it by combining multiple neural networks into a single neural network, although these networks could be separate in other embodiments. An ultrasound session is when a technician takes a set of images during a single interaction with a patient. Considering that many ultrasound sessions have over 100 images, this is useful for processing efficiency and accuracy. LSTMs are designed to be used with continuous (or time-series) data, where data points are correlated within a sequence. Although the ultrasound images in an ultrasound session have little, if any, order, this type of network can propagate information from previous steps to process one image and classify this combined information after each image in the ultrasound session has been processed. Because ultrasound technicians typically look at one anatomical feature and then move on to another, there is little order to the ultrasound session. It should be noted that the anatomical features used need not be limited to the mother or fetus, but could be a combination of both. It is possible to record a video of the entire session, and in that embodiment, the images in the video are continuous in nature, and this general technique can be employed.

number

[0103] Obtain data divided into training, validation, and test sets. The training data is created in an unconventional way: rather than using each study as a separate folder of images, create folders that randomly sample a range of images from the pool of all full-term studies and all preterm studies. It is also possible to sample images from each class pool at runtime for any desired sequence length. Other embodiments can use two, three, or more classes or values. There are several reasons for this:

[0104] First, the gestational ages of many ultrasound sessions overlapped the marginal error of 37 weeks, meaning that the training data contained a significant number of mislabeled studies.

[0105] Second, some investigations into preterm birth may be performed before the cause of preterm birth is present, or the cause of preterm birth may not be visible on ultrasound at any time point.

[0106] Random selection ensures that each training sample has an ultrasound that contains the information necessary to make a correct prediction.

[0107] An alternative method of prediction is to predict the time from the date of the ultrasound to birth and then calculate the class that time window falls into. This prediction is useful without subsequent classification.

[0108] It should also be noted that multiple ultrasound examination sessions are typically performed within a single pregnancy, and multiple sessions from a single pregnancy can be combined when making a prediction.

[0109] A single session may indicate preterm labor but may not be reliable, therefore the system may indicate that a follow-up session is advisable and when the follow-up session should occur.

[0110] Preterm birth is generally believed to have multiple underlying causes, each of which suggests different treatments. The network can inform physicians of causal anatomical views predicting preterm birth, allowing for more informed treatment choices.

[0111] The error in the neural network's predictions can be used to determine new treatments for preterm birth or to better understand preterm birth. For example, if the network predicts preterm birth but the baby is not born preterm, the patient's medical data can be mined to find commonalities and discover new treatments.

[0112] [Data Collection] Obtain the path to each folder containing images from one study.

number

[0113] In this example, the sequence length is 36 images, but smaller or larger ranges can be used. More images will likely work better, but with diminishing returns. The image size used is 400x400 pixels, which is significantly larger than the 224x224 pixels commonly used for image classification problems. However, smaller or larger images will also work.

number

[0114] The encoder class takes a model pre-trained on one ultrasound image, removes the final classification layer, and returns 512 features for each image. However, in other embodiments, the number of features can be as few as one or much more than 512.

number

[0115] The CNNLSTM module takes features from each image and classifies the entire sequence of images.

number

[0116] Create a Learner object that provides high-level control over training and inference. Using high values ​​for weight decay helps make correct predictions.

[0117] If you need to be GPU memory constrained, you can use GradientAccumulation, which accumulates 32-item gradients before updating the network weights.

[0118] ModelReseter resets the hidden state of the LSTM between batches.

[0119] cnnlstm_splitter allows you to freeze one part of the whole network while updating the weights of another part.

number

[0120] The scheduler trains the neural network for 5 epochs by gradually increasing and then decreasing the learning rate, with a maximum learning rate of 1e -3 These parameters are simply chosen for this embodiment, many other choices would also work.

number

[0121] Alternative Embodiments The individual predictions from the survey will be tallied and a tournament will be run to determine the final classification.

[0122] Reinforcement learning can be used instead of the CNN-LSTM shown.

[0123] It uses a number of models trained on different subsets of data to create an ensemble that typically improves accuracy.

[0124] Instead of the CNN-LSTM shown, we use ConvLSTM, which incorporates a CNN model into an LSTM cell.

[0125] Additional data about the patient is used in combination with the image, such as the patient's age or related events in their medical history.

[0126] In addition to the preterm birth class, additional outputs can be used. Some examples are days until birth, birth weight, current gestational age, etc. Having multiple outputs for a prediction can improve the overall accuracy of the prediction due to the inherent relationships between the various items being predicted. Any of these alternative predictions can be run without the preterm birth prediction if desired.

[0127] [Conclusion] The systems and methods disclosed herein have been applied to real data from clinical settings and have been shown to consistently yield positive predictive values ​​of greater than 90% and negative predictive values ​​of greater than 90%, in some cases achieving results including positive predictive values ​​of greater than 99% and negative predictive values ​​of greater than 97%.

[0128] Although some embodiments are specifically illustrated and / or described herein, it will be appreciated that modifications and variations are covered by the above teachings and within the scope of the claims without departing from the spirit and intended scope thereof. For example, while ultrasound images and preterm birth are taught herein as examples, the systems and methods described herein may be applied to other medical information and situations, such as predicting Alzheimer's disease, dementia, multiple sclerosis, long-term sequelae of infectious diseases, cervical cancer, ovarian cancer, uterine cancer, and / or any other medical condition for which precursor lesions may be present in ultrasound images. The disclosed methods and systems may be used to determine current clinical status, either individually or in combination with a prediction (optionally quantitative) of future status. The systems and methods disclosed herein may also be used to predict the future health status of a fetus after birth, such as future learning disabilities, cognitive ability, personality, poor development or impaired function of various organs, etc.

[0129] Although the teachings herein involve the use of medical images, e.g., ultrasound images, in various embodiments, the systems and methods may use raw data other than in the form of images. For example, the image analysis logic 130 is optionally trained to process raw ultrasound data rather than, or in addition to, images generated from the raw ultrasound data.

[0130] The embodiments discussed herein exemplify the present invention. Because these embodiments of the present invention have been described with reference to examples, various modifications or adaptations of the methods and specific structures described may become apparent to those skilled in the art. All such modifications, adaptations, or variations that rely on the teachings of the present invention and that have advanced the art through these teachings are considered to be within the spirit and scope of the present invention. Therefore, it is understood that the present invention is not limited to only the illustrated embodiments, and therefore these descriptions and drawings should not be considered in a limiting sense.

[0131] "Logic" as discussed herein is expressly defined to include hardware, software, or any combination thereof stored on a non-transitory computer-readable medium. This logic may be implemented in electronic and / or digital devices to create special-purpose computing systems. Any of the systems discussed herein optionally include a microprocessor that includes electronic and / or optical circuitry configured to execute any combination of the logic discussed herein. The methods discussed herein optionally include execution of logic by such a microprocessor.

[0132] The computing systems and / or logic referred to herein may include integrated circuits, microprocessors, personal computers, servers, distributed computing systems, communication devices, network devices, etc., and various combinations thereof. Additionally, the computing systems or logic may include volatile and / or nonvolatile memory, such as random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), magnetic media, optical media, nanomedia, hard drives, compact discs, digital versatile discs (DVDs), optical circuits, and / or other devices configured to store analog or digital information, such as databases. As used herein, computer-readable media explicitly excludes paper. Computer-implemented steps of methods described herein may include sets of instructions stored on computer-readable media that, when executed, cause a computing system to perform the steps. A computing system programmed to perform specific functions pursuant to instructions from program software is a special-purpose computing system for performing those specific functions. Data manipulated while a special-purpose computing system performs these specific functions is stored, at least electronically, in buffers of the computing system, and each change in the stored data physically changes the special-purpose computing system from one state to the next.

Claims

1. 1. A medical prediction system configured to predict preterm birth, the medical prediction system comprising: an image storage configured to store an ultrasound image, the ultrasound image including a fetus; and image analysis logic configured to provide a quantitative prediction that the fetus will be born preterm based on the ultrasound images, the image analysis logic configured to use a regression algorithm to provide the quantitative prediction including an estimate that the fetus will be born preterm within two or more time ranges; The image analysis logic includes: first logic configured to estimate, based on the ultrasound image, the gestational age of the fetus at the time the ultrasound image was generated; second logic configured to estimate a time to birth of the fetus based on the ultrasound image; and third logic configured to calculate an estimated gestational age of the fetus at birth. a user interface configured to provide a user with the prediction that at least the fetus will be born preterm; and a microprocessor configured to execute at least a portion of the image analysis logic.

2. The medical prediction system of claim 1 , further comprising an image generator configured to generate the ultrasound image.

3. 3. The medical prediction system of claim 2, wherein the image generator comprises a sound source, a sound detector, and logic configured to generate the ultrasound image based on sounds detected by the sound detector.

4. The medical prediction system of claim 2 , wherein the image generator is configured to adapt the generation of the ultrasound image based on an output of the image analysis logic.

5. The medical prediction system of claim 2 , wherein the image generator is configured to generate a sequence of images representing fetal movement.

6. The medical prediction system of claim 2 , wherein the image generator is configured to prompt a user to modify a position of a sound source configured to generate the ultrasound image.

7. a data input configured to receive clinical data relating to the fetus or the mother of the fetus; The medical prediction system of claim 1 , wherein the estimated time to birth of the fetus is based on the clinical data.

8. 10. The medical prediction system of claim 1, further comprising feedback logic configured to guide the acquisition of the ultrasound images based on the quality of the prediction that the fetus will be born preterm or a classification of the acquired images into object classes.

9. 9. The medical prediction system of claim 8, wherein the feedback logic is configured to guide positioning of an image generator to generate images that are more useful in predicting that the fetus will be born preterm.

10. 9. The medical prediction system of claim 8, wherein the feedback logic is configured to indicate a need to acquire additional ultrasound images useful in the prediction that the fetus will be born preterm.

11. The medical prediction system of claim 1 , further comprising training logic configured to train the second logic.

12. The medical prediction system of claim 1 , wherein the ultrasound images including the fetus are generated over a period comprising at least three months.

13. The medical prediction system of claim 1 , wherein the ultrasound images include Doppler information or fluid, bone, or tissue density information.

14. 2. The medical prediction system of claim 1, wherein the image analysis logic is configured to use the regression algorithm to output a range prediction to provide a prognosis that the fetus will be born preterm.

15. the image analysis logic is configured to use a classification algorithm to provide a prediction that the fetus will be born preterm; The medical prediction system of claim 1 , wherein the classification algorithm includes at least two birth timing classifications.

16. 2. The medical prediction system of claim 1, wherein the image analysis logic comprises a neural network configured to receive the ultrasound image and generate an output representing the prediction that the fetus will be born preterm.

17. 10. The medical prediction system of claim 1, wherein the image analysis logic comprises a neural network configured to receive the ultrasound images and generate an output indicative of time to birth of the fetus.

18. the quantitative prediction that the fetus will be born preterm comprises an estimate of the probability that the fetus will be born preterm; The medical prediction system of claim 1 , wherein the estimation is based on ultrasound images of the cervix and / or amniotic fluid index among the ultrasound images.

19. The medical prediction system of claim 1 , wherein the quantitative prediction is based on ultrasound images that include the endometrium and / or the uterine wall among the ultrasound images.

20. 2. The medical prediction system of claim 1, wherein the user interface is configured to provide at least two of the gestational age, the estimated time to birth, and the gestational age at birth of the fetus.

21. 8. The medical prediction system of claim 7, wherein the image analysis logic is further configured to make a prediction that the fetus will be born preterm based on clinical data received via the data input.

22. 22. The medical prediction system of claim 21, wherein the clinical data includes at least one of maternal genetics, maternal weight, maternal pregnancy history, maternal blood glucose level, maternal cardiac function, maternal kidney function, maternal blood pressure, placental condition, maternal infection, maternal nutrition, maternal medication use, maternal age, maternal cervical or uterine characteristics.

23. the image analysis logic includes machine learning logic; The medical prediction system of claim 11 , wherein the training logic is configured to pre-train the machine learning logic to recognize features in the ultrasound images.

24. the image analysis logic includes machine learning logic; 12. The medical prediction system of claim 11, wherein the training logic is configured to train the machine learning logic to make the quantitative prediction that the fetus will be born preterm.

25. the image analysis logic includes machine learning logic; 12. The medical prediction system of claim 11, wherein the training logic is configured to train the machine learning logic to make the prediction that the fetus will be born preterm using both the regression algorithm and the classification algorithm.

26. 1. A method of providing material for generating a quantitative prediction of preterm birth, the method comprising: acquiring a set of medical images including the fetus; and providing material using a machine learning system to analyze the medical images to generate the quantitative prediction for providing the quantitative prediction to a user, wherein the quantitative prediction comprises an estimate of time to birth of the fetus or an estimate of gestational age of the fetus at birth.

27. 1. A method for training a medical prediction system, the method comprising: receiving a plurality of medical images, the medical images including ultrasound images of a fetus during pregnancy; classifying the images according to views or features contained within the images; training a neural network to provide a quantitative prediction regarding the birth of the fetus, wherein the quantitative prediction comprises an estimate of the gestational age of the fetus at birth, or an estimate of the current gestational age of the fetus and an estimate of the time remaining until birth of the fetus; testing the trained neural network to determine the accuracy of the quantitative prediction.

Images