Premature birth prediction

Abstract

To provide systems and methods for prediction of premature birth and estimation of gestational age based on ultrasound images.SOLUTION: A method of providing specific corrective treatment in anticipation of a premature birth by optionally using quantitative prediction comprises: obtaining a set of one or more images; receiving additional clinical data regarding a patient (mother or fetus); analyzing the obtained images using image analysis logic so as to produce one or more quantitative predictions; providing a user (e.g., a caregiver) with feedback regarding acquisition of the images; and providing a user, e.g., a patient or caregiver, with the quantitative predictions generated.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a U.S. Provisional Patent Application (Application No. 63 / 041,360, filed June 2020). 19), and U.S. non-provisional patent application (application number 17 / 352,290, filed 202 The disclosure of the above patent application is incorporated herein by reference in its entirety. is incorporated herein by reference. [Technical Field]

[0002] The present invention provides a predictive diagnostic method applicable to a wide range of medical conditions, including but not limited to pregnancy. It is in the field. [Background technology]

[0003] Preterm birth is when a baby is born more than three weeks before the estimated due date. Preterm birth is birth before the start of the 37th week of pregnancy. Babies born prematurely often have complex medical problems. Symptoms vary, but the earlier the baby is born, the higher the risk of complications.

[0004] A fetal sonogram uses sound waves to create a picture of the fetus inside the uterus. Fetal ultrasound imaging is an imaging technique that allows healthcare providers to assess the baby's growth and development and In some cases, fetal ultrasound is used to help monitor the pregnancy. The first fetus can be examined to assess any problems that may be present or to confirm a diagnosis. Ultrasound is usually done in the first trimester to confirm pregnancy and estimate the length of the pregnancy. A wave scan is typically performed in the second trimester, when anatomical details become more visible. Summary of the Invention

[0005] Ultrasound, and optionally other medical data, can be used to generate predictions regarding the outcome of a medical condition, such as pregnancy. These predictions include, among other things, the expected date of birth, preterm birth, and / or This prediction may optionally be used to specifically This prediction allows for proactive treatment to prevent undesirable outcomes. This provides a heretofore unavailable method of treatment that prevents or otherwise alleviates the symptoms of rheumatoid arthritis. The systems and methods of the present invention facilitate prediction, anticipatory treatment, and / or predictive treatment development. include.

[0006] Various embodiments provide a mechanism configured to generate quantitative predictive output based on the images. This involves using medical images, e.g., ultrasound images, as input to the learning system. These machine learning systems use regression, classification, and / or other machine learning algorithms. For example, any regression algorithm that outputs a range, such as quantile regression, In some embodiments, multiple algorithms may be used in their own way to It is incorporated into I.

[0007] Various embodiments also include pre-processing of images and / or pre-training of machine learning systems. For example, image pre-processing can be used to reduce the risk of poor or variable quality medical images or image quality variations in size. This has proven useful when there is variability, for example when the images are ultrasound images. do.

[0008] Various embodiments include real-time feedback during image acquisition. The check may also be aimed at image selection, obtaining better images and / or obtaining more useful images. The feedback may be based on processing of the acquired images. For example: In some embodiments, evaluation of the initial image is used to generate an image having predictive and / or diagnostic value. Optionally, the image processing system is included in the image capture device. or using a communications network such as a local area network or the Internet. It is connected to the image processing device using

[0009] Various embodiments of the present invention comprise a medical prediction system configured to predict preterm birth. The system comprises: an image storage device configured to store ultrasound images; and an image storage device for storing ultrasound images of a fetus, the ultrasound images including, for example, a fetus. Image analysis logic configured to provide a prediction that a person will be born preterm. A user interface configured to provide a user with a prediction that the fetus will be born prematurely. A microprocessor configured to execute at least a portion of the image analysis logic. The image analysis logic may optionally be configured to estimate fetal gestational age based on ultrasound images. The first logic is configured to determine the number of weeks of pregnancy at the time of generating the ultrasound image. 1 logic and configured to estimate time to birth of a fetus based on ultrasound images and a second logic configured to calculate the estimated gestational age of the fetus at the time of birth. The number of days until delivery can be calculated directly without the number of weeks of pregnancy. Healthcare providers can calculate the number of days that delivery will occur sooner.

[0010] Various embodiments of the present invention include a method for generating a quantitative prediction of preterm birth, the method comprising: obtaining a set of medical images including a fetus; and using a machine learning system to analyzing the medical images to generate a quantitative prediction, the quantitative prediction being indicative of a risk of fetal birth; Steps include estimating the time to conception or estimating the gestational age of the fetus at birth. providing the same to the user.

[0011] Various embodiments of the present invention include a method for training a medical prediction system, the method comprising: receiving a plurality of medical images, the medical images optionally representing a fetus during pregnancy; Optionally, filtering the image. , classifying the image according to views or features contained in the image. Pre-training a neural network to recognize features within a class of images. Training the neural network to provide quantitative predictions regarding the birth of a child. wherein the quantitative prediction includes an estimate of the gestational age of the fetus at birth or the current gestational age of the fetus. Optionally, a trained testing the neural network to determine the accuracy of the quantitative predictions.

[0012] Various embodiments of the present invention collect ultrasound images for training a neural network. The method includes a method for scraping social media accounts, the method including: Identifying the fetal ultrasound image on the social media account. Identifying birth announcements in social media accounts. Post the birth announcement to your social media accounts first Using the ultrasound image and the calculated time, A program to train a neural network to generate predictions that an image is an indication of preterm birth. Optionally, the prediction is a quantitative prediction.

[0013] Various embodiments of the present invention include a method for identifying beneficial treatments based on medical predictions, The method includes determining a quantitative prediction of a future medical condition, The predictive prediction includes the probability of a medical condition occurring within a future time range, and is based on the analysis of the patient's medical images. Based on the analysis, providing the patient with a candidate treatment for the medical condition, Repeating the steps of determining a quantitative prediction of the condition and providing the candidate treatment to multiple patients. The candidate treatment provided statistically relevant benefit to multiple patients over each time range. determining whether the candidate treatment is a beneficial treatment based on the statistically relevant benefit; identifying the [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a medical prediction system configured to predict medical outcomes, according to various embodiments of the present invention. [Figure 2] FIG. 1 illustrates a method for making quantitative predictions according to various embodiments of the present invention. [Figure 3] FIG. 1 illustrates a method for training a medical prediction system according to various embodiments of the present invention. [Figure 4] FIG. 1 illustrates a method for identifying beneficial treatments based on medical predictions, according to various embodiments of the present invention. [Figure 5] 1 illustrates a method for acquiring ultrasound images in accordance with various embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Artificial intelligence systems, such as machine learning systems and expert systems, are used to analyze, for example, chest x-rays. It has been used to identify features in medical images, such as reading the Only limited progress has been made in the field of medical predictions. In contrast to applications, the systems and methods disclosed herein use artificial intelligence to demonstrate the predictive value of the objective image, and optionally other data processing. As an illustrative example, the prediction of such a process may be performed in connection with the prediction of preterm birth using ultrasound. The measured values will be explained.

[0016] The systems and methods disclosed herein address the risk of a fetus being born early (e.g., preterm birth). Quantitative prediction of disease, or some other medical event The quantitative nature of these predictions is in clear contrast to the prior art. For example, quantitative prediction does not simply identify that a pregnancy is "at risk." As used herein, "quantitative prediction" refers to the probability, distribution within multiple preterm birth classes, or In the case of preterm birth prediction, the time estimate is the expected time to birth. It may be expressed in terms of days and / or estimated gestational age at birth. These estimates are It may be expressed over two, three or more time horizons. The advantage of quantitative forecasting is that it allows for a simple provides much more actionable information compared to the common (and error-prone) "at risk" classification. It is something we can provide.

[0017] FIG. 1 illustrates a medical device configured to predict a medical outcome in accordance with various embodiments of the present invention. 1 shows a typical prediction system 100. The prediction system 100 is used for ultrasound systems and image processing. The prediction system 100 may include multiple devices, such as configured computing devices. communication between various components and / or external devices via a communications network such as the Internet It is configured to:

[0018] Processing the images to generate predictions can be used to predict whether a particular medical condition and / or event will occur in the future, e.g. This involves generating an estimate that the event will occur during a time period beginning at a future date. The estimate may be, optionally, absolute certainty. The estimate may be expressed as a rate or relative probability. Furthermore, the estimate may include a time component. For example, the estimate women have a 66% chance of premature birth and a 30% chance of giving birth between 33 and 34 weeks of pregnancy. Higher than expected chance of giving birth between 7 and 38 weeks and / or by 34 weeks of gestation In another example, the prediction may be that the probability of developing lung cancer in the future is 50%. Or it may include the probability of developing breast cancer.

[0019] The prediction system 100 includes an optional image generator 110 configured to generate an image. The image generator 110 may be a conventional ultrasound system or may be a computer system for processing. and further configured to provide the images to other elements of the prediction system 100 over a network. In various embodiments, the image generator 110 may include other imaging systems. A system including an image generating device and any combination of one or more elements of the prediction system 100. For example, the image generator 110 may include storage 120 and image analysis logic 13. 0, user interface 150, and feedback logic 170 (as described elsewhere herein). In various embodiments, the ultrasound device may include a The image generator 110 may be a radiographic (e.g., X-ray) based imaging device, a magnetic resonance imaging device, a nuclear imaging device, or any other suitable imaging device. Imaging device, ultrasound imaging system, elastography device, photoacoustic devices, tomography devices, echocardiography devices, magnetic particle imaging systems This includes instruments such as microscopes, spectroscopic (e.g., near-infrared) instruments, etc.

[0020] In some embodiments, other elements of the prediction system 100 may be directly connected to the image generator 110. The image generator 110 may be connected to or included within the image generator 110. For example, the image generator 110 may The ultrasound machine may include an ultrasound machine having image analysis logic 130, which may include filtering. Provides real-time feedback via feedback logic 170 to monitor ultrasound data In some embodiments, the image generator is configured to direct the acquisition of data and / or images. The generator 110 includes a sound source, a sound detector, and generates an ultrasound image based on the sound detected by the sound detector. and logic configured to generate an image. Optionally, the image generator 110 The image analysis logic 130 may be configured to adapt the generation of ultrasound images based on feedback from the image analysis logic 130. For example, sound generation, focus, and processing may be improved to better represent images containing such information. In response to an indication from the image analysis logic 130 that the image analysis logic 130 will provide better predictions and estimates, It can be adapted to better detect blood perfusion in the small capillaries of the fetus.

[0021] The image generator 110 may be an embodiment in which the prediction system 110 receives an externally acquired image. , or is optional in embodiments that use raw image data for prediction. For example, the image generator 110 processes raw ultrasound (sonogram) data rather than images to generate medical images. In some embodiments, generating a predictive prediction is optional. and / or the raw data is transmitted to the prediction system 1 via a communication network such as the Internet. 00. The images generated by the image generator 110 represent fetal movements. For example, the sequence may include images of blood flow, capillary abundance, Such sequences may also indicate the direction and speed of movement. It may also include Doppler information.

[0022] The prediction system 100 further comprises a storage 120. The storage 120 stores raw Sensor data, medical images, medical data, executable code (logic), neural networks The storage 12 includes a digital memory configured to store a network, etc. 0 configured to store raw sensor data generated by photon or acoustic detectors This may be used to generate X-ray or ultrasound images. As discussed above, storage 120 optionally includes memory circuitry and, optionally, The storage includes a data structure configured to manage and store any of the above data types. The ultrasound image stored in the image storage 120 is optionally 600 x 600 pixels, with 400 x 400 (or at least 300x300) random crops are optionally used as discussed herein. "Ultrasound images" as discussed herein are used for training and / or prediction. optionally includes three-dimensional rendering based on ultrasound data.

[0023] In some embodiments, storage 120 may include ultrasounds of the pregnant mother and / or fetus. Specifically, the ultrasound image includes circuitry configured to store the image. The ultrasound image may be acquired from one or more acquisition sessions. For example, a first set of ultrasound images may be generated in a first session. The ultrasound technician acquires a set of ultrasound images at a time, and the second set of ultrasound images is at least 1, 2, 5, 7, 15, 21, 30, 60, 90, or 180 days, or any range between these values Specific maternal and / or fetal ultrasounds may be obtained in a second session occurring at the The images may be generated over a period of time that includes any of the durations listed above. For example, a mother with a high-risk pregnancy may have an ultrasound scan once a week. Optionally, a sequence of images (e.g., video) representing Doppler data and / or fetal movements For example, an ultrasound image may show the heartbeat or blood flow of a fetus. It may also include information about the density of the baby's tissues, fluids, or bones. Images that were found to be useful included fetal heart rate, umbilical artery, uterus (including the lower uterine segment), and uterine Cervix, cervical view taken specifically to measure cervical length, amniotic fluid index view , abdominal circumference (AC), bifurcation diameter (BPD) and all other brain views, femur, humerus, fetal Intima (e.g., thickness and vascularization), placental margin against the cervix, fetal kidneys, placenta, appendages In some embodiments, images useful for estimating the gestational age of a fetus may be used. The images are then processed together with images useful for estimating the gestational age of the placenta. The difference is used to predict preterm birth.

[0024] The prediction system 100 predicts whether a fetus will be born preterm based on ultrasound images and, optionally, clinical data. and image analysis logic 130 configured to provide a quantitative prediction of the likelihood of the image being distorted. This prediction involves an estimate that can take a variety of forms. For example, the prediction may be based on the fetus's current gestational age. Estimates of prenatal (remaining) time (time to birth or days earlier than the standard 280 days) Alternatively, the estimation may be based on whether the fetus is at risk of preterm birth (e.g., low birth weight). ), this estimate may optionally be independent of the current gestational age. In addition, the prediction made by the image analysis logic 130 is not suitable for doctors who are still in the preterm birth stage. Including probability estimates of choosing to induce labor even if there is a risk or to induce labor at term. For example, predictions may be made if a doctor chooses to induce preterm labor because of a condition such as preeclampsia. Such predictions may include the probability of selecting at least one, two, three, or more of the following: It can be done 4 weeks prior, or 1, 2, 3, 4, 5, 6, or 7 months prior. In the case of nephropathy, if the condition manifests in a life-threatening way, caregivers should consider whether the fetus is premature or not. It may be necessary to induce the condition immediately regardless of the cause.

[0025] In some embodiments, the image analysis logic 130 determines the gestational age of the fetus based on the ultrasound image. a first logic configured to estimate a gestational age and a birth date of the fetus based on the ultrasound image; and second logic configured to estimate the time until The device is configured to use the ultrasound image to estimate the gestational age at the time the ultrasound image was generated. while the second logic is configured to estimate the time remaining until birth of the fetus. Then, in these embodiments, further logic in image analysis logic 130 , the estimated gestational age of the fetus at birth is calculated by adding the estimated number of weeks of gestation to the time remaining until birth. The first logic and the second logic are optionally configured to calculate the gestational age. It is placed in a machine learning system, for example, that receives ultrasound images as input and predicts the number of weeks pregnant and The remaining time can be output by the same neural network. By using a learning system to calculate the estimated gestational age and time remaining until birth, The errors in both of these calculations result in a larger overall error than would be expected if the two errors were independent. It may be related to less.

[0026] Image Analysis Logic 130 uses a wide range of machine learning algorithms to perform the above calculations For example, in some embodiments, the image analysis logic 130 uses a regression algorithm (e.g., quantile regression) to predict that a fetus will be born preterm. Quantile regression, etc., is a simple method for providing a time estimate for the population and, optionally, for estimating time to birth. Instead of a single value answer, predict a range that is likely to contain the actual answer. In an embodiment, any regression system that predicts a range rather than a point value can be used in the image analysis logic 13. It may be used as 0. Using a range as an estimate can lead to overfitting of the data. tting), which may result in mislabeled ultrasound images used for training. This is sometimes useful because image analysis logic typically analyzes a single ultrasound image. The present invention is configured to make predictions and estimates based on a set of ultrasound images rather than on a single image.

[0027] In some embodiments, the image analysis logic 130 uses a classification algorithm to identify the fetus. It is designed to provide an estimate of whether a baby will be born preterm. In contrast to the traditional classification, which divides preterm birth into two classes: early and late (preterm birth is 22 days or more early), The algorithm optionally includes three, four or more categories related to birth timing. The neural network using the algorithm classifies pregnancy into specific classes, e.g., estimated The time remaining until birth can be assigned to a range of birth times, and thus a quantitative estimate of the time remaining until birth can be obtained. These classifications are, for example, " if the number of days early is 29 or more, "Premature birth," "Borderline birth" if 14-28 days, "Full-term birth" if less than 14 days, etc. In practice, the date ranges for these classifications vary + / - 1 to 4 days in different implementations. In alternative embodiments, more and / or different classifications are used.

[0028] When using classification algorithms, we optionally apply a "label smoothing" function. In the case of the processed image, the label is not yet developed when the ultrasound image is generated. This may be incorrect, so a smoothing function may be useful. This smoothing function can be e.g. It can take the form described below, where optionally, epsilon (ε) is 0.05, 0.1, It is 0.3 or more.

[0029] JPEG2025114697000002.jpg25166JPEG2025114697000003.jpg40148JPEG2025114697000004.jpg130166

[0030] In some embodiments, label smoothing is performed when the loss function is cross-entropy. This model applies a softmax function to the logit vector z in the penultimate layer. We apply a label smoothing algorithm to calculate the output probability p. Label smoothing is used to smooth the labels during training. Regularizing classification problems to prevent overconfident predictions and poor generalization It is a technology. For example, see https: / / leimao.github.io / blog / Label-Smoothing / .

[0031] In some embodiments, both regression and classification algorithms are used to quickly For example, the image analysis logic 130 may analyze two separate One can include a neural network, the other a regression algorithm (which outputs a range) one configured to apply a classification algorithm, and the other configured to apply a classification algorithm. In this case, the classification algorithm is applied before the regression algorithm, The algorithm is optionally applied separately to each class.

[0032] Alternatively, both the regression and classification algorithms can be implemented using the same neural network. In such an embodiment, the neural network may be applied by It is trained to produce both classes and regression-based predictions, both of which are quantitative. The algorithm outputs one or more values for each selected percentile, e.g. , some embodiments may use a 1 for the output (which represents a percentile of the quantitative prediction) Use percentiles 0%, 25%, 50%, 75%, and 90% and Each of the images may be associated with a probability and / or confidence measure. From the input, the neural network of the image analysis logic 130 will typically select Generate one or more values for each percentile. The output may be used to confirm the prediction of preterm birth and / or time to birth. This scenario is arbitrarily unrelated because the algorithm and classification algorithm should produce the same results. It is used to establish the reliability of the actual predictions.

[0033] The image analysis logic 130 may perform other functions in addition to, or as an alternative to, regression and classification. For example, image analysis logic may be used. 130 may be configured to apply a regression that outputs an estimated range, where the range is a single However, many neural networks use data generated from different subsets of data (each trained on a different subset of images / data) ), single point predictions may be used if they are statistically analyzed to form averages and / or distributions. In some embodiments, a Bayesian convolutional neural network can be used. We use a framework to capture epistemic uncertainty, which is difficult due to the limited training data. Specifically, in neural networks, Instead of learning specific weight (and bias) values for a given Learn the weight distribution that is the sample that produces the output for the input, and encode the uncertainty of the weights. Bayesian networks are also used in the prediction classification methods discussed herein. can be used in a similar manner.

[0034] As noted elsewhere herein, the image analysis logic 130 may be configured to analyze the fetus if it is born prematurely. configured (using the above algorithms / machine learning techniques) to provide an estimate that This estimate may include the probability of preterm birth, the current estimated gestational age, the estimated time to fetal birth, and These estimates may include the estimated total gestational age and / or estimated total gestational age. Optionally, based on other factors associated with pregnancy. For example, the image analysis logic 130 may The estimation may be based on clinical data, e.g., For example, maternal genetics, maternal weight, maternal pregnancy history, maternal blood sugar levels, maternal cardiac function, maternal Kidney function, maternal blood pressure, placental status, maternal infection, maternal nutrition, maternal medication use (smoking) and alcohol use), maternal age, maternal socioeconomic status, maternal family environment, and maternal income. , maternal race, and / or maternal cervical or uterine characteristics. The image analysis logic 130 may optionally include any of these clinical data. It takes one or a combination of these as inputs and performs the estimation and prediction discussed herein in part on these. The method is designed to be based on these clinical data.

[0035] The prediction system 100 optionally performs a prediction process based on the estimations made by the image analysis logic 130. The system further includes calculation logic 140 configured to calculate an output useful for Sanlogic 140 calculates total pregnancy length based on current gestational age and estimated time to birth The calculation logic 140 may be configured to calculate a probability distribution (e.g., percentile The calculation logic may be configured to calculate the total gestational age based on the distribution represented by Q140 is designed to calculate the probability of preterm birth based on estimated time to birth or estimated total gestational age. For example, the calculation logic 140 may be configured to A distribution function may be applied to the estimates made, from which a probability distribution may be produced. In an embodiment, the image analysis logic 130 is configured to generate the characteristics of this distribution function. For example, in some embodiments, an estimate of the reliability of the predicted time to birth and a The estimate can be used to determine the width (e.g., standard deviation) of the distribution function. The image analysis logic 140 is optionally included in the image analysis logic 110.

[0036] The prediction system 100 optionally includes a function for analyzing the image and the prediction result. and / or a user interface 150 configured to provide the prediction to a user. The user interface 150 may optionally be a graphical user interface (and and logic associated therewith, an instance of the image generator 110, a mobile a device (in which case the user interface 150 may include a mobile app); or displayed on a computing device separate from the image generator 110 and / or image analysis logic 130. For example, the user interface 150 may display the gestational age of the fetus at birth, the fetus's age, the least likely cause of death is the estimated time to birth and / or estimated gestational age. In some embodiments, the user may be configured to display at least one or two of the following: The user interface 150 allows the image to be displayed by a remote user for processing by the image analysis logic 130. The system is configured to upload one or more ultrasound images.

[0037] As discussed further herein, in some embodiments, a user interface 150 is configured to provide feedback to the user in real time, e.g. The user interface 150 can be used to improve predictions and / or estimates related to preterm birth. The ultrasound technician may be instructed during the ultrasound session to generate images that result in .

[0038] The prediction system 100 optionally receives data about the pregnancy, such as the fetus and / or the mother of the fetus. It further includes a data input 160 configured to receive clinical data regarding the parent. Data input 160 is optionally configured to receive any of the clinical data discussed herein. The clinical data is used by the image analysis logic 130 to For example, this data may generate estimates and / or probabilities as discussed herein. The image generator 110 may include any of the clinical data discussed above or input from a user of the image generator 110. In some embodiments, the data input 160 can input medical images, such as ultrasound images. configured to receive from a remote source.

[0039] The prediction system 100 optionally further includes feedback logic 170. Feedback logic 170 may be configured to provide a feedback logic for evaluating ultrasound images based on the quality of the pregnancy-related estimates and / or predictions. The imaging device is configured to guide the acquisition of images, e.g., ultrasound images acquired during an imaging session. The analysis using the analysis logic 130 of the prediction and / or estimation may determine the accuracy, precision, and / or Or if reliability becomes insufficient, the feedback logic 170 may The user may also use the device 150 to inform the user that adding an ultrasound image is desirable. good.

[0040] Additionally, in some embodiments, the feedback logic may include feedback based on fetal heartbeat activity, fetal heart rate, and / or fetal heart rate. Fetal heart rate, placenta, cervix, fetal blood flow, fetal bone development, fetal spine, fetal kidneys, Fetal capillary blood perfusion, umbilical artery, uterus, lower uterine segment, and cervical length were measured. Cervical view, amniotic fluid index (AFI) view, abdominal circumference (AC) taken specifically to determine , BPD and all other brain views, femur, humerus, endometrium, intrauterine Ultrasound of specific features such as membrane vascularization, placental margin relative to the cervix, fetal kidneys, adnexa, etc. In some embodiments, the image is The analysis logic 130 separates the ultrasound images according to the objects and / or objects contained within the images. For example, distinct object classes may be configured to categorize the views and In such an embodiment, the image analysis logic 1 may include any of the following features: 30 identifies objects in ultrasound images and determines whether there are sufficient quality images of the objects in each object class. (Object classification may be based on a class of predicted gestational age at birth.) This should not be confused with the classification of ultrasound images by The user interface 150 can be used to allow the operator of the image generator 110 to include additional objects. Therefore, the feedback logic 17 may request that additional images be acquired. 0 indicates the need to acquire additional ultrasound images that may be useful in predicting that the fetus will be born preterm. In certain examples, the image analysis logic 130 may be configured to: At least one set of images showing the gestational age of the fetus ( fetal bone development and / or fetal heart movement) and optionally one segment indicating the state of the uterus. In some cases, the feedback loop may be configured to request an image of the The Gic 270 produces images that are more useful in predicting that a fetus will be born prematurely. The imaging device is configured to guide the positioning of an imaging device (e.g., an ultrasound probe) in this manner. Such guidance allows for the positioning of the ultrasound probe at a specific location or for the "full length of the femur." The message may include a text / audio request such as "Get the image shown."

[0041] In various embodiments, feedback logic 170 may train future models to It is configured to induce or request the acquisition of new images that are useful for obtaining high accuracy.

[0042] The prediction system 100 optionally further includes training logic 180. 0 is the image analysis logic 130, the feedback logic 170, the image acquisition logic 19 0, and / or configured to train any other machine learning system discussed herein. Such training is typically conducted to assess the quantitative risk associated with whether a fetus will be born preterm. The goal is to learn to make accurate predictions and / or estimates. The RenLogic 180 is designed to quantitatively predict and / or estimate fetal gestational age at birth. The image analysis logic 130 may be configured to train the image analysis logic 130. As mentioned above, this prediction can be achieved by using both the quantile regression algorithm and the classification algorithm together or They can be used separately.

[0043] The training logic 180 may be implemented using commonly known neural network training algorithms. However, the training logic 180 Optionally, various methods for better training the neural networks disclosed herein may be used. For example, in some embodiments, the training logic 180 may Pre-trained neural network of image analysis logic 130 for better feature recognition This pre-training is designed to train the user in various aspects such as orientation, contrast, resolution, and viewpoint. This can include training on images that are similar to the ultrasound images, allowing the system to recognize anatomical features in ultrasound images. Pre-training can be performed using optionally unlabeled data. do.

[0044] In some embodiments, training logic 180 may also generate training images for specific conditions. If the images are sparse or infrequent, additional training images may be generated. Once the most predictive images and features are known, the training logic 180 may select a subset of images as We take the data and use a generative adversarial network (GAN) to generate features that predict extreme preterm birth. New training images can be generated.

[0045] In some embodiments, training logic 180 is configured to train on a large set of images. The images are composed of multiple images, each of which may be from different mothers. Training on a set of can reduce overfitting of the data. Preferably, the set Each of the above is a set of at least some of the following that contain information useful for making the quantitative predictions discussed herein: It is large enough to ensure that several images are present in the set.

[0046] In some embodiments, the training logic 180 uses image analysis logic to enhance the image. For example, the image analysis logic 130 may be configured to train the image analysis logic 130 to analyze poor quality images. Enhanced ultrasound images or blood perfusion of fetal capillaries that are not normally visible on ultrasound images during processing and / or may be pre-trained to reveal features such as angiogenesis. Enhancement allows handheld ultrasound imaging to be used to generate processed images to identify conditions associated with preterm birth. It is possible to make quantitative predictions.

[0047] The prediction system 100 optionally further includes image acquisition logic 190. The 190th edition of the 1990 JSOC collects training images and other data from unusual sources, such as social media accounts. For example, the image collection logic 190 may be configured to collect birth information. Social media accounts (i.e., Instagram or Facebook) k®) to automatically identify prenatal ultrasound images and then These images will be correlated with birth announcements posted within the same social media accounts. The time from the ultrasound image to the birth announcement may be used to determine whether the ultrasound was collected. The time from conception to birth, i.e., the remaining gestational age until birth, can be approximated. Such social media collected information may optionally be used by training logic 180. In some embodiments, the social media Ultrasound images (or videos) retrieved from the database may be shared on the image or social media websites. It is also somewhat common to write the number of weeks of pregnancy on the image. do.

[0048] In some embodiments, the training logic 180 may perform multiple stages of training, such as transfer learning. The neural network is configured to train the neural network in stages. The machine is first trained to recognize relevant fetal features and then estimate gestational age. and then provide a quantitative estimate of gestational age at birth or length of time to birth. may be trained to provide

[0049] The prediction system 100 generally implements some or all of the logic described herein. The microprocessor 195 may further include a microprocessor configured to perform the The processor 195 includes the image analysis logic 130, the calculation logic 140, and the feedback logic. Logic 170, training logic 180 and / or image acquisition logic 190. The microprocessor 195 may be configured to perform these functions. The optical components may include integrated circuitry and / or optical components.

[0050] FIG. 2 illustrates a method for making quantitative (optionally medical) predictions according to various embodiments of the present invention. Quantitative predictions are optionally used to anticipate preterm birth and provide specific corrective measures. Thus, the method of FIG. 2 is optionally followed by appropriate therapy and / or treatment.

[0051] In the image acquisition step 210, a set of one or more images is acquired. These images are These images generally relate to a specific patient, such as a pregnant woman and her fetus. The image may be obtained from a source external to the prediction system 100 or may be generated using the image generator 110. For example, in some embodiments, ultrasound images may be acquired via a mobile device such as the internet. The electronic medical record is uploaded to the storage 120 via a computer network. In other embodiments, the image may be received from the system. The images are generated using a medical imaging system such as 120. An image may be any combination of views and / or features discussed herein. The images can include a combination of views, optionally classified based on their respective views and features (objects object classification).

[0052] In the optional data receiving step 220, additional clinical data relating to the patient (mother or fetus) may be received. Receive data, which again may be received from an electronic medical record system. , may be provided by the patient and / or caregiver. The received clinical data may be The clinical data may include any of the clinical data discussed in the previous section, optionally via data input section 160. and is received.

[0053] In the image analysis step 230, the image acquired in the image acquisition step 210 is analyzed by an image analysis engine. The images are analyzed using a SIC 130. The images are analyzed to produce one or more quantitative predictions. In the case of pregnancy, quantitative predictions generally involve quantitative estimates related to preterm birth of the fetus. For example, Quantitative prediction involves estimating the fetus's current gestational age (at the time of image recording) and the time until birth. Prediction may include estimating the fetus' gestational age, and / or estimating the fetus' gestational age at birth. In addition to images, quantitative predictions can be made using any and further based on the clinical data received in the data receiving step 220. The analysis methods in step 230 include those discussed with reference to image analysis logic 130. any set of algorithms and / or machine learning systems discussed elsewhere herein For example, analyzing the medical image may include a combination of a quantile regression algorithm. Using rhythm and / or classification algorithms to make quantitative predictions related to fetal preterm birth In another example, analyzing the medical image may include a step of: algorithm to provide an estimate that the fetus will be born preterm, and optionally, time to birth. The regression algorithm includes estimating the estimated gestational age of the fetus at birth. Both are configured to fall into one of two or three time ranges.

[0054] Examples of quantitative predictions that may be made in the image analysis step 230 include whether the fetus is preterm. Probability of birth, probability of one or more fetuses being born in the preterm range, probability of labor being induced probability of a fetus being born in the future ("future time" means a time period whose start date is in the future) "borderline preterm", "preterm birth" or "extremely preterm birth" (these classes are for a time period classification of births as defined by pregnancy risk, prediction of adverse pregnancy-related medical conditions, maternal and / or prediction of post-natal health problems for the fetus, prediction of the placenta not being expelled intact, etc. Examples include:

[0055] In an optional feedback providing step 240, the user (e.g., a caregiver) is prompted to provide feedback on the image. This feedback can be used to improve the quality of quantitative forecasts. The quality and / or classification of previously acquired images can be based on the specific example of ultrasound. During the session, the caregiver may collect additional images with different resolutions, different views, different features, etc. The image acquisition step 210 and the image analysis step 230 is optionally repeated, followed by a feedback providing step 240 .

[0056] In the prediction providing step 250, the quantitative predictions generated in the image analysis step 230 are provided, e.g., The predictions may be provided to a user, such as a patient or a caregiver. The predictions may optionally be stored in storage 120 and / or In various embodiments, the predictions are also placed in the electronic medical record (EMR) system. Through the interface, through the EMR system, through the mobile application, It may be provided on the display of the image generator 110, on the display of a computing device, or the like.

[0057] FIG. 3 illustrates a method for training a medical prediction system according to various embodiments of the present invention. The method illustrated in FIG. 3 can optionally be used to analyze the image, analyze the feedback logic 130, and 70 and / or image acquisition logic 190. These methods involve training logic 18 It may be implemented using 0.

[0058] In an image receiving step 310, a plurality of medical images is received as a training set. The received medical images optionally include ultrasound images of a fetus during pregnancy. 5 and stored in storage 120.

[0059] In an optional classification step 320, the received images are classified according to the views or Classify according to features. For example, an image may be classified as showing a fetal heart. The classification step 320 may be performed by any method. Optionally, the image analysis logic 130 may be implemented by a neural network, and and / or trained by training logic 180. Also, classification step 320 classifies each image fetus's (actual or estimated) gestational age at the time of conception, and / or the known outcome of each pregnancy The step of classifying the image accordingly may be included, for example, if the image was generated at 12 weeks of pregnancy. Sometimes the images are classified as unborn, sometimes as unborn fetuses born several weeks after the images were generated. may be classified as preterm and / or premature (with varying degrees of prematurity). may be classified as belonging to the fetus / mother.

[0060] An optional filter step 330 filters the received image. The filtering step may include filtering out features or views that are determined to have little or no predictive value. For example, the class of images of maternal bladders may be , may be determined to be of little value in determining quantitative predictions, and this class Images may be removed from the training set. Images may also be removed depending on their quality or resolution. It may be filtered.

[0061] In some embodiments, the filter step 330 filters multiple images in various classes. This includes a step of balancing the images, e.g., for training purposes, to compare extremely preterm births, premature births, and induced births. In some cases, it may be desirable to have roughly equal numbers of full-term births. Steps may be used to adjust the amount of images based on gestational age at birth. For training purposes, based on the classification of views and / or features determined in classification step 320, In some cases, it may be desirable to balance the number of images in the training set.

[0062] In an optional pre-training step 340, the neural network is optionally trained on images or The image analysis logic 130 may be pre-trained to recognize features within a class of images. The neural network recognizes various orientation, resolution, and / or quality features of ultrasound images. The device may be pre-trained to recognize

[0063] In the training step 350, the neural network is trained to make quantitative predictions about fetal birth. As discussed elsewhere in this document, quantitative predictions are Estimate the fetus's gestational age at birth, estimate the fetus's current gestational age, and / or the fetus's estimated gestational age until birth It may include an estimate of the time remaining.

[0064] In an optional testing step 360, test images are used to test the Test the predictions made by the trained neural network. Determining the accuracy and / or precision of quantitative predictions produced by neural networks good.

[0065] FIG. 4 illustrates a method for identifying beneficial treatments based on medical predictions, according to various embodiments of the present invention. The present invention provides a method for quantitatively predicting a medical condition and / or outcome using the method disclosed herein. The systems and methods described offer new opportunities for identifying beneficial treatments. and the like, which can prevent or ameliorate an undesirable medical condition or event. The quantitative nature of the predictions disclosed in allows for the detection of quantitative changes in outcomes and the assessment of treatment effectiveness. This approach can be used for any of the medical conditions discussed herein. Furthermore, the medical condition may be present or apparent at the time the quantitative prediction is determined. Treatment may include the administration of drugs or physical treatment. For pregnant women, based on ultrasound images taken at least 1, 2, 3, 4, or 6 months before birth The effectiveness of physical treatments, including rest, can be prescribed and / or evaluated. This may include changes in parental behavior, such as reducing physical activity and avoiding physically strenuous work. etc. Treatment may further include follow-up ultrasound examinations and evaluations.

[0066] Quantitative predictions of future medical conditions or events are optionally used to select populations for clinical trials. For example, assuming that extremely preterm birth occurs in less than 1% of pregnancies, , administering a candidate treatment to the general population of pregnant women to detect a non-identifiable 1% benefit in early pregnancy. However, it is inefficient in identifying the 1% of pregnancies most likely to end in extremely preterm birth. This allows us to investigate the benefit of candidate treatments within this population. It is much more efficient and is more likely to reveal benefits with better statistical relevance. Such preferred collections for clinical studies can be performed using the systems and methods disclosed herein. This approach may identify clusters of individuals with distinct symptoms, such as Alzheimer's disease or dementia. For diseases that begin to develop (and benefit from treatment) long before symptoms appear It is particularly useful.

[0067] In a prediction determination step 410, a quantitative prediction of a future medical condition is determined for a specific patient. As discussed elsewhere herein, quantitative prediction determines the likelihood that a medical condition will develop in the future. The quantitative prediction may include a probability of occurrence within a time range of As used herein, a "future" time range can be based on the This means that the time range begins at some time in the future, not just at the beginning. For example, at the time of your second trimester ultrasound, your future time range is 1-2 weeks into your full-term pregnancy. Quantitative predictions can be made in advance or 2-4 weeks in advance using the systems and methods described herein. In various embodiments, the quantitative prediction may be determined using any of the methods. The time delay from the step to the future time horizon may be weeks, months, or even years. For example, in the case of premature birth, the delay may be at least 1, 2, 3, 4 or 5 months.

[0068] In a candidate providing step 420, a candidate treatment is provided to a particular patient. The treatment may include administration of a drug. and / or rest, specific diet, physical exercise, physical therapy, mental exercise, dialysis, use of support garments. In some embodiments, the candidate treatments may include physical treatments such as applying a drug. Only patients who meet the criteria will be offered the candidate treatment. Optionally, the candidate treatment will be offered to patients whose medical condition is expected to progress in the future. Quantitative predictions are only provided for patients with >50%, 66%, or 75% of cases occurring within the 24-hour window. For example, some medicines may only be provided to pregnant women who have a greater than 75% risk of extremely preterm birth. The candidate treatment may be provided to the patient before the patient shows any outward symptoms of the medical condition. Good too.

[0069] In an iterative step 430, a quantitative prediction of the future medical condition is determined and candidate treatments are evaluated. The steps of providing the test to a number of patients are repeated. The number of patients is determined so as to include a statistically sufficient population. may be selected.

[0070] In the relevance determination step 440, the candidate treatment is statistically associated with multiple patients over each time range. The relevance determination step 440 generally determines whether the transaction resulted in a relevant benefit. Typically, at some point after the prediction determination step 410 and the candidate providing step 420, respectively: The relevance determination step 440 is performed for each patient. for a time range of , and / or for different respective time ranges for different patients The time range may be determined based on the time period over which a future medical condition is predicted in the prediction determination step 410. The association determination step 440 optionally includes a time range between patients who received the candidate treatment and patients who received the candidate treatment. As used herein, "statistical correlation" includes comparing patients who received cebo with patients who received cebo. "Relevant interests" include acceptance and / or government approval of a treatment, or the ability to further research / improve a treatment. This refers to benefits that motivate people to do better.

[0071] In the identifying step 450, the candidate treatment is identified as a beneficial treatment based on the statistically relevant benefit. Improvements statistically associated with a candidate treatment are identified as the likelihood that a population of patients will be eligible for the intervention. Note that this information is also an indicator of patient compliance. Improve (treatment).

[0072] FIG. 5 illustrates a method for acquiring ultrasound images according to various embodiments of the present invention. The method generates training data using posts on social media accounts. Optionally, the method shown in FIG. 5 may be implemented using the image acquisition It is implemented using logic 190 and may be applied to images other than ultrasound images.

[0073] In the scraping step 510, Facebook or Instagram review content from social media accounts such as ram(R) and Access to content. This access may occur through a browser or automated system such as a web crawler. This may be performed using a system.

[0074] In the image identification step 520, images containing fetuses within the accessed social media accounts are identified. Identify the ultrasound image containing the social account name (e.g., username and / or Optionally, but not necessarily, ultrasound images are The data is downloaded and stored in the storage 120 .

[0075] In the birth identification step 530, the social media recorded in the image identification step 520 is Revisit the count to identify birth announcements, which are then identified as birth announcements seen on the ultrasound images. It is hypothesized that this is associated with the fetus being born.

[0076] In a calculation step 540, the ultrasound image is posted to a social media account and then Calculate the time it takes to post the birth announcement on your social media account. The calculated time is assumed to represent the time from the generation of the ultrasound image to birth.

[0077] In a training step 550, the ultrasound images and the calculated time are used to train the neural network. The neural network is optionally included in the image analysis logic 130. Rarely, ultrasound images are trained to produce an optional quantitative prediction that indicates preterm birth.

[0078] The various methods shown in Figures 2-5 may be used in any combination, and may optionally be used in conjunction with the system shown in Figure 1. For example, the methods of Figures 2 and 3 and / or Figures 3 and 4 may be performed together. The method of FIG. 5 may be used to generate training data for other methods disclosed herein. It may be generated.

[0079] Examples: The following are illustrative examples that may be included in any of the embodiments discussed herein: .

[0080] The neural networks discussed herein require the use of all gestational ages. Use all ultrasounds, however, different models can be created for a range of gestational ages. can.

[0081] Note: The models below are just specific examples of how to create a working model. Many of the hyperparameters such as rate, batch size, and number of epochs can be easily adjusted.

[0082] Import all the libraries used in your code.

number

[0083] The following paths are possible: "Extremely Preterm," "Very Preterm," "Moderately Preterm," or "Normal." A folder containing one subfolder for each predicted class. This is merely an example, and any number of different classifications would work, e.g., two, three, four or more. There may be more classes.

[0084] Preterm births account for approximately 10% of the total data set used, but in a balanced data set like this, It is believed that training is more effective when using a validation set. This is unbalanced in this embodiment, but balanced in other embodiments. In some embodiments, the neural network may be divided into four categories: In various embodiments, training At least 50% or 75% of the set contains images that are balanced between the possible classes.

[0085] Optionally, set the image size to 400x400 pixels. Most are several times this size, and we have reduced the size to improve the efficiency of training and inference. Increasing the image size increases accuracy but with diminishing returns. Alternative implementations The display size must be at least 224x224 pixels, 640x640 pixels, or 2048x Use an image with a resolution of 2048 (pixels) or any range in between.

number

[0086] This is the object that feeds data to the neural network during training and validation. 10% of the surveys are put into a validation set that is used to monitor training. The data includes training data and a distribution of preterm infants that is typical of those found in developed countries. Although the validation set is imbalanced due to training, it is a natural distribution and cannot be balanced in any way. No. The batch size can be freely set to 28 images and adjusted as needed. Adding a function called transforms randomly augments each image, which helps prevent overfitting. Examples of enhancements include adjusting brightness, adjusting contrast, and adjusting the horizontal This embodiment uses the loss function We use Binary Cross Entropy, which is Multi-label classification problems, even though there is exactly one label for each image Other embodiments use categorical cross entropy, or training the data as a regression problem. Other loss functions, such as mean squared error, can be used if the

number

[0087] Create an object that provides high-level control over training and inference. Use high values for weight decay. The use of this method is merely one embodiment. Normalizing the weight values in other ways may have the same effect. There is a possibility that this may result.

[0088] Obtain a pre-trained network to use for transfer learning, but with an initial randomized It is also useful to train a neural network from the generated structure. We use Net-152. Other types of resnet also work, and the more layers there are, the better the accuracy. Many other neural networks will likely produce usable results. The example below shows an example of training. Alternative embodiments may vary the parameters and steps. .

number

[0089] This freezes the parameters of the convolutional part of the neural network and leaves only the linear layers. Other embodiments do not necessarily require the step of freezing the layer.

number

[0090] The scheduler gradually increases and then decreases the learning rate, running the neural network for 10 epochs. Train the network with a maximum learning rate of 1e -3 Other embodiments may use alternative training schedules. A module may also be used.

number

[0091] This allows for training of the entire neural network, including convolutional layers.

number

[0092] The scheduler gradually increases and then decreases the learning rate, running the neural network for 5 epochs. Further train the network with a maximum learning rate of 1e -5 This becomes:

number

[0093] The scheduler gradually increases and then decreases the learning rate, running the neural network for 5 epochs. Further train the network with a maximum learning rate of 1e -5 This becomes:

number

[0094] When the prediction threshold is set to 0.5, the best validation accuracy of this embodiment is 86%. do.

[0095] The network selects the most useful and specific anatomical view of the study to make this prediction. It allows the determination of the specific anatomical view or Other information can be omitted if it is not generally useful, or The different views can be weighted when calculating the prediction.

[0096] Each prediction outputs a score of the model's confidence, which is calculated in real time on the ultrasound machine. It is used to time and notify the sonographer when a view is taken, improving accuracy. Alternatively, the software can provide reliable views without the need for technical effort. This technology can be used to automatically retrieve updated A training set can be created to continuously improve the model, and this training set can be It is a feedback loop that incorporates better data to train future models.

[0097] Optionally, trained systems can be used to provide real-time feedback to ultrasound technicians. For example, the system may provide a timeline for when it acquires the most predictable image, or The system may notify the technician if the mother, fetus or their The technician may be requested to acquire images of both specific anatomical structures. When the system identifies the image as predictive of preterm birth, it will either confirm or deny the prediction of preterm birth. The technologist may be requested to acquire additional images to clarify the fetal anatomy. If an image of a portion of a structure (e.g., the heart) predicts preterm birth, the system or a further image of another part of the fetal anatomy (e.g., the hand or face). You may request the technician to obtain it.

[0098] Making predictions based on individual ultrasounds and then simply aggregating these individual predictions Although this method is useful, we believe it can be improved. The biggest challenge with this method is the During the ultrasound session, some ultrasound images confidently predicted preterm birth, while other anatomy Because ultrasound images of the anatomical site confidently predict a live birth outcome, simple aggregation is not very effective. Therefore, we have tried to use neural networks or networks to process a large number of images. A neural network that allows classification in one pass through a sequence of work The team decided to create a network that would allow them to simultaneously feed more images into the neural network. Therefore, we use a large number of images in one network (or network Develop a more efficient way to pass a sequence of workpieces through the system (in parallel or serially) Ta.

[0099] This does not necessarily have to be a classification problem: the value to be predicted is a number representing the desired target. values, and a neural network that performs the regression is created instead.

[0100] The training data is used to extract images from a class when the data is restricted to a specific class. It can be created by training a neural network to change to a different class. A neural network that can convert images from one class to another. An example of this work is CycleGAN.

[0101] It is not necessary to use an image. Sound waves, such as those from the ionospheric nucleus, can also be used instead of or in addition to images for this prediction. Cut.

[0102] [CNN-LSTM (Convolutional Neural Network - Long Short-Term Memory)] Then, useful information is extracted from each image and combined with a number of neural networks. Create neural networks that can be aggregated together into a single neural network Although it has been determined that these networks may be separate in other embodiments, An ultrasound session is when a technician completes a set of images in one interaction with the patient. Considering that many ultrasound sessions take more than 100 images, This is then beneficial for efficiency and accuracy. LSTMs use the fact that data points are related in a sequence. It is designed to be used with continuous (or time-series) data that are related to one another. Ultrasound images from a wave session are not in much order, if at all, but they are a part of this kind of network. The network transfers information from the previous step to process a single image and then processes each image of the ultrasound session. After the images are processed, this combined information can be classified. An ultrasound session involves looking at one anatomical feature and then moving on to another. There is no significant order of importance. The anatomical features used need not be limited to the mother or fetus. It should be noted that it may be a combination of both. In this embodiment, the images in the video are continuous in nature. and this general technique can be adopted.

number

[0103] Obtain data divided into training, validation, and test sets. Training data is usually Instead of using each survey as a separate folder of images, A range of images was randomly sampled from the pool of all full-term birth studies and all preterm birth studies. At runtime, the pool of each class with the desired sequence length is created. It is also possible to sample images from two, three or more classes. Either a symbol or a number can be used. There are several reasons for this.

[0104] First, the gestational ages of many ultrasound sessions overlapped the margin of error of 37 weeks. This means that the training data contains a significant number of mislabeled studies. Taste.

[0105] Second, whether any investigation into preterm birth was performed before the cause of preterm birth appeared or whether ultrasound examinations were performed at any time Even with an ultrasound, the cause of premature birth may not be apparent.

[0106] By choosing randomly, each training example will have enough information to make a correct prediction. It becomes possible to have ultrasound that includes

[0107] An alternative method of prediction is to predict the time from the date of ultrasound to birth, and then calculate the time frame. The goal is to calculate the class that a given object will fall into. This prediction is useful even without subsequent classification.

[0108] Also, keep in mind that multiple ultrasound sessions are typically performed during a single pregnancy. Also, when making predictions, it is important to combine multiple sessions from one pregnancy. It can also be combined.

[0109] A single session may indicate preterm labor, but this may not be reliable Therefore, the system determines whether a follow-up session is desired and if so, It may also indicate when a backup session should be conducted.

[0110] It is generally believed that there are multiple underlying causes of preterm birth, each with different treatment options. We propose that the network can inform physicians of the anatomical views responsible for predicting preterm birth. This allows for more informed treatment choices.

[0111] Using neural network prediction errors to determine new treatments for preterm birth For example, the network predicted preterm birth. However, if the child was not born prematurely, the patient's medical data can be mined and shared. You can find common ground and discover new treatments.

[0112] [Data Collection] Obtain the path to each folder containing images from one study.

number

[0113] In this example, the sequence length is 36 images, but a smaller or larger range can be used. It's likely to work better with more images, but with diminishing returns. The image size used is 224x224, which is commonly used for image classification problems. It is 400x400 pixels, which is much larger than the original. It also works for large images.

number

[0114] The encoder class takes a model pre-trained on one ultrasound image. We remove the layer and return 512 features for each image. However, in other embodiments, the number of features is It can be as few as 1 or much more than 512.

number

[0115] The CNNLSTM module takes features from each image and classifies the entire sequence of images. do.

number

[0116] Create a Learner object that provides high-level control over training and inference. Using a lower value helps to make correct predictions.

[0117] If you need GPU memory constraints, use GradientAccumulation. This allows us to perform a 32-item gradient before updating the network weights. is accumulated.

[0118] ModelReseter resets the hidden state of the LSTM between batches.

[0119] cnnlstm_splitter freezes part of the overall network while allowing other The weights of the parts can be updated.

number

[0120] The scheduler gradually increases and then decreases the learning rate, running the neural network for 5 epochs. Train the network with a maximum learning rate of 1e -3 These parameters are This was simply chosen for its appearance, many other options would also work.

number

[0121] Alternative Embodiments The individual predictions from the survey will be tallied and a tournament will be run to determine the final classification.

[0122] Reinforcement learning can be used instead of the CNN-LSTM shown.

[0123] Using multiple models trained on different subsets of data typically improves accuracy. Create an ensemble that can

[0124] Instead of the CNN-LSTM shown, we use Con Use vLSTM.

[0125] Combined with the image, additional data about the patient, such as the patient's age or related events in their medical history, Use the data.

[0126] In addition to the preterm class, additional outputs can be used. Some examples are It has multiple outputs for prediction, such as days to birth, birth weight, and current gestational age. This improves the overall accuracy of the forecast due to the inherent relationships between the various items being forecasted. Any of these alternative predictions can be performed without the preterm birth prediction if desired. can.

[0127] [Conclusion] The systems and methods disclosed herein are applied to real data obtained from clinical settings. has been shown to consistently yield positive predictive values of over 90% and negative predictive values of over 90% In some cases, positive predictive values of over 99% and negative predictive values of over 97% have been shown. It has achieved results including sexual neuter rates.

[0128] Some embodiments are specifically illustrated and / or described herein. However, modifications may be made. Many modifications and variations may be made in accordance with the above teachings without departing from the spirit and intended scope thereof. It will be recognized that the invention is within the scope of the present invention and the claims. Although imaging and preterm birth are taught herein as examples, the systems and methods described herein The Act may also apply to other medical information and situations, such as Alzheimer's disease, cognitive impairment, and multiple sclerosis, long-term sequelae of infections, cervical cancer, ovarian cancer, uterine cancer, and / or The present invention also includes the prediction of any other medical condition for which a precursor lesion may be present in an ultrasound image. The methods and systems described herein can be used to assess the current clinical state individually or to assess the future state (optionally defined) The determination may be made in combination with a quantitative prediction. The method may be used to predict the future health status of a fetus after birth. For example, future learning These include disabilities, cognitive abilities, personality, and underdevelopment or impaired function of various organs.

[0129] The teachings herein include the use of medical images, such as ultrasound images, in various embodiments. However, the systems and methods may use raw data other than in the form of images. For example, image resolution The analysis logic 130 may optionally generate images other than or derived from raw ultrasound data. In addition, it is trained to process raw ultrasound data.

[0130] The embodiments discussed herein are illustrative of the present invention. Because the embodiments have been described with reference to examples, various modifications of the methods and specific structures described may be made. or adaptations may become apparent to one skilled in the art. All such modifications, adaptations or variations which have advanced the art by the teachings are within the spirit of the present invention. It is understood that the present invention is not limited to only the illustrated embodiments. These descriptions and drawings should not be considered in a limiting sense, as it is understood that the present invention is not intended to It should not be.

[0131] The "logic" discussed herein may be stored on a non-transitory computer-readable medium. Expressly defined to include hardware, software, or any combination thereof This logic can be implemented in electronic and / or digital devices to create special-purpose computing systems. Any of the systems discussed herein may optionally be Electronic and / or optical circuits configured to perform any combination of the logic discussed The methods discussed herein optionally include a microprocessor including an electronic circuit. This involves the execution of logic by a microprocessor.

[0132] The computing systems and / or logic referred to herein may include integrated circuits, microprocessors, , personal computers, servers, distributed computing systems, communication devices, network devices etc., and various combinations thereof. Also, a computing system or logic The memory is divided into random access memory (RAM), dynamic random access memory (DR AM), static random access memory (SRAM), magnetic media, optical media, nano Media, hard drives, compact discs, digital versatile discs (DVDs), optical circuits and / or databases for storing analog or digital information It may contain volatile and / or non-volatile memory, such as other configured devices. As used herein, computer readable media expressly excludes paper. The computer-implemented steps of the method include executing a set of instructions stored on a computer-readable medium. It can contain programs that, when executed, cause a computing system to perform steps. A computing system programmed to perform specific functions according to instructions from system software. Systems are special-purpose computing systems for performing their specific functions. The data manipulated while the computing system performs these specific functions is, at a minimum, The data is stored electronically in a buffer in the computing system, and each time the stored data changes, it is updated for special use. The process physically transforms a computational system from one state to the next.

Claims

1. 1. A medical prediction system configured to predict preterm birth, the system comprising: an image storage configured to store ultrasound images, the ultrasound images being of a fetus; image storage, including a child; providing a quantitative prediction that the fetus will be born preterm based on the ultrasound images; Image analysis logic configured in configured to provide to a user the prediction that at least the fetus will be born prematurely. A user interface and a microprocessor configured to execute at least a portion of the image analysis logic; A system comprising:

2. The method of claim 1 , further comprising: an image generator configured to generate the ultrasound image. The system.

3. a data input device configured to receive clinical data relating to the fetus or the mother of the fetus; Further provided with a power section, 3. The system of claim 1 or 2, wherein the estimated time to birth of the fetus is based on the clinical data. Stem.

4. the quality of the prediction that the fetus will be born prematurely or the object classification of the collected images feedback configured to guide the acquisition of the ultrasound images based on the classification of the ultrasound images into The system of any one of claims 1 to 3, further comprising logic.

5. training the logic configured to estimate time to birth of the fetus. The system of any one of claims 1 to 4, further comprising configured training logic. 。

6. Image collection logistics configured to collect medical data from social media sources It also has a the medical data includes training images and birth information; The system of any one of claims 1 to 5, wherein the training images are generated using ultrasound. Stem.

7. The ultrasound images including the fetus are taken over a period of time including at least one month, two months, or three months. The system according to any one of claims 1 to 6 or claims 31 to 53, 10. The method according to any one of the preceding claims.

8. wherein the ultrasound image includes Doppler information or fluid, bone, or tissue density information. The system according to any one of claims 1 to 7 or any one of claims 31 to 53 How to do it.

9. the image analysis logic: a first logic device configured to estimate the gestational age of the fetus based on the ultrasound image; a first logic, wherein the gestational age is that at the time of generating the ultrasound image; 、 a second ultrasound imaging system configured to estimate a time to birth of the fetus based on the ultrasound image; The logic of logic configured to calculate an estimated gestational age of the fetus at the time of birth of the fetus; The system according to any one of claims 1 to 8 or any one of claims 31 to 53, 1. The method according to claim 1.

10. The image analysis logic uses a regression algorithm to identify when the fetus is present over two or more time ranges. Provides estimates of preterm birth within a certain range and optionally configures estimates of time to birth. And, 10. Any of claims 1 to 9, wherein the regression algorithm is optionally a quantile regression algorithm.

10. The system of claim 1.

11. The image analysis logic uses a regression algorithm to output a range prediction of the fetus. is constructed to provide an estimate that the child will be born preterm and optionally estimate the time to birth. The system according to any one of claims 1 to 10.

12. The image analysis logic uses a classification algorithm to determine if the fetus was born preterm. is configured to provide an inference that Claims wherein the classification algorithm includes at least two or three birth timing classifications. Item 12. The system according to any one of items 1 to 11.

13. The image analysis logic receives an ultrasound image and determines whether the fetus was born prematurely. The method of claim 1 includes a neural network configured to generate an output representing a prediction.

13. The system of any one of claims 12.

14. The image analysis logic receives an ultrasound image and outputs an output indicative of a time to birth of the fetus. Any of claims 1 to 13, comprising a neural network configured to generate 10. The system of claim 1.

15. The quantitative prediction that the fetus will be born preterm is a probability that the fetus will be born preterm. Including estimates, 15. The method of claim 1, wherein said estimation is optionally based on an image of the cervix and / or amniotic fluid index.

10. The system of claim 1 .

16. The quantitative prediction is based on ultrasound images including the endometrium and / or the uterine wall.

16. A system according to any one of claims 15.

17. The user interface displays the number of weeks of pregnancy, the estimated time until birth of the fetus, and the time of birth.

17. The method of claim 1, wherein the method is configured to provide the user with at least two of the pregnancy weeks.

10. The system of claim 1 .

18. The image generator includes a sound source, a sound detector, and a sound generator based on the sound detected by the sound detector. and logic configured to generate the ultrasound image using the 10. The system of claim 1 .

19. The image generator generates the ultrasound image based on the output of the image analysis logic. The system of any one of claims 2 to 18, configured to adapt:

20. the image generator is configured to generate a sequence of images representative of fetal movement; The sequence of images may optionally include measurements of fetal blood flow, capillary abundance, and / or cardiac function. The system of any one of claims 2 to 19, comprising a movement.

21. The image generator modifies the position of a sound source configured to generate the ultrasound image. The system of any one of claims 2 to 20, configured to prompt a user to Hmm.

22. The image analysis logic performs the image analysis based on clinical data received via a data input.

22. The method of claim 1, further configured to predict that the fetus will be born prematurely. The system according to any one of claims 1 to 5.

23. The clinical data includes maternal genetics, maternal weight, maternal pregnancy history, maternal blood sugar level, maternal cardiac maternal kidney function, maternal blood pressure, placental condition, maternal infection, maternal nutrition, maternal Drug use (smoking and alcohol use), maternal age, and / or maternal cervix or child 41. The system of claim 22 or claim 41, comprising at least one of the following characteristics: The method described.

24. The feedback logic is more effective in predicting that the fetus will be born prematurely. and configured to guide the positioning of the image generator to generate a more useful image. The system according to any one of claims 4 to 23.

25. The feedback logic may further include a predictor of whether the fetus will be born prematurely.

25. The method of claim 4, configured to indicate the need to acquire additional ultrasound images. Item 10. The system according to item 10.

26. the image analysis logic comprises a machine learning system; Training logic pre-trains the machine learning system to recognize features in the ultrasound images.

26. The system of any one of claims 1 to 25, configured to:

27. the image analysis logic comprises a machine learning system; training logic to train the machine to make the quantitative prediction that the fetus will be born preterm; A system according to any one of claims 1 to 26, adapted to train a learning system. Tem.

28. the image analysis logic comprises a machine learning system; The training logic uses both a regression algorithm and a classification algorithm to determine whether the fetus is premature. and configuring the machine learning system to make the prediction that the patient will be born at birth. The system according to any one of claims 1 to 27.

29. the image collection logic collects the training images from a social media account; Collect birth information, including date of birth, from each of the same social media accounts and and configured to determine a time from said generation of said training images to each of said dates of birth. The system according to any one of claims 6 to 28.

30. The image collection logic detects ultrasound images in a social media account. configured to detect relevant birth announcements in social media accounts. The system according to any one of claims 6 to 29,

31. 1. A method of generating a quantitative prediction of preterm birth, the method comprising: acquiring a set of medical images including the fetus; A machine learning system is used to analyze the medical images to generate the quantitative prediction. wherein the quantitative prediction is an estimate of the time to birth of the fetus or the fetus at birth. and estimating the gestational age of the baby. and providing the quantitative prediction to a user.

32. 1. A method for training a medical prediction system, the method comprising: receiving a plurality of medical images, said medical images optionally including an image of a fetus during pregnancy; and Optionally, filtering the images to identify images with poor predictive value from the plurality of medical images. removing the image; Optionally, classifying said images according to views or features contained within said images. 、 Optionally, a neural network is trained to recognize features within said image or type of image. a pre-training step; Training the neural network to provide a quantitative prediction regarding the birth of the fetus. wherein the quantitative prediction includes an estimation of the gestational age of the fetus at birth; or estimating the fetus's current gestational age and estimating the time remaining until the birth of the fetus. , steps, and Optionally, testing the trained neural network to determine the accuracy of said quantitative predictions. and

33. 1. A method of acquiring ultrasound images for training a neural network, said method comprising: The law is scraping social media accounts; identifying an ultrasound image of a fetus in the social media account; identifying a birth announcement within the social media account; Posting the ultrasound image to the social media account before the birth announcement calculating the time to post to said social media account; Using the ultrasound image and the calculated time, the ultrasound image is determined to be indicative of preterm labor. training a neural network to generate a prediction, said prediction being A method comprising the steps of:

34. 1. A method for identifying beneficial treatments based on medical predictions, the method comprising: determining a quantitative prediction of a future medical condition, said quantitative prediction comprising: The probability that a medical condition will occur within a future time range is based on an analysis of the patient's medical images. , steps, and providing a candidate treatment for said medical condition to said patient; determining a quantitative prediction of a future medical condition and providing a candidate treatment to a plurality of patients; Repeating steps; The candidate treatment provides a statistically relevant benefit to the plurality of patients over each time range. determining whether the Identifying the candidate treatment as the beneficial treatment based on the statistically relevant benefit. A method comprising the steps of:

35. 35. The method of claim 31, wherein the medical image comprises an ultrasound image of the fetus or the mother of the fetus. The method according to any one of claims 1 to 5.

36. The medical images include any of the image types or characteristics discussed herein, and optionally: Claim 3, classified to include any of the image types or characteristics discussed herein.

36. The method of any one of 1 to 35.

37. receiving clinical data relating to the fetus or the mother of the fetus; 37. The method of claim 31, wherein the estimated time to birth of the fetus is based on the clinical data.

10. The method according to any one of claims 1 to 9.

38. and determining whether to perform a method for acquiring ultrasound images based on the quality of the prediction or the object classification of previously acquired images. Any one of claims 31 to 37, further comprising the step of providing a feedback to the user.

1. The method according to claim 1.

39. time to birth of the fetus, the gestational age of the fetus at birth, and / or the current gestational age of the fetus Claims 31 to 35 further comprising training a machine learning system to estimate gestational age.

38. The method of any one of claims 38 to 38.

40. The step of analyzing the medical image may include using a quantile regression algorithm to predict whether the fetus was born preterm. providing an estimate of the number of children who will be born in the first quintile, and optionally estimating time to birth; a regression algorithm that estimates the gestational age of the fetus at birth over at least two or three time ranges; 40. The method according to claim 31, wherein the method is configured to classify the data into one of the following categories: method.

41. The quantitative prediction is based on clinical data relating to the mother of the fetus received via a data input unit. The method according to any one of claims 31 to 40, further based on

42. The step of analyzing the medical image may include using a quantile regression algorithm and / or a classification algorithm.

42. The method of claim 31, further comprising: predicting that a fetus will be born prematurely using the 10. The method according to any one of the preceding claims.

43. further comprising balancing the images based on object classification of views or features.

43. The method according to any one of claims 31 to 42.

44. 4. The method of claim 1 further comprising the step of balancing the amount of images based on gestational age at birth. 31 to 43. A method according to any one of claims 31 to 43.

45. Claims wherein the quantitative prediction is performed using one of the systems or methods described herein. The method according to any one of claims 31 to 44.

46. The candidate treatment has a quantitative prediction of 50%, 66%, or 75% that the medical condition will occur within a time range. % of patients.

47. 47. The method of any one of claims 31 to 46, wherein the medical condition is preterm birth of the fetus.

48. The medical condition is not present or apparent in the medical image at the time the quantitative prediction is determined. The method of any one of claims 31 to 47, wherein the

49. 49. The method according to any one of claims 31 to 48, wherein the proposed treatment comprises the administration of a drug or a physical treatment. How to post.

50. the candidate treatment is provided to a plurality of patients before the patients exhibit any symptoms of the medical condition; 50. The method of any one of claims 31 to 49.

51. The step of determining whether the candidate treatment provided a statistically relevant benefit may include, in part:

51. The method of claim 31, further comprising providing a placebo to said patient. method.

52. The time from the step of determining the quantitative prediction to the time range is at least 1, 2 or 52. The method of any one of claims 31 to 51, wherein the period is 3 months.

53. Claims 31 to 35, wherein the medical condition is any of the medical conditions discussed herein.

52. The method of any one of claims 52 to 52.

54. The system of any one of claims 1 to 30, wherein the image source comprises an ultrasound device.

54. The method of any one of claims 31 to 53.

Images