Systems and methods for improving the detection of fetal congenital heart defects - Patents.com

Abstract

Systems and methods are provided for assisting in the detection and diagnosis of significant cardiac defects during fetal ultrasound examinations, in which image data (e.g., motion video clips and / or image frames) are analyzed using machine learning algorithms to identify and select image frames within the image data that correspond to standard views recommended by fetal ultrasound guidelines, and the selected image frames are analyzed using machine learning algorithms to detect and identify morphological abnormalities indicative of significant cardiac dysfunction associated with the standard views. The results of the analysis are presented to a clinician for review, along with overlays on the selected image frames that identify the abnormalities using graphical or textual indicia. The overlays can be further annotated and stored by the clinician to create a documentation record of the fetal ultrasound examination.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 18 / 406,446, filed January 8, 2024, U.S. Provisional Application No. 63 / 584,117, filed September 20, 2023, U.S. Patent Application No. 18 / 183,937, filed March 14, 2023 (now U.S. Patent No. 11,869,188), and European Patent Application No. 23305235.6, filed February 22, 2023, the entire contents of each of which are incorporated herein by reference.

[0002] The present invention is directed to systems and methods for improving the detection of fetal congenital heart defects during and after ultrasound examinations by using machine learning algorithms to ensure the creation of complete datasets, perform pre-review of completed datasets, and determine which datasets should be designated for expert review. [Background technology]

[0003] Congenital heart disease (CHD) is the most common birth defect, with a prevalence of approximately 0.8–1% of all live births. As of 2014, CHD accounted for 4% of neonatal deaths and 30%–50% of deaths related to congenital anomalies in the United States. A study by Nayak et al., titled "Evaluation of fetal echocardiography as a routine antenatal screening tool for detection of congenital heart disease," Cardiovasc. Diagn. Ther. 6, 44–49 (2016), demonstrated that 92% of CHD cases occurring during pregnancy are defined as "low risk." In a study entitled "Effect of detailed fetal echocardiography as part of routine prenatal ultrasonographic screening on detection of congenital heart disease," The Lancet 348, 854-857 (1996), Stumpflen et al. observed that most CHDs are identified during the second trimester of pregnancy screening, supporting the need for universal fetal cardiac screening during the second trimester of pregnancy.

[0004] CHD is often asymptomatic during the fetal period but causes substantial morbidity and mortality after birth. In addition to adverse cardiac outcomes, CHD is associated with an increased risk for adverse neurodevelopmental outcomes, which are associated with factors such as associated chromosomal abnormalities, syndromes, postnatal cardiac dysfunction, and intrauterine hemodynamic abnormalities. Significant CHD (see Table 1), defined as requiring surgical or catheter-based intervention within the first year of life, accounts for approximately 25 percent of all CHD cases. See Oster, ME et al., "Temporal trends in survival among infants with critical congenital heart defects," Pediatrics 131, e1502-1508 (2013). In infants with significant cardiac involvement, the risk of morbidity and mortality increases when there is a delay in diagnosis and timely referral to a tertiary care center with the expertise to treat these patients. See Kuehl, KS, et al., "Failure to Diagnose Congenital Heart Disease in Infancy," Pediatrics, 103:743-7 (1999); Eckersley, L., et al., "Timing of diagnosis affects mortality in critical congenital heart disease," Arch. Dis. Child. 101, 516-520 (2016). [Table 1]

[0005] Compared with postnatal diagnosis, fetal diagnosis can dramatically improve neonatal outcomes by predicting obstetric medical, surgical, and / or early intervention planning and, in some cases, considering intrauterine therapy. Furthermore, accurate prenatal diagnosis allows parents to make informed decisions regarding the continuation of the pregnancy.

[0006] Distinguishing a normal fetal heart from one presenting with complex forms of CHD typically involves an initial screening test performed by physicians, nurse practitioners, physician assistants, sonographers, and other providers trained in diagnostic obstetric ultrasound. Licensed healthcare providers who meet specialty training guidelines are responsible for interpreting the ultrasound. If the ultrasound is abnormal, further testing via fetal echocardiography is warranted for confirmation and diagnostic refinement. Further testing may also be warranted in circumstances such as a family history of congenital heart defects, the presence of maternal diabetes, or the use of in vitro fertilization. Only well-trained and / or experienced pediatric cardiologists, maternal-fetal medicine specialists, obstetricians, or radiologists with the appropriate knowledge base and skills should oversee and perform such fetal echocardiograms. Low sensitivity in this task may limit palliative options, worsen postnatal outcomes, and hinder research into intrauterine therapy, while low specificity may result in unnecessary additional testing and referrals.

[0007] The World Health Organization (WHO) recommends that all pregnant women undergo one ultrasound scan by 24 weeks of gestation to estimate gestational age (GA), assess placental placement, determine single or multiple pregnancy, increase fetal anomaly detection, and improve pregnancy outcomes (WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience (World Health Organization, 2016)).

[0008] In 2013 and 2018, both the American Society of Ultrasound in Medicine (AIUM) and the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG) revised their practice guidelines for fetal cardiac screening in the second trimester. See Carvalho et al., "ISUOG Practice Guidelines (updated): sonographic screening examination of the fetal heart: ISUOG Guidelines," Ultrasound Obstet. Gynecol. 41, 348–359 (2013); "AIUM-ACR-ACOG-SMFM-SRU Practice Parameter for the Performance of Standard Diagnostic Obstetric Ultrasound Examinations," J. Ultrasound Med. 37, E13–E24 (2018). These updated guidelines stipulated a minimum of three views: a four-ventricle view (4C) and views of the left ventricular outflow tract (LVOT) and right ventricular outflow tract (RVOT) (2, 3). Unfortunately, some cardiac anomalies are not adequately detected prenatally with this approach. Although the three-vessel (3V) and three-vessel and tracheal (3VT) views are not required in the AIUM and ISUOG practice guidelines, both guidelines state that these views are desirable and should be attempted as part of routine screening. See Table 2. Many groups have already implemented additional views during routine screening and reported higher fetal cardiac anomaly detection rates of 62–87.5% compared with 40–74% using the recommended three views, as described in “Committee on Practice Bulletins—Obstetrics and the American Institute of Ultrasound in Medicine, Practice Bulletin No. 175: Ultrasound in Pregnancy,” Obstet. Gynecol. 128, e241–e256 (2016). [Table 2]

[0009] Some serious CHDs are more amenable to visualization through ultrasound screening during pregnancy than others. Using data from 1997 to 2007 from the Utah Birth Defect Network, Pinto et al., in "Barriers to prenatal detection of congenital heart disease: a population-based study," Ultrasound Obstet. Gynecol. Off. J. Int. Soc. Ultrasound Obstet. Gynecol. 40, 418-425 (2012), observed that defects most likely to be detected prenatally included those with an abnormal four-ventricle view, while defects presenting with an abnormal outflow tract were much less likely to be detected prenatally. In a study of members of a large health maintenance organization (HMO) in California from 2005 to 2010, Levy et al., "Improved prenatal detection of congenital heart disease in an integrated health care system," Pediatr. Cardiol. 34, 670-679 (2013), showed that women served by HMO clinics that had instituted a policy of examining the outflow tract during prenatal ultrasound had a much higher prenatal diagnosis rate (59%) compared with HMO clinics that did not have such a policy (28%).

[0010] In the current screening workflow, patients are typically presented at a first-line care site (OB-Gyn, midwife, or radiologist), where fetal assessment is performed, for example, by a health care professional or via fetal ultrasound screening performed by a sonographer. Image data is interpreted by the first-line practitioner in real time during the ultrasound examination or offline after the examination is performed. A report may be generated by the first-line practitioner and pre-filled by the sonographer. If a congenital heart defect is suspected, the patient is referred to a specialist who will review the report and perform specific tests (echocardiography, genetic testing) intended to confirm the presence or absence of a potential congenital defect. Depending on the results of that further examination or test, a decision is made regarding transfer of the patient to a treatment and / or follow-up care site.

[0011] The drawbacks of previously known CHD screening workflows are numerous and generally include inaccuracies and low specificity caused by inappropriate examination techniques, time constraints, maternal obesity, and simple misdiagnosis. In particular, CHD detection during second-trimester ultrasound examinations is often as low as 30%. Specificity is also suboptimal, as low as 40-50%, due to a lack of skill in adapting ultrasound images (i.e., ultrasound operators lack the necessary skills to obtain data from which a correct diagnosis can be made, resulting in approximately 49% of misdiagnoses), a lack of experience in formulating an accurate diagnosis (i.e., images acquired are sufficient and prenatal pathology findings are visible but not recognized by the operator, resulting in approximately 31% of misdiagnoses), and pathology findings that cannot be detected because they are not visible on ultrasound images, accounting for approximately 20% of missed diagnoses. Time constraints associated with achieving adequate patient throughput in clinical settings can exacerbate these problems, especially when patient transfer to a specialist is required.

[0012] Although some attempts have been made to improve CHD detection during routine prenatal ultrasound examinations, much work remains to be done. For example, numerous guidelines exist for sonographers that explain how to acquire complete, high-diagnostic-quality image datasets during an examination and how to confirm the presence of cardiac structures in real time during the examination. For example, U.S. Patent No. 7,672,491 to Krishnan et al. describes a system for assessing the diagnostic quality of images acquired during an ultrasound examination that uses machine learning to compare acquired images with expected images.

[0013] As discussed above, the ISUOG clinical practice guidelines, published in Ultrasound Obstet. Gynecol. 2013; 41:348-359, propose five axial locations that should be imaged during routine fetal cardiac ultrasound examinations, as well as the major organs and vessels and their respective orientations that should be identified at each location. European Patent Application Publication No. EP 3964136 to Voznyuk et al. describes a machine learning system that uses a first convolutional neural network (CNN) to analyze ultrasound images generated during the examination and compare the acquired images with views required by the guidelines, and a second CNN to analyze the images and identify potential abnormalities.

[0014] Ciofolo-Veit et al., U.S. Patent Application Publication No. US2021 / 0345987, describes an ultrasound imaging system that uses machine learning algorithms to analyze acquired images and detect abnormal features, and if abnormal features are detected, to determine and display other previously acquired ultrasound images that provide a complementary view of the potential abnormal features and enable improved diagnosis.

[0015] Additionally, fetal ultrasound screening typically generates thousands of image frames covering multiple structures per video "sweep," so diagnostic frames of interest for CHD may be few and easily missed. Furthermore, because the prevalence of CHD in the population is low (approximately 0.8-1%), non-specialists may encounter it infrequently and may discount or miss abnormal images. Together, these factors make CHD detection one of the most challenging diagnostic challenges in ultrasound, with dramatic implications for postnatal outcomes and quality of life.

[0016] In view of the foregoing, it would be desirable to provide a method and apparatus for screening prenatal ultrasound scans to improve the accuracy of birth defect detection and subsequent management.

[0017] Additionally, it would be desirable to provide a machine learning enabled system for prenatal fetal ultrasound examination that is configured to review recorded ultrasound videos and identify images from the videos that correspond to views recommended by guidelines.

[0018] Still further, it would be desirable to provide methods and systems for performing prenatal ultrasound examinations that assist the sonographer in collecting a high-quality data set in accordance with applicable guidelines, assist the interpreting physician and / or technician in identifying potential abnormalities in the acquired data, and guide the sonographer in real time to obtain additional views to enhance the image data set, for example, to facilitate specialist review.

[0019] Still further, it would be desirable to provide a method and system for objectively assessing sonographer performance across multiple exams. Summary of the Invention [Means for solving the problem]

[0020] The present invention is directed to a system and method for performing fetal ultrasound examinations that assists in the detection of significant cardiac defects during second-trimester ultrasound examinations. The system and method of the present invention assist trained and qualified physicians in interpreting ultrasound-recorded motion video clips by identifying standard views that appear within the motion video clips. In addition, the system and method of the present invention may assist in detecting and identifying morphological abnormalities that may indicate significant CHD.

[0021] Provided herein is a system for use with an ultrasound system to assist clinicians in detecting and diagnosing cardiac defects during fetal ultrasound examinations, the system including one or more computers configured to store non-transient program instructions, the system being programmed to: store view templates corresponding to standard guideline views, including view templates for at least 4C, LVOT, RVOT, 3V, and 3VT views; store data indicative of one or more potential abnormalities associated with each one of the view templates; receive multiple sets of image data generated by the ultrasound system during fetal ultrasound examinations, each set of image data comprising multiple frames; compare each frame of the multiple frames with a view template; identify and select a corresponding image frame, if present, for each view template; analyze each corresponding image frame; detect the presence of one or more potential abnormalities; and, in response to a request by a clinician, present the corresponding image frame for each standard view on a display screen, including an overlay indicating the presence of the one or more potential abnormalities.

[0022] The one or more computers may include a display computer and a server computer. The non-transient program instructions may include a user interface component and an interpretation component. The user interface component may be configured to receive, store, and display multiple sets of image data generated by the ultrasound system in real time. The user interface component may be configured to store and display analysis results returned by the interpretation component. The interpretation component includes a machine learning algorithm for identifying and selecting each corresponding image frame. The interpretation component may further include a machine learning algorithm for detecting the presence of one or more potential anomalies in each corresponding image frame. The overlay, which may indicate the presence of one or more potential anomalies, includes one or more of a bounding box, a graphical indicator, or a text indicator surrounding the potential anomaly.

[0023] The system may further include non-transient program instructions that allow a clinician to annotate the overlay. The system may further include non-transient program instructions for generating a report that documents the clinician's observations during the fetal ultrasound examination. The report may include the clinician's observations during the fetal ultrasound examination. 4C means four ventricles, LVOT means left ventricular outflow tract, RVOT means right ventricular outflow tract, 3V means three vessels, and 3VT means three vessels and trachea.

[0024] The one or more computers may be further programmed to determine, for each view template, data indicative of one or more potential abnormalities. The one or more computers may be further programmed to associate each of the data indicative of the one or more potential abnormalities with each respective view template. Each frame of the plurality of frames may be compared to the view template using a neural network. The one or more computers may be further programmed to determine, for each view template of the view templates, data indicative of one or more potential abnormalities, associate each of the data indicative of the one or more potential abnormalities with each respective view template, compare each of the one or more corresponding image frames with the data indicative of the one or more potential abnormalities for the view template corresponding to each of the one or more corresponding image frames using a neural network to detect the presence of the one or more potential abnormalities, generate a report indicative of the presence of the one or more potential abnormalities for each of the one or more corresponding image frames, and, in response to a request by a clinician, cause the report, including an overlay indicating the presence of the one or more potential abnormalities, to be presented on a display screen.

[0025]

[0009] Provided herein is a method for use with an ultrasound system to assist clinicians in detecting and diagnosing cardiac defects during fetal ultrasound examinations, the method may include providing a computer configured to store and execute non-transient program instructions; storing on the computer view templates corresponding to standard guideline views, including view templates for at least 4C, LVOT, RVOT, 3V, and 3VT views; storing on the computer data indicative of one or more potential abnormalities associated with each one of the view templates; generating animated video clips during the fetal ultrasound examination using the ultrasound system and storing the animated video clips; receiving the animated video clips by the computer; comparing, using the computer, each frame of the animated video clip with the view templates, and identifying and selecting corresponding image frames for each view template, if present; analyzing, using the computer, each corresponding image frame to detect the presence of one or more potential abnormalities; and presenting, upon request by a clinician, the corresponding image frames for each standard view, including an overlay, indicating the presence of the one or more potential abnormalities, on a display screen associated with the computer.

[0026] The step of providing a computer may include providing one or more computers, including a display computer and a server computer. The step of providing one or more computers may include providing non-transient program instructions for a user interface component to the display computer and non-transient instructions for an interpretation component to the server computer. The method may further include displaying multiple sets of image data generated by the ultrasound system in real time to a clinician using the display computer. The method may further include transmitting analysis results generated by the interpretation component from the server computer to the display computer. Using the computer to compare each frame of the multiple frames to a view template may include analyzing the multiple frames using a machine learning algorithm to identify and select each corresponding image frame. Using the computer to analyze each corresponding image frame to detect the presence of one or more potential anomalies may include analyzing each corresponding image frame using a machine learning algorithm to detect the presence of one or more potential anomalies.

[0027] Presenting the overlay on the display screen may include presenting graphical or textual indicia indicating the presence of one or more potential abnormalities. The method may further include creating an annotated overlay corresponding to the overlay, including additional graphical or textual information entered by the clinician, and storing the annotated overlay. The method may further include generating, using a computer, a report documenting the annotated overlay for the fetal ultrasound examination. The method may further include generating, using a computer, a report with an entry for each standard view and an overlay indicating the presence of one or more potential abnormalities. The method may further include using a computer to determine the presence of one or more abnormalities and to transmit at least a portion of the multiple sets of image data or one or more of the report to a specialist. The method may further include using a computer to determine a quality value for the fetal ultrasound examination based on the multiple sets of image data.

[0028] In one embodiment, the present system and method are embodied in computer-aided diagnostic aids for use in two-dimensional prenatal ultrasound examinations of the fetus, typically performed during the second trimester of pregnancy. Machine learning algorithms are employed to assist users in identifying and interpreting standard views in fetal cardiac ultrasound image data (e.g., motion video clips). In particular, the present system and method are embodied in software that can be executed to assist in the identification of significant CHD. Additionally, information generated during machine learning-enhanced analysis may be stored for later referral to a specialist (e.g., a medical specialist) to assist in further diagnosis and treatment planning.

[0029] In a preferred embodiment, the system of the present invention employs two components: a user interface component that provides clinician tools for analyzing and reviewing fetal ultrasound images and ultrasound image data (e.g., animated video clips); and a machine learning interpretation component that receives ultrasound image data (e.g., animated video clips and images) from a conventional fetal ultrasound screening system and identifies images within the image data (e.g., animated video clips) that correspond to fetal ultrasound screening guidelines. The interpretation component also analyzes the identified images to detect and identify the presence of morphological abnormalities and provides that information to the user interface component, highlighting such abnormalities for clinician review. The interpretation component may be partially or fully executed in real time on a local computer workstation. Alternatively, the interpretation component may reside on a cloud-based server and interact with the user interface component via a secure connection over a local or wide area network, such as the Internet.

[0030] According to another aspect of the present invention, the method and system provide a consistent process for ensuring that all views suggested by clinical practice guidelines for fetal testing are obtained. In particular, if machine learning-based review of image data (e.g., a motion video clip) from a fetal ultrasound scan does not identify an image frame appropriately determined for review, the system will flag the view as unavailable or of inadequate quality to enable analysis for anomaly detection, and a user interface will instruct the clinician to re-perform the ultrasound scan to obtain the missing data. The new image data (e.g., a new motion video clip) will then be transmitted to an interpretation component for analysis, and a supplemental analysis will be returned to the user interface for presentation to the clinician.

[0031] According to another aspect of the invention, the analysis results returned to the user interface component may be displayed and further annotated by a clinician to include additional graphical indicia or textual remarks, and the resulting analysis results and annotations may be stored for later referral to a specialist to develop a plan for further diagnosis or treatment.

[0032] According to another aspect of the invention, the analysis and / or results, including detected morphological abnormalities, may be used to generate a report. The report may be automatically populated with entries for each standard view using frames from the video clip, which may include bounding box overlays. Information about the view may be included in the report to add context to the image.

[0033] According to another aspect of the invention, the system may recommend referral to a clinician and / or specialist. According to another aspect of the invention, the system may perform an objective evaluation of the technician (e.g., sonographer) who performed the imaging. According to another aspect of the invention, the system may automatically organize the results with the most relevant information appearing first or otherwise most prominent. Additionally or alternatively, the results may be organized by patient in order of severity.

[0034] In another embodiment, a system and computer-implemented method for analysis of fetal ultrasound images is provided. The system and method may include receiving multiple sets of image data generated by an ultrasound system during a fetal ultrasound examination, each set of image data of the multiple sets of image data including multiple frames; analyzing the sets of image data of the multiple sets of image data and automatically determining that one or more frames of the sets of image data correspond to a standard view of a plurality of standard views; analyzing the sets of image data and automatically determining that one or more frames exhibit a first morphological abnormality of a plurality of morphological abnormalities; and generating a user interface for display, the user interface including: an image data viewer adapted to visually present the set of image data; a standard view indicator corresponding to the set of image data presented on the image data viewer and visually indicating whether each standard view of the plurality of standard views is present in the set of image data; and a morphological anomaly indicator corresponding to the set of image data presented on the image data viewer and visually indicating whether each morphological abnormality of the plurality of morphological abnormalities is present in the set of image data, wherein when the image data viewer visually presents the set of image data, the standard view indicator indicates that the first standard view is present in the set of image data and the morphological anomaly view indicator indicates that the first morphological abnormality is present.

[0035] The user interface may be generated on a display of the ultrasound system and / or on a display of a healthcare provider device. The standard view indicator may include a plurality of color indicators, each corresponding to one of the plurality of standard views, wherein each color indicator of the plurality of color indicators is adapted to present a first color when the individual standard view of the plurality of standard views is present in the set of image data and a second color when the individual standard view of the plurality of standard views is absent from the set of image data. The morphological anomaly indicator may include a plurality of color indicators, each corresponding to one of the plurality of morphological anomalies, wherein each color indicator of the plurality of color indicators is adapted to present a first color when the individual morphological anomaly of the plurality of morphological anomalies is present in the set of image data and a second color when the individual morphological anomaly of the plurality of morphological anomalies is absent from the set of image data. Each color indicator of the plurality of color indicators may further be adapted to present a third color indicating that the presence of the individual morphological anomaly of the plurality of morphological anomalies in the set of image data is uncertain.

[0036] The image data viewer may include a first time bar and a cursor on the time bar, the cursor adapted to move and cause the image data viewer to visually present multiple image frames corresponding to multiple time points along the time bar. The standard view indicator may include multiple second time bars, each corresponding to the first time bar, each accompanied by a first visual indicator corresponding to the cursor, and adapted to move synchronously with the cursor. Each of the multiple second time bars may be adapted to visually indicate one or more time points on the second time bar, each corresponding to the presence of a respective standard view of the multiple standard views. The morphological anomaly indicator may include multiple second time bars, each corresponding to the first time bar, each accompanied by a first visual indicator corresponding to the cursor, and adapted to move synchronously with the cursor. Each of the plurality of second time bars is adapted to visually indicate one or more time points on the second time bar that correspond to the presence of a respective morphological abnormality of the plurality of morphological abnormalities.

[0037] The multiple sets of image data may be generated by the ultrasound system and may include multiple image data (e.g., motion video clips) generated by the ultrasound system. The multiple standard views may include four-ventricle (4C), left ventricular outflow tract (LVOT), right ventricular outflow tract (RVOT), three-vessel (3V), and / or three-vessel and tracheal (3VT) views. The multiple morphological abnormalities may include enlarged cardiothoracic ratio, right ventricular to left ventricular size discrepancy, tricuspid to mitral annular size discrepancy, cardiac axis deviation, Crooks' septal defect, pulmonary valve to aortic annular size discrepancy, caudal artery, and / or abnormal outflow tract relationship.

[0038] The user interface may include an examination summary adapted to present a list of standard views of a plurality of standard views determined to be present in a plurality of sets of image data and a list of morphological abnormalities of a plurality of morphological abnormalities determined to be present in a plurality of sets of image data. [Brief explanation of the drawings]

[0039] [Figure 1A] 1A-1C are schematic diagrams of exemplary server-based and / or local models for implementing the methods and systems of the present invention. [Figure 1B] 1A-1C are schematic diagrams of exemplary server-based and / or local models for implementing the methods and systems of the present invention. [Figure 1C] 1A-1C are schematic diagrams of exemplary server-based and / or local models for implementing the methods and systems of the present invention.

[0040] [Figure 2] FIG. 2 is an exemplary screen display presented to the clinician showing the results returned by the interpretation component to the user interface.

[0041] [Figure 3A] 3A-3B are exemplary flowcharts and data flows illustrating the analysis process performed by the interpretation component to analyze a set of image data (eg, a motion video clip) generated during a fetal ultrasound examination. [Figure 3B] 3A-3B are exemplary flowcharts and data flows illustrating the analysis process performed by the interpretation component to analyze a set of image data (eg, a motion video clip) generated during a fetal ultrasound examination.

[0042] [Figure 4A]Figures 4A and 4B are example images presented to the clinician via the user interface module after the analysis results are returned from the interpretation module, with Figure 4A showing an image selected as corresponding to a four-ventricle view and Figure 4B being a similar view for a different fetus, including a bounding box identifying a large atrioventricular defect. [Figure 4B] Figures 4A and 4B are example images presented to the clinician via the user interface module after the analysis results are returned from the interpretation module, with Figure 4A showing an image selected as corresponding to a four-ventricle view and Figure 4B being a similar view for a different fetus, including a bounding box identifying a large atrioventricular defect.

[0043] [Figure 5] FIG. 5 is an exemplary graphic user interface that includes a representation of a patient's image and a list of standard views and abnormalities.

[0044] [Figure 6] FIG. 6 is an exemplary data flow for acquiring medical images, identifying red flags, annotating the images, and providing the flagged and annotated images in a graphical representation.

[0045] [Figure 7] FIG. 7 is an exemplary graphic user interface that includes a representation of a patient's image, patient information, and a list of abnormalities.

[0046] [Figure 8A] 8A-8C are exemplary user interfaces for displaying image data generated by an ultrasound system as well as standard views and abnormality indicators. [Figure 8B] 8A-8C are exemplary user interfaces for displaying image data generated by an ultrasound system as well as standard views and abnormality indicators. [Figure 8C]8A-8C are exemplary user interfaces for displaying image data generated by an ultrasound system as well as standard views and abnormality indicators.

[0047] [Figure 9] FIG. 9 is an exemplary flow chart for dynamically requesting additional image views and logging the presence and absence of standard views and morphological abnormalities. DETAILED DESCRIPTION OF THE INVENTION

[0048] Systems and methods for performing fetal ultrasound examinations are disclosed that aid in the detection of significant cardiac defects during fetal ultrasound examinations, typically performed during the second trimester. In particular, the systems and methods of the present invention assist trained and qualified physicians in interpreting sets of ultrasound-recorded image data (e.g., motion video clips, images, etc.) by identifying and selecting for presentation to the physician image frames corresponding to standard guideline views that appear within the image data set. More specifically, the systems and methods of the present invention assist in detecting and identifying morphological abnormalities that may indicate significant cardiac dysfunction. Table 3 provides exemplary correspondences between representative cardiac dysfunctions (CHDs), the views in which those CHDs typically appear, and the morphological abnormalities that may typically be identified in those views.

[0049] In the exemplary system depicted in FIG. 1A , the system may include a conventional ultrasound system 10, a display computer 20, and a server system 30, communicating with each other via a wide area network 40, illustratively the Internet or any other suitable network (e.g., a local area network). In a preferred embodiment, the system and method are embodied in a computer-aided diagnostic system for use in two-dimensional fetal ultrasound examinations, typically performed during the second trimester of pregnancy or similar. Machine learning algorithms are employed to assist a user in identifying and interpreting standard views in a set of fetal cardiac ultrasound image data (e.g., motion video clips). While an ultrasound system is described throughout, it should be understood that the same or similar approaches may be used in conjunction with any other suitable medical imaging system (e.g., computed tomography (CT) scans, magnetic resonance imaging (MRI), etc.).

[0050] In one preferred embodiment, the method and system of the present invention employ two software components: a user interface component and an interpretation component. The user interface computer preferably resides on the display computer 20 and provides clinician tools for analyzing and reviewing fetal ultrasound images and ultrasound motion video clips. The interpretation component preferably resides on the server computer 30 and receives sets of ultrasound motion video clips and images from the ultrasound system 10 or the display computer 20 and uses machine learning algorithms to identify images (e.g., image frames) within the motion video clips and / or sets of image data that correspond to fetal ultrasound screening guidelines. The interpretation component also analyzes the identified images, as well as any unidentified images (e.g., corresponding to non-standard or non-recommended views), to detect and identify the presence of morphological abnormalities and provide that information to the user interface component, highlighting such abnormalities for clinician review. In an alternative embodiment, the interpretation component may run partially or entirely in real time on a local computer workstation.

[0051] As is typical, ultrasound system 10 includes a handheld probe that a clinician moves across the patient's abdomen to generate sets of fetal image data (e.g., motion video clips) during a prenatal fetal examination, which may be transmitted to display computer 20 during the scanning process for storage and display on a display screen associated with display computer 20. The sets of image data (e.g., motion video clips) generated during the examination may be uploaded directly from ultrasound system 10 to server system 30 via wide area network 40, or transmitted by a user interface module executing on display computer 20.

[0052] The display computer 20 is preferably configured to display real-time video generated by the ultrasound system 10 and, in addition, to display to the clinician analysis results generated by the interpretation component executing on the server system 30. The display computer may include a display screen, storage, a CPU, input devices (e.g., keyboard, mouse), and network interface circuitry for bidirectional communication with the server system 30 over the wide area network 40. In a preferred embodiment, the display computer 20 executes the user interface component of the present invention, which receives and stores physiological information about the patient. The display computer 20 also receives and stores real-time ultrasound video from the ultrasound system 10 and relays that image data, along with the patient's physiological information, to the interpretation component executing on the server system 30.

[0053] The server system 30 includes an interpretation component of the system of the present invention, which includes a machine learning algorithm for analyzing a set of image data (e.g., an animated video clip) received from the display computer 20 and comparing the ultrasound video clip with a set of preferred image templates corresponding to fetal ultrasound screening guidelines. In a preferred embodiment, the interpretation component includes image templates corresponding to each of the views recommended in the fetal cardiac ultrasound screening guidelines listed in Table 2, including (1) transverse abdominal view, (2) four-ventricle view (4C), (3) left ventricular outflow tract view (LVOT), (4) right ventricular outflow tract view (RVOT), (5) three-vessel view (3V), and (6) three-vessel and tracheal view (3VT). As described in further detail below, the interpretation component preferably employs machine learning to compare each frame of the input image data (e.g., an animated video clip) with the six aforementioned view templates and selects one or more high-quality image frames as corresponding to the selected template. If an abnormality is detected, the image frame showing the abnormality may be selected. The interpretation component employs a machine learning model to analyze the image frames selected as representative of the guideline view, and optionally each of the other unselected image frames, for the presence of anomalies known to be present in those image templates as described in Table 3.

[0054] For example, once the interpretation component identifies and selects an image frame from an uploaded set of image data (e.g., a motion video clip) as representative of a 3VT view, the machine learning features will analyze the selected image frame for features identified in Table 3 as being visible in a 3VT view, namely, an aorta larger than the pulmonary artery, associated with aortic coarctation and conotruncal lesions, a right aortic arch, associated with transposition of the great arteries, abnormal vascular alignment, and associated with anomalous pulmonary venous return, which are associated with aortic coarctation and conotruncal lesions.

[0055] If the interpretation component of the system identifies one or more of the features described in Table 3 as present in the selected image frame, the system may further create an overlay on the selected image, including a bounding box surrounding the detected abnormality and, optionally, a text indicator associated with the suspected defect. The selected image frame and analysis results are then transmitted back to the display computer 20 for presentation to and consideration by the clinician. Because clinicians often have multiple patients, they may be sent or otherwise tasked with reviewing results from several patients. To facilitate efficient review by the clinician and / or specialist, the system may automatically organize the results with the most relevant information, such as detected morphological abnormalities, appearing first or otherwise most prominent. Additionally or alternatively, the results may be organized by patient in order of severity.

[0056] The display computer 20 may provide the ability to annotate selected image frames with additional graphical or textual notes, which are then saved with the results for later recall during preparation of a written report for the fetal ultrasound examination.

[0057] During analysis by the interpretation component, if any image frames corresponding to the image data (e.g., an animation video clip) are not identified as corresponding to a standard view template, or if the identified image frames are determined to be of too poor quality to permit analysis for potential defects, that image template is identified as missing when the analysis results are transmitted back to display computer 20. In this case, the clinician may be prompted by display computer 20 to rescan the fetus to obtain the missing views, and the image data (e.g., an animation video clip) may be resubmitted to the interpretation component for supplemental analysis. The results of the supplemental analysis may then be transmitted back to display computer 20 for presentation to and consideration by the clinician.

[0058] Referring now to FIG. 1B, the exemplary system of FIG. 1A is illustrated. System 15 may be identical to the system illustrated in FIG. 1A and may include ultrasound system 10, display computer 20, and server 30. As shown in FIG. 1B, ultrasound scanning system 10 may be any suitable ultrasound scanning system for performing fetal anatomy ultrasound examinations (e.g., second-trimester fetal anatomy ultrasound examinations at 18-24 weeks gestation, first-trimester examinations, third-trimester fetal examinations, or the like), however, the software / programming of the present invention described herein is stored and executed on the controller of ultrasound scanning system 10. In one embodiment, ultrasound scanning system 10 may be a Samsung WS80A ultrasound system or any other suitable ultrasound scanning system. Ultrasound scanning system 10 may include an ultrasound probe (e.g., probe 35), a display, one or more computers, inputs (e.g., keyboards, mice, knobs, dials, switches, toggles, and the like), speakers, microphones, and the like. System 15 may also optionally include a healthcare provider device 25 , which may include a display 22 .

[0059] Healthcare provider device 25 may be a stand-alone computing device that may display analysis results generated by an interpretation component executing on server system 30 to a healthcare provider (e.g., a doctor, technician, specialist, etc.). The display computer may include a display screen, storage, a CPU, input devices (e.g., keyboard, mouse), and network interface circuitry for bidirectional communication with server system 30 and / or computing device 20 via any suitable wired or wireless connection. Display computer 20, and optionally, healthcare provider device 25, may execute a user interface component of the present invention. For example, display computer 20 and / or healthcare provider device 25 may display graphic user interface 17, which may be any of the graphic user interfaces described herein (e.g., graphic user interface 200 of FIGS. 8A-8C ). Healthcare provider device 25 may communicate with server 30 and / or computing device 20 to provide input, comments, edits, and otherwise adjust or modify the graphic user interface.

[0060] 1A-1B。 Referring now to Figure 1C, the clinical workflow of the system shown in Figures 1A-1B is illustrated. As shown in Figure 1C, clinical complex 12 may communicate with backend 14, which may be running on a server (e.g., server 30 of Figure 1A). Clinical complex 12 may include ultrasound module 16, which may be running on an ultrasound scanning device (e.g., ultrasound scanning device 10 and / or display computer 20 of Figure 1A), picture archiving and communications (PACS) system 18, which may be running on the ultrasound scanning device or devices located in the same building or campus as the ultrasound scanning device and / or on a remote server, digital imaging and communications in medicine (DICOM) viewer 22, and a DICOM router 24.

[0061] The ultrasound module 16 may generate, receive, acquire, and / or store ultrasound images (e.g., image data such as motion video clips and image frames). The image data may be communicated from the ultrasound module 16 to the PACS system 18. The PACS system 18 may securely store the image data received from the ultrasound module 16. The image data stored in the PACS system 18 may electronically tag records based on user-selected input. Once the image data is stored and / or tagged in the PACS system 18, the DICOM router 24 may connect to the PACS system 18 and retrieve the image data, and may also connect to the backend 14, which may run on a server (e.g., server 30 of FIG. 1A). For example, the DICOM router 24 may be connected to the implementation module 26 and may send the image data to the implementation module 26. In one embodiment, the DICOM router 24 may pseudonymize files so that only pseudonymized files are sent to the backend 14. For example, all patient information may be removed except for certain required variables (e.g., gestational age), and a pseudonymous identifier may be added to the file for each study and / or record. Once the DICOM router 24 receives the output from the backend 14, it may perform re-identification by replacing the pseudonymous identifier with patient information. The implementation module 26 may upload the image data to the storage device 28. For example, the storage device 28 may store encrypted and otherwise secured image data.

[0062] The implementation module 26 may read certain image data from the storage device 28 and may communicate such image data to the analysis module 29. The analysis module 29 may process the image data using machine learning algorithms to identify the presence of morphological abnormalities in the image data, as described in more detail herein with respect to FIGS. 3A and 3B. In one example, the results and / or output of the analysis module 29 may be stored in the storage device 28 as annotated DICOM files. The annotated DICOM files (e.g., indicating morphological anomalies) may be communicated back to the DICOM router 24 and stored in the PACS 18. Once stored in the PACS 18, a healthcare provider (e.g., a physician) may use the DICOM viewer 22 to access the annotated DICOM files from the PACS 18 and view the annotated DICOM files (e.g., using a healthcare provider device).

[0063] 2, an exemplary display suitable for summarizing results returned to display computer 20 by an interpretation component resident on server system 30 is illustrated. Display 50 includes three columns 51-53, which may contain links that may be activated using an input device, e.g., a mouse, associated with display computer 20. Column 51, labeled "View," describes a standard guideline view (e.g., 4C, LVOT, etc.). Column 52, labeled "Image," includes a checkbox that indicates whether the interpretation component of the system identified and selected an image frame as corresponding to a standard guideline view. Column 53, labeled "Observation," indicates whether any observations, such as potential abnormalities, were detected in the selected image frame.

[0064] Activating a link in the View column, i.e., column 51, such as by clicking the view title with a mouse, will display an idealized generic image of a standard guideline view, such as that shown in Table 2. In column 52, the presence of a checkbox indicates that an image frame was selected by the interpretation component on server computer 30. Clicking that checkbox will cause the display computer to display the raw image selected by the interpretation component. The absence of a checkbox in column 52 indicates that the interpretation component was unable to locate an image within the image data (e.g., an animated video clip) suitable for analysis by machine learning features. For example, clicking an empty checkbox for the RVOT in FIG. 2 may be configured to display a prompt to the clinician to rescan a portion of the patient's abdomen to obtain new image data (e.g., an animated video clip) containing the desired view, which may then be transmitted to server computer 30 for supplemental analysis.

[0065] Column 53 may include a text description of any observations noted by the interpretation component in the selected image frame. For example, in FIG. 2 , column 53 presents the label “No Findings” for all views except the RVOT view. Because display 50 may be visible to the patient, column 53 preferably uses the neutral labeling “Review” to indicate to the clinician that a potential abnormality has been detected in that view, rather than a more descriptive label that may cause undue concern to the patient. In one embodiment, the text indicator for each standard view may be a clickable link. For example, clicking the label stating “No Findings” for the 4C, LVOT, 3V, and 3VT views in FIG. 2 may display the image frame selected by the interpretation component along with labels indicating where the interpretation component determined the anatomical landmarks are located. On the other hand, clicking the label “Review” for the RVOT view in FIG. 2 may display the selected image frame, labeled anatomical landmarks, and a bounding box surrounding the suspected abnormality, along with a text description of the potential defect.

[0066] In a fetal ultrasound examination performed in accordance with the principles of the present invention, following review of the real-time ultrasound image data (e.g., a motion video clip) generated by ultrasound scanning device 10 as displayed on display computer 20, the clinician may then review the analysis results generated and returned by the interpretation component resident on server computer 30. Thus, the clinician may review the contents of display 50 of FIG. 2 , review selected raw image data corresponding to each standard guideline view (by clicking a checkbox in column 52), and review a detailed machine learning analysis of that selected image frame by clicking an indicator in column 53. The clinician may thus be able to confirm their own observations during review of the real-time video clip or adjust their findings based on the machine learning analysis results. As described above, display computer 20 may include the ability to open additional boxes associated with any of the image frames presented in column 53 to record and store additional findings for later recall when preparing a written or electronic report documenting the fetal ultrasound examination.

[0067] 3A, an exemplary flowchart 60 for the interpretation component of the analysis software is described. It should be understood that the tasks and / or operations performed in flowchart 60 may be performed on a server (e.g., server 30 of FIG. 1), an ultrasound scanning system (e.g., ultrasound scanning system 10 of FIG. 1), and / or a display (e.g., display 20 of FIG. 1). In step 61, image data (e.g., motion video clips, image frames, and the like) is received from and / or determined by ultrasound system 10 or display computer 20. In step 62, a template corresponding to a standard guideline view, e.g., 4C, LVOT, RVOT, 3V, or 3VT, is selected. The template may consist of an idealized version of the image shown in Table 2. In step 63, the received image data, which may be a video image clip, is analyzed using a machine learning algorithm, e.g., a convolutional neural network, a deep neural network, and / or one or more suitable neural networks, to compare each frame (e.g., image frame) of the received image data (e.g., a motion video clip) to the standard template and determine the frame or frames that best match the standard view guideline template. It should be understood that a video clip formed from a set of image frames, or individual image frames, may be analyzed in step 63. The interpretation component may also analyze the selected image frame or frames to ensure that the image meets specified quality requirements for sharpness. If no frames in the image data are determined to correspond to the standard view, the process proceeds to step 65, where a flag is set indicating that no suitable image frames are available, in decision box 64, and the process continues with selecting the next standard view template in step 62.

[0068] If the interpretation component determines that a corresponding frame is available in the received image data, the process proceeds to step 66, where the selected image frame, and optionally, non-selected image frames, are analyzed by another machine learning algorithm (e.g., one or more suitable neural networks) to detect the presence or absence of anomalies associated with that standard view. For example, if the selected image frame corresponds to a 4C standard view template, the algorithm will analyze the selected frame for the presence of any of the defects and / or anomalies listed in Table 3 for that standard view. If a defect is detected in the selected image frame, the algorithm may look to adjacent frames of the video clip to confirm the presence of the same defect.

[0069] Morphological abnormalities include, in one example, riding arteries (e.g., arteries exiting the left ventricle positioned over a ventricular septal defect), septal defects at the crook of the heart (e.g., a septal defect positioned at the crook of the heart in either the primum atrial septum or the inlet ventricular septum), parallel aorta, enlarged cardiothoracic ratio (e.g., a ratio of cardiac area to thorax measured at end diastole greater than 0.33), right to left ventricular size discrepancy (e.g., a ratio of right and left ventricular area at end diastole greater than 1.4 or less than 0.5). ), tricuspid-to-mitral annular size mismatch (e.g., a ratio between the tricuspid and mitral valves at end diastole greater than 1.5 or less than 0.65), pulmonary-to-aortic annular size mismatch (e.g., a ratio between the pulmonary and aortic valves at end systole greater than 1.6 or less than 0.85), abnormal outflow tract relationship (e.g., absence of the typical anterior-posterior crossing pattern of the aorta and pulmonary artery), and cardiac axis deviation (e.g., a cardiac axis (the angle between the line bisecting the rib cage and the interventricular septum) less than 25° or more than 65°). Alternatively, or in addition, any other morphological abnormality may be detected in step 66.

[0070] In optional step 67, an overlay may be created for the selected image frame, including graphical pointers to the detected anatomical landmarks and bounding boxes surrounding the abnormalities detected in the image frame. The overlay may also, in addition or alternatively, include text information describing the specific abnormality and / or the associated class of CHD, as described in Table 3. In step 68, the information generated by the interpretation component, i.e., the overlay and graphical / explanatory information, is associated with the selected image frame and stored within server computer 30 for later transmission to display computer 20. In optional decision box 69, a determination is made whether all image data received in step 61 has been analyzed and / or whether all standard views have been determined to exist. If it is determined that not all standard views exist and / or all image data received in step 61 has not been analyzed, the process may return to step 62, and the next standard view template is selected for analysis. Alternatively, if in decision box 69 it is determined that all standard views are present and / or all image data has been analyzed, the process may proceed to step 71 and results are returned to display computer 20 for presentation and review by the clinician. Alternatively, decision 69 may be optional and may be bypassed in favor of beginning blocks 71 and / or 72. For example, the user may decide to return the analysis results to the user interface for display and / or generate a report even if all standard views have not been determined to be present and / or all image data has not been analyzed.

[0071] In optional step 72, the analysis and / or results may be used to generate a report. For example, the report may identify detected morphological abnormalities corresponding to an image frame and / or may include an entry for each standard view. Alternatively, only entries for standard views determined to be present may be included in the report. For example, detected anomalies may include one or more of abnormal ventricular asymmetry, aortic coarctation, pulmonary artery or aortic valve stenosis, ventricular hypoplasia, or single ventricle, and / or any other cardiovascular abnormality. The report may be pre-populated so that a representative image may be selected for each standard view entry. If a morphological abnormality is detected, an image representative of the morphological abnormality for a given standard view may be included in the report in the entry for the corresponding view. If a bounding box is generated for a given frame, such image with a bounding box overlay may be used in the report. Information about the view, anatomical structures, any textual descriptions of detected morphological defects and / or abnormalities, and / or any other relevant information may additionally be included in the report to add context to the image and otherwise generate a more informative report. The resulting analysis, results, annotations, and / or report may be stored for later reference. The system may cause the report to be presented on a display screen. For example, in response to a request by a clinician, the report may be displayed or otherwise presented. The report may include an overlay indicating the presence of one or more potential abnormalities.

[0072] The images, image frames, video clips, analysis, results, annotations, and / or reports may be shared with or otherwise made available to a specialist or clinician (e.g., in response to a referral to a specialist or clinician). Each type of morphological abnormality may be associated with a specialist or clinician and their contact information. If a morphological abnormality is detected in step 66, a specialist or clinician who addresses the morphological abnormality may optionally be recommended.

[0073] In addition to performing steps 61-72 illustrated in FIG. 3A , the system may also perform an objective evaluation of the examination, the images generated, and / or the technician (e.g., sonographer) performing the imaging. The system may consider data such as the average duration of the examination and / or image acquisition, the quality of the images obtained, the percentage of standard views for which suitable images are obtained, and / or any other information indicative of the quality of the examination, images, and / or sonographer. For example, a model trained to generate one or more quality values ​​indicative of the quality of the examination and / or images may be used to process such data. The data may be determined over the course of several examinations. For example, the percentage of standard views for which suitable images are obtained for multiple examinations by the same technician may be used to determine a quality value indicative of the technician's rate of performing complete examinations.

[0074] Turning now to FIG. 3B, the neural network output and post-processing steps of the analysis software are described. As shown in FIG. 3B, input 32 may be input to model 33, which may be one or more neural networks. Input 32 may be image data, such as motion video clips, image frames, and the like. In one embodiment, input 32 may be raw digital data representing or forming the image data. Model 33 may include a convolutional base 34, a classification head 35, a segmentation head 36, and a keypoint detection head 37. Convolutional base 34 may be a convolutional neural network (CNN) that may process input 32. Convolutional base 34 may include a classification head 35, a segmentation head 36, and a keypoint detection head 37.

[0075] The classification head 35 may be a classification neural network that may be trained to process the input 32 to determine the probability of the presence or absence of one or more morphological anomalies and / or the likelihood that one or more of the morphological anomalies are indeterminate. The segmentation head 36 may be a segmentation neural network that may be trained to determine contours, perimeters, and / or areas that are interpretable to or otherwise correspond to certain anatomical structures within the image data represented by the input 32. The keypoint detection head 37 may be a neural network that may be trained to determine the location of certain anatomical structures and / or points within the image data represented by the input 32.

[0076] As shown in FIG. 3B , model 33 may output neural network outputs 39, which may include outputs 38, 40, and 41. Output 38 may be output from classification head 35 and may be the probability of the presence or absence of a morphological abnormality (e.g., a lateral artery, a septal defect in the Crooks' heart, parallel aorta, etc.). Output 40 may be output from segmentation head 36 and may identify data points corresponding to image data forming the contours of anatomical structures represented in the image data (e.g., contours of the left ventricle, right ventricle, heart, rib cage, etc.). Output 41 may be output from keypoint detection head 37 and may identify data points corresponding to the locations in the image data of features of anatomical structures represented in the image data (e.g., the edge of the tricuspid valve, the edge of the mitral valve, the edge of the pulmonary valve, the edge of the aortic valve, the long axis of the heart, and / or the anterior-posterior axis of the thorax).

[0077] The neural network output 39 may then be processed by a post-processing module 42. For example, the output 38 may be processed by module 43 to determine whether a morphological abnormality is absent, present, or uncertain. For example, the output 38 may be one or more vectors, including values ​​indicating the probability of presence, absence, and / or uncertainty of presence or absence for each morphological abnormality. Module 43 may process the vectors by comparing each to a threshold value to determine whether each morphological abnormality is absent, present, or uncertain. For example, for the morphological abnormality "rider artery," a vector may be output having values ​​of 0.95 for presence, 0.1 for absence, and 0.1 for uncertainty. The threshold may be set to 0.9 for presence, absence, and uncertainty, and a value of 0.95 for presence may then satisfy the threshold. As a result, module 43 may determine that an abnormal "rider artery" is present. It should be understood that other thresholds and / or limits may be used to determine the presence, absence, and / or uncertainty of a morphological abnormality.

[0078] Output 40 may be processed by module 44, which may determine measurements (e.g., areas, lengths, diameters, circumferences, and the like) related to the contours of anatomical structures represented in the image data, such as, for example, left ventricular area, right ventricular area, cardiac circumference, and / or thoracic circumference. These measurements may then be provided to and processed by module 47, which may determine ratios and / or comparisons of measurements (e.g., right ventricular area divided by left ventricular area, cardiac circumference divided by thoracic circumference, etc.). The ratios and / or comparisons may then be provided to and processed by module 50, which may compare such ratios and comparisons to thresholds and / or limits to determine the absence, presence, or uncertainty of certain abnormalities based on the ratios and / or comparisons determined in module 47. In one example, module 50 may determine the presence, absence, or uncertainty of right ventricular / left ventricular size mismatch or the presence, absence, or uncertainty of an enlarged cardiothoracic ratio. For example, the valves determined by module 46 may be compared against threshold values ​​to determine whether such values ​​exceed the threshold values. For example, the valves determined by module 47 may be compared against threshold values ​​to determine whether such values ​​exceed the threshold values.

[0079] Output 41 may be processed by module 45, which may determine certain measurements (e.g., lengths, angles, areas, etc.) based on anatomical features represented in the image data. For example, module 45 may determine values ​​such as tricuspid valve size (e.g., length, width, area), mitral valve size, pulmonary valve size, aortic valve size, and / or heart axis angle (e.g., the angle between the long axis of the heart and the anterior-posterior axis of the chest). The values ​​determined by module 45 may be provided to module 46, which may determine ratios and / or comparisons based on the values. For example, ratios such as tricuspid valve size divided by mitral valve size and / or pulmonary valve size divided by aortic valve size may be determined. The ratios and / or comparisons may then be provided to and processed by module 48, which may determine the absence, presence, or uncertainty of certain abnormalities based on the ratios and / or comparisons determined in module 46 by comparing such ratios and comparisons to thresholds and / or limits. In one example, module 48 may determine the presence, absence, or uncertainty of tricuspid-to-mitral valve size mismatch, pulmonary / aortic valve size mismatch, and cardiac axis prolapse. For example, the valves determined by module 46 may be compared against threshold values ​​to determine whether such values ​​exceed the threshold values.

[0080] 4A and 4B, examples of screen displays that may be presented on display computer 20 corresponding to clicking on a text indicator identified in FIG. 2 are described. As described above, if a standard view is identified by the interpretation component as being present in the image data (e.g., a motion video clip) transmitted to server computer 30, a check mark will appear in column 52 of FIG. 2. As previously described, clicking the check box will cause the raw image selected by the interpretation component to be displayed on display computer 20. Clicking on the "No Findings" indicator in column 53 will cause the selected image frame annotated with an overlay generated by the interpretation component to be displayed on display computer 20. FIG. 4A is an example of an image frame corresponding to a 4C standard view with no abnormalities and in which the foramen ovale is identified. FIG. 4B is an example of a selected image frame corresponding to a standard 4C view in which a large atrioventricular defect is detected. In FIG. 4B, the defect is surrounded by a white bounding box, and other cardiac structures are identified by text and graphical indicators.

[0081] 5, an exemplary graphical user interface is depicted that includes at least a portion of images and / or video captured by an imaging system (e.g., an ultrasound imaging system), a list of standard views, and a list of abnormalities. Graphical user interface 80 may be a digital display and may be presented on a healthcare provider's device (e.g., a computing device, laptop, desktop, tablet, smartphone, display, etc.). Graphical user interface 80 may present information about whether the images correspond to certain types of standard views and / or whether certain abnormalities or conditions are present in the images from the imaging system.

[0082] As shown in FIG. 5 , image 81 may be an image frame from a video clip (e.g., ultrasound video frames and / or clips) generated by an imaging system. Time bar 87 may include cursor 90, which may move along the time bar and indicate a time point along the video clip corresponding to the image presented in image 81. Time bar 87 may also include indicator 91, which may visually indicate (e.g., via color or marker) the presence and location of an abnormality within the video clip. Cursor 90 may be moved along time bar 87 to an image frame or other time point within the time bar that corresponds to an abnormality and / or a standard view. Moving cursor 90 may then change image 81 to an image frame at a discrete time point. In one example, image 81 may be annotated with color, a bounding box, text, or other visual indicator to identify the location of the abnormality within image 81.

[0083] The standard view list 82 may include a list of standard imaging views (e.g., 4C, LVOT, RVOT, 3V, 3VT, etc.). Any other image views other than those listed in FIG. 5 may alternatively or additionally be included. For each standard view, the image 81 includes a record and frame indicator 83, which identifies whether a record exists for each view and whether a representative frame is identified for each view. A time bar 84 is also included for each view and is equal in length to the time length of a given video clip. For each time bar, a visual indicator (e.g., visual indicator 85) is included to indicate the location within the video clip where the given standard view appears. If no visual indicator is provided for a given time bar, the respective standard view does not appear within the video clip. A cursor bar 86 is also included on the time bar to indicate the location on the time bar 84 corresponding to the image frame presented in the image 81.

[0084] The anomaly list 89 may include a list of anomalies and / or conditions corresponding to the image 81. For example, the anomaly list 89 may include an enlarged CTR, cardiac axis deviation, RV / LV size mismatch, TM / MV size mismatch, septal defect in cardiac Crooks, riding artery, parallel aorta, PV / AV size mismatch, abnormal outflow tract relationship, and / or any other anomaly and / or condition. For each anomaly and / or condition, the graphical user interface 80 includes a record and frame indicator 83, which identifies whether a record exists for each view and whether a representative frame is identified for each view.

[0085] A time bar 95 is also included for each view and is equal in length to the duration of a given video clip. For each time bar of the time bars 95, a visual indicator 96 is included to indicate the location within the video clip where the given view appears. If no visual indicator is provided for a given time bar, then the given anomaly or condition corresponding to the time bar does not appear within the respective video clip. A cursor bar 86 may also be included on the time bar to indicate the location on the time bar 95 that corresponds to the image frame presented in the image 80.

[0086] A time bar 94 may also be included below time bar 84 and time bar 95 and may, via cursor 93, indicate the location of cursor bar 86 along the length of the respective video clip. Moving cursor 90, cursor bar 86, and / or cursor 93 may individually move the other cursors and / or cursor bars. Time bar 94 may include play and / or pause buttons. When the play button is activated, the video clip may play and show various image frames of the video clip within image 81. As the video clip progresses within image 81, cursor bar 86 and cursors 90 and 93 may advance along their respective time bars. When the pause button is activated, the video clip may be paused. Graphic user interface 80 may optionally include buttons 98 for moving to the next or previous video clip.

[0087] Referring now to FIG. 6 , an exemplary process flow for acquiring medical images, identifying red flags, annotating the images, and providing the flagged and annotated images in a graphical representation is depicted. As shown in FIG. 6 , fetal images (e.g., 2T fetal ultrasound images, videos, and / or recordings) are generated and transferred to a platform, which may be a cloud-based platform on a server that may be a remote server or a local server. The platform may process the images using techniques described herein to identify red flags (e.g., the presence of anomalies, abnormalities, and / or conditions in the images). The images may be annotated to indicate the presence of anomalies, abnormalities, and / or conditions. The annotated images may be presented to a healthcare provider (e.g., a clinician) on a healthcare provider device for immediate or near-immediate (e.g., 1 minute, 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, etc.) review by the healthcare provider. Review by the healthcare provider may occur while the patient is on-site. The device generating the image and / or the healthcare provider device may be in wireless communication with the platform (e.g., via the Internet). The healthcare provider device and the imaging device may be the same or different devices.

[0088] 7, an exemplary graphic user interface is depicted that includes a representation of a patient's image, patient information, and a list of abnormalities. Graphic user interface 100 may include patient information 101, including patient ID, healthcare provider complex, date, time, and any other relevant information. Graphic user interface 100 may further include image 102, which may include play and / or pause buttons for playing a video of the image (e.g., an ultrasound video). A user may scroll to the next video or image by moving up or down the graphic user interface.

[0089] The graphic user interface 100 may further include an anomaly analysis 103 for each image 102, which may include a list of anomalies, and for each anomaly, a time bar may be provided on which a visual indicator may indicate whether that particular anomaly is present in the video. The visual indicator may be a color bar that may extend over the portion of the time bar where the anomaly is present. The time bar may include a cursor to indicate the location along the time bar that corresponds to the image frame present on the image 102. The anomaly analysis may further include a time bar with a cursor and pause and play buttons. Moving the cursor and / or operating the pause or play may cause the image 102 to move to a certain point in time, pause, or play.

[0090] 8A-8C, exemplary user interfaces for displaying image data and standard views and abnormality indicators generated by an ultrasound system are illustrated. For example, user interface 200 of FIG. 8A may be generated by an ultrasound system (e.g., ultrasound system 10 of FIG. 1B). User interface 200 may be presented on display computer 20 of ultrasound system 10 and / or display 22 of healthcare provider device 25.

[0091] The user interface 200 may include a user section 202, which may include a study title, which may be an identifier for the ultrasound study, and user information 206, which may include a user identifier (ID), date, general facility data, fetal age, maternal age, status (e.g., processed), and the like. The user section 202 may include a comments section 208 for a technician or other healthcare provider to describe the study and / or a set of image data (e.g., a video clip).

[0092] The user interface 200 may further include a thumbnail viewer 210, a detail viewer 230, and an exam summary 214. The thumbnail viewer 210 may be a collection of thumbnail images, each corresponding to a video clip and / or image frame generated by the ultrasound device 10. For example, image data, such as image clips and / or image frames, may be generated during an ultrasound exam. In one example, a thumbnail image for each video clip generated for a given exam may be included in the thumbnail viewer 210. The thumbnail viewer 210 may further include a video indicator 222, which may visually indicate whether the image data includes a video clip, and / or an indicator 224, which may visually indicate whether the fetal heart is interpretable within at least one frame of the corresponding image data set. For example, if the image data set is corrupted or if the fetal heart is not present in the image data set, the indicator 224 may not be included in the thumbnail viewer 210, or otherwise, the indicator 224 may visually indicate that the fetal heart is not interpretable.

[0093] Each set of image data (e.g., a video clip and / or one or more image frames) generated during the inspection may be viewed in detail viewer 230 (e.g., by clicking on a thumbnail image (e.g., thumbnail image 220) in thumbnail viewer 210). For example, detail viewer 230 may correspond to thumbnail image 220. A user may click on a different thumbnail image in thumbnail viewer 210 to update detail viewer 230 and present the set of image data corresponding to thumbnail image 220.

[0094] The detail viewer 230 may include an image data viewer 232, a standard view indicator 234, and a morphological anomaly indicator 236. The image data viewer 232 may present video clips and / or still frames of the image data (e.g., a set of image data corresponding to the thumbnail image 220). The standard view indicator 238 may include a list of standard views 242 and a color indicator 238 that indicates whether each standard view in the list of standard views 242 is present in the image data or whether the presence of such a standard view is uncertain. For example, the color indicator may indicate whether a certain standard view is present in the image data (e.g., using different colors for presence and absence).

[0095] The morphological anomaly indicator 236 may include a list of morphological anomalies 244 and a color indicator 240 that indicates whether each morphological anomaly in the list of morphological anomalies is present in the image data. For example, the color indicator 240 may indicate whether a morphological anomaly is present in the image data, or alternatively, whether the presence of the morphological anomaly is uncertain. The color indicator for the standard view indicator 234 may be different from the color indicator for the morphological anomaly indicator 236 (e.g., using different unique colors for presence and absence, respectively). A different unique color may also be used for uncertainty.

[0096] 8B , user interface 200 may be adjusted by a user to present a detail viewer 250, a thumbnail viewer 210, and an examination summary 270. A user may select a thumbnail image 240 in thumbnail viewer 210 and / or otherwise navigate user interface 200 to view detail viewer 250, which may correspond to thumbnail image 240. Detail viewer 250 may be identical to or similar to detail viewer 230 of FIG. 8A . Detail viewer 250 may include an image data viewer 251, which may include still frames or video clips of image data (e.g., a set of image data corresponding to thumbnail image 240). A time bar 252 may be included in detail viewer 250, as well as a cursor 254, which may indicate a point along the time bar corresponding to an image frame presented in image data viewer 251.

[0097] The detail viewer 250 may further include a standard view indicator 253 and a morphological anomaly indicator 255, which may be the same as or similar to the standard view indicator 234 and the morphological anomaly indicator 236 of FIG. 8A . As shown in FIG. 8B , the standard view “4C” may be present in the standard view indicator 253, and a color indicator 261 indicates “yes” for presence, which is presented in blue. Some morphological anomalies, such as an enlarged CTR, RV / LV size mismatch, TV / MV size mismatch, and cardiac Crookes septal defect, are shown as absent in the morphological anomaly viewer 255. Other anomalies in the morphological anomaly viewer 255 are shown as indeterminate. The color indicator 263 may display green to indicate the presence of each anomaly in the image data presented in the image data viewer 251, and white to indicate indeterminate.

[0098] The standard view indicator may also include a time bar for each standard view in the list of standard views. For example, time bar 256 may correspond to standard view "4C." Similarly, the morphological anomaly indicator 255 may include a time bar for each anomaly in the morphological anomaly viewer 255. For example, time bar 258 may correspond to a magnified CTR. Each time bar may exhibit a color along some or all of the time bar when the corresponding standard view or anomaly is determined to be present or absent in the image data. For example, time bar 256 may be blue to indicate the presence of standard view "4C," and time bar 258 may be green to indicate the absence of a magnified CTR.

[0099] Each time bar for standard view indicator 253 and morphological anomaly viewer 255 may include a visual indicator that moves with cursor 254. For example, time bar 256 may include visual indicator 262, and time bar 258 may include visual indicator 260. Additionally, below each time bar for standard view indicator 253 and morphological anomaly viewer 255, a time bar 264 may be included that may be aligned with each time bar (e.g., time bar 256 and time bar 258) and that may include a cursor that is aligned with visual indicators 262 and 260 and that may be used by a user to move cursor 254 to different points along time bar 252.

[0100] The user interface 200 may further include an examination summary 270, which may include a standard view summary 272 and an abnormality summary 276, which may summarize the standard views and morphological abnormalities determined to be present, absent, or indeterminate in the set of image data uploaded from the ultrasound system. The standard view summary 272 may include a list of standard views and a color indicator to show whether each standard view is present, absent, or indeterminate. The standard view summary 272 may include a forward button 274 for each standard view, which may be used by the user to cause the user interface 200 to advance to a detail viewer containing the image data for which the standard view is present, or may automatically adjust the image data viewer so that the image frame for which the standard view is present is displayed. For example, each time the forward button 274 is operated, the next image frame determined to correspond to a particular standard view will be displayed in the detail viewer, allowing the user to efficiently browse the image frames and / or image data sets corresponding to the standard views. The exam summary 270 may allow a user to efficiently determine whether a view and / or anomaly is absent, present, or indeterminate (e.g., whether an anomaly is indeterminate).

[0101] The inspection summary 276 may further include an anomaly summary 272. The anomaly summary 276 may include a list of anomalies and a color indicator to indicate whether each anomaly is present. The anomaly summary 276 may include a forward button 278 for each anomaly, which may be used by the user to cause the user interface 200 to advance to a detail viewer containing the image data in which the anomaly is present, or may automatically adjust the image data viewer so that the image frame in which the anomaly is present is displayed. For example, each time the forward button 278 is actuated, the next image frame determined to correspond to the particular anomaly will be displayed in the detail viewer, allowing the user to efficiently browse through the image frames and / or sets of image data corresponding to the anomaly. The user interface 200 may further include a save button 279 for saving any image data, images, decisions, and / or data, settings, notes, comments, or the like from the user interface 200.

[0102] 8C , the detail viewer 250 of the user interface 200 may be updated to indicate that an abnormality has been determined to be present in the image data. For example, the morphological abnormality indicator 255 may be updated to change the color indicator 284 corresponding to the deviated cardiac axis to red to indicate that the image data presented in the image data viewer 251 is a “deviated cardiac axis.” The time bar 282 may be changed to the same color (e.g., red) as the color indicator 255 to indicate the presence of a morphological abnormality. The time bar 282 may alternatively be changed to the same color (e.g., green) as the “absence” indicator to indicate one or more frames along the time bar that correspond to the absence of an abnormality. Alternatively, only the portion of the time bar 282 corresponding to the time point in the image data associated with the image frame showing the abnormality will be colored.

[0103] 9, a process flow 300 for dynamically requesting additional image data based on the presence or absence of standard views and morphological abnormalities in the image data is illustrated. Some or all of the process flow blocks in this disclosure may be performed in a distributed manner across any number of devices. Some or all of the operations in the process flow may be optional and may be performed in a different order.

[0104] In optional block 302, computer-executable instructions stored on a memory of a device, such as an ultrasound system server and / or computer (e.g., display computer 20 of FIG. 1B ), may be executed to present a request for image data (e.g., image frames, video clips, etc.) having a standard view (e.g., 4C) from a predetermined set of standard views. For example, a list of standard views to be collected in an ultrasound examination may be predetermined. In block 304, computer-executable instructions stored on a memory of a device, such as an ultrasound system server and / or computer, may be executed to determine the image data (e.g., image frames, video clips, etc.). For example, the ultrasound system may present a prompt for image data, and a user may generate the image data using the ultrasound system, which may include an ultrasound probe.

[0105] In block 306, computer-executable instructions stored on a memory of a device, such as a server and / or computer of an ultrasound system, may be executed to analyze the image data and determine the presence or absence of a standard view from a set of standard views. For example, the image data may be processed using the approach described above with respect to FIG. 3A. In decision 307, computer-executable instructions stored on a memory of a device, such as a server and / or computer of an ultrasound system, may be executed to determine whether the image data satisfies a standard view (e.g., whether a standard view is present in the image data).

[0106] If a standard view is determined to be absent or if the presence of a standard view in the image data is uncertain, then in block 310, computer-executable instructions stored on a memory of a device, such as an ultrasound system server and / or computer, may be executed to log or otherwise describe the absence of the standard view in the image data, or, if the presence or absence of a standard view is uncertain, to log or otherwise describe the uncertainty of the presence of the standard view. Further, one or more points in time (e.g., timestamps) in the image data may be associated with the absence or uncertainty of a standard view in the image data. In block 311, computer-executable instructions stored on a memory of a device, such as an ultrasound system server and / or computer, may be executed to submit a request for additional image data corresponding to the standard view.

[0107] Alternatively, if a standard view is determined to be present in the image data, in block 308, computer-executable instructions stored on a memory of a device, such as an ultrasound system server and / or computer, may be executed to log or otherwise describe the presence of a standard view in the image data and associate one or more time points (e.g., timestamps) in the image data with the standard view. In block 312, computer-executable instructions stored on a memory of a device, such as an ultrasound system server and / or computer, may be executed to analyze the image data and determine the presence or absence of a morphological anomaly from a set of morphological anomalies. For example, the image data may be processed using the approach described above with respect to FIG. 3A. For each view template, data indicative of one or more potential anomalies may be determined. Optionally, block 312 may be initiated after block 310 as well as after block 308.

[0108] In decision 316, computer-executable instructions stored on memory of a device, such as an ultrasound system server and / or computer, may be executed to determine whether a morphological anomaly is present or absent. If it is unclear whether a morphological anomaly is present or absent, in block 314, computer-executable instructions stored on memory of a device, such as an ultrasound system server and / or computer, may be executed to log or otherwise mark the morphological anomaly as indeterminate because it is unclear whether the anomaly is present or absent at one or more time points (e.g., timestamps) associated with the image data. Alternatively, if it is determined that a morphological anomaly is present or absent, in block 318, computer-executable instructions stored on memory of a device, such as an ultrasound system server and / or computer, may be executed to log or otherwise mark the morphological anomaly as present or absent, as appropriate, and / or associate the presence or absence of such an anomaly with one or more time points in the image data. For example, data indicative of one or more morphological anomalies may be associated with a view template corresponding to the image data.

[0109] At decision 320, computer-executable instructions stored on memory of a device, such as a server and / or computer of an ultrasound system, may be executed to determine whether additional views of the abnormality are requested. For example, when an abnormality is determined to be present, it may be desirable to generate additional imaging to further analyze the abnormality. Conversely, when an abnormality is determined to be absent, it may be desirable to generate additional images to further confirm the absence of the abnormality. Whether additional views are requested may depend on the type of abnormality detected and may be predetermined (e.g., if an abnormality is detected, the system may automatically request an additional view). If additional views are not requested at decision 320, blocks 302 and / or 304 may be resumed. Alternatively, if additional views of the abnormality are requested, at block 322, computer-executable instructions stored on memory of a device, such as a server and / or computer of an ultrasound system, may be executed to submit a request for additional views of the abnormality, and block 304 may be resumed.

[0110] While various illustrative embodiments of the present invention are described above, it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the invention. It is intended that the appended claims cover all such changes and modifications that fall within the true scope of the invention.

Claims

1. 1. A system for use with an ultrasound system to assist a clinician in detecting and diagnosing cardiac defects during fetal ultrasound examination, said system comprising: One or more computers configured to store non-transient program instructions, the one or more computers comprising: storing view templates corresponding to standard guideline views, including view templates for at least 4C, LVOT, RVOT, 3V, and 3VT views; storing data indicative of one or more potential anomalies associated with each one of the view templates; receiving a plurality of sets of image data generated by the ultrasound system during a fetal ultrasound examination, each set of image data of the plurality of sets of image data comprising a plurality of frames; comparing each frame of the plurality of frames with the view templates to identify and select a corresponding frame for each view template, if any; analyzing each corresponding frame to detect the presence of said one or more potential anomalies; presenting, in response to a request by the clinician, on a display screen the corresponding frame for each standard view including an overlay indicating the presence of the one or more potential abnormalities; one or more computers programmed to A system comprising:

2. The system of claim 1 , wherein the one or more computers comprise a display computer and a server computer.

3. The system of claim 2 , wherein the non-transient program instructions include a user interface component and an interpretation component.

4. The system of claim 3 , wherein the user interface component is configured to receive, store, and display the multiple sets of image data generated by the ultrasound system in real time.

5. The system of claim 4 , wherein the user interface component is configured to store and display analysis results returned by the interpretation component.

6. The system of claim 3 , wherein the interpretation component includes a machine learning algorithm for identifying and selecting each corresponding image frame.

7. The system of claim 3 , wherein the interpretation component further comprises a machine learning algorithm for detecting the presence of the one or more potential anomalies in each corresponding image frame.

8. The system of claim 1 , wherein the overlay indicating the presence of the one or more potential anomalies includes one or more of a bounding box, a graphical indicator, or a text indicator surrounding the potential anomaly.

9. The system of claim 1 , further comprising non-transient program instructions that enable the clinician to annotate the overlay.

10. 10. The system of claim 1, further comprising non-transient program instructions for generating a report documenting the clinician's observations during the fetal ultrasound examination.

11. The system of claim 1 , wherein the report includes the clinician's observations during the fetal ultrasound examination.

12. 2. The system of claim 1, wherein 4C means four ventricles, LVOT means left ventricular outflow tract, RVOT means right ventricular outflow tract, 3V means three vessels, and 3VT means three vessels and the trachea.

13. The system of claim 1 , wherein the one or more computers are further programmed to determine, for each template of views, data indicative of one or more potential anomalies.

14. The system of claim 1 , wherein the one or more computers are further programmed to associate each of the data indicative of one or more potential anomalies with a respective individual view template.

15. The system of claim 1 , wherein each frame of the plurality of frames is compared to the view template using a neural network.

16. The one or more computers further comprise: determining the data indicative of one or more potential anomalies for each of the view templates; associating each of the data indicative of one or more potential anomalies with a respective individual view template; using a neural network to compare each corresponding image frame with the data indicative of the one or more potential anomalies for the view template corresponding to each corresponding image frame to detect the presence of the one or more potential anomalies; generating a report indicating said presence of one or more potential anomalies for each corresponding image frame; and in response to a request by the clinician, causing the display screen to present the report including an overlay indicating the presence of the one or more potential abnormalities. The system of claim 1 , programmed to:

17. 1. A method for use with an ultrasound system to assist a clinician in detecting and diagnosing cardiac defects during fetal ultrasound examination, the method comprising: providing a computer configured to store and execute non-transient program instructions; storing, on the computer, view templates corresponding to standard guideline views, including view templates for at least 4C, LVOT, RVOT, 3V, and 3VT views; storing, on the computer, data indicative of one or more potential anomalies associated with each one of the view templates; generating, using an ultrasound system, a plurality of sets of image data during a fetal ultrasound examination and storing the plurality of sets of image data, each set of image data comprising a plurality of frames; receiving the plurality of sets of image data by the computer; using the computer to compare each frame of the plurality of frames to the view templates to identify and select a corresponding image frame for each view template, if any; analyzing, with the computer, each corresponding image frame to detect the presence of the one or more potential anomalies; presenting, upon request by the clinician, on a display screen associated with the computer, the corresponding image frames for each standard view including an overlay indicating the presence of the one or more potential abnormalities; A method comprising:

18. 20. The method of claim 17, wherein providing a computer comprises providing one or more computers including a display computer and a server computer.

19. 20. The method of claim 18, wherein providing the one or more computers includes providing the display computer with non-transient program instructions for a user interface component and providing a server computer with non-transient instructions for an interpretation component.

20. 20. The method of claim 19, further comprising displaying to the clinician, in real time, the multiple sets of image data generated by the ultrasound system using the display computer.

21. 20. The method of claim 19, further comprising transmitting analysis results generated by the interpretation component from the server computer to the display computer.

22. 20. The method of claim 17, wherein using the computer to compare each frame of the plurality of frames to the view template comprises using a machine learning algorithm to analyze the plurality of frames to identify and select each corresponding image frame.

23. 20. The method of claim 17, wherein using the computer to analyze each corresponding image frame to detect the presence of the one or more potential anomalies comprises using a machine learning algorithm to analyze each corresponding image frame to detect the presence of the one or more potential anomalies.

24. 20. The method of claim 17, wherein presenting the overlay on the display screen comprises presenting graphical or textual indicia indicating the presence of the one or more potential anomalies.

25. 18. The method of claim 17, further comprising using the computer to create an annotated overlay corresponding to the overlay that includes additional graphical or textual information entered by the clinician, and storing the annotated overlay.

26. 18. The method of claim 17, further comprising generating, with the computer, a report documenting the annotated overlay for the fetal ultrasound examination.

27. 18. The method of claim 17, further comprising generating, with the computer, a report with an entry for each standard view and the overlay indicating the presence of the one or more potential anomalies.

28. 28. The method of claim 27, further comprising using the computer to determine the presence of one or more abnormalities and transmitting at least a portion of the multiple sets of image data or one or more of the report to a specialist.

29. 18. The method of claim 17, further comprising: using the computer to determine a quality value for the fetal ultrasound examination based on the plurality of sets of image data.

30. 1. A computer-implemented method for analysis of fetal ultrasound images, the computer-implemented method comprising: receiving a plurality of sets of image data generated by an ultrasound system during a fetal ultrasound examination, each set of image data of the plurality of sets of image data comprising a plurality of frames; analyzing a set of image data of the plurality of sets of image data to automatically determine that one or more frames of the set of image data correspond to a standard view of a plurality of standard views; analyzing the set of image data to automatically determine that the one or more frames exhibit a first morphological abnormality of a plurality of morphological abnormalities; generating a user interface for display, said user interface comprising: (a) an image data viewer adapted to visually present said set of image data; (b) a standard view indicator corresponding to the set of image data presented on the image data viewer, visually indicating whether each standard view of the plurality of standard views is present in the set of image data; (c) a morphological anomaly indicator corresponding to the set of image data presented on the image data viewer, the morphological anomaly indicator visually indicating whether each morphological anomaly of the plurality of morphological anomalies is present in the set of image data; To have Including, A computer-implemented method, wherein when the image data viewer visually presents the set of image data, the standard view indicator indicates that a first standard view is present in the set of image data and the morphological anomaly view indicator indicates that a first morphological anomaly is present.

31. 31. The computer-implemented method of claim 30, wherein the user interface is generated on a display of the ultrasound system.

32. 31. The computer-implemented method of claim 30, wherein the user interface is generated on a display of a healthcare provider device.

33. 31. The computer-implemented method of claim 30, wherein the standard view indicator comprises a plurality of color indicators, each of the plurality of color indicators corresponding to one of the plurality of standard views, and wherein each color indicator of the plurality of color indicators is adapted to present a first color when a respective standard view of the plurality of standard views is present in the set of image data, and to present a second color when the respective standard view of the plurality of standard views is not present in the set of image data.

34. 31. The computer-implemented method of claim 30, wherein the morphological anomaly indicator comprises a plurality of color indicators, each of the plurality of color indicators corresponding to one of the plurality of morphological anomalies, and wherein each color indicator of the plurality of color indicators is adapted to present a first color when an individual morphological anomaly of the plurality of morphological anomalies is present in the set of image data, and to present a second color when the individual morphological anomaly of the plurality of morphological anomalies is absent from the set of image data.

35. 35. The computer-implemented method of claim 34, wherein each color indicator of the plurality of color indicators is further adapted to exhibit a third color that indicates uncertainty about the presence of the individual morphological anomaly of the plurality of morphological anomalies in the set of image data.

36. 31. The computer-implemented method of claim 30, wherein the image data viewer comprises a first time bar and a cursor on the time bar, the cursor adapted to move and cause the image data viewer to visually present multiple image frames corresponding to multiple time points along the time bar.

37. 37. The computer-implemented method of claim 36, wherein the standard view indicator comprises a plurality of second time bars, each corresponding to the first time bar, each accompanied by a first visual indicator corresponding to the cursor and adapted to move synchronously with the cursor.

38. 38. The computer-implemented method of claim 37, wherein each of the plurality of second time bars is adapted to visually indicate one or more time points on the second time bar, each of the one or more time points on the second time bar corresponding to the presence of a respective standard view of the plurality of standard views.

39. 37. The computer-implemented method of claim 36, wherein the morphological anomaly indicator comprises a plurality of second time bars, each corresponding to the first time bar, each accompanied by a first visual indicator corresponding to the cursor and adapted to move synchronously with the cursor.

40. 40. The computer-implemented method of claim 39, wherein each of the plurality of second time bars is adapted to visually indicate one or more time points on the second time bar that correspond to the presence of a respective morphological abnormality of the plurality of morphological abnormalities.

41. 31. The computer-implemented method of claim 30, wherein the plurality of sets of image data produced by the ultrasound system comprises a plurality of motion video clips produced by the ultrasound system.

42. 31. The computer-implemented method of claim 30, wherein the plurality of standard views comprises a four-ventricle (4C), a left ventricular outflow tract (LVOT), a right ventricular outflow tract (RVOT), a three-vessel (3V), and / or a three-vessel and tracheal (3VT) view.

43. 31. The computer-implemented method of claim 30, wherein the plurality of morphological abnormalities comprise an enlarged cardiothoracic ratio, right to left ventricular size discrepancy, tricuspid to mitral annular size discrepancy, cardiac axis deviation, Crookes' septal defect, pulmonary to aortic annular size discrepancy, riding artery, and / or abnormal outflow tract relationship.

44. 31. The computer-implemented method of claim 30, wherein the user interface further comprises an examination summary adapted to present a list of standard views of the plurality of standard views determined to be present in the plurality of sets of image data and a list of morphological abnormalities of the plurality of morphological abnormalities determined to be present in the plurality of sets of image data.

45. 1. A system for analysis of fetal ultrasound images, the system comprising: a memory configured to store computer-executable instructions; at least one computer processor, said at least one computer processor accessing memory and executing said computer-executable instructions to: receiving a plurality of sets of image data generated by an ultrasound system during a fetal ultrasound examination, each set of image data of the plurality of sets of image data comprising a plurality of frames; analyzing a set of image data of the plurality of sets of image data to automatically determine that one or more frames of the set of image data correspond to a standard view of a plurality of standard views; analyzing the set of image data to automatically determine that the one or more frames exhibit a first morphological abnormality of a plurality of morphological abnormalities; generating a user interface for display, said user interface comprising: (a) an image data viewer adapted to visually present said set of image data; (b) a standard view indicator corresponding to the set of image data presented on the image data viewer, visually indicating whether each standard view of the plurality of standard views is present in the set of image data; (c) a morphological anomaly indicator corresponding to the set of image data presented on the image data viewer, the morphological anomaly indicator visually indicating whether each morphological anomaly of the plurality of morphological anomalies is present in the set of image data; To have at least one computer processor configured to perform Equipped with When the image data viewer visually presents the set of image data, the standard view indicator indicates that a first standard view is present in the set of image data, and the morphological anomaly view indicator indicates that a first morphological anomaly is present.

46. 46. ​​The system of claim 45, wherein the user interface is generated on a display of the ultrasound system.

47. 46. ​​The system of claim 45, wherein the user interface is generated on a display of a healthcare provider device.

48. 46. ​​The system of claim 45, wherein the standard view indicator comprises a plurality of color indicators, each of the plurality of color indicators corresponding to one of the plurality of standard views, and each color indicator of the plurality of color indicators is adapted to present a first color when a respective standard view of the plurality of standard views is present in the set of image data, and to present a second color when the respective standard view of the plurality of standard views is not present in the set of image data.

49. 46. ​​The system of claim 45, wherein the morphological anomaly indicator comprises a plurality of color indicators, each of the plurality of color indicators corresponding to one of the plurality of morphological anomalies, and wherein each color indicator of the plurality of color indicators is adapted to present a first color when an individual morphological anomaly of the plurality of morphological anomalies is present in the set of image data and to present a second color when the individual morphological anomaly of the plurality of morphological anomalies is not present in the set of image data.

50. 50. The system of claim 49, wherein each color indicator of the plurality of color indicators is further adapted to exhibit a third color to indicate uncertainty about the presence of the individual morphological anomaly of the plurality of morphological anomalies in the set of image data.

51. 46. ​​The system of claim 45, wherein the image data viewer comprises a first time bar and a cursor on the time bar, the cursor adapted to move and cause the image data viewer to visually present multiple image frames corresponding to multiple time points along the time bar.

52. 52. The system of claim 51 , wherein the standard view indicator comprises a plurality of second time bars, each corresponding to the first time bar, each accompanied by a first visual indicator corresponding to the cursor and adapted to move synchronously with the cursor.

53. 53. The system of claim 52, wherein each of the plurality of second time bars is adapted to visually indicate one or more time points on the second time bar, each of the one or more time points on the second time bar corresponding to the presence of a respective standard view of the plurality of standard views.

54. 52. The system of claim 51 , wherein the morphological anomaly indicator comprises a plurality of second time bars, each corresponding to the first time bar, each accompanied by a first visual indicator corresponding to the cursor and adapted to move synchronously with the cursor.

55. 55. The system of claim 54, wherein each of the plurality of second time bars is adapted to visually indicate one or more time points on the second time bar that correspond to the presence of a respective morphological abnormality of the plurality of morphological abnormalities.

56. 46. ​​The system of claim 45, wherein the plurality of sets of image data produced by the ultrasound system comprises a plurality of motion video clips produced by the ultrasound system.

57. 46. ​​The system of claim 45, wherein the plurality of standard views comprises a four-ventricle (4C), a left ventricular outflow tract (LVOT), a right ventricular outflow tract (RVOT), a three-vessel (3V), and / or a three-vessel and tracheal (3VT) view.

58. 46. ​​The system of claim 45, wherein the plurality of morphological abnormalities comprise an enlarged cardiothoracic ratio, right to left ventricular size discrepancy, tricuspid to mitral annular size discrepancy, cardiac axis deviation, Crookes' septal defect, pulmonary to aortic annular size discrepancy, riding artery, and / or abnormal outflow tract relationship.

59. 46. ​​The system of claim 45, wherein the user interface further comprises an examination summary adapted to present a list of standard views of the plurality of standard views determined to be present in the plurality of sets of image data and a list of morphological abnormalities of the plurality of morphological abnormalities determined to be present in the plurality of sets of image data.

Images