System and method for detecting cardiovascular anomalies using spatiotemporal neural networks

Abstract

Systems and methods are provided for processing image data generated by a medical imaging device, such as an ultrasound or echocardiography device, and using artificial intelligence and machine learning to process the image data and determine the presence of one or more congenital heart defects (CHDs) and / or other cardiovascular anomalies within the image data, and / or to determine keypoint and / or contour detection. The image processing system may be used to detect CHDs and / or other cardiovascular anomalies in a fetus. The image data may be processed using a spatiotemporal convolutional neural network (CNN). The spatiotemporal CNN may include a spatial CNN for image recognition and a temporal CNN for processing optical flow data based on the image data. The outputs of the spatial CNN and the temporal CNN may be fused (e.g., using late fusion) to generate a likelihood of CHDs and / or other cardiovascular anomalies.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 18 / 412,325, filed January 12, 2024, U.S. Patent Application No. 18 / 183,942 (now U.S. Patent No. 11,875,507), filed March 14, 2023, and European Patent Application No. 23305236.4, filed February 22, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates generally to image processing systems, for example, image processing systems with artificial intelligence and machine learning functionality for detecting cardiovascular anomalies. [Background technology]

[0003] Today's imaging technology allows healthcare providers to look inside a patient's body and even detect abnormalities and conditions without the need for surgical procedures. For example, imaging technologies such as ultrasound imaging allow medical technicians to obtain two-dimensional views of a patient's anatomy, such as the patient's heart chambers. For example, echocardiography uses high-frequency sound waves to produce pictures of the patient's heart. Various views can be obtained by manipulating the orientation of the ultrasound sensor relative to the patient.

[0004] Medical imaging can be used by healthcare providers to perform medical examinations of a patient's anatomy without the need for surgery. For example, healthcare providers can examine generated images for visible deviations from normal anatomy. Additionally, healthcare providers can use medical images to take measurements and compare the measurements to known normal ranges to identify anomalies.

[0005] In one example, a healthcare provider may use echocardiography to identify a heart defect, such as a ventricular septal defect, which is an abnormal connection between the lower chambers (i.e., ventricles) of the heart. The healthcare provider may visually identify the connection in a medical image and make a diagnosis based on the medical image. This diagnosis may then lead to surgical intervention or other treatment.

[0006] Although healthcare providers frequently detect anomalies such as cardiac defects through medical imaging, defects and various other abnormalities remain undetected due to human error, inadequate training, subtle visual cues, and various other reasons. This is especially true with complex anatomical structures and prenatal imaging. For example, congenital heart defects (CHDs) in fetuses are particularly difficult to detect. CHDs during pregnancy are estimated to occur in approximately 1 percent of pregnancies. However, 50 to 70 percent of CHD cases are not properly detected by practitioners. Detecting CHDs during pregnancy allows healthcare providers to make a diagnosis and / or provide interventional treatment immediately, which can lead to improved fetal and infant health and fewer infant deaths.

[0007] Therefore, a need exists for improved methods and systems for analyzing and / or processing medical images, including ultrasound images, for detecting anomalies and defects such as CHD. Summary of the Invention [Means for solving the problem]

[0008] Provided herein are systems and methods for analyzing medical imaging using a spatiotemporal neural network to detect cardiovascular anomalies and / or conditions, such as CHD. The systems and methods may include using a spatiotemporal convolutional neural network (CNN) to process medical device imaging, such as single-frame images and / or video clips generated by an ultrasound system. Optical flow data may be generated based on the images and / or video clips and may indicate pixel movement in the images and / or video clips. The images and / or video clips may be processed by a spatial CNN, and the optical flow data may be processed using a temporal CNN. The spatial output from the spatial CNN and the temporal output from the temporal CNN may be fused to generate a combined spatiotemporal output that may indicate the possible presence of one or more CHDs or other cardiovascular anomalies in a patient (e.g., a fetus in a pregnant patient).

[0009] Provided herein is a method for determining the presence of one or more congenital heart defects (CHDs) in a patient, the method comprising: determining, by a server, first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the first image data, the optical flow data indicating movement of pixels in the series of image frames; processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks trained to process the image data; and processing the optical flow data using a temporal model, the temporal model processing the optical flow data. the one or more second convolutional neural networks trained to process the image data using the spatial model; generating a spatial output based on the image data using the spatial model, the spatial output indicating a first likelihood of the presence of one or more CHDs in the patient; generating a temporal output based on the plurality of optical flow data using the temporal model, the temporal output indicating a second likelihood of the presence of one or more CHDs in the patient; determining a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of one or more CHDs in the patient; and causing the first device to display a user interface corresponding to the fused output.

[0010] A third possibility for the presence of one or more CHDs in the patient may include one or more of the following: the possibility of the presence of an atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, abnormal heart size, ventricular septal defect, abnormal atrioventricular junction, abnormal area posterior to the left atrium, abnormal left ventricular junction, abnormal aortic junction, abnormal right ventricular junction, abnormal pulmonary artery junction, arterial size discrepancy, right aortic arch anomaly, abnormal pulmonary artery size, abnormal transverse aortic arch size, or abnormal superior vena cava size.

[0011] The method may further include comparing the fused output to a threshold, determining that the fused output satisfies the threshold, and determining the presence of one or more CHDs in the patient based on the fused output satisfying the threshold. The method may further include determining a request from the first device to generate a report corresponding to the fused output and causing the first device to generate the report corresponding to the fused output. The method may further include training the spatial model and the temporal model using a plurality of second image data different from the first image data. The method further includes removing at least a portion of the first image data from each of the image frames in the series of image frames. The method further includes receiving the first image data from an imaging system. The imaging system further includes an ultrasound or echocardiography device. The image data further includes a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device. The method further includes sampling the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model.

[0012]

[0009] Provided herein is a system for determining the presence of one or more congenital heart defects (CHDs) in a patient, the system including: a memory configured to store computer-executable instructions; and at least one computer processor configured to execute the computer-executable instructions to: access the memory; determine first image data representing a portion of the patient's cardiovascular system, the first image data including a series of image frames; determine optical flow data based on the image data, the optical flow data indicating movement of pixels in the series of image frames; generate a spatial output by processing the image data using a spatial model, the spatial model including one or more first convolutional neural networks, the spatial output indicating a first likelihood of the presence of the one or more CHDs in the patient; generate a temporal output by processing the optical flow data using a temporal model, the temporal model including one or more second convolutional neural networks, the temporal output indicating a second likelihood of the presence of the one or more CHDs in the patient; determine a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of the one or more CHDs in the patient; and cause a first device to display a user interface corresponding to the fused output.

[0013] A third possibility for the presence of one or more CHDs in the patient may include one or more of the following: the possibility of the presence of an atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, abnormal heart size, ventricular septal defect, abnormal atrioventricular junction, abnormal area posterior to the left atrium, abnormal left ventricular junction, abnormal aortic junction, abnormal right ventricular junction, abnormal pulmonary artery junction, arterial size discrepancy, right aortic arch anomaly, abnormal pulmonary artery size, abnormal transverse aortic arch size, or abnormal superior vena cava size.

[0014] The computer processor may be configured to execute the computer-executable instructions to compare the fused output to a threshold, determine that the fused output satisfies the threshold, and determine the presence of one or more CHDs in the patient based on the fused output satisfying the threshold. The computer processor may be further configured to execute the computer-executable instructions to determine a request from the first device to generate a report corresponding to the fused output and cause the first device to generate a report corresponding to the fused output. The computer processor may be further configured to execute the computer-executable instructions to train the spatial and temporal models using a plurality of second image data different from the first image data. The computer processor may be further configured to execute the computer-executable instructions to remove at least a portion of the first image data from each image frame in the series of image frames.

[0015] The computer processor may be further configured to execute the computer-executable instructions to receive first image data from an imaging system. The imaging system may include an ultrasound or echocardiography device. The image data includes a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device. The computer processor may be further configured to execute the computer-executable instructions to sample the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model. The patient may be a fetus during pregnancy. The spatial output includes a matrix of values ​​indicating view orientations for individual image frames in the series of image frames. The computer processor may be further configured to determine the fused output using late fusion. The late fusion is one of a sum fusion approach, a maximal fusion approach, a concatenation fusion approach, a conventional fusion approach, or a bilinear fusion approach.

[0016] Provided herein is a method for determining the presence of one or more CHDs and / or other cardiovascular anomalies in a patient, the method including the steps of: determining, by a server, first image data representing a portion of the patient's cardiovascular system, the first image data including a series of image frames; determining optical flow data based on the first image data, the optical flow data indicating pixel movement in the series of image frames; processing the image data using a spatial model, the spatial model including one or more first convolutional neural networks trained to process the image data; and processing the optical flow data using a temporal model, the temporal model including one or more second convolutional neural networks trained to process the optical flow data. The method may include using the temporal model to generate a spatial output based on the image data, the spatial output indicating a first possibility of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient; using the temporal model to generate a temporal output based on the plurality of optical flow data, the temporal output indicating a second possibility of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient; determining a fused output based on the spatial output and the temporal output, the fused output indicating a third possibility of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient; and causing the first device to display a user interface corresponding to the fused output.

[0017] The third possibility of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient may include one or more of the following: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch syndrome, ventricular disproportion, abnormal heart size, ventricular septal defect, abnormal atrioventricular junction, abnormal area posterior to the left atrium, abnormal left ventricular junction, abnormal aortic junction, abnormal right ventricular junction, abnormal pulmonary artery junction, arterial size discrepancy, right aortic arch anomaly, abnormal pulmonary artery size, abnormal transverse aortic arch size, or abnormal superior vena cava size. The method may further include comparing the fused output to a threshold, determining that the fused output satisfies the threshold, and determining the patient's risk or presence of one or more CHDs and / or other cardiovascular anomalies based on the fused output satisfying the threshold. The method may further include determining a request from the first device to generate a report corresponding to the fused output and causing the first device to generate the report corresponding to the fused output. The method may further include training the spatial model and the temporal model using a plurality of second image data different from the first image data. The method may further include removing at least a portion of the first image data from each of the image frames in the series of image frames.

[0018] The method may further include receiving first image data from an imaging system, which may include an ultrasound or echocardiography device. The image data may include a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device. It should be understood that multiple series of image frames may be processed using the imaging system. The method may include sampling the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model. Image data from adjacent and other image sequences and / or image frames may be used to process and / or generate output for an image sequence or image frame. Such other image sequences and / or image frames may provide context for the image sequence and / or frame for which the output is generated. One or more of the spatial outputs may further indicate one or more of keypoint data or contour data. One or more of the temporal outputs may further indicate one or more of keypoint data or contour data. The method may further include determining one or more of the keypoint data or contour data based on the spatial output and the time output and / or causing the first device to further display one or more of the keypoint data or contour data.

[0019] Provided herein is a system for determining the presence of one or more CHDs and / or other cardiovascular anomalies in a patient. The system may include a memory designed to store computer-executable instructions; and at least one computer processor designed to execute the computer-executable instructions to access the memory and execute the computer-executable instructions to: determine first image data representing a portion of the patient's cardiovascular system, the first image data including a series of image frames; determine optical flow data based on the image data, the optical flow data indicating movement of pixels in the series of image frames; process the image data using a spatial model to generate a spatial output, the spatial model including one or more first convolutional neural networks, the spatial output indicating a first likelihood of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient; process the optical flow data using a temporal model to generate a temporal output, the temporal model including one or more second convolutional neural networks, the temporal output indicating a second likelihood of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient; determine a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient; and cause the first device to display a user interface corresponding to the fused output.

[0020] The third possibility of the presence of one or more CHDs and / or other cardiovascular anomalies in the patient may include one or more of the following: an atrial septal defect, an atrioventricular septal defect, coarctation of the aorta, a double-outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, or an interrupted aortic arch. The computer processor may be further configured to execute the computer-executable instructions to compare the fused output to a threshold, determine that the fused output satisfies the threshold, and determine the presence of one or more CHDs and / or other cardiovascular anomalies in the patient based on the fused output satisfying the threshold. The computer processor may be further configured to execute the computer-executable instructions to determine a request from the first device to generate a report corresponding to the fused output and to cause the first device to generate a report corresponding to the fused output. The computer processor may be further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data. The computer processor may further be designed to execute the computer-executable instructions to remove at least a portion of the first image data from each of the image frames in the sequence of image frames.

[0021] The computer processor may be further designed to execute computer-executable instructions to receive first image data from an imaging system, which may include an ultrasound or echocardiography device. The image data may include a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device. The computer processor may be further designed to execute computer-executable instructions to sample the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model. One or more of the spatial outputs may further indicate one or more of the keypoint data or contour data. One or more of the time outputs may further indicate one or more of the keypoint data or contour data. The system may be further designed to execute computer-executable instructions to determine one or more of the keypoint data or contour data based on the spatial output and the time output and / or cause the first device to further display one or more of the keypoint data or contour data.

[0022] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the following drawings and detailed description. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an image processing system for determining the presence of a cardiovascular anomaly, in accordance with some aspects of the present invention.

[0024] [Figure 2A]2A-2B illustrate a schematic diagram of data flow between the imaging system, analyst device, and backend of the image processing system. [Figure 2B] 2A-2B illustrate a schematic diagram of data flow between the imaging system, analyst device, and backend of the image processing system.

[0025] [Figure 3A] 3A-3C illustrate schematic diagrams of the spatial streams, the temporal streams, and the fused spatiotemporal output. [Figure 3B] 3A-3C illustrate schematic diagrams of the spatial streams, the temporal streams, and the fused spatiotemporal output. [Figure 3C] 3A-3C illustrate schematic diagrams of the spatial streams, the temporal streams, and the fused spatiotemporal output.

[0026] [Figure 4] Figure 4 illustrates the process flow for the spatial CNN, the temporal CNN, and the fused spatiotemporal output.

[0027] [Figure 5A] 5A-5B illustrate a process flow for determining whether CHD and / or other cardiovascular anomalies are present based on the likelihood of the presence of CHD and / or other cardiovascular anomalies. [Figure 5B] 5A-5B illustrate a process flow for determining whether CHD and / or other cardiovascular anomalies are present based on the likelihood of the presence of CHD and / or other cardiovascular anomalies.

[0028] [Figure 6] FIG. 6 is a schematic block diagram of a computing device in accordance with one or more exemplary embodiments of the present disclosure.

[0029] The foregoing and other features of the present invention will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings, in which: The present disclosure will be described with additional specificity and detail through the use of the accompanying drawings, with the understanding that these drawings depict only some embodiments in accordance with the present disclosure and are therefore not to be considered limiting of its scope. DETAILED DESCRIPTION OF THE INVENTION

[0030] Detailed Description of the Invention The present invention is directed to an image processing system that uses artificial intelligence and machine learning to determine the probable presence or absence, or uncertainty of such presence or absence, of CHD and / or other cardiovascular anomalies in a patient, such as a fetus during pregnancy. For example, medical imaging, such as images (e.g., still frames and / or video clips), may be generated using an ultrasound system (e.g., an echocardiography system) and processed by a spatiotemporal neural network to generate the probable presence or absence of one or more CHD and / or other cardiovascular anomalies. The images may also be processed by a spatiotemporal neural network to determine detection of key points (e.g., apex, etc.) corresponding to cardiovascular anatomical structures (such data is referred to as key point data) and / or contours and / or segmentations of elements and / or features of the cardiovascular anatomical structures (e.g., contours of one or more ventricles, one or more atria, etc.) (such data is referred to as contour data), and this information may be used to calculate measurements (e.g., lengths, areas, ratios) and / or for detection of fetal features and / or anatomical structures (e.g., detection of parts of the heart such as the heart, lungs, atria, septum, ventricles, and the like).

[0031] Medical imaging may include a continuous series of still-frame images. The still-frame images may be preprocessed to remove excess or unwanted portions. For example, during preprocessing, spatial, temporal, and / or spatiotemporal filters may be used to remove noise. The still-frame images may be sampled, segmented, or analyzed so that only a certain number of frames may be selected (e.g., every two, three, or four frames). Optical flow data may be generated from the image data and may represent the movement of pixels in the image data. The optical flow data and the image data (e.g., a single frame of image data) may be processed in parallel using two neural networks, one for the image and the other for the optical flow data. The architectures of these two networks may be fused at one or more levels (e.g., late fusion and / or final feature map).

[0032] The two parallel neural networks may be two CNNs. Specifically, the first CNN may be a spatial network trained to process image data (e.g., a single frame of RGB data). The second CNN may be a temporal neural network trained to process optical flow data. Alternatively, or in addition, one or more of the neural networks may be a deep neural network (DNN) and / or any other suitable neural network. Each neural network may output a probability of the presence, absence, and / or uncertainty of CHD and / or other cardiovascular anomalies, and / or the output may indicate key points and / or contours of fetal anatomical structures. The architectures of the two neural networks may be fused to produce superior results compared to either network individually. For example, outputs determined using both networks may be merged via late fusion to produce a single spatiotemporal output indicating the presence, absence, and / or uncertainty of CHD and / or other anomalies in the image data (e.g., based on the visual appearance of the anatomical structures or the absence or absence of certain anatomical structures). It should be understood that one or more CNNs may optionally be attention-based neural networks. It should further be understood that the spatial network and the temporal network may be a single network or may be two networks. For example, the imaging system may include a dual-stream network having a two-stream architecture with a spatial CNN and a temporal CNN, or may combine CNNs. Although the imaging processing system described herein is described as a CNN, it should be understood that such imaging processing systems are not limited to CNNs, and other embodiments of the imaging processing system may alternatively use any combination of neural networks, such as one or more of a CNN, a residual neural network, an attention neural network, a region-based convolutional neural network (RCNN), and / or any other suitable neural network.

[0033] 1, an image processing system 100 is illustrated. Image processing system 100 may be designed to receive medical images and process the medical images using artificial intelligence and machine learning to determine the presence, absence, or probable uncertainty of one or more CHDs and / or other cardiovascular anomalies, and / or image processing system 100 may be used to determine output indicative of key points and / or contours of fetal anatomy. For example, image processing system 100 may receive image data indicative of fetal anatomy and process the image data using a spatiotemporal CNN to automatically determine the presence and / or absence of one or more CHDs and / or other cardiovascular anomalies.

[0034] Image processing system 100 may include one or more imaging systems 102, each of which may be in communication with server 104. For example, imaging systems 102 may be any well-known medical imaging system that generates image data (e.g., still frames and / or video clips comprising RGB pixel information), such as an ultrasound system, an echocardiography system, an X-ray system, a computed tomography (CT) system, a magnetic resonance imaging (MRI) system, a positron emission tomography (PET) system, and the like.

[0035] The imaging system 102 may be any suitable ultrasound scanning system for performing fetal ultrasound examinations (e.g., second-trimester fetal anatomy ultrasound examinations at 18-24 weeks gestation, first-trimester examinations, third-trimester fetal examinations, fetal echocardiography, or others), however, the software / programming of the present invention described herein is stored and executed on the imaging system 102, the server 104, and / or the data store 112. In one example, the imaging system may be a Samsung WS80A ultrasound system or any other suitable ultrasound scanning system. The image processing system 100 may optionally be designed to be independent of the manufacturer, model, and / or type of the imaging system 102. For example, the image processing system 100 may implement and / or incorporate the system and / or method for independent analysis provided in U.S. Pat. No. 11,861,838 (the entire contents of which are incorporated herein by reference). Although an ultrasound system will be described throughout, it should be understood that the same or similar approaches may be used in conjunction with any other suitable medical imaging system (e.g., a CT system, an MRI system, a PET system, or any other imaging and / or diagnostic system).

[0036] 1, the imaging system 102 may be an ultrasound imaging system including an ultrasound sensor 108 and an ultrasound device 106. The ultrasound sensor 108 may include a piezoelectric sensor device or any known ultrasound sensing device. The ultrasound device 106 may be any known computing device including a processor and a display and may have a wired or wireless connection to the ultrasound sensor 108.

[0037] The ultrasound sensor 108 may be used by a healthcare provider to acquire image data of the anatomy of a patient (e.g., patient 110). The ultrasound sensor 108 may generate a two-dimensional image corresponding to the orientation of the ultrasound sensor 108 relative to the patient 110. The image data generated by the ultrasound sensor 108 may be communicated to the ultrasound device 106. The ultrasound device 106 may transmit the image data to the remote server 104 via any known wired or wireless system (e.g., Wi-Fi, cellular network, Bluetooth, Bluetooth Low Energy (BLE), short-range wireless communication protocols, etc.). Additionally or alternatively, the image data may be received and / or retrieved from one or more picture archiving and communication systems (PACS). For example, the PACS system may use the Digital Imaging and Communications in Medicine (DICOM) format. Any results from the system (e.g., the spatiotemporal output 232 and / or the analyzed output 236) may be shared with the PACS.

[0038] The remote server 104 may be any computing device with one or more processors capable of performing the operations described herein. In the example illustrated in FIG. 1, the remote server 104 may be one or more servers, desktop or laptop computers, or the like, and / or may be located at a different location than the imaging system 102. The remote server 104 may run one or more local applications to facilitate communication between the imaging system 106, the data store 112, and / or the analyst device 116.

[0039] The data store 112 may be one or more drives having memory dedicated to storing digital information, such as information unique to a patient, professional, facility, and / or device. For example, the data store 112 may include, without limitation, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read-only memory (ROM)), flash memory, or any combination thereof. The data store 112 may be incorporated into the server 104 or may be separate and distinct from the server 104. In one example, the data store 112 may be a picture archiving and communication system (PACS).

[0040] The remote server 104 may communicate with the data store 112 and / or the analyst device 116 via any known wired or wireless system (e.g., Wi-Fi, cellular networks, Bluetooth, Bluetooth Low Energy (BLE), short-range wireless communication protocols, etc.). The data store 112 may receive and store image data (e.g., image data 118) received from the remote server 104. For example, the imaging system 102 may generate image data (e.g., ultrasound image data) and transmit such image data to the remote server 104, which may transmit the image data to the data store 112 for storage. It should be understood that the data store 112 may be optional and / or more than one imaging system 102, remote server 104, data store 112, and / or analyst device 116 may be used.

[0041] The analyst device 116 may be any computing device having a processor and a display, and capable of communicating with at least the remote server 104 and performing the operations described herein. The analyst device 116 may be any well-known computing device, such as a desktop, laptop, smartphone, tablet, wearable, or the like. The analyst device 116 may launch one or more local applications to facilitate communication between the analyst device 116 and the remote server 104 and / or any other computing device or server described herein.

[0042] The remote server 104 may receive image data (e.g., RGB image data from an ultrasound system) from the data store 112 and / or the imaging system 106 and may process the image data to determine the presence or absence of CHDs and / or any other cardiovascular anomalies in the patient (e.g., in the fetus of a pregnant person) and / or key points and / or contours of the fetal anatomy. For example, the remote server 104 may process one or more trained models, such as CNNs, that are trained to detect one or more CHDs and / or anomalies.

[0043] The remote server 104 may use two parallel spatiotemporal convolutional neural networks (CNNs) and fuse their outputs to produce a superior output with improved accuracy over the individual CNNs. The first CNN may be a spatial CNN and the second may be a temporal CNN. Image data, which may be ultrasound image frames, may be processed by the spatial CNN.

[0044] Optical flow data may be generated based on the images and / or video clips and may indicate the movement of pixels in the images and / or video clips. The optical flow data may be processed using a temporal CNN. The spatial output from the spatial CNN and the temporal output from the temporal CNN may be fused to generate a combined spatiotemporal output that may indicate the probable presence or absence of one or more CHDs and / or other cardiovascular anomalies in a patient (e.g., a fetus in a pregnant patient) and / or key points and / or contours of the fetal anatomy. The remote server 104 may cause the analyst device 116 to display information about the probable presence of one or more CHDs and / or other cardiovascular anomalies. For example, the analyst device may display a patient ID number and a probability percentage for one or more CHDs and / or other cardiovascular anomalies.

[0045] In one example, system 100 may be the same as or similar to the systems and methods for computer-aided diagnostic support for use in fetal ultrasound examinations provided in U.S. Patent No. 11,869,188, issued January 9, 2024, and U.S. Patent Application No. 18 / 406,446, filed January 8, 2024 (the entire contents of each of which are incorporated herein by reference).

[0046] 2A-2B, schematic diagrams of data flow between an imaging system, an analyst device, and a backend of an image processing system are depicted. As shown in FIG. 2A, an imaging system 202, which may be the same as or similar to the imaging system 102 of FIG. 1, may include an image generator 204 that may generate image data 206. The image data 206 may include still frames and / or video clips and may include RGB and / or grayscale pixel information. For example, the image data 206 may include a two-dimensional representation of an ultrasound scan of a patient's anatomy. Additionally or alternatively, the image data 206 may include Doppler image information (e.g., color Doppler, power Doppler, spectral Doppler, duplex Doppler, and the like). It should be understood that various types of image data 206 may be processed simultaneously by the imaging system 202. In one example, Doppler image data may be generated simultaneously with ultrasound image data.

[0047] The imaging system 202 may send image data 206 to a backend 208, which may be the same as or similar to the server 104 of FIG. 2A. The image data 206 may be processed by a preprocessor 210. The preprocessor 210 may focus on unnecessary areas of the image data 206 and crop, resize, and / or otherwise remove it to generate preprocessed image data 212. For example, black backgrounds and text in still frames generated by the imaging system 202 may be removed. The preprocessor may additionally or alternatively generate a series of consecutive still frame images from the video clip.

[0048] The pre-processed image data 212 may optionally be sent to a sampling generator 214, which may sample, analyze, and / or segment the pre-processed image data 212 to generate sampled image data 216. For example, the sampling generator 214 may determine the interval of frames to be sampled (e.g., intervals of two, three, four, etc.). In this manner, only sampled frames of the image data 212 may be processed by the neural network in the backend 208. Sampling the image data 212 may allow the network to process image frames over a longer time period of the image data 212.

[0049] The preprocessed image data 212, the image data 206, and / or the sampled image data 216 may be processed by an optical flow generator 218 to generate optical flow data 220 corresponding to the preprocessed image data 212, the image data 206, and / or the sampled image data 216. The optical flow data 220 may allow the network to better account for movement of the image data over time.

[0050] To generate optical flow data 220, successive image frames of the image data 212, image data 206, and / or sampled images 216 may be input to the optical flow generator 218. From the successive image frames, horizontal and vertical optical flow data may be calculated for each adjacent frame, resulting in an output size of H×W×2L, where H and W are the height and width of the image frame, and L is the length (e.g., the time between frames). The optical flow generator 218 may thereby encode the movement of individual pixels across the frames of the image data 212, image data 206, and / or sampled images 216, capturing the movement depicted in the images across time.

[0051] The sampled image data 216, the preprocessed image data 212, and / or the image data 206 may then be applied to a spatial model 222 to generate a spatial output 226, which may be a spatial CNN, such as a spatial CNN trained for image processing. The spatial model 222 may be trained to analyze the image data (e.g., RGB data) and determine the presence of one or more CHDs and / or other cardiovascular anomalies within each frame. It should be understood that the spatial model 222 may optionally take as input a temporal output 228 from the temporal model 224.

[0052] The spatial output 226 may include a vector or matrix including scores or values ​​for one or more frames corresponding to the likelihood of CHD and / or other cardiovascular anomalies. The spatial output 226 may optionally further include scores or values ​​indicating the likelihood of one or more views or orientations of the sensor device to which the image data corresponds. For example, various views may include anatomical standard views (e.g., four-ventricle view, left ventricular outflow tract, right ventricular outflow tract, etc.). Such views may have standard orientations for individual anatomical structures (e.g., top view, bottom view, left view, right view, superior, inferior, etc.). Each view and likelihood value may be depicted in a vector or matrix. In one example, the spatial output 226 may include low likelihoods for bottom, right, and left views, but a high likelihood for a top-down view. This would indicate a high likelihood that the view is from the top.

[0053] Similarly, the optical flow data 220 may be applied to a temporal model 224, which may be a temporal CNN, such as a temporal CNN trained for image processing and / or trained to process optical flow data, to generate a temporal output 228. For example, the temporal model 224 may generate, for each optical flow dataset, a temporal output 228 that may indicate a score or value indicative of the likelihood of the presence of one or more CHDs and / or other cardiovascular anomalies. The temporal output 228 may optionally further include a score or value indicative of the likelihood of one or more views or orientations of the sensor device to which the image data corresponds. It should be understood that the temporal model 224 may optionally take as an input the spatial output 226 from the spatial model 222.

[0054] Both spatial output 226 and temporal output 228 may be input into fuser 230, which fuses spatial model 222 and temporal model 224 to generate spatiotemporal output 232, which may be similar to spatial output 226 and temporal output 228, but with improved accuracy. For example, fuser 230 may combine the architectures of spatial model 222 and temporal model 224 at several levels (e.g., in the final feature map). Alternatively, or in addition, a weighted average of spatial output 226 and temporal output 228 may be determined to generate spatiotemporal output 232.

[0055] It should be understood that various well-known fusion approaches can be used, such as sum, max, concatenation, convolution, and bilinear. While late fusion can be used, it should be further understood that other techniques can be used, such as early fusion (changing the first convolutional layer of each stream to a 3D convolution) or late fusion (changing all convolutional layers in each stream to a 3D convolution with a smaller time span compared to early fusion).

[0056] The spatiotemporal output 232 may be processed by an analyzer 234, which may process the spatiotemporal output 232 and generate an analyzed output 236, which may indicate the presence or absence, or uncertainty of the presence or absence, of one or more CHDs and / or cardiovascular anomalies in the image data 206, and / or may indicate key points and / or contours of fetal anatomical structures. For example, the analyzer 234 may calculate a weighted average and / or filter a portion of the spatiotemporal output 232 based on the spatiotemporal output 232. In one example, the analyzed output 236 and / or the spatiotemporal output 232 may indicate a risk of the possible presence or absence of one or more morphological abnormalities or defects, and / or may indicate the presence or absence of one or more pathological findings.For example, the analyzed output 236 and / or the spatiotemporal output 232 may include any of the following: riding arteries (e.g., an artery exiting the left ventricle positioned across a ventricular septal defect), a septal defect at the heart crux (e.g., a septal defect positioned at the heart crux in either the primum atrial septum or the inlet ventricular septum), parallel aortas, an enlarged cardiothoracic ratio (e.g., a ratio of the area of ​​the heart to the thorax measured at end diastole greater than 0.33), right ventricular to left ventricular size mismatch (e.g., a large area at end diastole greater than 0.33), atrial fibrillation ... These include: a ratio of right and left ventricular area greater than 1.4 or less than 0.5 at end diastole; a tricuspid-to-mitral annular size discrepancy (e.g., a ratio between the tricuspid and mitral valves at end diastole greater than 1.5 or less than 0.65 at end diastole); a pulmonary valve-to-aortic annular size discrepancy (e.g., a ratio between the pulmonary and aortic valves at end systole greater than 1.6 or less than 0.85 at end systole); an abnormal outflow tract relationship (e.g., absence of the typical anterior-posterior crossing pattern of the aorta and pulmonary artery); and cardiac axis deviation ( For example, a cardiac axis (the angle between the line bisecting the rib cage and the interventricular septum) below 25° or above 65°, atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion (e.g., one left or right ventricle larger than the other), abnormal heart size, ventricular septal defect, abnormal atrioventricular junction, increased or abnormal area posterior to the left atrium, abnormal left ventricle and / or aortic junction, abnormal right ventricle and / or pulmonary artery junction, aortic size discrepancy (e.g., an aorta larger or smaller than the pulmonary artery), right aortic arch abnormality, abnormal sizes of the pulmonary artery, transverse aortic arch, and / or superior vena cava, visible additional vessels, abnormal ventricular asymmetry, pulmonary artery and / or aortic valve stenosis, ventricular hypoplasia, and / or single ventricle, and / or any other morphological abnormality, defect, and / or pathology. Alternatively, or in addition, analyzed output 236 and / or spatiotemporal output 232 may indicate the presence of, or may be used to determine the presence or likely presence of, any other morphological abnormality, condition, and / or disorder.

[0057] Backend 208 may communicate information based on analyzed output 236 and / or spatiotemporal output 232 to analyst device 240, which may be the same as or similar to analyst device 116. Analyst device 240 may be different or the same as the devices in imaging system 202. Display module 238 may generate a user interface on analyst device 240 to generate and display a representation of analyzed output 244 and / or spatiotemporal output 232. The representation may be the same as or similar to the graphic user interface described and / or illustrated in U.S. Application No. 18 / 406,446, filed January 8, 2024, the entire contents of which are incorporated herein by reference. For example, the display may show a representation of image data (e.g., ultrasound images) with an overlay indicating locations of detected risk or likelihood of CHD and / or other cardiovascular anomalies. In one example, the overlay may be a box or any other visual indicator (e.g., an arrow).

[0058] The user input module 242 may receive user input 244 and may communicate the user input 244 to the backend 208. The user input 244 may be instructions from a user to generate a report or other information, such as instructions that the results produced by one or more of the spatial model 222, the temporal model 224, and / or the fuser 230 are inaccurate. For example, if the user input 244 indicates an inaccuracy, the user input 244 may be used to further train the spatial model 222, the temporal model 224, and / or the fuser 230.

[0059] If the user input 244 indicates a request for a report, the user input 244 may be communicated to a report generator 246, which may generate a report. For example, the report may include some or all of the analyzed output 236, the spatiotemporal output 232, the user input 244, and / or analyses, graphs, plots, tables related thereto. The report 248 may then be communicated to the analyst device 240 for display (e.g., by the display module 238) of the report 248, which may also be printed out by the analyst device 240.

[0060] Referring now to Figure 2B, a clinical workflow of the system illustrated in Figure 1 is illustrated. As shown in Figure 2B, a clinical complex 250 may communicate with a backend 260, which may be running on a server (e.g., server 104 of Figure 1). Clinical complex 250 may include an ultrasound module 252, as well as a picture archiving and communications (PACS) system 254, a digital imaging and communications in medicine (DICOM) viewer 256, and a DICOM router 258, which may be running on an imaging system (e.g., imaging system 102 of Figure 1).

[0061] The ultrasound module 252 may generate, receive, acquire, and / or store ultrasound images (e.g., image data such as motion video clips and image frames). The image data may be communicated directly from the ultrasound module 252 to the PACS system 254 and / or to the implementation module 262 of the back end 260. The PACS system 254 may securely store the image data received from the ultrasound module 252. The image data stored in the PACS system 254 may electronically tag the records based on user-selected input. Once the image data is stored and / or tagged in the PACS system 254, a DICOM router 258 may connect to the PACS system 254 and retrieve the image data, and may also connect to the back end 260, which may run on a server (e.g., the server 104 and / or the data store 112 of FIG. 1 ). For example, the DICOM router 258 may be connected to the implementation module 262 and send the image data to the implementation module 262.

[0062] In one embodiment, the DICOM router 258 may pseudonymize the files so that only pseudonymized files are sent to the backend 260. For example, all patient information may be removed except for certain required variables (e.g., gestational age), and a pseudonymous identifier may be added to the file for each study and / or record. Once the DICOM router 258 receives the output from the backend 260, it may perform re-identification by replacing the pseudonymous identifier with the patient information. The implementation module 262 may upload the image data to the storage device 264. For example, the storage device 264 may store encrypted and otherwise secured image data.

[0063] The implementation module 262 may retrieve certain image data from the storage device 264 and may communicate such image data to the analysis module 266. The analysis module 266 may process the image data using machine learning algorithms to identify the presence, absence, or uncertainty of the presence or absence of one or more CHDs and / or cardiovascular anomalies in the image data, and / or delineate key points and / or contours of fetal anatomy. For example, the analysis module 266 may invoke one or more modules or models described with respect to the back end 260 of FIG. 2A . For example, the analysis module 266 may invoke the spatial model 222, the temporal model 224, and / or the fuser 230.

[0064] The results and / or output of the analysis module 266 may be stored in the storage device 264. The results and / or output (e.g., spatiotemporal output 232 and / or analyzed output 236 of FIG. 2A ) and any reports (e.g., report 248) may be communicated back to the DICOM router 258 and stored in the PACS 254. Once stored in the PACS 254, a healthcare provider (e.g., a physician) may use the DICOM viewer 256 to access the results and / or output from the PACS 254 and view the results and / or output (e.g., using a healthcare provider device).

[0065] 3A-3C, a spatio-temporal neural network (e.g., CNN) is illustrated. Referring now to FIG. 3A, a spatio-temporal CNN system 300 is illustrated. The spatio-temporal CNN system 300 may be either a single CNN or an independent CNN that may have a two-stream architecture. The spatio-temporal CNN system 300 may be the same as or similar to the CNN system used by the backend 208 of FIG. 2A. As shown in FIG. 3A, the spatio-temporal CNN system 300 may include a spatial stream 306 and a temporal stream 308, which may be parallel streams that may be combined in fusion 310.

[0066] As shown in Figure 3A, image data 302 may be input into and processed by spatial stream 306, and optical flow data 304 may be input into and processed by temporal stream 308. Spatial stream 306 and temporal stream 308 may be different CNNs or may be streams in the same CNN. Image data 302 may be the same as or similar to image data 206, preprocessed image data 212, and / or sampled image data 216 of Figure 2A. Optical flow data 220 may be the same as or similar to optical flow data 220 of Figure 2A.

[0067] The spatial stream 306 may receive a single image frame of image data 302, and the temporal stream 306 may receive fixed-size chunks of optical flow data 304. For example, the single frame of image data 302 may include RGB pixel information, and / or the fixed-size chunks of optical flow data 304 may include fixed-size maps and / or plots of the optical flow data 304. The spatial stream 306 may process the image data 302 at the same time that the temporal stream 306 processes the optical flow data 304. The optical flow data processed by the temporal stream 308 may correspond to or be based on the image data processed by the spatial stream.

[0068] If the CNN system 300 includes multiple CNNs, the spatial stream 306 may include one or more spatial CNNs, such as a spatial CNN trained for image processing. A spatial CNN may include one or more neural networks (e.g., CNNs) trained to analyze image data (e.g., RGB pixel data) generally (e.g., not specific to medical imaging) and / or one or more neural networks trained to analyze image data in medical imaging (e.g., ultrasound images). For example, a spatial CNN may be trained to analyze ultrasound image data (e.g., RGB pixel data) and determine, within each frame, the likelihood of the presence or absence of one or more CHDs and / or other cardiovascular anomalies and / or the likelihood of a certain oriented view corresponding to the image data.

[0069] The temporal stream 308 may include one or more temporal CNNs, such as a temporal CNN, trained for image processing and / or trained to process optical flow data to generate a temporal output. For example, the temporal CNN may generate, for each optical flow dataset, a temporal output that may indicate the presence of one or more CHDs and / or other cardiovascular anomalies and / or the likelihood of a certain view or orientation corresponding to the optical flow data.

[0070] Fusion 310 may combine the architectures and / or outputs of the architectures of the spatial stream 306 and the temporal stream 308, resulting in a spatiotemporal output 312. The spatial stream 306 and the temporal stream 308 may be fused at one or more levels. As shown in FIG. 3 , late fusion may be used such that outputs from both CNNs and / or both streams are merged to generate a single fused output, such as a single spatiotemporal representation, that may indicate the likelihood of the presence or absence of one or more CHDs and / or other cardiovascular anomalies, or the uncertainty of such presence or absence, the likelihood of a certain view or orientation corresponding to the image data, and / or indicate key points and / or contours of fetal anatomy. In one example, the late fusion is one of a sum fusion approach, a maximal fusion approach, a concatenation fusion approach, a conventional fusion approach, or a bilinear fusion approach.

[0071] It should be understood that the 2D CNN illustrated in FIG. 3A can be extended to take as input not a single image, but instead multiple images (e.g., multiple frames) by stacking filters in the time dimension and dividing the weights. For example, filters may be stacked K times in the time dimension for K image frames, and the weights may be divided by K. While the two streams in FIG. 3A are illustrated as parallel streams, the time stream 308 may alternatively take the output of the spatial stream 306 as input to the time stream 308. It should further be understood that other representations may be determined and / or processed along with the spatial and temporal representations.

[0072] 3B-3C, the spatiotemporal neural network comprises a fusion neural network. For example, neural network systems (e.g., CNN systems 330 and 340) may include fusion CNNs 336 and 344, respectively, which may be similar to fusion 310 of FIG. 3A but may be separate neural networks. For example, fusion CNN 336 and / or fusion 334 may be CNNs trained to output such spatiotemporal representations based on outputs from spatial and temporal streams 308.

[0073] As shown in FIG. 3B , the spatial stream 306 may be a CNN 332, which may be one or more spatial CNNs trained for image processing and / or trained to analyze image data (e.g., RGB pixel data) and determine, within each frame, the likelihood of the presence of one or more CHDs and / or other cardiovascular anomalies and / or the likelihood of a view with an orientation corresponding to the image data, the likelihood of a view or orientation corresponding to the image data, and / or indicate key points and / or contours of fetal anatomical structures. Similarly, the temporal stream 308 may be a CNN 334, which may be one or more temporal CNNs trained for image processing and / or trained to process optical flow data to generate a temporal output. The outputs of the spatial stream 306 and the temporal stream 308 may be input into a fusion CNN 336, which may be one or more CNNs trained to output a spatiotemporal representation of the spatial and temporal outputs. As shown in FIG. 3B, each of the spatial stream 306 and the temporal stream 308 may be an independent CNN, which together with the fused CNN 336 may result in a total of three or more neural networks (e.g., three or more CNNs).

[0074] As shown in FIG. 3C , the spatial stream 306 and the temporal stream 308 may be included in a CNN 342, which may be one or more CNNs with the streams trained to analyze the image data (e.g., RGB pixel data) and determine, within each frame, the likelihood of the presence of one or more CHDs and / or other cardiovascular anomalies, the likelihood of a view with an orientation corresponding to the image data, and / or indicate key points and / or contours of fetal anatomy. The outputs of the spatial streams 306 and the temporal streams 308 may be input into a fusion CNN 344, which may be one or more CNNs that may be trained to output spatiotemporal representations of the spatial and temporal streams. As shown in FIG. 3C , the spatial streams 306 and the temporal streams 308 may each be included in a CNN 342, which, together with the fusion CNN 336, may total two or more neural networks (e.g., two or more CNNs).

[0075] 4, a process flow for generating a spatiotemporal output indicating a likelihood of CHD and / or other cardiovascular anomalies and / or indicating a likelihood of a certain view of an orientation of an imaging device (e.g., an ultrasound sensor) is depicted. Some or all of the blocks of the process flow of the present disclosure may be performed in a distributed manner across any number of devices (e.g., servers such as server 104 of FIG. 1, computing devices, imaging or sensor devices, or the like). Some or all of the operations of the process flow may be optional and may be performed in a different order.

[0076] In block 402, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine image data. For example, the image data may be the same as or similar to image data 202 of FIG. 2A and may include still-frame images and / or video clips. In optional block 404, computer-executable instructions stored on a memory of a device, such as a server, may be executed to preprocess the image data (e.g., focus, resize, and / or crop the image data), as described with respect to preprocessor 210 and preprocessed image data 212 of FIG. 2A. Additionally or alternatively, in block 404, spatial, temporal, and / or spatiotemporal filters may be used to remove noise.

[0077] In optional block 406, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine sample image data, as described with reference to sampling generator 214 and sampled image data 216 of FIG. 2A . In optional block 408, computer-executable instructions stored on a memory of a device, such as a server, may be executed to create and train a spatial model. For example, a CNN may be trained for image processing, detection, and / or recognition using a large set of images. For example, images from everyday life (e.g., cars, bicycles, apples, etc.) may generally be used to train a CNN for image recognition.

[0078] Additionally or alternatively, the CNN may be trained or fine-tuned using specific data sets corresponding to cardiovascular anatomies with or without CHDs and / or anomalies to ultimately recognize CHDs and / or cardiovascular anomalies in input image data. The network may also be trained to identify image views, angles, and / or orientations. For example, an echocardiographer may consistently generate standardized views, angles, or certain anatomical structures, and the CNN may be trained to recognize such views, angles, and / or orientations. It should be understood that the images and data used for training purposes may be different from, and / or from different patients than, the image data input into the CNN being trained.

[0079] In block 410, computer-executable instructions stored on a memory of a device, such as a server, may be executed to process image data using the trained spatial model. The processed image data may be pre-processed and / or sampled imaging data. In block 412, computer-executable instructions stored on a memory of a device, such as a server, may be executed to generate a spatial output using the image data and the trained spatial model. The spatial output may be the same as or similar to spatial output 226 of FIG. 2A.

[0080] In block 414, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine optical flow data, as described with respect to optical flow generator 218 and optical flow data 220 of FIG. 2A . It should be understood that blocks 414-420 may be executed simultaneously or nearly simultaneously with blocks 406-412. In optional block 416, computer-executable instructions stored on a memory of a device, such as a server, may be executed to train a temporal model using the image data, similar to optional block 408. It should be understood that optional block 416 and optional block 408 may occur simultaneously, and / or the spatial and temporal streams may be trained together, and thus optional block 408 and optional block 416 may be the same step. Additionally or alternatively, a temporal model may ultimately be trained using the optical flow data to recognize CHDs and / or cardiovascular anomalies in the optical flow data and / or identify image views, angles, and / or orientations in the optical flow data.

[0081] At block 418, computer-executable instructions stored on a memory of a device, such as a server, may be executed to process the optical flow data using the trained temporal model. At block 420, computer-executable instructions stored on a memory of a device, such as a server, may be executed to generate a temporal output using the optical flow data and the trained temporal model. The temporal output may be the same as or similar to the temporal output 228 of FIG. 2A. At block 422, fusion may be performed on the temporal output and the spatial output to determine a spatiotemporal output, as described with respect to fuser 230 and spatiotemporal output 232 of FIG. 2A.

[0082] 5A and 5B, a process flow for determining whether a CHD and / or cardiovascular anomaly is present in a data flow is depicted. Figures 5A-5B may begin immediately after block 422 of Figure 4. Some or all of the blocks of the process flow of the present disclosure may be performed in a distributed manner across any number of devices (e.g., servers such as server 104 of Figure 1, computing devices, imaging or sensor devices, or the like). Some or all of the operations of the process flow may be optional and may be performed in a different order.

[0083] 5A, at block 504, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the likelihood of one or more CHDs and / or cardiovascular anomalies for each of the sampled image data and / or each frame or video clip input into the spatiotemporal CNN. For example, each output may include a likelihood of a CHD and / or cardiovascular anomaly, and each output may correspond to a frame of image data and / or a video clip (e.g., multiple frames of image data).

[0084] In block 506, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine an average likelihood of a CHD and / or cardiovascular anomaly based on the likelihood of a CHD and / or cardiovascular anomaly for each sampled image data. For example, the likelihood of each CHD and / or cardiovascular anomaly in each output may be averaged. It should be understood that other types of aggregation, modeling, and / or filtering calculations other than average calculations may be used alternatively or in addition. For example, the system may determine the highest likelihood detected and use that value for further processing and / or analysis. Alternatively, or in addition, key points and / or contours of fetal anatomical structures may be determined.

[0085] In decision 508, computer-executable instructions stored on a memory of a device, such as a server, may be executed to compare the average likelihood of CHD and / or cardiovascular anomalies to a threshold. For example, the threshold may be 51%, 75%, 90%, 99%, or any other threshold. If the threshold is not met by any average value (e.g., each average value is below the threshold), in block 510, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that no CHD and / or cardiovascular anomalies are present.

[0086] Alternatively, if the threshold is met for one or more CHDs and / or cardiovascular anomalies, then in block 510, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that a CHD and / or cardiovascular defect corresponding to the mean value meeting the threshold is present. For example, the spatiotemporal output may be a vector or matrix including several likelihood values ​​between 0 and 1, each corresponding to a different CHD and / or cardiovascular anomaly, with values ​​higher than a threshold (e.g., 0.9) being determined to be present. It may be desirable to set different thresholds for different abnormalities, conditions, morphological abnormalities, pathological findings, and the like.

[0087] 5B, an alternative or additional process flow for determining whether CHD and / or cardiovascular anomalies are present in image data is illustrated. At block 520, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the likelihood of one or more views, angles, and / or orientations corresponding to each frame of image data and / or video clip. The view values ​​may correspond to the likelihood of one or more views, angles, and / or orientations corresponding to each frame of image data and / or video clip. For example, the view values ​​may be between 0 and 1.

[0088] In block 522, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that a view value satisfies a view threshold. For example, the view threshold may be any value, such as 51%, 75%, 90%, 99%, etc. In one embodiment, if the view value is greater than 0.9, it may be determined that there is a high likelihood or confidence that the associated image data corresponds to a view.

[0089] At block 524, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the likelihood of the presence of CHD and / or cardiovascular anomalies for outputs having view values ​​that satisfy a threshold. Alternatively, or in addition, key points and / or contours of fetal anatomical structures may be determined. At decision 526, computer-executable instructions stored on a memory of a device, such as a server, may be executed to compare each likelihood of CHD and / or cardiovascular anomalies corresponding to outputs with a satisfied view threshold to a defect threshold. For example, the defect threshold may be 51%, 75%, 90%, 99%, or any other threshold. If the threshold is not satisfied by any average values ​​(e.g., all average values ​​are below the threshold), at block 528, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that no CHD and / or cardiovascular anomalies are present.

[0090] If the deficit threshold is not met by any value (e.g., all values ​​are below the deficit threshold), then in block 528, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that a CHD and / or cardiovascular anomaly is not present. Alternatively, if the deficit threshold is met for one or more CHDs and / or cardiovascular anomalies, then in block 530, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that a CHD and / or cardiovascular anomaly corresponding to a value above the deficit threshold is present.

[0091] 6, a schematic block diagram of server 600 is shown. Server 600 may be the same as or similar to server 104 of FIG. 1 or one or more of the servers of FIGS. 1-5B. It should be understood that the imaging system, analyst device, and / or data store may additionally or alternatively include one or more of the components illustrated in FIG. 6, and server 600 may perform one or more of the operations of server 600 described herein, either alone or in conjunction with any of the foregoing.

[0092] Server 600 may be designed to communicate with one or more servers, imaging systems, analyst devices, data stores, other systems, or the like. Server 600 may be designed to communicate over one or more networks. Such networks may include, but are not limited to, any one or more different types of communications networks, such as, for example, a cable network, a public network (e.g., the Internet), a private network (e.g., a frame relay network), a wireless network, a cellular network, a telephone network (e.g., the public switched telephone network), or any other suitable private or public packet-switched or circuit-switched network.

[0093] In the illustrative configuration, server 600 may include one or more processors 602, one or more memory devices 604 (also referred to herein as memory 604), one or more input / output (I / O) interfaces 606, one or more network interfaces 608, one or more transceivers 610, one or more antennas 634, and data storage 620. Server 600 may further include one or more buses 618 that operatively couple the various components of server 600.

[0094] The bus 618 may include at least one of a system bus, a memory bus, an address bus, or a message bus and may enable exchange of information (e.g., data (including computer-executable code), signaling, etc.) between various components of the server 600. The bus 618 may include, but is not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, etc. The bus 618 may be associated with any suitable bus architecture.

[0095] Memory 604 may include volatile memory (memory that maintains its state when powered), such as random access memory (RAM), and / or non-volatile memory (memory that maintains its state even when powered off), such as read-only memory (ROM), flash memory, ferroelectric RAM (FRAM®), etc. Persistent data storage, as that term is used herein, may include non-volatile memory. In various implementations, memory 604 may include multiple different types of memory, such as various types of static random access memory (SRAM), various types of dynamic random access memory (DRAM), various types of non-alterable ROM, and / or writable variants of ROM, such as electrically erasable programmable read-only memory (EEPROM), flash memory, etc.

[0096] Data storage device 620 may include removable and / or non-removable storage devices, including, but not limited to, magnetic storage devices, optical disk storage devices, and / or tape storage devices. Data storage device 620 may provide non-volatile storage of computer-executable instructions and other data. Memory 604 and data storage device 620, whether removable and / or non-removable, are examples of computer-readable storage media (CRSM), as that term is used herein. Data storage device 620 may store computer-executable code, instructions, or the like, that may be loadable into memory 604 and executable by processor 602 to cause processor 602 to perform or initiate various operations. Data storage device 620 may additionally store data that may be copied to memory 604 for use by processor 602 during execution of the computer-executable instructions. Output data generated as a result of execution of computer-executable instructions by processor 602 may also be initially stored in memory 604 and eventually copied to data storage device 620 for non-volatile storage.

[0097] The data storage device 620 may store one or more operating systems (O / S) 622, one or more optional database management systems (DBMS) 624, and one or more program modules, applications, engines, computer-executable code, scripts, or the like, such as, for example, one or more implementation modules 626, image processing modules 627, communications modules 628, optical flow modules 629, and / or spatio-temporal CNN modules. Some or all of these modules may be sub-modules. Any of the components depicted as stored in the data storage device 620 may include any combination of software, firmware, and / or hardware. The software and / or firmware may include computer-executable code, instructions, or the like that may be loaded into memory 604 for execution by one or more of the processors 602. Any of the components depicted as stored in the data storage device 620 may support functionality described with reference to correspondingly named components above in this disclosure.

[0098] With reference to other illustrative components depicted herein as stored within data storage device 620, O / S 622 may be loaded into memory 604 from data storage device 620 and may provide an interface between other application software executing on server 600 and the hardware resources of server 600. More specifically, O / S 622 may include a set of computer-executable instructions for managing the hardware resources of server 600 and providing common services to other application programs (e.g., managing memory allocation among various application programs). In an exemplary embodiment, O / S 622 may control the execution of other program modules for content rendering. O / S 622 may include any operating system now known or that may be developed in the future, including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

[0099] An optional DBMS 624 may be loaded into memory 604 and may support functionality for accessing, retrieving, storing, and / or manipulating data stored in memory 604 and / or data stored in data storage device 620. DBMS 624 may use any of a variety of database models (e.g., relational model, object model, etc.) and may support any of a variety of query languages. DBMS 624 may access data represented in one or more data schemas and stored in any suitable data repository, including, without limitation, a database (e.g., relational, object-oriented, etc.), a file system, a flat file, a distributed data store in which data is stored on more than one node of a computer network, a peer-to-peer network data store, or the like.

[0100] An optional input / output (I / O) interface 606 may facilitate the receipt of input information by server 600 from one or more I / O devices, and the output of information from server 600 to one or more I / O devices. The I / O devices may include any of a variety of components, such as a display or display screen having a touch surface or touch screen, an audio output device for producing sound, such as a speaker, an audio capture device, such as a microphone, an image and / or video capture device, such as a camera, etc. Any of these components may be integrated into server 600 or may be separate.

[0101] Server 600 may further include one or more network interfaces 608 through which server 600 may communicate with any of a variety of other systems, platforms, networks, devices, etc. Network interface 608 may, for example, enable communication with one or more wireless routers, one or more host servers, one or more web servers, and the like, over one or more of the networks.

[0102] The antenna 634 may include any suitable type of antenna, depending, for example, on the communication protocol used to transmit or receive signals via the antenna 634. Non-limiting examples of suitable antennas may include a directional antenna, an omnidirectional antenna, a dipole antenna, a folded dipole antenna, a patch antenna, a multiple-input multiple-output (MIMO) antenna, or the like. The antenna 634 may be communicatively coupled to one or more transceivers 612 or wireless components to or from which signals may be transmitted or received. The antenna 634 may include, without limitation, a cellular antenna for transmitting or receiving signals to / from a cellular network infrastructure, an antenna for transmitting or receiving Wi-Fi signals to / from an access point (AP), a Global Navigation Satellite System (GNSS) antenna for receiving GNSS signals from GNSS satellites, a Bluetooth® antenna for transmitting or receiving Bluetooth® signals, including BLE signals, a Near Field Communication (NFC) antenna for transmitting or receiving NFC signals, a 900 MHz antenna, etc.

[0103] The transceiver 612 may include any suitable radio components for cooperating with the antennas 634 to transmit or receive radio frequency (RF) signals within a bandwidth and / or channel corresponding to a communication protocol utilized by the server 600 to communicate with other devices. The transceiver 612 may potentially include hardware, software, and / or firmware for cooperating with any of the antennas 634 to modulate, transmit, or receive communication signals according to any of the communication protocols discussed above, including, but not limited to, one or more Wi-Fi and / or Wi-Fi Direct protocols, such as those standardized by the IEEE 802.11 standard, one or more non-Wi-Fi protocols, or one or more cellular communication protocols or standards. The transceiver 612 may further include hardware, firmware, or software for receiving GNSS signals. The transceiver 612 may include any known receiver and baseband suitable for communication via the communication protocol utilized by the server 600. The transceiver 612 may further include a low noise amplifier (LNA), an additional signal amplifier, an analog-to-digital (A / D) converter, one or more buffers, a digital baseband, or the like.

[0104] Referring now to the functionality supported by the various program modules depicted in FIG. 6, the implementation module 626 may include computer-executable instructions, code, or the like that, in response to execution by one or more of the processors 602, may perform functions including, but not limited to, overseeing the coordination and interaction between one or more modules and computer-executable instructions in the data storage device 620, determining user-selected actions and tasks, determining actions associated with user interactions, determining actions associated with user input, initiating commands locally or on remote devices, and the like.

[0105] The image processing module 627 may include computer-executable instructions, code, or the like that, in response to execution by one or more of the processors 602, may perform functions including, but not limited to, analyzing and processing image data (e.g., still frames and / or video clips) and cropping, segmenting, parsing, sampling, resizing, and / or modifying same.

[0106] The communications module 628 may include computer-executable instructions, code, or the like that, in response to execution by one or more of the processors 602, may perform functions including, but not limited to, for example, communicating with one or more devices via wired or wireless communication, communicating with a server (e.g., a remote server), communicating with a data store and / or database, communicating with an imaging system and / or analyst device, sending or receiving notifications or commands / instructions, communicating with cache memory data, communicating with computing devices, and the like.

[0107] The optical flow module 629 may include computer-executable instructions, code, or the like that, in response to execution by one or more of the processors 602, may perform functions including generating optical flow data, including, but not limited to, horizontal and vertical optical flow data, optical flow plots and / or representations, and other optical flow information from image data.

[0108] The spatio-temporal CNN module 630 may include computer-executable instructions, code, or the like that, in response to execution by one or more of the processors 602, may perform functions including generating, initiating, and executing one or more spatio-temporal CNNs, including, but not limited to, one or more spatial CNNs and one or more temporal CNNs.

[0109] Although specific embodiments of the present disclosure have been described, those skilled in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the present disclosure. For example, any of the functionality and / or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Furthermore, while various example implementations and architectures have been described in accordance with embodiments of the present disclosure, those skilled in the art will recognize that numerous other modifications of the example implementations and architectures described herein are also within the scope of the present disclosure.

[0110] Certain aspects of the present disclosure are described above with reference to block and flow diagrams of systems, methods, apparatuses, and / or computer program products according to example embodiments. It should be understood that one or more blocks of the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and flow diagrams, respectively, can be implemented by the execution of computer-executable program instructions. Similarly, some blocks of the block diagrams and flow diagrams may not necessarily be performed in the order presented, or may not necessarily be performed at all, according to some embodiments. Furthermore, additional components and / or operations other than those depicted in the blocks and / or flow diagrams may be present in some embodiments.

[0111] Thus, the blocks in the block diagrams and flow diagrams support combinations of means for performing the specified functions, combinations of elements or steps for performing the specified functions, and program instruction means for performing the specified functions. It should also be understood that each block of the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and flow diagrams, can be implemented by a dedicated hardware-based computer system that performs the specified functions, elements, or steps, or a combination of dedicated hardware and computer instructions.

[0112] A program module, application, or equivalent disclosed herein may include one or more software components, including, for example, software objects, methods, data structures, or the like. Each such software component may include computer-executable instructions that, upon execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein).

[0113] The software components may be coded in any of a variety of programming languages. An exemplary programming language may be a lower-level programming language, such as assembly language, associated with a particular hardware architecture and / or operating system platform. Software components that include assembly language instructions may require conversion by an assembler into executable machine code prior to execution by the hardware architecture and / or platform.

[0114] Another exemplary programming language may be a higher-level programming language that may be portable across multiple architectures. Software components containing higher-level programming language instructions may require conversion to an intermediate representation by an interpreter or compiler prior to execution.

[0115] Other examples of programming languages ​​include, but are not limited to, a macro language, a shell or command language, a job control language, a scripting language, a database query or search language, or a report writing language. In one or more exemplary embodiments, a software component that includes instructions in one of the foregoing examples of programming languages ​​may be executed directly by an operating system or other software component without first having to be converted into another form.

[0116] Software components may be stored as files or other data storage structures. Software components of similar type or functionally related may be stored together, for example, in a particular directory, folder, or library. Software components may be static (e.g., pre-established or fixed) or dynamic (e.g., created or modified at run time).

[0117] Software components may call, or be called by, other software components through any of a wide variety of mechanisms. The called or calling software components may include other custom-developed application software, operating system functionality (e.g., device drivers, data storage (e.g., file management) routines, other common routines and services, etc.), or third-party software components (e.g., middleware, encryption or other security software, database management software, file transfer or other network communication software, mathematical or statistical software, image processing software, and format conversion software).

[0118] The software components associated with a particular solution or system may reside and execute on a single platform or may be distributed across multiple platforms. The multiple platforms may be associated with more than one hardware vendor, underlying chip technology, or operating system. Furthermore, the software components associated with a particular solution or system may be initially written in one or more programming languages, but may call software components written in other programming languages.

[0119] The computer-executable program instructions may be loaded onto a special-purpose computer or other specific machine, processor, or other programmable data processing apparatus to produce a specific machine, such that execution of the instructions on the computer, processor, or other programmable data processing apparatus causes the computer, processor, or other programmable data processing apparatus to perform one or more functions or operations defined in the flow diagrams. These computer program instructions may also be stored in a computer-readable storage medium (CRSM) that, upon execution, may direct the computer or other programmable data processing apparatus to function in a particular manner to produce an article of manufacture, the instructions stored in the computer-readable storage medium including instruction means that implement one or more functions or operations defined in the flow diagrams. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational elements or steps to be performed on the computer or other programmable data processing apparatus to produce a computer-implemented process.

[0120] Additional types of CRSM that may be present in any of the devices described herein may include, but are not limited to, programmable random access memory (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device, or other magnetic storage device, or any other medium that can be used to store and access information. Combinations of any of the above are also included within the scope of CRSM. Alternatively, a computer-readable communication medium (CRCM) may include computer-readable instructions, program modules, or other data transmitted within a data signal or other transmission, such as a carrier wave. However, as used herein, CRSM does not include CRCM.

[0121] While the embodiments are described in language specific to structural features and / or methodological acts, it should be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the embodiments. In particular, conditional language such as "can," "could," "might," or "may," unless specifically stated otherwise or understood otherwise within the context as used, is intended to generally convey that certain embodiments may include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is generally not intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included or should be performed in any particular embodiment, with or without user input or prompting.

[0122] It should be understood that any of the computer operations described herein above may be implemented, at least in part, as computer-readable instructions stored on a computer-readable memory. Of course, it should be understood that the embodiments described herein are illustrative, and that components may be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are contemplated and fall within the scope of the present disclosure.

[0123] The foregoing description of exemplary embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or to be limited to the precise form disclosed, as modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed embodiments. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

1. 1. A method for determining the presence of one or more congenital heart defects (CHD) in a patient, the method comprising: a server determining first image data representative of a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the first image data, the optical flow data indicative of pixel movement in the series of image frames; processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks trained to process the image data; processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks trained to process the optical flow data; generating a spatial output based on the image data using the spatial model, the spatial output indicating a first likelihood of the presence of one or more CHDs in the patient; generating a temporal output based on the plurality of optical flow data using the temporal model, the temporal output indicating a second likelihood of the presence of one or more CHDs in the patient; and determining a fused output based on the spatial output and the temporal output, the fused output indicating a third possibility of the presence of one or more CHDs in the patient; and causing a first device to display a user interface corresponding to the fused output; and A method comprising:

2. 2. The method of claim 1, wherein the third possibility of the presence of one or more CHDs in the patient comprises one or more of the following: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, abnormal heart size, ventricular septal defect, abnormal atrioventricular junction, abnormal area posterior to the left atrium, abnormal left ventricular junction, abnormal aortic junction, abnormal right ventricular junction, abnormal pulmonary artery junction, arterial size discrepancy, right aortic arch anomaly, abnormal pulmonary artery size, abnormal transverse aortic arch size, or abnormal superior vena cava size.

3. comparing the fused output to a threshold; determining that the fused output satisfies the threshold; and determining the presence of the one or more CHDs in the patient based on the fused output satisfying the threshold; and The method of claim 1 further comprising:

4. determining a request from the first device to generate a report corresponding to the fused output; causing the first device to generate the report corresponding to the fused output; The method of claim 1 further comprising:

5. The method of claim 1 , further comprising training the spatial model and the temporal model using a plurality of second image data different from the first image data.

6. The method of claim 1 , further comprising removing at least a portion of the first image data from each of the image frames in the sequence of image frames.

7. The method of claim 1 , further comprising receiving the first image data from an imaging system.

8. The method of claim 7 , wherein the imaging system comprises an ultrasound or echocardiography device.

9. 9. The method of claim 8, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device.

10. The method of claim 1 , further comprising sampling the image data such that only non-adjacent image frames in the sequence of image frames are processed by the spatial model.

11. 1. A system for determining the presence of one or more congenital heart defects (CHD) in a patient, the system comprising: a memory configured to store computer-executable instructions; at least one computer processor, said at least one computer processor having access to a memory; determining first image data representative of a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the image data, the optical flow data indicative of pixel movement in the series of image frames; generating a spatial output by processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks, the spatial output indicating a first likelihood of the presence of one or more CHDs in the patient; generating a temporal output by processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks, the temporal output indicating a second likelihood of the presence of one or more CHDs in the patient; and determining a fused output based on the spatial output and the temporal output, the fused output indicating a third possibility of the presence of one or more CHDs in the patient; and causing a first device to display a user interface corresponding to the fused output; and at least one computer processor configured to execute the computer-executable instructions to A system comprising:

12. 12. The system of claim 11, wherein the third possibility of the presence of one or more CHDs in the patient comprises one or more of the following: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, abnormal heart size, ventricular septal defect, abnormal atrioventricular junction, abnormal area posterior to the left atrium, abnormal left ventricular junction, abnormal aortic junction, abnormal right ventricular junction, abnormal pulmonary artery junction, arterial size discrepancy, right aortic arch anomaly, abnormal pulmonary artery size, abnormal transverse aortic arch size, or abnormal superior vena cava size.

13. The computer processor further comprises: comparing the fused output to a threshold; determining that the fused output satisfies the threshold; and determining the presence of the one or more CHDs in the patient based on the fused output satisfying the threshold; and The system of claim 11 , configured to execute the computer-executable instructions to:

14. The computer processor further comprises: determining a request from the first device to generate a report corresponding to the fused output; causing the first device to generate the report corresponding to the fused output; The system of claim 11 , configured to execute the computer-executable instructions to:

15. 12. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data.

16. The system of claim 11 , wherein the computer processor is further configured to execute the computer-executable instructions to remove at least a portion of the first image data from each of the image frames in the sequence of image frames.

17. The system of claim 11 , wherein the computer processor is further configured to execute the computer-executable instructions to receive the first image data from an imaging system.

18. 20. The system of claim 17, wherein the imaging system comprises an ultrasound or echocardiography device.

19. 20. The system of claim 18, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device.

20. 12. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to sample the image data such that only non-adjacent image frames in the sequence of image frames are processed by the spatial model.

21. The system of claim 11 , wherein the patient is a fetus during pregnancy.

22. The system of claim 11 , wherein the spatial output comprises a matrix of values ​​indicating view orientations at individual image frames of the sequence of image frames.

23. The system of claim 1 , wherein the computer processor is further configured to determine the fused output using late fusion.

24. 24. The system of claim 23, wherein the late fusion is one of a sum fusion approach, a maximum fusion approach, a concatenation fusion approach, a conventional fusion approach, or a bilinear fusion approach.

25. 1. A method for determining the presence of one or more congenital heart defects (CHD) in a fetus during pregnancy, said method comprising: determining, by a computing device, image data representative of a portion of the fetal cardiovascular system, the image data comprising a series of image frames; determining a neural network system comprising a spatial model trained to process image data and an optical model trained to process optic flow data corresponding to said image data; determining a spatial output based on the image data using the spatial model, the spatial output corresponding to a first likelihood of the presence of one or more CHDs in the fetus, the spatial output comprising a matrix of values ​​indicative of a view orientation at a particular image frame of the series of image frames; determining a time output based on the image data using the temporal model, the time output corresponding to a second likelihood of the presence of one or more CHDs in the patient; and determining a third likelihood of the presence of one or more CHDs in the patient based on the spatial output and the temporal output; and causing the first device to display a user interface corresponding to the third possibility of the presence of one or more CHDs; and A method comprising:

26. 26. The method of claim 25, wherein the third possibility of the presence of one or more CHDs in the patient comprises one or more of the following: a riding artery, a septal defect in cardiac Crooks', parallel aorta, an enlarged cardiothoracic ratio, right to left ventricular size discrepancy, tricuspid to mitral annulus size discrepancy, pulmonary to aortic annulus size discrepancy, an abnormal outflow tract relationship, cardiac axis deviation, an atrial septal defect, an atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch syndrome, ventricular disproportion, abnormal heart size, a ventricular septal defect, an abnormal atrioventricular junction, an abnormal area posterior to the left atrium, an abnormal left ventricular junction, an abnormal aortic junction, an abnormal right ventricular junction, an abnormal pulmonary artery junction, an arterial size discrepancy, a right aortic arch abnormality, an abnormal pulmonary artery size, an abnormal transverse aortic arch size, or an abnormal superior vena cava size.

27. comparing the third likelihood of the presence of one or more CHDs to a threshold; determining that the third likelihood of the presence of one or more CHDs meets the threshold; determining the presence of the one or more CHDs in the patient based on the third likelihood of the presence of the one or more CHDs satisfying the threshold; and 26. The method of claim 25, further comprising:

28. determining a request from the first device to generate a report corresponding to the third possibility of the presence of one or more CHDs; causing the first device to generate the report corresponding to the third possibility of the presence of one or more CHDs; 26. The method of claim 25, further comprising:

29. 26. The method of claim 25, further comprising training the spatial model and the temporal model using a plurality of second image data different from the image data.

30. 26. The method of claim 25, further comprising removing at least a portion of the image data from each of the image frames in the sequence of image frames.

31. The method of claim 25 , further comprising receiving the image data from an imaging system.

32. 32. The method of claim 31 , wherein the imaging system comprises an ultrasound or echocardiography device.

33. 34. The method of claim 33, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device.

34. 26. The method of claim 25, further comprising sampling the image data such that only non-adjacent image frames in the sequence of image frames are processed by the spatial model.

35. 26. The method of claim 25, wherein one or more of the spatial outputs represent one or more of keypoint data or contour data.

36. 26. The method of claim 25, wherein one or more of the time outputs indicate one or more of keypoint data or contour data.

37. 26. The method of claim 25, further comprising determining one or more of keypoint data or contour data based on the spatial output and the temporal output.

38. 38. The method of claim 37, further comprising causing the first device to further display the one or more of keypoint data or contour data.

39. 1. A system for determining the presence of one or more congenital heart defects (CHD) in a fetus during pregnancy, the system comprising: a memory configured to store computer-executable instructions; at least one computer processor, said at least one computer processor having access to a memory; determining, by a computing device, image data representative of a portion of the fetal cardiovascular system, the image data comprising a series of image frames; determining a neural network system comprising a spatial model trained to process image data and an optical model trained to process optic flow data corresponding to said image data; determining a spatial output based on the image data using the spatial model, the spatial output corresponding to a first likelihood of the presence of one or more CHDs in the fetus, the spatial output comprising a matrix of values ​​indicative of a view orientation at a particular image frame of the series of image frames; determining a time output based on the image data using the temporal model, the time output corresponding to a second likelihood of the presence of one or more CHDs in the patient; and determining a third likelihood of the presence of one or more CHDs in the patient based on the spatial output and the temporal output; and causing the first device to display a user interface corresponding to the third possibility of the presence of one or more CHDs; and at least one computer processor configured to execute the computer-executable instructions to A system comprising:

40. 40. The system of claim 39, wherein the third possibility of the presence of one or more CHDs in the patient comprises one or more of the following: a riding artery, a septal defect in cardiac Crooks', parallel aorta, an enlarged cardiothoracic ratio, right to left ventricular size discrepancy, tricuspid to mitral annulus size discrepancy, pulmonary to aortic annulus size discrepancy, an abnormal outflow tract relationship, cardiac axis deviation, an atrial septal defect, an atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, dextro-transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch syndrome, ventricular disproportion, abnormal heart size, a ventricular septal defect, an abnormal atrioventricular junction, an abnormal area posterior to the left atrium, an abnormal left ventricular junction, an abnormal aortic junction, an abnormal right ventricular junction, an abnormal pulmonary artery junction, an arterial size discrepancy, a right aortic arch abnormality, an abnormal pulmonary artery size, an abnormal transverse aortic arch size, or an abnormal superior vena cava size.

41. The computer processor further comprises: comparing the third likelihood of the presence of one or more CHDs to a threshold; determining that the third likelihood of the presence of one or more CHDs meets the threshold; determining the presence of the one or more CHDs in the patient based on the third likelihood of the presence of the one or more CHDs satisfying the threshold; and 40. The system of claim 39, configured to execute the computer-executable instructions to:

42. The computer processor further comprises: determining a request from the first device to generate a report corresponding to the third possibility of the presence of one or more CHDs; causing the first device to generate the report corresponding to the third possibility of the presence of one or more CHDs; 40. The system of claim 39, configured to execute the computer-executable instructions to:

43. 40. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data.

44. 40. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to remove at least a portion of the image data from each of the image frames in the sequence of image frames.

45. 40. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to receive the first image data from an imaging system.

46. 46. ​​The system of claim 45, wherein the imaging system comprises an ultrasound or echocardiography device.

47. 47. The system of claim 46, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device.

48. 40. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to sample the image data such that only non-adjacent image frames in the sequence of image frames are processed by the spatial model.

49. 40. The system of claim 39, wherein one or more of the spatial outputs represent one or more of keypoint data or contour data.

50. 40. The system of claim 39, wherein one or more of the time outputs indicate one or more of keypoint data or contour data.

51. 40. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to determine one or more of keypoint data or contour data based on the spatial output and the temporal output.

52. 40. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to cause the first device to further display the one or more of keypoint data or contours.

Images