System and method for detecting cardiovascular malformations using spatiotemporal neural networks

Abstract

This disclosure provides for processing image data generated by a medical imaging device, such as an ultrasound or echocardiography device, and processing the image data using artificial intelligence and machine learning to determine one or more congenital heart defects (CHDs) in the image data and / or the presence of other cardiovascular malformations and / or determination of key points and / or contour detection. The image processing system may be used to detect CHD and / or other cardiovascular malformations in the body of a fetus. The image data may be processed using a spatio-temporal convolutional neural network (CNN). The spatiotemporal CNNs may include a spatial CNN for image recognition and a temporal CNN for processing optical flow data based on the image data. The outputs of the spatial CNNs and the temporal CNNs may be fused (e.g., using late fusion) to generate the likelihood of CHD and / or other cardiovascular malformations.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. patent application No. 18 / 412,325 filed on January 12, 2024, U.S. patent application No. 18 / 183,942 filed on March 14, 2023 (now U.S. Patent No. 11,875,507), and EP patent application No. 23305236.4 filed on February 22, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention generally relates to an image processing system, such as an image processing system having artificial intelligence and machine learning functionality for detecting cardiovascular malformations. Background Art

[0004] With today's imaging technology, medical providers can view the inside of a patient's body and detect deformities and conditions even without the need for surgery. For example, imaging technologies such as ultrasound imaging allow medical technicians to obtain two-dimensional views of a patient's anatomical structures, such as the chambers of a patient's heart. For example, an echocardiogram uses high-frequency sound waves to generate a picture of a patient's heart. Various views can be obtained by manipulating the orientation of the ultrasound transducer relative to the patient.

[0005] Medical imaging can be used by healthcare providers to perform medical examinations of a patient's anatomy without the need for surgery. For example, a healthcare provider can examine a generated image for visible deviations from normal anatomy. Additionally, a healthcare provider can use medical images to take measurements and compare the measurements to known normal ranges to identify deformities.

[0006] In one example, a healthcare provider can use echocardiography to identify heart defects, such as a ventricular septal defect, which is an abnormal connection between the lower chambers of the heart (i.e., the ventricles). The healthcare provider can visually identify the connection in the medical image and, based on the medical image, can make a diagnosis. This diagnosis can then lead to surgical intervention or other treatment.

[0007] Although healthcare providers often detect malformations such as heart defects via medical imaging, defects and various other malformations go undetected due to human error, inadequate training, subtle visual cues, and a variety of other reasons. This is especially true for complex anatomical structures and prenatal imaging. For example, congenital heart defects (CHDs) in fetuses are particularly difficult to detect. It is estimated that CHD during pregnancy occurs in approximately 1% of pregnancies. However, between 50% and 70% of CHD cases are not correctly detected by doctors. Detection of CHD during pregnancy allows healthcare providers to make a diagnosis and / or provide timely interventional treatment, which can lead to improved fetal and infant health and fewer infant deaths.

[0008] Therefore, there is a need for improved methods and systems for analyzing and / or processing medical imaging, including ultrasound imaging, for detecting deformities and defects such as CHD. Summary of the Invention

[0009] Provided herein are systems and methods for analyzing medical imaging using spatiotemporal neural networks to detect cardiovascular malformations and / or conditions, such as CHD. The systems and methods may include processing medical device imaging, such as single-frame images and / or video clips generated by an ultrasound system, using a spatiotemporal convolutional neural network (CNN). Optical flow data may be generated based on the image and / or video clip and may indicate the movement of pixels in the image and / or video clip. The image and / or video clip may be processed by a spatial CNN, and the optical flow data may be processed using a temporal CNN. The spatial output from the spatial CNN and the temporal output from the temporal CNN may be fused to generate a combined spatiotemporal output that may indicate the likelihood of the presence of one or more CHDs or other cardiovascular malformations in a patient (e.g., a fetus of a pregnant patient).

[0010] Provided herein is a method for determining the presence of one or more congenital heart defects (CHDs) in a patient, the method comprising: determining, by a server, first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the first image data, the optical flow data indicating movement of pixels in the series of image frames; processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks trained to process image data; processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks trained to process optical flow data; generating a spatial output based on the image data using the spatial model, the spatial output indicating a first likelihood of the presence of the one or more CHDs in the patient; generating a temporal output based on the multiple optical flow data using the temporal model, the temporal output indicating a second likelihood of the presence of the one or more CHDs in the patient; determining a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of the one or more CHDs in the patient; and causing a first device to display a user interface corresponding to the fused output.

[0011] The third likelihood of the presence of one or more CHDs in the patient may include one or more of the following possibilities of presence: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein's anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, heart size abnormalities, ventricular septal defect, atrioventricular junctional anomaly, left atrial posterior area anomaly, left ventricular junctional anomaly, aortic junctional anomaly, right ventricular junctional anomaly, pulmonary junctional anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size anomaly, transverse aortic arch size anomaly, or superior vena cava size anomaly.

[0012] The method may further include comparing the fused output to a threshold; determining that the fused output satisfies the threshold; and determining the presence of the one or more CHDs in the patient based on the fused output satisfying the threshold. The method may further include: determining a request from a first device to generate a report corresponding to the fused output; causing the first device to generate the report corresponding to the fused output. The method further includes training the spatial model and the temporal model using a plurality of second image data different from the first image data. The method further includes removing at least a portion of the first image data from each of the image frames in the series of image frames. The method further includes receiving the first image data from an imaging system. The imaging system further includes an ultrasound or echocardiographic device. The image data further includes a first series of image frames corresponding to a first orientation of the ultrasound or echocardiographic device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiographic device. The method further includes sampling the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model.

[0013] Provided herein is a system for determining the presence of one or more congenital heart defects (CHDs) in a patient, the system comprising: a memory configured to store computer-executable instructions; and at least one computer processor configured to access the memory and execute the computer-executable instructions to: determine first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determine optical flow data based on the image data, the optical flow data indicating movement of pixels in the series of image frames; generate a spatial output by processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks, the spatial output indicating a first likelihood of the presence of the one or more CHDs in the patient; generate a temporal output by processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks, the temporal output indicating a second likelihood of the presence of the one or more CHDs in the patient; determine a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of the one or more CHDs in the patient; and cause a first device to display a user interface corresponding to the fused output.

[0014] The third likelihood of the presence of one or more CHDs in the patient may include one or more of the following possibilities of presence: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, heart size abnormalities, ventricular septal defect, atrioventricular junctional anomaly, left atrial posterior area anomaly, left ventricular junctional anomaly, aortic junctional anomaly, right ventricular junctional anomaly, pulmonary junctional anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size anomaly, transverse aortic arch size anomaly, or superior vena cava size anomaly.

[0015] The computer processor may be configured to execute the computer-executable instructions to: compare the fused output to a threshold; determine that the fused output satisfies the threshold; and determine the presence of the one or more CHDs in the patient based on the fused output satisfying the threshold. The computer processor may be further configured to execute the computer-executable instructions to: determine a request from a first device to generate a report corresponding to the fused output; and cause the first device to generate the report corresponding to the fused output. The computer processor may be further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data. The computer processor may be further configured to execute the computer-executable instructions to remove at least a portion of the first image data from each of the image frames in the series of image frames.

[0016] The computer processor may be further configured to execute the computer-executable instructions to receive the first image data from an imaging system. The imaging system may include an ultrasound or echocardiography device. The image data includes a first series of image frames corresponding to a first orientation of the ultrasound or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiography device. The computer processor may be further configured to execute the computer-executable instructions to sample the image data so that only non-adjacent image frames in the series of image frames are processed by the spatial model. The patient may be a fetus during pregnancy. The spatial output includes a matrix of values ​​indicating the view orientation in corresponding image frames in the series of image frames. The computer processor may be further configured to use late fusion to determine the fused output. The late fusion is one of a sum fusion method, a maximum fusion method, a concatenation fusion method, a conventional fusion method, or a bilinear fusion method.

[0017] A method for determining the presence of one or more CHDs and / or other cardiovascular malformations in a patient is provided herein. The method may include: determining, by a server, first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the first image data, the optical flow data indicating movement of pixels in the series of image frames; processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks trained to process image data; processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks trained to process optical flow data; generating a spatial output based on the image data using the spatial model, the spatial output indicating a first likelihood of the presence of one or more CHDs and / or other cardiovascular malformations in the patient; generating a temporal output based on the plurality of optical flow data using the temporal model, the temporal output indicating a second likelihood of the presence of the one or more CHDs and / or other cardiovascular malformations in the patient; determining a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of the one or more CHDs and / or other cardiovascular malformations in the patient; and causing a first device to display a user interface corresponding to the fused output.

[0018] The third likelihood of the presence of one or more CHDs and / or other cardiovascular malformations in the patient may include one or more of the following likelihoods of the presence of: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, aortic arch interruption, ventricular disproportion, heart size anomaly, ventricular septal defect, atrioventricular junction anomaly, left atrial posterior region anomaly, left ventricular junction anomaly, aortic junction anomaly, right ventricular junction anomaly, pulmonary junction anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size anomaly, transverse aortic arch size anomaly, or superior vena cava size anomaly. The method may further include comparing the fused output to a threshold; determining that the fused output satisfies the threshold; and determining the risk or presence of the one or more CHDs and / or other cardiovascular malformations in the patient based on the fused output satisfying the threshold. The method may further include determining a request from a first device to generate a report corresponding to the fused output; and causing the first device to generate the report corresponding to the fused output. The method may further include training the spatial model and the temporal model using a plurality of second image data different from the first image data. The method may further include removing at least a portion of the first image data from each of the image frames in the series of image frames.

[0019] The method may further include receiving the first image data from an imaging system, and the imaging system may include an ultrasound or echocardiographic device. The image data may include a first series of image frames corresponding to a first orientation of the ultrasound or echocardiographic device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiographic device. It should be understood that the multiple series of image frames can be processed using the imaging system. The method may include sampling the image data so that only non-adjacent image frames in the series of image frames are processed using the spatial model. Image data from adjacent and other image series and / or image frames may be used to process and / or generate output related to a particular image series or image frame. Such other image series and / or image frames may provide context for the image series and / or frame for which the output is generated. One or more of the spatial outputs may further indicate one or more keypoint data or contour data. One or more of the temporal outputs may further indicate one or more keypoint data or contour data. The method may further include determining one or more of keypoint data or contour data based on the spatial output and the temporal output and / or causing the first device to further display the one or more of keypoint data or contour data.

[0020] Provided herein is a system for determining the presence of one or more CHDs and / or other cardiovascular malformations in a patient. The system includes a memory configured to store computer-executable instructions and at least one computer processor configured to access the memory and execute the computer-executable instructions to: determine first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determine optical flow data based on the image data, the optical flow data indicating movement of pixels in the series of image frames; generate a spatial output by processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks, the spatial output indicating a first likelihood of the presence of one or more CHDs and / or other cardiovascular malformations in the patient; generate a temporal output by processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks, the temporal output indicating a second likelihood of the presence of one or more CHDs and / or other cardiovascular malformations in the patient; determine a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of one or more CHDs and / or other cardiovascular malformations in the patient; and cause a first device to display a user interface corresponding to the fused output.

[0021] The third likelihood of the presence of one or more CHDs and / or other cardiovascular malformations in the patient may include one or more of the following likelihoods of the presence of: atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, or interrupted aortic arch. The computer processor may be further configured to execute the computer-executable instructions to compare the fused output with a threshold; determine that the fused output meets the threshold; and determine the presence of the one or more CHDs and / or other cardiovascular malformations in the patient based on the fused output meeting the threshold. The computer processor may be further configured to execute the computer-executable instructions to determine a request from a first device to generate a report corresponding to the fused output, and cause the first device to generate the report corresponding to the fused output. The computer processor may be further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data. The computer processor may be further designed to execute the computer-executable instructions to remove at least a portion of the first image data from each of the image frames in the series of image frames.

[0022] The computer processor may be further configured to execute the computer-executable instructions to receive the first image data from an imaging system, and the imaging system may include an ultrasound or echocardiographic device. The image data may include a first series of image frames corresponding to a first orientation of the ultrasound or echocardiographic device and a second series of image frames corresponding to a second orientation of the ultrasound or echocardiographic device. The computer processor may be further configured to execute the computer-executable instructions to sample the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model. One or more of the spatial outputs may further indicate one or more of key point data or contour data. One or more of the temporal outputs may further indicate one or more of the key point data or contour data. The system may be further configured to execute the computer-executable instructions to determine one or more of the key point data or contour data based on the spatial output and the temporal output and / or cause the first device to further display the one or more of the key point data or contour data.

[0023] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the following drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1An image processing system for determining the presence of cardiovascular malformations according to aspects of the present invention is described.

[0025] Figures 2A to 2B Schematic diagram illustrating the data flow between the imaging system, analysis device, and the backend of the image processing system.

[0026] Figures 3A to 3C Schematic diagram illustrating spatial flow, temporal flow, and fused spatiotemporal output.

[0027] Figure 4 Describe the process flow of spatial CNN, temporal CNN, and fused spatiotemporal output.

[0028] Figures 5A to 5B A process flow for determining the presence or absence of CHD and / or other cardiovascular malformations based on the likelihood of their presence is described.

[0029] Figure 6 is a schematic block diagram of a computing device according to one or more example embodiments of the present disclosure.

[0030] The foregoing and other features of the present invention will become apparent from the following description and appended claims taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several embodiments in accordance with the present disclosure and are therefore not to be considered limiting of its scope, the present disclosure will be described in more detail and in greater detail through use of the accompanying drawings. DETAILED DESCRIPTION

[0031] The present invention relates to an image processing system that uses artificial intelligence and machine learning to determine the likelihood of the presence or absence of CHD and / or other cardiovascular malformations in a patient (e.g., a fetus during pregnancy), or where such presence or absence is uncertain. For example, medical imaging, such as images (e.g., still frames and / or video clips), can be generated using an ultrasound system (e.g., an echocardiography system) and processed by a spatiotemporal neural network to generate a likelihood of the presence or absence of one or more CHD and / or other cardiovascular malformations. The images can also be processed by the spatiotemporal neural network to detect keypoints corresponding to cardiovascular anatomical structures (e.g., the apex of the heart, etc.) (this data is referred to as keypoint data) and / or contours and / or segmentations of elements and / or features of cardiovascular anatomical structures (e.g., contours of one or more ventricles, one or more atria, etc.) (this data is referred to as contour data). This information can be used to calculate measurements (e.g., lengths, areas, ratios) and / or detect features and / or anatomical structures of the fetus (e.g., detection of the heart, lungs, or portions of the heart (e.g., atria, septum, ventricles, etc.)).

[0032] Medical imaging may include a series of consecutive still frame images. The still frame images may be preprocessed to remove redundant or unwanted portions. For example, during preprocessing, spatial, temporal, and / or spatiotemporal filters may be used to remove noise. The still frame images may be sampled, segmented, or parsed so that only a certain number of frames (e.g., every second, third, or fourth frame) may be selected. Optical flow data may be generated from the image data and may represent the movement of pixels in the image data. Two neural networks may be used to process the optical flow data and the image data (e.g., a single frame of image data) in parallel, one for the image and the other for the optical flow data. The architectures of the two networks may be fused at one or more levels (e.g., late fusion and / or final feature mapping).

[0033] The two parallel neural networks may be two CNNs. Specifically, the first CNN may be a spatial network trained to process image data (e.g., a single frame of RGB data). The second CNN may be a temporal neural network trained to process optical flow data. Alternatively or additionally, one or more neural networks may be deep neural networks (DNNs) and / or any other suitable neural networks. Each neural network may output a probability of the presence, absence, and / or uncertainty of CHD and / or other cardiovascular malformations and / or output key points and / or contours that may indicate fetal anatomical structures. The architectures of the two neural networks may be fused to produce results that are superior to either network individually. For example, the outputs may be determined using the two networks and combined via late fusion to produce a single spatiotemporal output that indicates the probability of the presence, absence, and / or uncertainty of CHD and / or other malformations in the image data (e.g., based on the visual appearance of the anatomical structures or the absence or absence of certain anatomical structures). It should be understood that the one or more CNNs may optionally be attention-based neural networks. It should further be understood that the spatial network and the temporal network may be a single network or two networks. For example, the imaging system may include a two-stream network having a two-stream architecture with a spatial CNN and a temporal CNN, and may fuse the CNNs. Although the imaging processing system described herein is described as a CNN, it should be understood that such an imaging processing system is not limited to a CNN, and other embodiments of the imaging processing system may alternatively use a neural network, such as a CNN, a residual neural network, an attention neural network, a region-based convolutional neural network (RCNN), and / or any other suitable neural network. Any combination of one or more.

[0034] Now refer to Figure 1, illustrating an image processing system 100. The image processing system 100 can be designed to receive medical images, process the medical images using artificial intelligence and machine learning, and determine the likelihood of the presence, absence, or uncertainty of one or more CHDs and / or other cardiovascular malformations and / or the image processing system 100 can be used to determine outputs indicating key points and / or contours of fetal anatomy. For example, the image processing system 100 can receive image data showing fetal anatomy and can process the image data using a spatiotemporal CNN to automatically determine the presence and / or absence of one or more CHDs and / or other cardiovascular malformations.

[0035] The image processing system 100 may include one or more imaging systems 102, each of which may be in communication with a server 104. For example, the imaging system 102 may be any well-known medical imaging system that generates image data (e.g., still frames and / or video clips including RGB pixel information), such as an ultrasound system, an echocardiography system, an x-ray system, a computed tomography (CT) system, a magnetic resonance imaging (MRI) system, a positron emission tomography (PET) system, and the like.

[0036] The imaging system 102 can be any suitable ultrasound scanning system for performing fetal ultrasound examinations (e.g., mid-trimester fetal anatomy ultrasound examinations between 18 and 24 weeks of gestation, first-trimester examinations, third-trimester fetal examinations, fetal echocardiography, or other). However, the software / programming of the present invention described herein is stored and executed on the imaging system 102, the server 104, and / or the data storage device 112. In one example, the imaging system can be a Samsung WS80A ultrasound system or any other suitable ultrasound scanning system. The image processing system 100 can optionally be designed to be agnostic to the manufacturer, model, and / or type of the imaging system 102. For example, the image processing system 100 can implement and / or incorporate the systems and / or methods for agnostic analysis provided in U.S. Patent No. 11,861,838, the entire contents of which are incorporated herein by reference. Although ultrasound systems are described throughout, it should be understood that the same or similar methods can be used with any other suitable medical imaging system (e.g., a CT system, an MRI system, a PET system, or any other imaging and / or diagnostic system).

[0037] like Figure 1 , the imaging system 102 may be an ultrasound imaging system including an ultrasound sensor 108 and an ultrasound device 106. The ultrasound sensor 108 may include a piezoelectric sensor device and may be any well-known ultrasound sensing device. The ultrasound device 106 may be any well-known computing device including a processor and a display and may have a wired or wireless connection to the ultrasound sensor 108.

[0038] The ultrasound sensor 108 can be used by a healthcare provider to obtain image data of a patient's anatomy (e.g., patient 110). The ultrasound sensor 108 can generate a two-dimensional image corresponding to the orientation of the ultrasound sensor 108 relative to the patient 110. The image data generated by the ultrasound sensor 108 can be communicated to the ultrasound device 106. The ultrasound device 106 can send the image data to the remote server 104 via any well-known wired or wireless system (e.g., Wi-Fi, cellular network, Bluetooth, Bluetooth Low Energy (BLE), near-field communication protocol, etc.). Additionally or alternatively, the image data can be received and / or retrieved from one or more picture archiving and communication systems (PACS). For example, a PACS system can use the Digital Imaging and Communications in Medicine (DICOM) format. Any results from the system (e.g., spatiotemporal output 232 and / or analyzed output 236) can be shared with the PACS.

[0039] The remote server 104 may be any computing device having one or more processors capable of performing the operations described herein. Figure 1 In the example illustrated in , the remote server 104 may be one or more servers, desktop or laptop computers, or the like and / or may be located at a different location than the imaging system 102. The remote server 104 may run one or more local applications to facilitate communication between the imaging system 106, the data storage device 112, and / or the analysis device 116.

[0040] The data storage device 112 may be one or more drives having memory dedicated to storing digital information (e.g., information specific to a particular patient, professional, facility, and / or device). For example, the data storage device 112 may include, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read-only memory (ROM)), flash memory, or any combination thereof. The data storage device 112 may be incorporated into the server 104 or may be separate and distinct from the server 104. In one example, the data storage device 112 may be a picture archiving and communication system (PACS).

[0041] The remote server 104 may communicate with the data storage device 112 and / or the analysis device 116 via any well-known wired or wireless system (e.g., Wi-Fi, cellular network, Bluetooth, Bluetooth Low Energy (BLE), near field communication protocol, etc.). The data storage device 112 may receive and store image data (e.g., image data 118) received from the remote server 104. For example, the imaging system 102 may generate image data (e.g., ultrasound image data) and may send this image data to the remote server 104, which may send the image data to the data storage device 112 for storage. It should be understood that the data storage device 112 may be optional and / or that more than one imaging system 102, remote server 104, data storage device 112, and / or analysis device 116 may be used.

[0042] Analysis device 116 may be any computing device having a processor and a display and capable of communicating with at least remote server 104 and performing the operations described herein. Analysis device 116 may be any well-known computing device, such as a desktop computer, laptop computer, smartphone, tablet computer, wearable device, or the like. Analysis device 116 may run one or more local applications to facilitate communication between analysis device 116 and remote server 104 and / or any other computing devices or servers described herein.

[0043] The remote server 104 may receive image data (e.g., RGB image data from an ultrasound system) from the data storage device 112 and / or the imaging system 106 and may process the image data to determine the presence or absence of CHD and / or any other cardiovascular malformation in a patient (e.g., a fetus in a pregnant woman) and / or key points and / or contours of the fetus's anatomical structures. For example, the remote server 104 may process one or more trained models, such as a CNN trained to detect one or more CHDs and / or malformations.

[0044] The remote server 104 may use two parallel spatiotemporal convolutional neural networks (CNNs) and may fuse the outputs to generate a superior output with improved accuracy over the individual CNNs. The first CNN may be a spatial CNN and the second CNN may be a temporal CNN. Image data, which may be ultrasound image frames, may be processed by the spatial CNN.

[0045] Optical flow data can be generated based on images and / or video clips and can indicate the movement of pixels in the images and / or video clips. The optical flow data can be processed using a temporal CNN. The spatial output from the spatial CNN and the temporal output from the temporal CNN can be fused to generate a combined spatiotemporal output that can indicate the likelihood of the presence or absence of one or more CHDs and / or other cardiovascular malformations and / or key points and / or contours of fetal anatomical structures within a patient (e.g., within a fetus of a pregnant patient). The remote server 104 can cause the analysis device 116 to display information regarding the likelihood of the presence of one or more CHDs and / or other cardiovascular malformations. For example, the analysis device can display a patient ID number and a likelihood percentage of one or more CHDs and / or other cardiovascular malformations.

[0046] In one instance, system 100 may be the same as or similar to the systems and methods for computer-aided diagnostic assistance in fetal ultrasound examinations provided in U.S. Patent No. 11,869,188, issued on January 9, 2024, and U.S. Patent Application No. 18 / 406,446, filed on January 8, 2024, the entire contents of each of which are incorporated herein by reference.

[0047] Now refer to Figures 2A to 2B , a schematic diagram depicting the data flow between the imaging system, the analysis device and the backend of the image processing system. Figure 2A Displayed in Figure 1 An imaging system 202 that is the same as or similar to the imaging system 102 may include an image generator 204 that may generate image data 206. The image data 206 may include still frames and / or video clips and may include RGB and / or grayscale pixel information. For example, the image data 206 may include a two-dimensional representation of an ultrasound scan of a patient's anatomy. Additionally or alternatively, the image data 206 may include Doppler image information (e.g., color Doppler, power Doppler, spectral Doppler, duplex Doppler, and the like). It should be understood that various types of image data 206 may be processed simultaneously by the imaging system 202. In one example, the Doppler image data may be generated simultaneously with the ultrasound image data.

[0048] The imaging system 202 may send image data 206 to a backend 208 which may communicate with Figure 2A The image data 206 may be processed by a pre-processor 210. The pre-processor 210 may focus, crop, resize, and / or otherwise remove unnecessary areas of the image data 206 to generate pre-processed image data 212. For example, the black background and text in the still frames generated by the imaging system 202 may be removed. The pre-processor may additionally or alternatively generate a series of continuous still frame images from the video clip.

[0049] The pre-processed image data may optionally be sent to a sampling generator 214, which may cause the pre-processed image data 212 to be sampled, parsed, and / or segmented to generate sampled image data 216. For example, the sampling generator 214 may determine the intervals at which frames are to be sampled (e.g., intervals of two, three, four, etc.). In this manner, only sampled frames of the image data 212 may be processed by the neural network at the backend 208. Sampling the image data 212 may allow the network to process image frames of the image data 212 over a longer period of time.

[0050] Pre-processed image data 212, image data 206, and / or sampled image data 216 may be processed by optical flow generator 218 to generate optical flow data 220 corresponding to pre-processed image data 212, image data 206, and / or sampled image data 216. Optical flow data 220 may allow the network to better account for movement of image data over time.

[0051] To generate optical flow data 220, successive image frames of image data 212, image data 206, and / or sampled image 216 may be input to optical flow generator 218. Horizontal and vertical optical flow data may be calculated from the successive image frames for each adjacent frame, resulting in an output size of H×W×2L, where H and W are the height and width of the image frame, and L is the length (e.g., the time between frames). Optical flow generator 218 may thereby encode the motion of individual pixels across frames of image data 212, image data 206, and / or sampled image 216 to capture movement illustrated in the image over time.

[0052] The sampled image data 216, the pre-processed image data 212, and / or the image data 206 can then be applied to a spatial model 222 to generate a spatial output 226, which can be a spatial CNN, such as a spatial CNN trained for image processing. The spatial model 222 can be trained to analyze image data (e.g., RGB data) to determine the presence of one or more CHDs and / or other cardiovascular malformations in each frame. It should be understood that the spatial model 222 can optionally use the temporal output 228 from the temporal model 224 as input.

[0053] The spatial output 226 may include a vector or matrix that includes scores or values ​​corresponding to one or more frames of the likelihood of CHD and / or other cardiovascular malformations. Optionally, the spatial output 226 may further include scores or values ​​indicating the likelihood of one or more views or orientations of the sensor device to which the image data corresponds. For example, various views may include anatomical standard views (e.g., 4-chamber views, left ventricular outflow tract, right ventricular outflow tract, etc.). Such views may have standard orientations relative to the corresponding anatomical structures (e.g., top view, bottom view, left view, right view, above, below, etc.). Each view and likelihood value may be depicted in a vector or matrix. In one example, the spatial output 226 may include a low likelihood of bottom, right, and left views, but a high likelihood of a top-down view. This will indicate that the view is likely to be from the top down.

[0054] Similarly, optical flow data 220 can be applied to a temporal model 224, which can be a temporal CNN, such as a temporal CNN trained for image processing and / or trained to process optical flow data to generate temporal outputs 228. For example, temporal model 224 can generate temporal outputs 228 that can indicate, for each optical flow data set, a score or value indicating the likelihood of the presence of one or more CHDs and / or other cardiovascular malformations. Optionally, temporal outputs 228 can further include scores or values ​​indicating the likelihood of one or more views or orientations of the sensor device to which the image data corresponds. It should be understood that temporal model 224 can optionally take as input the spatial outputs 226 from spatial model 222.

[0055] Both the spatial output 226 and the temporal output 228 may be input to a fuser 230 to fuse the spatial model 222 and the temporal model 224 to generate a spatiotemporal output 232, which may be similar to the spatial output 226 and the temporal output 228, but with improved accuracy. For example, the fuser 230 may combine the architecture of the spatial model 222 and the temporal model 224 at several levels (e.g., the last feature map). Alternatively or in addition, a weighted average of the spatial output 226 and the temporal output 228 may be determined to generate the spatiotemporal output 232.

[0056] It should be understood that various well-known fusion methods can be used, such as summation, maximum, concatenation, convolution, and bilinear. It should be further understood that although late fusion can be used, other techniques can also be used, such as early fusion (changing the first convolution layer of each stream to a three-dimensional convolution) or slow fusion (changing all convolution layers within each stream to a three-dimensional convolution with a smaller time range than early fusion).

[0057] The spatiotemporal output 232 may be processed by an analyzer 234, which may process the spatiotemporal output 232 to generate an analyzed output 236, which may indicate the presence or absence, or the inclusiveness of the presence or absence, of one or more CHDs and / or cardiovascular malformations in the image data 206, and / or may indicate key points and / or contours of the fetal anatomy. For example, the analyzer 234 may calculate a weighted average based on the spatiotemporal output 232 and / or may filter certain portions of the spatiotemporal output 232. In one example, the analyzed output 236 and / or the spatiotemporal output 232 may indicate a risk of likelihood of the presence or absence of one or more morphological abnormalities or defects and / or may indicate the presence or absence of one or more conditions. For example, the analyzed output 236 and / or the spatiotemporal output 232 may indicate the presence of, or be used to determine the presence or likelihood of, the following: an overriding aorta (e.g., an artery exiting the left ventricle is positioned above a ventricular septal defect), a septal defect at a junction of the heart (e.g., a septal defect located at a junction of the heart, whether it is a septal defect of the first atrial septum or an inlet ventricular septum), parallel great arteries, an increased cardiothoracic ratio (e.g., a heart to thoracic area ratio greater than 0.33 measured at end diastole), a right ventricular to left ventricular size difference (e.g., a right ventricular to left ventricular area ratio greater than 1.4 or less than 0.5 at end diastole), a tricuspid to mitral annulus diameter difference (e.g., a ratio between the tricuspid to mitral valves at end diastole greater than 1.5 or less than 0.65), a pulmonary to aortic annulus diameter difference (e.g., a ratio between the pulmonary to aortic valves at end systole greater than 1.6 or less than 0.85), an outflow tract relationship abnormality (e.g., an incomplete Typical anterior-posterior crossing pattern of the aorta and pulmonary arteries) and cardiac axis deviation (e.g., the angle between the cardiac axis (the line bisecting the chest cavity and the ventricular septum) is less than 25° or greater than 65°), atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion (e.g., the left or right ventricle is larger than the other), abnormal heart size, ventricular septal defect, atrioventricular crossover The present invention also includes but is not limited to the following: abnormalities of the left ventricular junction, enlargement or abnormality of the posterior region of the left atrium, abnormalities of the left ventricular and / or aortic junction, abnormalities of the right ventricular and / or pulmonary junction, large arterial size discrepancies (e.g., aorta larger or smaller than the pulmonary artery), right aortic arch malformation, abnormalities of the size of the pulmonary artery, transverse aortic arch, and / or superior vena cava, visible extra vessels, abnormal ventricular asymmetry, pulmonary and / or aortic valve stenosis, ventricular hypoplasia and / or univentricular heart, and / or any other morphological abnormality, defect, and / or condition. Alternatively or additionally, the analyzed output 236 and / or spatiotemporal output 232 may indicate the presence of, or be used to determine the presence or likelihood of, any other morphological abnormality, condition, and / or disease.

[0058] The backend 208 can communicate the analyzed output 236 and / or information based on the spatiotemporal output 232 to an analysis device 240, which can be the same as or similar to the analysis device 116. The analysis device 240 can be different from or the same as the device in the imaging system 202. The display module 238 can generate a user interface on the analysis device 240 to generate and display a representation of the analyzed output 244 and / or the spatiotemporal output 232. The representation can be the same as or similar to the graphical user interface described and / or illustrated in U.S. application Ser. No. 18 / 406,446, filed on Jan. 8, 2024, the entire contents of which are incorporated herein by reference. For example, the display can show a representation of image data (e.g., an ultrasound image) with an overlay indicating the location of the detected risk or likelihood of CHD and / or other cardiovascular malformations. In one example, the overlay can be a box or any other visual indicator (e.g., an arrow).

[0059] User input module 242 may receive user input 244 and may communicate user input 244 to backend 208. User input 244 may be an instruction from a user to generate a report or other information, such as an indication that a result generated by one or more of spatial model 222, temporal model 224, and / or fuser 230 is inaccurate. For example, if user input 244 indicates an inaccuracy, user input 244 may be used to further train spatial model 222, temporal model 224, and / or fuser 230.

[0060] If user input 244 indicates a request for a report, user input 244 may be communicated to report generator 246, which may generate a report. For example, the report may include some or all of analyzed output 236, spatiotemporal output 232, user input 244, and / or analyses, graphs, charts, and tables therefor. Report 248 may then be communicated to analysis device 240 for display (e.g., via display module 238), or may be printed by analysis device 240.

[0061] Now refer to Figure 2B ,illustrate Figure 1 The clinical workflow of the system is described in Figure 2B As shown in FIG, the clinical center 250 can communicate with a backend 260, which can be on a server (e.g., Figure 1 The clinical center 250 may include a server that can run on an imaging system (e.g., Figure 1 An ultrasound module 252 running on the imaging system 102 of the present invention, as well as a picture archiving and communication (PACS) system 254, a digital imaging and communications in medicine (DICOM) viewer 256 and a DICOM router 258.

[0062] The ultrasound module 252 may generate, receive, obtain, and / or store ultrasound images (e.g., image data such as motion video clips and image frames). The image data may be communicated from the ultrasound module 252 to a PACS system 254 and / or directly to an implant module 262 of a back end 260. The PACS system 254 may securely store the image data received from the ultrasound module 252. The image data stored in the PACS system 254 may be electronically tagged based on user selected input records. Once the image data is stored and / or tagged in the PACS system 254, the DICOM router 258 may connect to the PACS system 254 to retrieve the image data, and may also connect to the back end 260, which may be connected to a server (e.g., Figure 1 For example, the DICOM router 258 can be connected to the implementation module 262 and can send image data to the implementation module 262.

[0063] In one example, the DICOM router 258 can pseudonymize the files so that only pseudonymized files are sent to the backend 260. For example, all patient information except certain necessary variables (e.g., gestational age) can be removed, and a pseudonym identifier can be added to the file for the examination and / or for the respective records. Once the DICOM router 258 receives the output from the backend 260, it can then perform re-identification by replacing the pseudonym identifier with the patient information. The implementation module 262 can upload the image data to the storage device 264. For example, the storage device 264 can store encrypted and otherwise secured image data.

[0064] The implementation module 262 may retrieve certain image data from the storage device 264 and may communicate this image data to the analysis module 266. The analysis module 266 may process the image data using a machine learning algorithm to identify the presence, absence, or uncertainty of the presence or absence of one or more CHDs and / or cardiovascular malformations in the image data, and / or key points and / or contours that may indicate fetal anatomy. For example, the analysis module 266 may run a machine learning algorithm on Figure 2A The analysis module 266 may execute one or more modules or models described in the backend 260 of the embodiment. For example, the analysis module 266 may execute the spatial model 222, the temporal model 224, and / or the fuser 230.

[0065] The results and / or output of the analysis module 266 may be stored in the storage device 264. The results and / or output (e.g., Figure 2AThe spatiotemporal output 232 and / or analyzed output 236) and any reports (e.g., report 248) can be communicated back to the DICOM router 258 and stored in the PACS 254. Once stored in the PACS 254, a healthcare provider (e.g., a physician) using a DICOM viewer 256 can access the results and / or output from the PACS 254 and view the results and / or output (e.g., using a healthcare provider device).

[0066] Now refer to Figures 3A to 3C , illustrating a spatiotemporal neural network (e.g., CNN). Now refer to Figure 3A , illustrating the spatiotemporal CNN system 300. The spatiotemporal CNN system 300 may be a single CNN with a dual-stream architecture, or may be an independent CNN. The spatiotemporal CNN system 300 may be coupled with a Figure 2A The CNN system used by the backend 208 is the same or similar. Figure 3A As shown in , a spatiotemporal CNN system 300 may include a spatial stream 306 and a temporal stream 308, which may be parallel streams that may be combined at fusion 310.

[0067] like Figure 3A As shown in FIG, image data 302 may be input into and processed by spatial stream 306, and optical flow data 304 may be input into and processed by temporal stream 308. Spatial stream 306 and temporal stream 308 may be streams in different CNNs or may be streams in the same CNN. Figure 2A The optical flow data 220 may be the same as or similar to the image data 206, the pre-processed image data 212, and / or the sampled image data 216. Figure 2A The optical flow data 220 are the same or similar.

[0068] The spatial stream 306 may receive a single image frame of the image data 302, and the temporal stream 306 may receive fixed-size groups of optical flow data 304. For example, the single frame of the image data 302 may include RGB pixel information and / or the fixed-size groups of the optical flow data 304 may include fixed-size maps and / or graphs of the optical flow data 304. The spatial stream 306 may process the image data 302 concurrently with the temporal stream 306 processing the optical flow data 304. The optical flow data processed by the temporal stream 308 may correspond to or be based on the image data processed by the spatial stream.

[0069] In the case where the CNN system 300 includes multiple CNNs, the spatial stream 306 may include one or more spatial CNNs, such as spatial CNNs trained for image processing. The spatial CNN may include one or more neural networks (e.g., CNNs) trained to analyze image data (e.g., RGB pixel data) generally (e.g., not specific to medical imaging) and / or one or more neural networks trained to analyze image data in medical imaging (e.g., ultrasound images). For example, the spatial CNN may be trained to analyze ultrasound image data (e.g., RGB pixel data) to determine the likelihood of the presence or absence of one or more CHDs and / or other cardiovascular malformations in each frame and / or the likelihood of a certain view corresponding to the orientation of the image data.

[0070] The temporal stream 308 may include one or more temporal CNNs, such as those trained for image processing and / or trained for processing optical flow data to generate temporal outputs. For example, the temporal CNN may generate a temporal output that may indicate, for each optical flow data set, the presence of one or more CHDs and / or other cardiovascular malformations and / or the likelihood of corresponding to a certain view or orientation of the optical flow data.

[0071] Fusion 310 can combine the architectures and / or outputs of the architectures of spatial stream 306 and temporal stream 308 to produce a spatiotemporal output 312. Spatial stream 306 and temporal stream 308 can be fused at one or more levels. As shown in FIG3 , late fusion can be used so that the outputs from two CNNs and / or two streams are combined to produce a single fused output, such as a single spatiotemporal representation that indicates the likelihood of the presence or absence of one or more CHDs and / or other cardiovascular malformations or the uncertainty of such presence or absence, the likelihood of corresponding to a certain view or orientation of the image data, and / or can indicate key points and / or contours of fetal anatomy. In one example, late fusion is one of a sum fusion method, a maximum fusion method, a splicing fusion method, a convolution fusion method, or a bilinear fusion method.

[0072] It should be understood that this can be extended by stacking filters and dividing the weights in the time dimension Figure 3A The two-dimensional CNN described in

[15] takes multiple images (e.g., multiple frames) as input instead of a single image. For example, for K image frames, the filters can be stacked K times in the time dimension, and the weights can be divided by K. Although Figure 3A The two streams in are illustrated as parallel streams, but alternatively, the temporal stream 308 may take the output of the spatial stream 306 as input to the temporal stream 308. It should be further understood that other representations may be determined and / or processed along with the spatial and temporal representations.

[0073] Now refer to Figures 3B to 3C, the spatiotemporal neural network has a fusion neural network. For example, the neural network system (e.g., CNN system 330 and 340) may include fusion CNN 336 and 344, respectively, which may be similar to Figure 3A The fusion 310 of FIG. 310 may be a CNN trained to output this spatiotemporal representation based on the output from the spatial and temporal streams 308, but may be a separate neural network. For example, the fusion CNN 336 and / or the fusion CNN 334 may be a CNN trained to output this spatiotemporal representation based on the output from the spatial and temporal streams 308.

[0074] like Figure 3B , the spatial stream 306 may be a CNN 332, which may be one or more spatial CNNs trained for image processing and / or trained to analyze image data (e.g., RGB pixel data) to determine in each frame the likelihood of the presence of one or more CHDs and / or other cardiovascular malformations and / or the likelihood of a certain view corresponding to an orientation of the image data, the likelihood of a certain view or orientation corresponding to the image data, and / or key points and / or contours indicative of anatomical structures of the fetus. Similarly, the temporal stream 308 may be a CNN 334, which may be one or more temporal CNNs, such as a temporal CNN trained for image processing and / or trained to process optical flow data to generate a temporal output. The outputs of the spatial stream 306 and the temporal stream 308 may be input into a fused CNN 336, which may be one or more CNNs trained to output a spatiotemporal representation of the spatial output and the temporal output. As shown in FIG. Figure 3B As shown in , each of the spatial stream 306 and the temporal stream 308 may be an independent CNN, and together with the fused CNN 336 may total three or more neural networks (e.g., three or more CNNs).

[0075] like Figure 3C As shown in , the spatial stream 306 and the temporal stream 308 may be included in a CNN 342, which may be one or more CNNs having streams trained to analyze image data (e.g., RGB pixel data) to determine in each frame the likelihood of the presence of one or more CHDs and / or other cardiovascular malformations, the likelihood of a certain view corresponding to the orientation of the image data, and / or key points and / or contours indicating the anatomy of the fetus. The outputs of the spatial stream 306 and the temporal stream 308 may be input to a fused CNN 344, which may be one or more CNNs trained to output a spatiotemporal representation of the spatial and temporal streams. Figure 3C As shown in , each of the spatial stream 306 and the temporal stream 308 may be included in a CNN 342, which together with the fused CNN 336 may amount to two or more neural networks (e.g., two or more CNNs).

[0076] Now refer to Figure 4 , depicts a process flow for generating a spatiotemporal output indicating the likelihood of CHD and / or other cardiovascular malformations and / or indicating the likelihood of a certain view of an orientation of an imaging device (e.g., an ultrasound sensor). Some or all of the blocks of the process flow in this disclosure may span any number of devices (e.g., Figure 1 The processes may be executed in a distributed manner (eg, by a server 104, a computing device, an imaging or sensor device, or the like). Some or all of the operations of the process flow may be optional and may be performed in a different order.

[0077] At block 402, computer executable instructions stored on a memory of a device, such as a server, may be executed to determine image data. For example, the image data may be associated with Figure 2A The image data 202 may be the same or similar to the image data 202 of FIG. 4 and may include still frame images and / or video clips. At optional block 404, computer-executable instructions stored on a memory of a device, such as a server, may be executed to pre-process the image data (e.g., focus, resize, and / or crop the image data), as described with respect to FIG. Figure 2A

[0046] As described above with respect to the preprocessor 210 and preprocessed image data 212 of FIG. Additionally or alternatively, at block 404, spatial, temporal, and / or spatiotemporal filters may be used to remove noise.

[0078] At optional block 406, computer executable instructions stored on a memory of a device, such as a server, may be executed to determine sample image data, such as with respect to Figure 2A

[0046] As described above with respect to the sample generator 214 and the sampled image data 216 of FIG. 40, at optional block 408, computer-executable instructions stored on a memory of a device, such as a server, may be executed to create and train a spatial model. For example, a large image set may be used to train a CNN for image processing, detection, and / or recognition. For example, images from everyday life (e.g., cars, bicycles, apples, etc.) may be used to generally train a CNN for image recognition.

[0079] Additionally or alternatively, a specific dataset corresponding to cardiovascular anatomy (including with and / or without CHD and / or malformations) can be used to train or fine-tune the CNN to ultimately identify CHD and / or cardiovascular malformations in input image data. The network can be further trained to recognize image views, angles, and / or orientations. For example, echocardiographers can consistently generate standardized views, angles, or a certain anatomical structure, and the CNN can be trained to recognize such views, angles, and / or orientations. It should be understood that the images and data used for training purposes can be different and / or can be from different patients than the image data input into the trained CNN.

[0080] At block 410, computer executable instructions stored on a memory of a device, such as a server, may be executed to process image data using the trained spatial model. The processed image data may be pre-processed and / or sampled imaging data. At block 412, computer executable instructions stored on a memory of a device, such as a server, may be executed to generate a spatial output using the image data and the trained spatial model. The spatial output may be Figure 2A The spatial output 226 is the same or similar.

[0081] At block 414, computer executable instructions stored on a memory of a device, such as a server, may be executed to determine the Figure 2A 4 and 416. The optical flow generator 218 and the optical flow data 220 may be configured to generate an optical flow image. It should be understood that blocks 414 through 420 may be performed simultaneously or nearly simultaneously with blocks 406 through 412. At optional block 416, computer-executable instructions stored on a memory of a device, such as a server, may be executed similarly to optional block 408 to train a temporal model using the image data. It should be understood that optional block 416 and optional block 408 may occur simultaneously and / or the spatial and temporal streams may be trained together, such that optional block 408 and optional block 416 may be the same step. Additionally or alternatively, the temporal model may be trained using the optical flow data to ultimately identify CHD and / or cardiovascular malformations in the optical flow data and / or identify image views, angles, and / or orientations in the optical flow data.

[0082] At block 418, computer executable instructions stored on a memory of a device, such as a server, may be executed to process the optical flow data using the trained temporal model. At block 420, computer executable instructions stored on a memory of a device, such as a server, may be executed to generate a temporal output using the optical flow data and the trained temporal model. The temporal output may be compared to Figure 2A The temporal output 228 is the same or similar. At block 422, a fusion may be performed on the temporal output and the spatial output to determine a spatiotemporal output, such as Figure 2A The fuser 230 and the spatiotemporal output 232 are described.

[0083] Now refer to Figure 5A and 5B , depicting a process flow for determining the presence of CHD and / or cardiovascular malformations in a data stream. Figures 5A to 5B Available in Figure 4 Some or all of the blocks of the process flow in this disclosure may span any number of devices (e.g., Figure 1 The processes may be executed in a distributed manner (eg, by a server 104, a computing device, an imaging or sensor device, or the like). Some or all of the operations of the process flow may be optional and may be performed in a different order.

[0084] Now refer to Figure 5A At block 504, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine a likelihood of one or more CHDs and / or cardiovascular malformations for each sampled image data and / or each frame or video segment input into the spatiotemporal CNN. For example, each output may include a likelihood of CHD and / or cardiovascular malformation, and each output may correspond to a frame of image data and / or video segment (e.g., multiple frames of image data).

[0085] At block 506, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine an average probability of CHD and / or cardiovascular malformation based on the probability of CHD and / or cardiovascular malformation for each sampled image data. For example, the probability of each CHD and / or cardiovascular malformation in each output may be averaged. It should be understood that other types of aggregation, modeling, and / or filtering calculations may alternatively or additionally be used in addition to averaging. For example, the system may determine the highest probability detected and may use that value for further processing and / or analysis. Alternatively or additionally, key points and / or contours of the fetal anatomy may be determined.

[0086] At decision 508, computer-executable instructions stored on a memory of a device, such as a server, may be executed to compare the average likelihood of CHD and / or cardiovascular malformation to a threshold. For example, the threshold may be 51%, 75%, 90%, 99%, or any other threshold. If the threshold is not met by any of the averages (e.g., each average is below the threshold), then at block 510, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that CHD and / or cardiovascular malformation is not present.

[0087] Alternatively, if a threshold is met for one or more CHD and / or cardiovascular malformations, then at block 510, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the presence of CHD and / or cardiovascular defects corresponding to an average value that meets the threshold. For example, the spatiotemporal output may be a vector or matrix containing a number of likelihood values ​​between 0 and 1, each likelihood value corresponding to a different CHD and / or cardiovascular malformation, with values ​​above a threshold (e.g., 0.9) being determined to be present. It may be desirable to set different thresholds for different abnormalities, conditions, morphological abnormalities, disorders, and the like.

[0088] Now refer to Figure 5B, illustrating an alternative or additional process flow for determining the presence of CHD and / or cardiovascular malformations in image data. At block 520, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine a likelihood of one or more views, each corresponding to sampled data and / or other image data input to the network. A view value may correspond to a likelihood of the presence of one or more views, angles, and / or orientations corresponding to each frame and / or video clip of the image data. For example, a view value may be between 0 and 1.

[0089] At block 522, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine that certain view values ​​satisfy a view threshold. For example, the view threshold may be any value, such as 51%, 75%, 90%, 99%, etc. In one example, it may be determined that if the view value is greater than 0.9, then there is a high likelihood or confidence that the associated image data corresponds to a certain view.

[0090] At block 524, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine a likelihood of the presence of CHD and / or cardiovascular malformations for outputs having a view value that satisfies a threshold. Alternatively or additionally, key points and / or contours of the fetal anatomy may be determined. At decision 526, computer-executable instructions stored on a memory of a device, such as a server, may be executed to compare each likelihood of CHD and / or cardiovascular malformation corresponding to outputs having a view threshold that satisfies a threshold to a defect threshold. For example, the defect threshold may be 51%, 75%, 90%, 99%, or any other threshold. If the threshold is not satisfied by any of the average values ​​(e.g., all average values ​​are below the threshold), then at block 528, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the absence of CHD and / or cardiovascular malformations.

[0091] If the defect threshold is not met by any value (e.g., all values ​​are below the defect threshold), then at block 528, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the absence of CHD and / or cardiovascular malformations. Alternatively, if the defect threshold is met for one or more CHD and / or cardiovascular malformations, then at block 530, computer-executable instructions stored on a memory of a device, such as a server, may be executed to determine the presence of CHD and / or cardiovascular malformations corresponding to values ​​above the defect threshold.

[0092] Now refer to Figure 6 , which illustrates a schematic block diagram of the server 600. The server 600 can be connected to Figure 1 Server 104 or otherwise Figures 1 to 5BIt should be understood that the imaging system, analysis device and / or data storage device may additionally or alternatively include Figure 6 One or more of the components described in , and the server 600 can perform one or more of the operations of the server 600 described herein, alone or in conjunction with any of the foregoing.

[0093] The server 600 may be designed to communicate with one or more servers, imaging systems, analysis devices, data storage devices, other systems, or the like. The server 600 may be designed to communicate via one or more networks. Such networks may include, but are not limited to, any one or more different types of communication networks, such as, for example, a cable network, a public network (e.g., the Internet), a private network (e.g., a frame relay network), a wireless network, a cellular network, a telephone network (e.g., a public switched telephone network), or any other suitable private or public packet-switched or circuit-switched network.

[0094] In an illustrative configuration, the server 600 may include one or more processors 602, one or more memory devices 604 (also referred to herein as memory 604), one or more input / output (I / O) interfaces 606, one or more network interfaces 608, one or more transceivers 610, one or more antennas 634, and a data storage device 620. The server 600 may further include one or more buses 618 that functionally couple the various components of the server 600.

[0095] The bus 618 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may allow information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between the various components of the server 600. The bus 618 may include, but is not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, etc. The bus 618 may be associated with any suitable bus architecture.

[0096] The memory 604 may include volatile memory (memory that maintains its state when powered) such as random access memory (RAM) and / or non-volatile memory (memory that maintains its state even when not powered) such as read-only memory (ROM), flash memory, ferroelectric RAM (FRAM), etc. As the term is used herein, a persistent data storage device may include non-volatile memory. In various embodiments, the memory 604 may include a variety of different types of memory, such as various types of static random access memory (SRAM), various types of dynamic random access memory (DRAM), various types of unchangeable ROM and / or writable variants of ROM such as electrically erasable programmable read-only memory (EEPROM), flash memory, etc.

[0097] Data storage device 620 may include removable and / or non-removable storage devices, including but not limited to magnetic storage devices, optical disk storage devices, and / or tape storage devices. Data storage device 620 may provide non-volatile storage of computer-executable instructions and other data. Memory 604 and data storage device 620 (removable and / or non-removable) are examples of the term computer-readable storage medium (CRSM) as used herein. Data storage device 620 may store computer-executable code, instructions, or the like, which may be loaded into memory 604 and executed by processor 602 to cause processor 602 to perform or initiate various operations. Data storage device 620 may also store data that may be copied to memory 604 for use by processor 602 during execution of the computer-executable instructions. Furthermore, output data generated as a result of execution of the computer-executable instructions by processor 602 may initially be stored in memory 604 and may ultimately be copied to data storage device 620 for non-volatile storage.

[0098] The data storage device 620 may store one or more operating systems (O / S) 622; one or more optional database management systems (DBMS) 624; and one or more program modules, applications, engines, computer executable code, scripts, or the like, such as, for example, one or more implementation modules 626, an image processing module 627, a communication module 628, an optical flow module 629, and / or a spatiotemporal CNN module. Some or all of these modules may be sub-modules. Any component depicted as stored in the data storage device 620 may include any combination of software, firmware, and / or hardware. The software and / or firmware may include computer executable code, instructions, or the like that can be loaded into the memory 604 for execution by one or more of the processors 602. Any component depicted as stored in the data storage device 620 may support the functionality described with reference to the correspondingly named components previously in this disclosure.

[0099] Referring now to other illustrative components depicted as being stored in data storage device 620, O / S 622 can be loaded from data storage device 620 into memory 604 and can provide an interface between other application software executing on server 600 and the hardware resources of server 600. More specifically, O / S 622 can include a set of computer-executable instructions for managing the hardware resources of server 600 and for providing common services to other applications (e.g., managing memory allocation among various applications). In certain example embodiments, O / S 622 can control the execution of other program modules for content rendering. O / S 622 can include any operating system now known or that may be developed in the future, including but not limited to any server operating system, any host operating system, or any other proprietary or non-proprietary operating system.

[0100] An optional DBMS 624 can be loaded into memory 604 and can support functionality for accessing, retrieving, storing, and / or manipulating data stored in memory 604 and / or data stored in data storage 620. The DBMS 624 can use any of a variety of database models (e.g., a relational model, an object model, etc.) and can support any of a variety of query languages. The DBMS 624 can access data represented in one or more data schemas and stored in any suitable data repository, including, but not limited to, a database (e.g., a relational database, an object-oriented database, etc.), a file system, a flat file, a distributed data storage device in which data is stored on more than one node of a computer network, a peer-to-peer network data storage device, or the like.

[0101] Optional input / output (I / O) interface 606 can facilitate receiving input information from one or more I / O devices by server 600 and outputting information from server 600 to one or more I / O devices. I / O devices can include any of a variety of components, such as a display or screen with a touch surface or touch screen; an audio output device for producing sound, such as a speaker; an audio capture device, such as a microphone; an image and / or video capture device, such as a camera; etc. Any of these components can be integrated into server 600 or can be separate.

[0102] The server 600 may further include one or more network interfaces 608 via which the server 600 can communicate with any of a variety of other systems, platforms, networks, devices, etc. The network interface 608 may enable, for example, communication via one or more of the networks with one or more wireless routers, one or more host servers, one or more web servers, and the like.

[0103] The antenna 634 may include any suitable type of antenna, depending on, for example, the communication protocol used to transmit or receive signals via the antenna 634. Non-limiting examples of suitable antennas may include directional antennas, non-directional antennas, dipole antennas, folded dipole antennas, patch antennas, multiple-input, multiple-output (MIMO) antennas, or the like. The antenna 634 may be communicatively coupled to one or more transceivers 612 or radio components, to which signals may be transmitted or received. The antenna 634 may include, but is not limited to, a cellular antenna for transmitting or receiving signals to or from a cellular network infrastructure, an antenna for transmitting or receiving Wi-Fi signals to or from an access point (AP), a Global Navigation Satellite System (GNSS) antenna for receiving GNSS signals from a GNSS satellite, a Bluetooth antenna for transmitting or receiving Bluetooth signals, including BLE signals, a Near Field Communication (NFC) antenna for transmitting or receiving NFC signals, a 900 MHz antenna, and the like.

[0104] The transceiver 612 may include any suitable radio components for cooperating with the antenna 634 to transmit or receive radio frequency (RF) signals in a bandwidth and / or channel corresponding to the communication protocol utilized by the server 600 to communicate with other devices. The transceiver 612 may include hardware, software, and / or firmware for modulating, transmitting, or receiving communication signals according to any of the communication protocols discussed above (possibly in conjunction with any of the antennas 634), including but not limited to one or more Wi-Fi and / or Wi-Fi Direct protocols standardized by the IEEE 802.11 standard, one or more non-Wi-Fi protocols, or one or more cellular communication protocols or standards. The transceiver 612 may further include hardware, firmware, or software for receiving GNSS signals. The transceiver 612 may include any known receiver and baseband suitable for communicating via the communication protocol utilized by the server 600. The transceiver 612 may further include a low noise amplifier (LNA), additional signal amplifiers, an analog-to-digital (A / D) converter, one or more buffers, a digital baseband, or the like.

[0105] Now refer to Figure 6 6. The implementation module 626 may include computer-executable instructions, codes, or the like that, in response to execution of executable functions by one or more of the processors 602, include, but are not limited to, supervising the coordination and interaction between one or more modules in the data storage device 620 and the computer-executable instructions, determining user-selected actions and tasks, determining actions associated with user interactions, determining actions associated with user input, initiating commands locally or at a remote device, and the like.

[0106] The imaging processing module 627 may include computer-executable instructions, code, or the like that, in response to execution of executable functions by one or more of the processors 602, include, but are not limited to, analyzing and processing image data (e.g., still frames and / or video clips) and cropping, segmenting, parsing, sampling, resizing, and / or altering the image data.

[0107] The communication module 628 may include computer-executable instructions, code, or the like that are responsive to execution of executable functions by one or more of the processors 602, including, but not limited to, for example, communicating with one or more devices via wired or wireless communications, communicating with a server (e.g., a remote server), communicating with a data storage device and / or database, communicating with an imaging system and / or analysis device, sending or receiving notifications or commands / instructions, communicating with cache memory data, communicating with a computing device, and the like.

[0108] The optical flow module 629 may include computer-executable instructions, code, or the like that, in response to execution of executable functions by one or more of the processors 602, include, but are not limited to, generating optical flow data (including horizontal and vertical optical flow data), optical flow graphs and / or representations, and other optical flow information from image data.

[0109] The spatiotemporal CNN module 630 may include computer-executable instructions, code, or the like that, in response to execution of executable functions by one or more of the processors 602 , include, but are not limited to, generating, running, and executing one or more spatiotemporal CNNs, including one or more spatial CNNs and one or more temporal CNNs.

[0110] While specific embodiments of the present disclosure have been described, those skilled in the art will recognize that many other modifications and alternative embodiments are within the scope of the present disclosure. For example, any functionality and / or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Furthermore, while various illustrative implementations and architectures have been described in accordance with embodiments of the present disclosure, those skilled in the art will recognize that many other modifications to the illustrative implementations and architectures described herein are within the scope of the present disclosure.

[0111] Certain aspects of the present disclosure are described above with respect to block diagrams and flow charts of systems, methods, apparatus, and / or computer program products according to example embodiments. It should be understood that one or more blocks in the block diagrams and flow charts, and combinations of blocks in the block diagrams and flow charts, respectively, may be implemented by the execution of computer-executable program instructions. Similarly, according to some embodiments, some blocks in the block diagrams and flow charts may not necessarily need to be executed in the order presented, or may not need to be executed at all. Furthermore, in some embodiments, there may be additional components and / or operations beyond those depicted in the blocks of the block diagrams and / or flow charts.

[0112] Therefore, the blocks of the block diagrams and flowcharts support combinations of means for performing the specified functions, combinations of elements or steps for performing the specified functions, and program instruction means for performing the specified functions. It should also be understood that each block in the block diagrams and flowcharts, and combinations of blocks in the block diagrams and flowcharts, can be implemented by a special-purpose hardware-based computer system that performs the specified functions, elements or steps, or a combination of special-purpose hardware and computer instructions.

[0113] The program modules, applications, or the like disclosed herein may include one or more software components, including, for example, software objects, methods, data structures, or the like. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0114] Software components can be coded in any of a variety of programming languages. The declarative programming language can be a lower-level programming language, such as assembly language, that is associated with a specific hardware architecture and / or operating system platform. Software components containing assembly language instructions may need to be converted into executable machine code by an assembler before being executed by the hardware architecture and / or platform.

[0115] Another example programming language may be a higher-level programming language that is portable across multiple architectures. Software components that include higher-level programming language instructions may need to be converted into an intermediate representation by an interpreter or compiler before execution.

[0116] Other examples of programming languages ​​include, but are not limited to, macro languages, shell or command languages, job control languages, scripting languages, database query or search languages, or report writing languages. In one or more example embodiments, a software component including instructions in one of the foregoing examples of a programming language may be directly executed by an operating system or other software component without first being converted to another form.

[0117] Software components can be stored as files or other data storage structures. Software components of similar type or related functions can be stored together, such as, for example, in a specific directory, folder, or library. Software components can be static (e.g., pre-built or fixed) or dynamic (e.g., created or modified at execution time).

[0118] Software components can call other software components or be called by other software components through any of a variety of mechanisms. The called or calling software components may include other custom-developed application software, operating system functionality (e.g., device drivers, data storage (e.g., file management) routines, other common routines and services, etc.), or third-party software components (e.g., middleware, encryption or other security software, database management software, file transfer or other network communication software, mathematical or statistical software, image processing software, and format conversion software).

[0119] The software components associated with a particular solution or system may reside and execute on a single platform, or may be distributed across multiple platforms. Multiple platforms may be associated with more than one hardware vendor, underlying chip technology, or operating system. Furthermore, the software components associated with a particular solution or system may be initially written in one or more programming languages ​​but may call software components written in another programming language.

[0120] Computer-executable program instructions can be loaded onto a special-purpose computer or other specific machine, processor or other programmable data processing device to produce a specific machine, so that execution of the instructions on the computer, processor or other programmable data processing device causes one or more functions or operations specified in the flowchart to be performed. These computer program instructions can also be stored in a computer-readable storage medium (CRSM), which, when executed, can direct the computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the computer-readable storage medium produce an article of manufacture containing instruction means that implement one or more functions or operations specified in the flowchart. The computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operating elements or steps to be executed on the computer or other programmable device to produce a computer-implemented process.

[0121] Additional types of CRSMs that may be present in any of the devices described herein may include, but are not limited to, programmable random access memory (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage device, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store information and can be accessed. Combinations of any of the foregoing are also included within the scope of a CRSM. Alternatively, a computer-readable communication medium (CRCM) may include computer-readable instructions, program modules, or other data transmitted within a data signal (e.g., a carrier wave or other transmission). However, as used herein, a CRSM does not include a CRCM.

[0122] Although embodiments have been described in language specific to structural features and / or method actions, it should be understood that the present disclosure is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as illustrative forms of implementing embodiments. Unless otherwise specifically stated, or otherwise understood within the context of use, conditional terms, in particular, for example, "may," "can," "might," or "could" are generally intended to convey that certain embodiments may include and other embodiments do not include certain features, elements, and / or steps. Therefore, such conditional terms are generally not intended to mean that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements, and / or steps are included in any particular embodiment or are to be performed in any particular embodiment with or without user input or prompting.

[0123] It should be understood that any computer operations described above herein may be implemented at least in part as computer-readable instructions stored on a computer-readable memory. Of course, it should be understood that the embodiments described herein are illustrative, and that components may be arranged, replaced, combined, and designed in a variety of different configurations, all of which are contemplated and fall within the scope of the present disclosure.

[0124] The foregoing description of illustrative embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or limiting with respect to the precise forms disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed embodiments. It is intended that the scope of the invention be defined by the following claims and their equivalents.

Claims

1. A method for determining the presence of one or more congenital heart defects (CHDs) in a patient, the method comprising: determining, by a server, first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the first image data, the optical flow data indicating movement of pixels in the series of image frames; processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks trained to process image data; processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks trained to process the optical flow data; generating a spatial output using the spatial model and based on the image data, the spatial output indicating a first likelihood of the presence of one or more CHDs of the patient; generating a temporal output using the temporal model and based on the plurality of optical flow data, the temporal output indicating a second likelihood of the presence of one or more CHDs of the patient; determining a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of one or more CHDs of the patient; and The first device is caused to display a user interface corresponding to the fused output.

2. The method of claim 1 , wherein the third likelihood of the presence of one or more CHDs in the patient comprises one or more of the following likelihoods of the presence of atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, heart size abnormalities, ventricular septal defect, atrioventricular junctional anomaly, left atrial posterior area anomaly, left ventricular junctional anomaly, aortic junctional anomaly, right ventricular junctional anomaly, pulmonary junctional anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size abnormality, transverse aortic arch size abnormality, or superior vena cava size abnormality.

3. The method according to claim 1, further comprising: comparing the fused output to a threshold; determining that the fused output satisfies the threshold; and The presence of the one or more CHDs of the patient is determined based on the fused output satisfying the threshold.

4. The method according to claim 1, further comprising: determining a request from a first device to generate a report corresponding to the fused output; and The first device is caused to generate the report corresponding to the fused output. 5 . The method of claim 1 , further comprising training the spatial model and the temporal model using a plurality of second image data different from the first image data. 6 . The method of claim 1 , further comprising removing at least a portion of the first image data from each of the image frames in the series of image frames. The method of claim 1 , further comprising receiving the first image data from an imaging system.

8. The method of claim 7, wherein the imaging system comprises an ultrasound or echocardiography device.

9. The method of claim 8, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound device or echocardiographic device and a second series of image frames corresponding to a second orientation of the ultrasound device or echocardiographic device.

10. The method of claim 1, further comprising sampling the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model.

11. A system for determining the presence of one or more congenital heart defects (CHDs) in a patient, the system comprising: a memory configured to store computer-executable instructions; and at least one computer processor configured to access the memory and execute the computer-executable instructions to: determining first image data representing a portion of the patient's cardiovascular system, the first image data comprising a series of image frames; determining optical flow data based on the image data, the optical flow data indicating movement of pixels in the series of image frames; generating a spatial output by processing the image data using a spatial model, the spatial model comprising one or more first convolutional neural networks, the spatial output indicating a first likelihood of the presence of one or more CHDs of the patient; generating a temporal output by processing the optical flow data using a temporal model, the temporal model comprising one or more second convolutional neural networks, the temporal output indicating a second likelihood of the presence of one or more CHDs of the patient; determining a fused output based on the spatial output and the temporal output, the fused output indicating a third likelihood of the presence of one or more CHDs of the patient; and The first device is caused to display a user interface corresponding to the fused output.

12. The system of claim 11 , wherein the third likelihood of the presence of one or more CHDs in the patient comprises one or more of the following likelihoods of the presence of atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, heart size abnormalities, ventricular septal defect, atrioventricular junctional anomaly, left atrial posterior area anomaly, left ventricular junctional anomaly, aortic junctional anomaly, right ventricular junctional anomaly, pulmonary junctional anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size abnormality, transverse aortic arch size abnormality, or superior vena cava size abnormality.

13. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to: comparing the fused output to a threshold; determining that the fused output satisfies the threshold; and The presence of the one or more CHDs of the patient is determined based on the fused output satisfying the threshold.

14. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to: determining a request from a first device to generate a report corresponding to the fused output; and The first device is caused to generate the report corresponding to the fused output.

15. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data.

16. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to remove at least a portion of the first image data from each of the image frames in the series of image frames.

17. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to receive the first image data from an imaging system.

18. The system of claim 17, wherein the imaging system comprises an ultrasound or echocardiography device.

19. The system of claim 18, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound device or echocardiographic device and a second series of image frames corresponding to a second orientation of the ultrasound device or echocardiographic device.

20. The system of claim 11, wherein the computer processor is further configured to execute the computer-executable instructions to sample the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model.

21. The system of claim 11, wherein the patient is a fetus during pregnancy.

22. The system of claim 11, wherein the spatial output comprises a matrix of values ​​indicative of view orientations in corresponding image frames in the series of image frames.

23. The system of claim 1, wherein the computer processor is further configured to determine the fused output using late fusion.

24. The system of claim 23, wherein late fusion is one of a sum fusion method, a maximum fusion method, a concatenation fusion method, a regular fusion method, or a bilinear fusion method.

25. A method for determining the presence of one or more congenital heart defects (CHDs) in a fetus during pregnancy, the method comprising: determining, by a computing device, image data representing a portion of a cardiovascular system of the fetus, the image data comprising a series of image frames; determining a neural network system comprising a spatial model trained to process image data and an optical model trained to process optical flow data corresponding to the image data; determining a spatial output based on the image data using the spatial model, the spatial output corresponding to a first likelihood of the presence of one or more CHDs of the fetus, the spatial output comprising a matrix of values ​​indicating view orientations in respective image frames in the series of image frames; determining a temporal output using the temporal model and based on the image data, the temporal output corresponding to a second likelihood of the presence of one or more CHDs of a patient; determining a third likelihood of the presence of one or more CHDs in the patient based on the spatial output and the temporal output; and The first device is caused to display a user interface corresponding to the third possibility of the existence of one or more CHDs.

26. The method of claim 25, wherein the third likelihood of the presence of one or more CHDs in the patient comprises one or more of the following likelihoods of the presence of: overriding aorta, septal defect at the cardiac junction, parallel great arteries, increased cardiothoracic ratio, right ventricular to left ventricular size discrepancy, tricuspid to mitral valve annulus size discrepancy, pulmonary to aortic valve annulus size discrepancy, outflow tract relationship abnormality, cardiac axis deviation, atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, heart size abnormality, ventricular septal defect, atrioventricular junctional anomaly, left atrial posterior region anomaly, left ventricular junctional anomaly, aortic junctional anomaly, right ventricular junctional anomaly, pulmonary junctional anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size abnormality, transverse aortic arch size abnormality, or superior vena cava size abnormality.

27. The method of claim 25, further comprising: comparing the third likelihood of the presence of one or more CHDs to a threshold value; determining that the third likelihood of the presence of one or more CHDs satisfies the threshold; and The presence of the one or more CHDs in the patient is determined based on the third likelihood of the presence of one or more CHDs satisfying the threshold.

28. The method of claim 25, further comprising: determining a request from the first device to generate a report corresponding to the third likelihood of the presence of one or more CHDs; and The first device is caused to generate the report corresponding to the third likelihood of the presence of one or more CHDs.

29. The method of claim 25, further comprising training the spatial model and the temporal model using a plurality of second image data different from the image data.

30. The method of claim 25, further comprising removing at least a portion of the image data from each of the image frames in the series of image frames.

31. The method of claim 25, further comprising receiving the image data from an imaging system.

32. The method of claim 31 , wherein the imaging system comprises an ultrasound or echocardiography device.

33. The method of claim 33, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound device or echocardiographic device and a second series of image frames corresponding to a second orientation of the ultrasound device or echocardiographic device.

34. The method of claim 25, further comprising sampling the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model.

35. The method of claim 25, wherein one or more of the spatial outputs indicates one or more of keypoint data or silhouette data.

36. The method of claim 25, wherein one or more of the temporal outputs indicates one or more of keypoint data or silhouette data.

37. The method of claim 25, further comprising determining one or more of keypoint data or silhouette data based on the spatial output and the temporal output.

38. The method of claim 37, further comprising causing the first device to further display the one or more of keypoint data or outline data.

39. A system for determining the presence of one or more congenital heart defects (CHDs) in a fetus during pregnancy, the system comprising: a memory configured to store computer-executable instructions; and at least one computer processor configured to access the memory and execute the computer-executable instructions to: determining, by a computing device, image data representing a portion of a cardiovascular system of the fetus, the image data comprising a series of image frames; determining a neural network system comprising a spatial model trained to process image data and an optical model trained to process optical flow data corresponding to the image data; determining a spatial output based on the image data using the spatial model, the spatial output corresponding to a first likelihood of the presence of one or more CHDs of the fetus, the spatial output comprising a matrix of values ​​indicating view orientations in respective image frames in the series of image frames; determining a temporal output using the temporal model and based on the image data, the temporal output corresponding to a second likelihood of the presence of one or more CHDs of a patient; determining a third likelihood of the presence of one or more CHDs in the patient based on the spatial output and the temporal output; and The first device is caused to display a user interface corresponding to the third possibility of the existence of one or more CHDs.

40. The system of claim 39, wherein the third likelihood of the presence of one or more CHDs in the patient comprises one or more of the following likelihoods of the presence of: overriding aorta, septal defect at the cardiac junction, parallel great arteries, increased cardiothoracic ratio, right ventricular to left ventricular size discrepancy, tricuspid to mitral valve annulus size discrepancy, pulmonary to aortic valve annulus size discrepancy, outflow tract relationship abnormalities, cardiac axis deviation, atrial septal defect, atrioventricular septal defect, coarctation of the aorta, double outlet right ventricle, complete transposition of the great arteries, Ebstein anomaly, hypoplastic left heart syndrome, interrupted aortic arch, ventricular disproportion, heart size abnormalities, ventricular septal defect, atrioventricular junctional anomaly, left atrial posterior region anomaly, left ventricular junctional anomaly, aortic junctional anomaly, right ventricular junctional anomaly, pulmonary junctional anomaly, arterial size discrepancy, right aortic arch anomaly, pulmonary artery size abnormality, transverse aortic arch size abnormality, or superior vena cava size abnormality.

41. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to: comparing the third likelihood of the presence of one or more CHDs to a threshold value; determining that the third likelihood of the presence of one or more CHDs satisfies the threshold; and The presence of the one or more CHDs in the patient is determined based on the third likelihood of the presence of one or more CHDs satisfying the threshold.

42. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to: determining a request from the first device to generate a report corresponding to the third likelihood of the presence of one or more CHDs; The first device is caused to generate the report corresponding to the third likelihood of the presence of one or more CHDs.

43. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to train the spatial model and the temporal model using a plurality of second image data different from the first image data.

44. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to remove at least a portion of the image data from each of the image frames in the series of image frames.

45. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to receive the first image data from an imaging system.

46. ​​The system of claim 45, wherein the imaging system comprises an ultrasound or echocardiography device.

47. The system of claim 46, wherein the image data comprises a first series of image frames corresponding to a first orientation of the ultrasound device or echocardiography device and a second series of image frames corresponding to a second orientation of the ultrasound device or echocardiography device.

48. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to sample the image data such that only non-adjacent image frames in the series of image frames are processed by the spatial model.

49. The system of claim 39, wherein one or more of the spatial outputs indicates one or more of keypoint data or silhouette data.

50. The system of claim 39, wherein one or more of the temporal outputs indicates one or more of keypoint data or silhouette data.

51. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to determine one or more of keypoint data or silhouette data based on the spatial output and the temporal output.

52. The system of claim 39, wherein the computer processor is further configured to execute the computer-executable instructions to cause the first device to further display the one or more of keypoint data or outlines.

Images