Multiple pregnancy / gestation detection based on graphing and skeletonization of fetal anatomy detections from ultrasound imaging blind sweep protocol

Abstract

A system includes a processor that communicates with an ultrasound probe. The processor controls the ultrasound probe to obtain a series of ultrasound image frames during a blind sweep protocol on a patient with a pregnancy. The processor provides the ultrasound image frames as an input to a deep learning network trained to detect fetal anatomy. The processor generates, as an output of the deep learning network, detections of at least one fetal anatomical part within the ultrasound image frames. The processor places the detections into a 3D graph. The processor determines, using the 3D graph, at least one skeletonization. The processor, based on the at least one skeletonization, determines whether the pregnancy is a multiple gestation. The processor provides, to a display in communication with the processor, an output representative of the determination of whether the pregnancy is multiple gestation.

Description

MULTIPLE PREGNANY / GESTATION DETECTION BASED ON GRAPHING AND SKELETONIZATION OF FETAL ANATOMY DETECTIONS FROM ULTRASOUND IMAGING BLIND SWEEP PROTOCOLTECHNICAL FIELD

[0001] The subject matter described herein relates to devices, systems, and methods for using ultrasound data from a blind abdominal imaging sweep. In particular, detections of fetal anatomies can be graphed and skeletonized to detect multiple pregnancies / gestations.BACKGROUND

[0002] Ultrasound imaging is often used for diagnostic purposes in an office or hospital setting, but may also be used in resource-constrained care settings (e.g., homes, accident sites, ambulances, mobile health facilities, etc.) by emergency personnel, home health nurses, midwives, etc., who may lack ultrasound expertise. To facilitate ultrasound image acquisition by untrained or minimally trained users, a “blind sweep” protocol is often employed, in which the user follows pre-determined probe paths (e.g., sweeping out a pattern on the patient’s abdomen) during imaging.

[0003] Ultrasound imaging is a vital component of high-quality obstetric care. For example, detection of multiple pregnancies through ultrasound can allow for appropriate referral for delivery care in highly resourced centers with providers trained to handle any associated risks and complications. However, in rural and under-resourced communities, the scarcity of ultrasound imaging results in a considerable gap in the healthcare of pregnant mothers.

[0004] Multiple gestation can constitute a significant risk to both mother and fetuses, and is therefore clinically categorized as a type of high-risk pregnancy. Both diagnosis and management of these pregnancies are a challenge. Antepartum complications - including preterm labor, preterm premature rupture of the membranes, intrauterine growth restriction (IUGR), intrauterine fetal demise, gestational diabetes, and preeclampsia - develop in over 80% of multiple pregnancies as compared with approximately 25% of singleton gestation. While ultrasound can be very useful in identifying multiple gestation access to ultrasound may be out of reach in low-resource settings, where the expertise required to identify multiple pregnancies is also limited. Access and skill to perform the ultrasound in low-resource setting is therefore deficient in many parts of the world.

[0005] The information included in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be regarded as subject matter by which the scope of the disclosure is to be bound.SUMMARY

[0006] Disclosed is an ultrasound blind sweep multiple pregnancy detection system that, following a blind sweep protocol (sweeps at particular locations along body without trying to find specific anatomy), detects anatomical features in the captured images and uses them to identify patients who may need to be referred for evaluation by human experts. For example, the system may detect multiple pregnancy (e.g., twins, triplets, etc.), also known as multiple gestation. A pregnant patient can be referred to an obstetrician trained to deal with multiple pregnancy / gestation based on the output of the system. A deep learning network, such as a convolutional neural network, is used to detect anatomy of the pregnant patient or the one or multiple fetuses inside the patient’s uterus (e.g., fetal head, fetal heart, etc.). Graphing and skeletonization are then performed on these detections, placing them in a 3D context with respect to other related anatomies (e.g., a centerline including the head, heart, and abdomen of a fetus). The skeletonizations are then used to determine whether more than one fetus is present in the womb.

[0007] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0008] One general aspect includes a system. The system includes a processor configured for communication with an ultrasound probe, where the processor is configured to: control the ultrasound probe to obtain a plurality of ultrasound image frames during a blind sweep protocol on a patient with a pregnancy; provide the plurality of ultrasound image frames as an input to a deep learning network trained to detect fetal anatomy; generate, as an output of the deep learning network, detections of a plurality of detections of at least one fetal anatomical part within the plurality of ultrasound image frames; place the detections into a 3D graph; determine, using the 3D graph, at least one skeletonization; based on the at least one skeletonization, determine whether the pregnancy may include a multiple gestation; andprovide, to a display in communication with the processor, an output representative of the determination of whether the pregnancy may include the multiple gestation. Other examples of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0009] Implementations may include one or more of the following features. In some aspects, the processor is further configured to, based on the at least one skeletonization: determine an orientation of at least one fetus; and provide, to a display in communication with the processor, an output representative of the determination of the orientation of the at least one fetus. In some aspects, the processor is further configured to, based on the at least one skeletonization, update the deep learning network. In some aspects, the plurality of detections of at least one fetal anatomical part includes detections of at least a head, a heart, and an abdomen of at least one fetus. In some aspects, the determining the at least one skeletonization involves identifying a centerline between the head, heart, and abdomen of the at least one fetus. In some aspects, the processor is further configured to merge or split the at least one skeletonization based on anatomical context. In some aspects, the processor is further configured to merge or split the at least one skeletonization based on geometric positioning. In some aspects, the deep learning network is a multi-class anatomy detector. In some aspects, placing the detections into the 3D graph involves inertial measurement unit (IMU) data associated with movements of the ultrasound probe. In some aspects, determining whether the pregnancy may include a multiple gestation involves counting a number of skeletons formed by the at least one skeletonization. In some aspects, the system may include the ultrasound probe. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0010] One general aspect includes a method. The method includes, with a processor configured for communication with an ultrasound probe: controlling the ultrasound probe to obtain a plurality of ultrasound image frames during a blind sweep protocol on a patient with a pregnancy; providing the plurality of ultrasound image frames as an input to a deep learning network trained to detect fetal anatomy; generating, as an output of the deep learning network, a plurality of detections of at least one fetal anatomical part within the plurality of ultrasound image frames; placing the detections into a 3D graph; determining, using the 3D graph, at least one skeletonization; based on the at least one skeletonization, determining whether the pregnancy may include a multiple gestation; and providing, to a display in communication with the processor, an output representative of the determination of whetherthe pregnancy may include the multiple gestation. Other examples of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0011] Implementations may include one or more of the following features. In some aspects, the method may include, based on the at least one skeletonization: determining an orientation of at least one fetus; and providing, to a display in communication with the processor, an output representative of the determination of the orientation of the at least one fetus. In some aspects, the method may include, based on the at least one skeletonization, updating a training of the deep learning network. In some aspects, the plurality detections of at least one fetal anatomical part includes detections of at least a head, a heart, and an abdomen of at least one fetus. In some aspects, the determining the at least one skeletonization involves identifying a centerline through the head, heart, and abdomen of the at least one fetus. In some aspects, the method may include merging or splitting the at least one skeletonization based on anatomical context. In some aspects, the method may include merging or splitting the at least one skeletonization based on geometric positioning. In some aspects, the deep learning network is a multi-class anatomy detector. In some aspects, placing the detections into the 3D graph involves inertial measurement unit (IMU) data associated with movements of the ultrasound probe. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer- accessible medium.

[0012] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the ultrasound blind sweep multiple pregnancy detection system, as defined in the claims, is provided in the following written description of various aspects of the disclosure and illustrated in the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Illustrative aspects of the present disclosure will be described with reference to the accompanying drawings, of which:

[0014] Figure l is a schematic, diagrammatic representation of an ultrasound imaging system, according to aspects of the present disclosure.

[0015] Figure l is a schematic diagram of a processor circuit, according to aspects of the present disclosure.

[0016] Figure 3A is a set of schematic, diagrammatic, cross-sectional views of two fetuses surrounded by an amniotic sac and chorionic sac within a uterus of a patient, representing different types of twins, according to aspects of the present disclosure.

[0017] Figure 3B is a set of schematic, diagrammatic, cross-sectional views of two fetuses arranged in different positions within a uterus of a patient as seen by an ultrasound imaging plane, according to aspects of the present disclosure.

[0018] Figure 4 is a schematic, diagrammatic representation of a patient on whose abdomen the ultrasound blind sweep protocol will be followed, according to aspects of the present disclosure.

[0019] Figure 5 is a schematic, diagrammatic representation, in hybrid block diagram / flow diagram form, of an example ultrasound blind sweep multiple pregnancy detection system, according to aspects of the present disclosure.

[0020] Figure 6 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example ultrasound blind sweep multiple pregnancy detection system, according to aspects of the present disclosure.

[0021] Figure 7A is a schematic, diagrammatic overview, in block diagram form, of a training mode for an untrained neural network, according to aspects of the present disclosure.

[0022] Figure 7B is a schematic, diagrammatic overview, in block diagram form, of an inference mode or clinical usage mode for the trained neural network, according to aspects of the present disclosure.

[0023] Figure 8 is a schematic, diagrammatic illustration, in block diagram form, of the detection of anatomy (e.g., the head, heart, abdomen, pelvis, placenta, membrane, etc.), according to aspects of the present disclosure.

[0024] Figure 9 is a graph indicating, on the Y-axis, the number of higher trimester anatomies detected vs. the number of lower trimester anatomies detected and, on the X-axis, the gestational age in weeks, according to aspects of the present disclosure.

[0025] Figure 10 is a graph indicating, on the Y-axis, the number of higher trimester anatomies detected vs. the number of lower trimester anatomies detected and, on the X-axis, the gestational age in weeks, according to aspects of the present disclosure.

[0026] Figure 11 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example ultrasound blind sweep multiple pregnancy detection system, according to aspects of the present disclosure.

[0027] Figure 12 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example individual sweep processing module, according to aspects of the present disclosure.

[0028] Figure 13 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example inter-sweep processing module, according to aspects of the present disclosure.

[0029] Figure 14 is a 3D graph of head detections, heart detections, and abdomen detections, according to aspects of the present disclosure.

[0030] Figure 15 is an ultrasound image frame showing a fetal head, according to aspects of the present disclosure.

[0031] Figure 16 is a 3D graph of head detections for a single fetus, according to aspects of the present disclosure.

[0032] Figure 17 is a pair of ultrasound image frames from a single sweep, each showing a different fetal head, according to aspects of the present disclosure.

[0033] Figure 18 is a 3D graph of head detections for two fetuses, according to aspects of the present disclosure.DETAILED DESCRIPTION

[0034] In accordance with at least one aspect of the present disclosure, an ultrasound blind sweep multiple pregnancy detection system is provided which can identify anatomy of interest such as anatomy indicating two or more fetuses are present - based on a blind sweep protocol with an ultrasound imaging probe. The ultrasound blind sweep multiple pregnancy detection system presents a novel approach to imaging quality control by ensuring that features such as multiple leads, multiple spines, etc. are detected, and using the detected locations of these features to, for example, determine whether the pregnancy is multiple and thus potentially in need of expert care.

[0035] Ultrasound imaging is a vital component of high-quality obstetric care. In rural and under-resourced communities, the scarcity of ultrasound imaging results in a considerable gap in healthcare for pregnant mothers. Increased detection of pregnancy complications through ultrasound can allow for appropriate referral for delivery care in more- resourced centers with more highly trained providers. The present disclosure seeks to overcome this barrier to ultrasound access in a locally sustainable and resource-conscious way, using standardized blind-sweep scanning protocols combined with artificial intelligence, obviating the need for an interpreting provider (e.g., a radiologist or obstetrician) and an experienced sonographer in the remote location.

[0036] Monitoring intrauterine structures is essential for normal fetal development and perinatal outcome. Women with multiple pregnancy / multiple gestation may be at increased risk of maternal, fetal and postnatal adverse outcomes. The present disclosure provides a computer assisted simple triaging (CAST) setting for ultrasound systems, to assist novice users (e.g., midwives with minimal training) with a method for obstetrical ultrasound screening for multiple gestation that pre-selects appropriate referral cases for experts / trained providers from community health, for delivery care in more resourced centers. Automated CAST can help rural health providers like midwives with minimal training / non-expertise to screen out benign cases (e.g., low-risk or non-emergent cases that may not need referral), to avoid additional steps in the workflow, and to increase exam capabilities in community medicine.

[0037] The obstetric blind sweep protocol can be taught to health care workers in a very short period, enabling them to acquire high-quality ultrasound images of pregnant mothers. There have been encouraging results on determination of gestational age and fetalpresentation from this type of blind sweep protocol. A need exists for detection of multiple gestation from this automated process, as it is one type of high-risk pregnancy.

[0038] Aspects of the present disclosure can include features described in U.S. Provisional Application No. 63 / 540,740, filed September 27, 2023, titled “Ultrasound Imaging With Ultrasound Probe Guidance In Blind Sweep Protocol” and / or U.S. Provisional Application No. 63 / 540,755, filed September 27, 2023, titled “Ultrasound Imaging with Follow Up Sweep Guidance After Blind Sweep Procedure”, which are incorporated by reference as though fully set forth herein.

[0039] There is a need for automated detection of multiple gestation, as it can be considered a high-risk pregnancy. While there have been initial attempts to classify pregnancies as single or multiple, a literature survey shows limited utility for existing methods. Further, identification of the fetal lie in multiple gestation can be very important for clinicians to make a decision on the mode of delivery of the fetus.

[0040] The present disclosure provides a solution to the problem of identifying the number of fetuses in the maternal womb at a time. Since multiple gestation is a high-risk pregnancy that can result in maternal fetal complications, it needs to be diagnosed and referred to higher centers for confirmation, determining the chronicity and amnionicity and follow-up. While the obstetric sweep protocol takes the images of the fetus in a simplified manner which can be easily taught to the users, the artificial intelligence (Al)-guided automatic detection of the fetus allows the user know the number of fetuses inside the mother’s uterus. Once the detection of multiple gestations is done, the user can refer the patient to higher centers for further management or follow-up of the patient. Diagnosis of multiple gestation will help manage maternal and fetal complications if any and reduce perinatal mortality. Further, the Al could give a possible indication of the fetal lie of each fetus, thus facilitating decision-making for the mode of delivery for late twin gestation.

[0041] Literature on automated identification of multiple gestation cases via blind sweep is limited. Most of the work in the blind sweep features is via statistical approaches, e.g., identifying the presence of fetal parts, formulating this as a black-box Al approach (see for example Viswanathan et al., “Deep learning to assess twin gestation from blindly obtained ultrasound sweeps”, American Journal of Obstetrics & Gynecology, 2024 622). The present disclosure instead provides temporal inspection of fetal orientation, navigating through the sweep, identifying repeated anatomies, and mapping it back to the overall protocol. Knowing the geometric position of anatomies over the maternal womb helps the sonographer to assess the presence of multiple gestation and their orientation(s). Hence, a need exists foridentifying multiple gestation and fetal orientation via a geometric process that is more aligned with current inspection and reporting methods.

[0042] The present disclosure provides geometric processing of anatomical structures to address multiple gestation and fetal presentation. In some aspects, multiple gestation identification involves identifying repeated anatomies sequentially over the sweep, as well as graph formulations of fetal structures and shaping up their skeletonization both within a single sweep and across multiple sweeps. This can help in assessing multiple gestations through anatomical appearance. Individual fetuses are mapped (e.g., skeletonized) as individual entities.

[0043] In some aspects, the system may detect fetal orientation (e.g., fetal lie, fetal presentation) or. Temporal correlation of detected anatomical structures in relation to the uterus can help determine the spatial location of identified features, and assign them to the respective fetus, in order to derive fetal presentation / lie. Geometric tagging of anatomy such as the head, heart, abdomen, spine, and pelvis can provide information regarding the lie, orientation, or presentation of each fetus.

[0044] In some aspects, the graphing and skeletonization process can boost, assist, or update a single-frame-based detector module. As a by-product leveraging semantic mapping of fetal structures, the single frame-based detector module can also be refined (e.g., supplemented, retrained, etc.). This may for example involve refining a single-frame-based detector as a by-product of graphing / skeletonization, thus leveraging the semantic mapping of fetal structures. Leveraging temporal mapping of different fetal parts with logical sequencing, the present disclosure can reduce false detections on the part of the Al model.

[0045] Intra-sweep identifying of repeated fetal anatomies sequentially in a sweep, and temporal / spatial mapping of fetal structures (identified by frame-level Al models e.g., multi class YOLO detector targeting fetal parts) can be used for constructing possible single / multiple fetal centerlines / graphs. Graph-based approaches can be useful for the splitting and / or merging of entities, e.g., determining whether anatomical structures and their associations belong to one fetus or multiple fetuses. Each associated fetal structure should be a part of a single entity, whereas if there exist multiple fetuses in a sweep, they can be split into multiple fetal centerlines / entities. These intra-sweep-constructed fetal centerlines / graphs will be mapped across multiple sweeps, leveraging the defined blind sweep protocol to further refine the prediction or determination of singleton pregnancy vs. multiple gestation.

[0046] Fetal movement can also be detected via this fetal skeletonization, both within and across sweeps. For example, any abrupt changes of centerline of the skeleton can act as anindicator of fetal movement. Detection of multiple pregnancies by blind sweeps using an Al solution will be useful for novice users. Referral of patients with multiple gestation to a tertiary care center for further diagnosis and management may improve outcomes.Determination of the lie of the individual fetuses in multiple pregnancies may be extremely useful in deciding the mode of delivery in late gestation.

[0047] The present disclosure aids substantially in the capture of high-quality ultrasoundbased diagnoses by minimally trained users, by automatically detecting multiple pregnancy / multiple gestation. Implemented on a processor in communication with an ultrasound probe, the ultrasound blind sweep multiple pregnancy detection system disclosed herein provides practical improvements in the quality of care available to patients in underserved areas. This improved imaging methodology transforms a process that is heavily reliant on professional experience into one that is accurate and repeatable even for minimally trained personnel, without the normally routine need to train clinicians such as emergency department personnel to recognize multiple gestation and other prenatal conditions. This unconventional approach improves the functioning of the ultrasound imaging system, by providing reliable, repeatable imaging and diagnosis in hospital, office, vehicle, field, and home settings, as well as referral recommendations for patients suspected to have a multiple pregnancy.

[0048] The ultrasound blind sweep multiple pregnancy detection system may be implemented as a process at least partially viewable on a display, and operated by a control process executing on a processor that accepts user inputs from a keyboard, mouse, or touchscreen interface, and that is in communication with one or more sensor probes. In that regard, the control process performs certain specific operations in response to different inputs or selections made at different times. Certain structures, functions, and operations of the processor, display, sensors, and user input systems provide novel features or aspects of the present disclosure.

[0049] These descriptions are provided for exemplary purposes only, and should not be considered to limit the scope of the ultrasound blind sweep multiple pregnancy detection system. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

[0050] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the aspects illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of thepresent disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one aspect may be combined with the features, components, and / or steps described with respect to other aspects of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately.

[0051] Figure l is a schematic, diagrammatic representation of an ultrasound imaging system 100, according to aspects of the present disclosure. The ultrasound imaging system 100 may for example be used to acquire ultrasound video sweeps, which can then be analyzed by a human clinician or an artificial intelligence to diagnose medical conditions.

[0052] The ultrasound imaging system 100 is used for scanning an area or volume of a subject’s body. A subject may include a patient of an ultrasound imaging procedure, or any other person, or any suitable living or non-living organism or structure. The ultrasound imaging system 100 includes an ultrasound imaging probe 110 in communication with a host 130 over a communication interface or link 120. The probe 110 may include a transducer array 112, a beamformer 114, a processor circuit 116, and a communication interface 118. The host 130 may include a display 132, a processor circuit 134, a communication interface 136, and a memory 138 storing subject information.

[0053] In some aspects, the probe 110 is an external ultrasound imaging device including a housing 111 configured for handheld operation by a user. The transducer array 112 can be configured to obtain ultrasound data while the user grasps the housing 111 of the probe 110 such that the transducer array 112 is positioned adjacent to or in contact with a subject’s skin. The probe 110 is configured to obtain ultrasound data of anatomy within the subject’s body while the probe 110 is positioned outside of the subject’s body. In some aspects, the probe 110 can be an external ultrasound probe, a transthoracic probe, and / or a curved array probe.

[0054] In other aspects, the probe 110 can be an internal ultrasound imaging device and may comprise a housing 111 configured to be positioned within a subject’s body. In some aspects, the probe 110 may be a curved array probe. Probe 110 may be of any suitable form for any suitable ultrasound imaging application including both external and internal ultrasound imaging.

[0055] For an ultrasound imaging device, the transducer array 112 emits ultrasound signals towards an anatomical object 105 of a subject and receives echo signals reflected from the object 105 back to the transducer array 112. The ultrasound transducer array 112 can include any suitable number of acoustic elements, including one or more acousticelements and / or a plurality of acoustic elements. In some instances, the transducer array 112 includes a single acoustic element. In some instances, the transducer array 112 may include an array of acoustic elements with any number of acoustic elements in any suitable configuration. For example, the transducer array 112 can include between 1 acoustic element and 10000 acoustic elements, including values such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1000 acoustic elements, 3000 acoustic elements, 8000 acoustic elements, and / or other values both larger and smaller. In some instances, the transducer array 112 may include an array of acoustic elements with any number of acoustic elements in any suitable configuration, such as a linear array, a planar array, a curved array, a curvilinear array, a circumferential array, an annular array, a phased array, a matrix array, a one-dimensional (ID) array, a 1.x dimensional array (e.g., a 1.5D array), or a two- dimensional (2D) array. The array of acoustic elements (e.g., one or more rows, one or more columns, and / or one or more orientations) can be uniformly or independently controlled and activated. The transducer array 112 can be configured to obtain one-dimensional, two- dimensional, and / or three-dimensional images of a subject’s anatomy. In some aspects, the transducer array 112 may include a piezoelectric micromachined ultrasound transducer (PMUT), capacitive micromachined ultrasonic transducer (CMUT), single crystal, lead zirconate titanate (PZT), PZT composite, other suitable transducer types, and / or combinations thereof.

[0056] The object 105 may include any anatomy or anatomical feature, such an abdomen of a pregnant patient, one or multiple fetuses inside the abdomen of the pregnant patient, etc.

[0057] The beamformer 114 is coupled to the transducer array 112. The beamformer 114 controls the transducer array 112, for example, for transmission of the ultrasound signals and reception of the ultrasound echo signals. In some aspects, the beamformer 114 may apply a time-delay to signals sent to individual acoustic transducers within an array in the transducer 112 such that an acoustic signal is steered in any suitable direction propagating away from the probe 110. The beamformer 114 may further provide image signals to the processor circuit 116 based on the response of the received ultrasound echo signals. The beamformer 114 may include multiple stages of beamforming. The beamforming can reduce the number of signal lines for coupling to the processor circuit 116. In some aspects, the transducer array 112 in combination with the beamformer 114 may be referred to as an ultrasound imaging component.

[0058] The processor 116 is coupled to the beamformer 114. The processor 116 may also be described as a processor circuit, which can include other components in communication with the processor 116, such as a memory, beamformer 114, communication interface 118, and / or other suitable components. The processor 116 may include a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 116 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 116 is configured to process the beamformed image signals. For example, the processor 116 may perform filtering and / or quadrature demodulation to condition the image signals. The processor 116 and / or 134 can be configured to control the array 112 to obtain ultrasound data associated with the object 105.

[0059] The communication interface 118 is coupled to the processor 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 118 can include hardware components and / or software components implementing a particular communication protocol suitable for transporting signals over the communication link 120 to the host 130. The communication interface 118 can be referred to as a communication device or a communication interface module.

[0060] The communication link 120 may be any suitable communication link. For example, the communication link 120 may be a wired link, such as a universal serial bus (USB) link or an Ethernet link. Alternatively, the communication link 120 may be a wireless link, such as an ultra-wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0061] At the host 130, the communication interface 136 may receive the image signals. The communication interface 136 may be substantially similar to the communication interface 118. The host 130 may be any suitable computing and display device, such as a workstation, a personal computer (PC), a laptop, a tablet, or a mobile phone.

[0062] The processor 134 is coupled to the communication interface 136. The processor 134 may also be described as a processor circuit, which can include other components in communication with the processor 134, such as the memory 138, the communication interface 136, an optional speaker 139, and / or other suitable components. The processor 134may be implemented as a combination of software components and hardware components. The processor 134 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 134 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor 134 can be configured to generate image data from the image signals received from the probe 110. The processor 134 can apply advanced signal processing and / or image processing techniques to the image signals. In some aspects, the processor 134 can form a three-dimensional (3D) volume image from the image data. In some aspects, the processor 134 can perform real-time processing on the image data to provide a streaming video of ultrasound images of the object 105. In some aspects, the host 130 includes a beamformer. For example, the processor 134 can be part of and / or otherwise in communication with such a beamformer. The beamformer in the in the host 130 can be a system beamformer or a main beamformer (providing one or more subsequent stages of beamforming), while the beamformer 114 is a probe beamformer or micro-beamformer (providing one or more initial stages of beamforming).

[0063] The memory 138 is coupled to the processor 134. The memory 138 may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor 134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory.

[0064] The memory 138 can be configured to store subject information, measurements, data, or files relating to a subject’s medical history, history of procedures performed, anatomical or biological features, characteristics, or medical conditions associated with a subject, computer readable instructions, such as code, software, or other application, as well as any other suitable information or data. The memory 138 may be located within the host 130. Subject information may include measurements, data, files, other forms of medical history, such as but not limited to ultrasound images, ultrasound videos, and / or any imaging information relating to the subject’s anatomy. The subject information may include parameters related to an imaging procedure such as an anatomical scan window, a probeorientation, and / or the subject position during an imaging procedure. The memory 138 can also be configured to store information related to the training and implementation of machine learning models (e.g., neural networks) and / or information related to implementing image recognition methods for detecting / segmenting anatomy, image quantification methods, and / or image acquisition guidance methods, including those described herein.

[0065] The display 132 is coupled to the processor circuit 134. The display 132 may be a monitor or any suitable display. The display 132 is configured to display the ultrasound images, image videos, and / or any imaging information of the object 105.

[0066] The ultrasound imaging system 100 may be used to assist a sonographer in performing an ultrasound scan. The scan may be performed in a point-of-care setting. In some instances, the host 130 is a console or movable cart. In some instances, the host 130 may be a mobile device, such as a tablet, a mobile phone, or portable computer. During an imaging procedure, the ultrasound system can acquire an ultrasound image of a particular region of interest within a subject’s anatomy. The ultrasound imaging system 100 may then analyze the ultrasound image to identify various parameters associated with the acquisition of the image such as the scan window, the probe orientation, the subject position, and / or other parameters. The ultrasound imaging system 100 may then store the image and these associated parameters in the memory 138. At a subsequent imaging procedure, the ultrasound imaging system 100 may retrieve the previously acquired ultrasound image and associated parameters for display to a user which may be used to guide the user of the ultrasound imaging system 100 to use the same or similar parameters in the subsequent imaging procedure, as will be described in more detail hereafter.

[0067] In some aspects, the processor 134 may utilize deep learning-based prediction networks to identify parameters of an ultrasound image, including an anatomical scan window, probe orientation, subject position, identify and location of anatomical features, and / or other parameters. In some aspects, the processor 134 may receive metrics or perform various calculations relating to the region of interest imaged or the subject’s physiological state during an imaging procedure. These metrics and / or calculations may also be displayed to the sonographer or other user via the display 132.

[0068] In some aspects, the host 130 may also include a speaker 180. The speaker 180 may for example be used to provide advisory tones, beeps, or other auditory feedback to the user.

[0069] Before continuing, it should be noted that the examples described above are provided for purposes of illustration, and are not intended to be limiting. Other devices and / or device configurations may be utilized to carry out the operations described herein.

[0070] Figure l is a schematic diagram of a processor circuit 250, according to aspects of the present disclosure. The processor circuit 250 may be implemented in the ultrasound imaging system 100, or other devices or workstations (e.g., third-party workstations, network routers, etc.), or on a cloud processor or other remote processing unit, as necessary to implement the method. As shown, the processor circuit 250 may include a processor 260, a memory 264, and a communication module 268. These elements may be in direct or indirect communication with each other, for example via one or more buses.

[0071] The processor 260 may include a central processing unit (CPU), a digital signal processor (DSP), a controller, or any combination of general-purpose computing devices, reduced instruction set computing (RISC) devices, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other related logic devices, including mechanical and quantum computers. The processor 260 may also comprise another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor 260 may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0072] The memory 264 may include a cache memory (e.g., a cache memory of the processor 260), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, other forms of volatile and nonvolatile memory, or a combination of different types of memory. In an aspect, the memory 264 includes a non-transitory computer-readable medium. The memory 264 may store instructions 266. The instructions 266 may include instructions that, when executed by the processor 260, cause the processor 260 to perform the operations described herein. Instructions 266 may also be referred to as code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instructions” and “code” may refer to one or more programs, routines, subroutines, functions, procedures, etc. “Instructions” and “code” may include a single computer-readable statement or many computer-readable statements.

[0073] The communication module 268 can include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processor circuit 250, and other processors or devices. In that regard, the communication module 268 can be an input / output (I / O) device. In some instances, the communication module 268 facilitates direct or indirect communication between various elements of the processor circuit 250 and / or the ultrasound imaging system 100. The communication module 268 may communicate within the processor circuit 250 through numerous methods or protocols. Serial communication protocols may include but are not limited to United States Serial Protocol Interface (US SPI), Inter-Integrated Circuit (I2C), Recommended Standard 232 (RS- 232), RS-485, Controller Area Network (CAN), Ethernet, Aeronautical Radio, Incorporated 429 (ARINC 429), MODBUS, Military Standard 1553 (MIL-STD-1553), or any other suitable method or protocol. Parallel protocols include but are not limited to Industry Standard Architecture (ISA), Advanced Technology Attachment (ATA), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI), Institute of Electrical and Electronics Engineers 488 (IEEE-488), IEEE-1284, and other suitable protocols. Where appropriate, serial and parallel communications may be bridged by a Universal Asynchronous Receiver Transmitter (UART), Universal Synchronous Receiver Transmitter (USART), or other appropriate subsystem.

[0074] External communication (including but not limited to software updates, firmware updates, model sharing between the processor and central server, or readings from the ultrasound imaging system 100) may be accomplished using any suitable wireless or wired communication technology, such as a cable interface such as a universal serial bus (USB), micro USB, Lightning, or FireWire interface, Bluetooth, Wi-Fi, ZigBee, Li-Fi, or cellular data connections such as 2G / GSM (global system for mobiles) , 3G / UMTS (universal mobile telecommunications system), 4G, long term evolution (LTE), WiMax, or 5G. For example, a Bluetooth Low Energy (BLE) radio can be used to establish connectivity with a cloud service, for transmission of data, and for receipt of software patches. The controller may be configured to communicate with a remote server, or a local device such as a laptop, tablet, or handheld device, or may include a display capable of showing status variables and other information. Information may also be transferred on physical media such as a USB flash drive or memory stick.

[0075] Figure 3A is a set of diagrammatic views of two fetuses 200, 201 surrounded by a at least one amniotic sac or amnion 210 and at least one chorionic sac or chorion 315 within a uterus of a patient, representing different types of twins, according to aspects of the presentdisclosure. At least one placenta 240 is also visible. Automated detection of multiple pregnancy or multiple gestation (e.g., twins) is an object of the present disclosure.

[0076] In a first example 305, the multiple pregnancy / gestation is monochorionic (e.g., having only one chorion 315) and monoamniotic (e.g., having only one amniotic sac 210 that contains both fetuses 200, 201, and has a single placenta 240.

[0077] In a second example 325, the multiple pregnancy / gestation is monochorionic (e.g., having one chorion 315) but diamniotic (e.g., having two amniotic sacs 210, 211, separated by a fetal membrane 350, with each amniotic sac holding one fetus 200, 201), and a single placenta 240. Detection of the fetal membrane is one way to detect a multiple gestation or multiple pregnancy.

[0078] In a third example 335, the multiple pregnancy / gestation is dichorionic (e.g., having two chorions 315, 316, separated by respective membranes 350) and diamniotic (e.g., having two amniotic sacs 210, 211, with each sac holding one twin 200, 201), and two placentas 240 that are fused together.

[0079] In a fourth example 345, the multiple pregnancy / gestation is dichorionic (e.g., having two separate chorions 315, 316, separated by respective membranes 350) and diamniotic (e.g., having two separate amniotic sacs 200, 201, each holding one twin 200, 201), and two separate placentas 240, 241. This type of twin pregnancy is associated with a lower rate of complications than monochorionic or monoamniotic pregnancies.

[0080] 70% of twin pregnancies are dizygotic (e.g., resulting from two fertilized eggs), and dizygotic pregnancies are believed to be always dichorionic and diamniotic. 30% of pregnancies are monozygotic (e.g., resulting from a single fertilized egg), and 20% of these pregnancies will also be dichorionic and diamniotic. Thus, detection of the fetal membrane 350 separating the two amniotic sacs can be effective in detecting up to 76% of twin pregnancies. The remaining 24% of detections may rely on other features of the pregnancy such as multiple occurrences of a unique anatomical feature (e.g., two heads, two hearts, two pelvises, etc.) and / or the spatial / geometrical relationship between fetal parts, as described below.

[0081] The uterus can be considered maternal anatomy. Depending on the context, the placenta may be considered fetal anatomy, maternal anatomy, or an interface between the two.

[0082] Figure 3B is a set of schematic, diagrammatic, cross-sectional views of two fetuses 200, 201 arranged in different positions within a uterus of a patient as seen by an ultrasound imaging plane 360, according to aspects of the present disclosure. The imagingplane 360 is example of an imaging plane (e.g., for one frame during one sweep), and the same imaging plane shown in all fetal positions for twins, and indicates that different fetal parts will be visible in the same imaging plane depending on which fetal position twins are in.

[0083] In a first example 370, both twins 200, 201 are in a vertex (head-down) position. Approximately 45% of twin pregnancies fall into this category. In example 370, the imaging plane 360 may not include any duplicate anatomy.

[0084] In a second example 375, one twin 200 is in a breech (head-up) position, and the other is in a vertex (head-down) position. Approximately 37% of twin pregnancies fall into this category. Breech births are associated with a higher rate of complications. In example 375, the imaging plane 360 includes the heads of both fetuses 200, 201.

[0085] In a third example 380, both twins 200, 201 are in a breech (head-up) position. Approximately 10% of twin pregnancies fall into this category. In example 380, the imaging plane 360 may not contain any duplicate anatomy.

[0086] In a fourth example 385, one twin 200 is in a transverse (sideways) position, and the other twin 201 is in a vertex position. Approximately 5% of twin pregnancies fall into this category. Vertex births are associated with a higher rate of complications. In example 385, the imaging plane 360 may include the hearts of both fetuses 200, 201.

[0087] In a fifth example 390, one twin 200 is in the breech (head-up) position, and the other twin 201 is in the transverse (sideways) position. Approximately 2% of twin pregnancies fall into this category. In example 390, the imaging plane 360 may include the hearts of both fetuses 200, 201.

[0088] In a sixth example 395, both twins 200, 201 are in the vertex position.Approximately 0.5% of twin pregnancies fall into this category. In example 395, the imaging plane 360 may include the hearts of both fetuses 200, 201.

[0089] The positions shown in Figure 3B may also be referred to as fetal orientation, fetal presentation, or fetal lie.

[0090] Figure 4 is a schematic, diagrammatic representation of a patient 300 on whose abdomen the ultrasound blind sweep protocol will be followed, according to aspects of the present disclosure. Visible on the abdomen 310 of the patient 300 is a desired sweep pattern 320 intended to capture images of desired features of the patient’s anatomy. The sweep pattern 320 includes multiple vertical sweep lines 330 and multiple horizontal sweep lines 340. Each sweep line 330, 340 represents a desired path for one imaging sweep of the abdomen 310. In the example shown in Figure 4, the sweep pattern includes three verticalsweep lines 330 labeled L (patient’s left), M (patient’s middle), and R (patient’s right), all in an upward direction with respect to the patient, and three horizontal sweep lines 340 labeled Cl (bottom), C2 (middle), and C3 (top), all in a right-to-left direction with respect to the patient. However, it is understood that a sweep pattern 320 may include more or fewer sweep lines 330, including vertical sweep lines 330, horizontal sweep lines 330, or combinations thereof, in any combination of upward, downward, left, or right directions based on the patient’s fundal height. For example, if the patient’s belly is bigger in size, more sweeps may be needed. Furthermore, a sweep pattern 320 may cover other portions of the patient’s body, including but not limited to the head, neck, spine, limbs, etc. Types of blind sweep protocol include but are not limited to obstetric sweep imaging (OSI), volume sweep imaging (VSI), 6-Stage, Fetal Age Machine Learning Initiative (FAMLI), and Philips.

[0091] These sweep patterns represent desired probe motion information, including desired positions, a desired velocity or velocities, and / or a desired orientation of the ultrasound probe while the ultrasound probe is obtaining a plurality of ultrasound image frames during the sweep. It is noted that the desired sweep patterns or blind sweep protocols stored in a memory of the processor may include only vertical sweeps, only horizontal sweeps, may include a grid (e.g., 3x3, 5x5, etc.) of vertical and horizontal sweeps, and may also include associated parameters such as desired probe motion (e.g., positions, velocities, and / or orientations) stored in the memory(e.g., blind sweep protocol 440 in Fig. 5).Depending on the implementation, sweeps may include curved, diagonal, and other types of sweeps. The protocols and their associated parameters can for example be based on standards established by authorities in the field (physician organizations, sonographer organizations, etc.), published in scholarly journals / textbooks, etc.

[0092] Figure 5 is a schematic, diagrammatic representation, in hybrid block diagram / flow diagram form, of an example ultrasound blind sweep multiple pregnancy detection system 400, according to aspects of the present disclosure. An ultrasound probe 110 operated by a novice user 410 performs a blind sweep protocol 440 on the body of a patient 300 and sends ultrasound imaging data to a host 130 such as a tablet, smartphone, ultrasound cart, etc. In step 415, the host 130 generates ultrasound images using the ultrasound image data obtained by the ultrasound probe 110. The host 130 can control the ultrasound probe 110 to obtain the ultrasound image data (e.g., the host 130 establishes communication with the ultrasound probe 110, the host 130 sends control signals to start and / or stop acquisition of ultrasound image data, the host 130 sends power signals to power the ultrasound probe 110, etc.).

[0093] In step 420, the host 130 uses the ultrasound images generated in step 415 and / or the ultrasound image data used to generate the ultrasound images to determine if the pregnant mother has multiple pregnancy or multiple gestation, as described in more detail below. In step 430, the host generates a visual representation on a display, showing an indication of whether there is a multiple pregnancy and / or a graphic associated with the determination that there is a multiple pregnancy.

[0094] In step 450, the host 130 determines whether the planned path of the blind sweep protocol 440 has been followed adequately. If no, execution proceeds to steps 430 and 460, wherein the host 130 provides feedback (e.g., audio feedback through a speaker and / or visual feedback from a display) to repeat one or multiple sweeps of the blind sweep protocol 440. In some aspects, the determination in step 450 can be based on the results of analysis in step 420 of whether the patient has multiple pregnancy. For example, if there is insufficient ultrasound image data to perform the determinations in step 420 or if the ultrasound image data that has been obtained is not suitable to perform the determination in step 420, then step 450 can determine that the one or multiple sweeps of the blind sweep protocol 440 has not been completed correctly. The one or multiple sweeps that need to be performed again can be communicated to the user through visual or audio feedback in steps 430 and 460.

[0095] In some aspects, the planned path determined in step 450 of the blind sweep protocol 440 can be used by the host 130 to determine whether the patient has multiple pregnancy in step 420. For example, the planned path determined in step 450 can provide absolute or relative spatial information (e.g., vertical location, horizontal location) about the ultrasound image data obtained by the ultrasound probe and / or the ultrasound images generated by the host. For example, referring again to Fig. 4, the planned path determined in step 450 can indicate that the M sweep is horizontally located (left-right direction in Fig. 4) between the R sweep and the L sweep. The host 130 can use this spatial information from step 450 to process the ultrasound image data and / or ultrasound images to determine whether there is multiple pregnancy in step 420.

[0096] Based on the visual representation 430 and / or the audio guidance 460, the novice user 410 makes a triage decision to either rule out or rule in multiple pregnancy / gestation. In step 470, the novice user 410 rules out multiple pregnancy, and in step 480, the novice user 410 performs or recommends normal periodic evaluations for the patient 300. In step 490, the novice user 410 rules in multiple pregnancy, and in step 495, the novice user 410 refers the patient to an expert user such as an experienced sonographer, radiologist, obstetrician, or other physician for follow-up imaging and diagnosis.

[0097] Flow diagrams and block diagrams are provided herein for exemplary purposes; a person of ordinary skill in the art will recognize myriad variations that nonetheless fall within the scope of the present disclosure. For example, any of the steps described herein may optionally include an output to a user of information relevant to the step, and may thus represent an improvement in the user interface over existing art by providing information not otherwise available. Similarly, block diagrams may show a particular arrangement of components, modules, services, steps, processes, or layers, resulting in a particular data flow. It is understood that some embodiments of the systems disclosed herein may include additional components, that some components shown may be absent from some embodiments, and that the arrangement of components may be different than shown, resulting in different data flows while still performing the methods described herein. The logic of flow diagrams may be shown as sequential. However, similar logic could be parallel, massively parallel, object oriented, real-time, event-driven, cellular automaton, or otherwise, while accomplishing the same or similar functions. In order to perform the methods described herein, a processor may divide each of the steps described herein into a plurality of machine instructions, and may execute these instructions at the rate of several hundred, several thousand, several million, or several billion per second, in a single processor or across a plurality of processors. Such rapid execution may be necessary in order to execute the method in real time or near-real time as described herein. For example, to provide an assessment of fetal anatomy locations and / or an indication of whether the pregnancy is multiple, the system may need to generate the visual representation 430 within one second of completing the blind sweep protocol.

[0098] Figure 6 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example ultrasound blind sweep multiple pregnancy detection system 400, according to aspects of the present disclosure. A set of ultrasound image sequences 610, also known as cineloops or cine scans (e.g., one cine scan per sweep of the blind sweep protocol) are received (whether one at a time, simultaneously, or in groups), and are mapped by a 3D mapping module 615 to locations within the maternal abdomen.

[0099] A set of ultrasound image / frame sequences (as depicted in 610) are obtained from different sweeps as per the defined blind sweep protocol. An objective of the present disclosure is to cover the entire abdomen from both orthogonal direction and, more preciously the entire uterus with a defined sweep protocol. As per Figure 4, the vertical (310) and horizontal (320) sweeps are captured following a predefined path consecutively, where these individual sweeps / cine loops (330 and 340 respectively) and their respective frames aremapped to the abdominal spread in an order. In other words, if the user adheres to the acquisition protocol, different parts of the sweeps and their respective frames can be roughly mapped to abdominal / uterine regions. This helps in forming a rough / relative 3D grid from the acquired data 615 representing / mapping to the maternal abdomen.

[0100] However, stacking these individual horizontal (or vertical) sweeps in parallel and constructing a 3*3 (or 5*5) mega grid will fetch two sets of 3 (or 5) x (image height x image width x #frames) matrix for both horizontal and vertical sweeps. However, these uniform approximations might not meet the practical appearances. Hence, the below modifications will help in adapting to these practical considerations.

[0101] In Sweep level: To make this 3D grid more realistic from the individual sweep, the system may leverage the (9-axis) inertial measurement unit (IMU) information (e.g., from an IMU located within the ultrasound probe) to correct the inter-alignment of frames, which in turn will help to form a more realistic skeletonization by compensating for any non- uniform movement of the probe across frames in a sweep.

[0102] In Exam level (intra direction): Craniocaudal (and lateral) sweeps are performed such that the ultrasound probe overlaps laterally -50% with the consecutive parallel sweeps (as per defined protocol). With this prerequisite, following the guidelines and performing affine registration between successive parallel sweeps (via B-mode or on detection levels), the system can contextually stitch the inter-sweeps in a particular direction (either in vertical / craniocaudal, or horizontal / lateral direction) to form two sets of refined matrix spaces from the individual sweep-level.

[0103] In Exam level (inter direction): Formation of unified matrix space following a direction (either in vertical / craniocaudal, or horizontal / lateral direction) via correlation / affine- registration on B-Mode or salient detections of different anatomies can also be performed. Applying the same method between orthogonal frames / sweeps in B-Mode may for example rely on salient detections of different anatomies considering minimal fetal movement.

[0104] Thus, to alleviate this confusion, the system treats individual sweeps separately to shape this skeleton formulation of individual / multiple fetuses (full or partial) and register across parallel sweeps. It is noted that, rather than merging both matrices coming from both directions, merging / splitting the obtained skeleton(s) between both directions can form a final skeleton of the fetus(es), as described below.

[0105] The localized image frames are then received by a multi-class anatomy detector or object detector 620. Each cine scan can include a plurality of ultrasound image frames (e.g., 100-1000 ultrasound image frames, and / or other values both larger and smaller). The objectdetector 620 may for example be a machine learning (ML) neural network as described below, although other types of object detectors may be used instead or in addition, including classical image recognition methods. The object detector 620 can be a deep learning network (e.g., convolutional neural network or CNN) trained to detect maternal anatomy and / or fetal anatomy. An output of the object detector may for example include annotated versions of the cineloops 610 that include bounding boxes for each detected anatomy in each frame where anatomy was detected. The object detector 620 can generate, as its output, a detection of one or multiple fetal anatomical parts within the ultrasound image frames of the cine scan(s). Detected anatomy may for example include the fetal head, abdomen, heart, abdomen, spine, or pelvis - unique features of which each fetus is expected to have only one.

[0106] The outputs of the object detector 620 are then passed to a multiple pregnancy / gestation determination module 630. The multiple pregnancy / gestation determination module 630 determines, using the plurality of fetal anatomical parts (from object detector 620), whether the pregnancy is a multiple pregnancy / gestation. In the example shown in Figure 6, the multiple pregnancy / gestation determination 630 includes a graphing and skeletonization of fetal parts within a given sweep 1180, and a graphing and skeletonization of fetal parts across multiple sweeps 1190. In an example, the multiple pregnancy / gestation determination module may have, as its output, a binary (e.g., yes / no) indication of whether a multiple pregnancy is detected. This detection may occur individually in every sweep of every cineloop 610, or may occur across all of the cineloops collectively.

[0107] A fetal orientation determination module 640 may receive inputs either from the graph and skeletonization steps 1180, 1190 and / or may receive inputs directly from the multiclass anatomy detector 620. The fetal orientation (e.g., fetal lie, fetal presentation) determination module 640 determines the orientation(s) of one or more fetuses within the womb, based on locations of the detected anatomy. Different examples of fetal orientations are shown in Fig. 3B. For example, if a heart and abdomen of a given fetus are detected above the head, then that fetus may be determined to be in a vertex position, whereas if the heart and abdomen are detected below the head, then the fetus may be determined to be in a breech position.

[0108] Outputs of the graph and skeletonization steps 1180, 1190 may be used in a refinement step 645, to refine the anatomy detections (e.g., on a per frame basis) of the multiclass anatomy detector 620 (e.g., by using the skeletonizations as training data to refine the training of a neural network, as described below).

[0109] Due to deployment complexity and compute limitations on resource-constrained solutions, and also to achieve real-time processing, the system can employ a single framebased detection module (e.g. 2D YOLO). In other instances, 3D detection modules can be used (e.g. ROLO or Recurrent YOLO). In the absence of temporal learning, sometimes the outcome of these detection modules can fetch false positives, which can be problematic for later steps that rely on this raw detection information. Heuristically, some of these false positives can be eliminated by post-processing. However, the output of blocks 1180 or 1190 can help to clear any false anatomical appearances considering anatomical context and its presence from its surroundings. These false detection corrections not only solidify the skeletonization steps in 1180 and / or 1190, but can also retrain the anatomy detector with negative classes to boost the detector outcome in block 645 of Figure 6 or block 1230 of Figure 12.

[0110] The processor (e.g., processor 134 of Fig. 1, processor 260 of Fig. 2, and / or other processors) provide, to a display (e.g., display 132 of Fig. 1 and / or other displays) in communication therewith, an output representative of the determination of whether the pregnancy is a multiple gestation / pregnancy. For example, the output of the multiple pregnancy / gestation determination 630, and / or the outputs of the individual determinations 1180, 1190, are presented as a visual representation 430 (e.g., a screen display on the display 132 of Fig. 1). The visual representation 430 can be a binary indicator of whether or not a multiple pregnancy is detected. The visual representation 430 can include an indication 650 (e.g., text or symbols) of whether a multiple pregnancy / gestation is suspected or not suspected. The screen display 430 may also include a graphical representation 660 associated with the determination of whether a multiple pregnancy / gestation is suspected. The graphical representation 660 may for example include drawings, ultrasound image frames, cineloops, or generated graphics, either with or without text or symbols as annotations, including graphics, images, text, and / or other visual representations described herein (e.g., the output of the fetal lie determination module 640, the output of the multiple pregnancy / gestation determination 630, and / or the outputs of the individual determinations 1180, 1190).

[0111] The visual representation 430 may also include text or graphic 670 associated with the fetal lie determination. This text or graphic 670 may for example include a representation of the fetus(es), or arrows indicating the lie of each fetus, or other graphical representations as would occur to a person of ordinary skill in the art.

[0112] Any of the modules 615, 620, 630, 640, 645, or their subcomponents, may include software, hardware, firmware, anal og / digi tai logic, analog / digital circuitry, or combinationsthereof, and may for example be implemented or executed by a processor circuit (e.g., processor circuit 250 of Figure 2).

[0113] Figure 7A is a schematic, diagrammatic overview, in block diagram form, of a training mode 700 for an untrained neural network 710a, according to aspects of the present disclosure. In the example shown in Figure 7A, a set of training data 705a includes ultrasound cineloops of probe sweeps annotated with the corresponding anatomy (head, heart, abdomen, pelvis, membrane, placenta, etc.). The training data 705a is fed into an untrained neural network 710a in an iterative training process that will be familiar to a person of ordinary skill in the art.

[0114] The parameters of a network model (e.g., the weights at each artificial neuron) are initialized with initial values A that may be random values or with results from training on prior datasets. In an iterative process, the network is used to make detection inferences on the training images, the results are compared with the ground truth annotations, and an optimizer is used to adjust the network parameters B until a metric of accuracy is maximized.

[0115] Thus, an output of this training process 700 is a trained neural network 710b, wherein the parameters B (e.g., weights) are optimized for generating accurate bounding boxes for the anatomy imaged in the training data 705a.

[0116] Figure 7B is a schematic, diagrammatic overview, in block diagram form, of an inference mode or clinical usage mode 704 for the trained neural network 710b, according to aspects of the present disclosure. In clinical usage, an ultrasound video, cineloop, or cine sweep 720 of the blind sweep is fed to the trained and validated neural network 710b for analysis. The trained and validated neural network 710b then produces, as an output, anatomy detection bounding boxes 740 for each image (or the entire sweep). In some aspects, a confidence value can be determined as a normalized value in the range [0-1], where 0 indicates lowest confidence, and 1 indicates highest confidence that the detection is correct.

[0117] Figure 8 is a schematic, diagrammatic illustration, in block diagram form, of the detection of anatomy (e.g., the head, heart, abdomen, pelvis, placenta, membrane, etc.), according to aspects of the present disclosure. A cineloop 810 comprising multiple frames 820 is fed into a trained object detector 830.

[0118] The object detector 830 may implement or include any suitable type of learning network. For example, in some aspects, the object detector 830 could include a neural network, such as a convolutional neural network (CNN). In addition, the convolutional neural network may additionally or alternatively be an encoder-decoder type network, or may utilize a backbone architecture based on other types of neural networks, such as an objectdetection network, classification network, etc. One example backbone network is the Darknet YOLO backbone, (e.g., Yolov3) which can be used for object detection. The CNN may for example include a set of N convolutional layers, where N may be any positive integer. Fully connected layers can be omitted when the CNN is a backbone. The CNN may also include max pooling layers and / or activation layers. Each convolutional layer may include a set of filters configured to extract features from an input (e.g., from a frame of the ultrasound video). The value N and the size of the filters may vary depending on the aspects. In some instances, the convolutional layers may utilize any non-linear activation function, such as for example a leaky rectified non-linear (ReLU) activation function and / or batch normalization. The max pooling layers gradually shrink the high-dimensional output to a dimension of the desired result (e.g., bounding boxes of a detected feature). Outputs of detection network may include numerous bounding boxes, with most having very low confidence scores and thus being filtered out or ignored. Fully connected layers may be referred to as perception or perceptive layers. In some aspects, perception / perceptive and / or fully connected layers may be found in object detector 830 (e.g., a multi-layer perceptron).

[0119] These descriptions are included for exemplary purposes; a person of ordinary skill in the art will appreciate that other types of learning models, with features similar to or dissimilar to those described above, may be used instead or in addition, without departing from the spirit of the present disclosure.

[0120] Outputs of the object detector 830 may include an annotated cineloop 840 made up of a plurality of annotated image frames 842, possibly including per-frame metrics 845 such as the confidence level of the detections.

[0121] The systems and methods disclosed herein are broadly applicable to different types of features, and can for example draw boxes around the head, heart, placenta, or other anatomical features depending on the implementation. The object detector can be one class or multi-class, depending how the model is built. If another detector is trained separately, then both models can be run separately (e.g., one model for each feature type). Otherwise, multiple feature classes can be identified, and enclosed in detection boxes, at the same time. In an example, the ML model for placenta detection can use exactly the same structure as a model for heart detection. One can either train / run a single detector that detects multiple feature types (a multi-class detector) and provides their locations as an output, along with the confidence score and feature type (class) of each detection. Alternatively, one could run several single-class detectors, each trained to detect a single feature type / class. These separate single-class detectors may have the same architecture (e.g., layers and connections),but would have been trained with different data (e.g., different images and / or annotations) and thus have different weights.

[0122] Figure 9 is a graph 900 indicating, on the Y-axis 910, the number of higher trimester anatomies detected vs. the number of lower trimester anatomies detected and, on the X-axis 920, the gestational age in weeks, according to aspects of the present disclosure. The graph 900 includes both raw detections 930 and a curve fit 940. In the graph 900, negative Y-values indicate a preponderance of detections of lower trimester anatomies such as the embryo, yolk sac, etc., whereas positive Y-values indicate a preponderance of detection of higher-trimester, non-cranial anatomies such as the heart, abdomen, spine, pelvis, etc. As can be seen in the graph 900, a preponderance of higher-trimester anatomies are detected after approximately 11 weeks of gestation. In some cases, 13 weeks may be considered the boundary between low and high trimester, but 11 weeks may be used for systems that rely on on yolk sac identification for the yolo models, which may be difficult after 11 weeks. This information, along with detections from the multi-class anatomy detector, can be used for automatic estimation of the trimester or gestation duration of the pregnancy, as described below.

[0123] Figure 10 is a graph 1000 indicating, on the Y-axis 1010, the number of higher trimester anatomies detected vs. the number of lower trimester anatomies detected and, on the X-axis 1020, the gestational age in weeks, according to aspects of the present disclosure. The graph 1000 includes both raw detections 1030 and a curve fit 1040. In the graph 1000, negative Y-values indicate a preponderance of detections of lower trimester anatomies such as the embryo, yolk sac, etc., whereas positive Y-values indicate a preponderance of detection of higher-trimester, anatomies such as the head, heart, abdomen, spine, pelvis, etc. As can be seen in the graph 1000, a preponderance of higher-trimester anatomies are detected after approximately 11 weeks of gestation. This information, along with detections from the multi-class anatomy detector, can be used for automatic estimation of the trimester or gestation duration of the pregnancy, as described below.

[0124] Figure 11 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example ultrasound blind sweep multiple pregnancy detection system 1100, according to aspects of the present disclosure. In the example of Figure 11, blind sweep data 1110 is received by an automatic trimester estimation module or automatic gestational age estimation module 1120. In a rural setting, there may not be a precise calculation of gestational age. Having the automatic trimester estimation module 1120 advantageously allows the user (e.g., a midwife) and the system to be able to understand ageneral estimate of the gestational age. The gestational age being higher trimester can then be used to determine whether there is multiple gestation or not, as described herein.

[0125] The gestational age estimation module 1120 then reports a lower trimester exam 1130 based on the anatomy detector 620 detecting anatomy of interest 1150 for an embryo (e.g., embryo, yolk sac, etc.), or a higher-trimester exam 1160 based on the anatomy detector 620 detecting anatomy of interest 1170 for a fetus (e.g., head, heart, abdomen, spine, pelvis, etc.). For a higher trimester exam 1160, the system 1100 then performs individual sweep processing or intra-sweep processing 1180 and / or multiple sweep or inter-sweep processing 1190, based on which a multiple gestation decision 1195 (e.g., a yes / no determination of whether multiple fetuses are present) is made.

[0126] Figure 12 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example individual sweep processing module 1180, according to aspects of the present disclosure. First, from raw detections, the system will form individual class-level connected graphs (e.g., clusters of head detections, clusters of heart detections, etc.) in a graph formation step. The individual anatomical / class graphs are then merged or connected, following an anatomical sequence based on proximity, to form the individual skeletal structures in a skeletonization step. As shown above in Figure 11, the individual sweep processing module 1180 receives outputs from the anatomy detector 620. Based on this information, the individual sweep processing module 1180 performs a graph formation step 1210 for the individual structures of the fetus(es). This may for example involve placing one point in a 3D graph for the center or other location of each detection box in frames of the individual sweep, based on a known or estimated position of the ultrasound probe for each frame of the sweep (see Figure 6, step / module 615). Examples of 3D graph formation are shown in Figures 14, 16, and 18. This may involve placement of head detections, heart detections, abdomen detections, etc. into a 3D space representing the maternal abdomen, and connecting the points that are determined to be likely part of the same anatomy.

[0127] The individual sweep processing module 1180 then performs a skeletonization step 1240 of individual structures following sub-plane classes identified by the anatomy detector. Skeletonization is the process of compact / core structural representation of any foreground object. In the present context, identifying the mid-point of individual detections of certain anatomies and connecting them internally, knowing their orientations via view plane-based detection classes (e.g. different sub-planes of head, heart, abdomen, and spine to aid this formation), is a step to clustering individual anatomical centerlines. This may befollowed by the skeletal structure (partial / full) formation of the fetus connecting different above-computed individual anatomical centerlines following their position and orientation.

[0128] Skeletonization thus involves identifying a geometric relationship (e.g., a centerline or other line) between different expected anatomical features. An example of skeletonization is shown in Figure 14. Skeletonization groups all or a subset of the detections that were graphed in graph formation step 1210. For an example, an individual skeleton is the subset of detections that are associated with an individual fetus. When there is multiple gestation / multiple pregnancy, the skeletonization step can result in grouping multiple subsets of detections (e.g., one skeleton for each individual fetus). For example, it may be expected that the heart of a fetus is located between the head and the abdomen, and that a centerline defined by the head, heart, and abdomen may indicate the position and orientation of the skeleton of a given fetus. In some aspects, a fetal orientation identification step 1220 may make use of this information to report fetal orientation, also known as fetal lie or fetal presentation.

[0129] The individual sweep processing module 1180 then performs a connecting / splitting 1250 of the anatomical graphs or skeletons based on anatomical context. For example, if two skeletons or centerlines are identified, it may be necessary to reassign points from one skeleton to the other, e.g., based on which centerline the point is closer to, or based on threshold distances, or otherwise. If two hearts are detected on either side of a head, then the connecting / splitting module may assign each heart to a different skeletonization. Proximity is a factor that rules in / out single / multiple fetus parts. However, to make this 3D grid from the individual sweep, the (9-axis) IMU information can be leveraged to correct the inter-alignment of frames, which in turn will help to form a more realistic skeletonization, by compensating for any non-uniform movement of the probe. This will help in refining the skeleton more realistically.

[0130] Once the initial connecting / splitting step 1250 is complete, the skeletons may be considered reportable single-sweep skeletonizations. In some aspects, the fetal orientation identification step 1220 (See Figure 6, step 1220) may make use of the single-sweep skeletonizations to determine or update fetal orientation.

[0131] In some aspects, a detector module refinement step 1230 (see step 645 of Figure 6) makes use of the single-sweep skeletonizations to refine the anatomy detector 620. This may be done, for example, by using the final skeletonizations as training data for the machine learning network.

[0132] Once the connecting / splitting step is done, execution may proceed to the visual representation or display step 430 (to display the results of a single sweep) and / or to the intersweep processing step 1190. Due to deployment complexity and compute limitations on resource-constrained solutions, and to achieve real-time processing, the system uses a single frame-based detection module (e.g. 2D YOLO). In other instances, 3D detection modules can be used (e.g. ROLO or Recurrent YOLO). In the absence of temporal learning, sometimes the outcome of these detection modules can fetch false positives, which can be corrected using information from the skeletonizations. These identified false detections can be used to retrain with negative classes to boost the detector outcomes in block 645 of Figure 6 and / or block 1230 of Figure 12.

[0133] To elaborate, identifying the mid-point of individual detections of certain anatomies and connecting them internally knowing their orientations via view plane-based detection classes (e.g. different sub-planes of head, heart, abdomen, and spine to aid this formation) is a step to cluster individual anatomical centerlining. This will form a raw connected graph (from top-down or vice versa). Due to (slight) fetal movement, non-tight detection outcomes, or false detections, these graphs of individual anatomies can be nonlinear. With the presence of surrounding anatomical graphs following an anatomical context / ordering, these graphs can be refined / combined following a more realistic (non)linear trajectory representing a skeletonization of a single fetus.

[0134] The addition of IMU data can bring a more realistic appearance to the constructed skeletons, by compensating for nonuniform probe movements.

[0135] Figure 13 is a schematic, diagrammatic representation, in block diagram form, of at least a portion of an example inter-sweep processing module 1190, according to aspects of the present disclosure. As shown above in Figure 11, the inter-sweep processing module 1190 receives outputs from the individual sweep processing module 1180, for multiple sweeps (in the example shown in Figure 13, for sweeps A and B). Based on this information, the inter-sweep processing module 1190 combines anatomical graphs and / or skeletonizations from multiple sweeps in a combining step 1305, then performs an additional connecting / splitting step 1310 based on geometric positioning within the 3D space. In the present context, identifying the mid-point of individual detections of certain anatomies and connecting them internally, knowing their orientations via view plane-based detection classes (e.g. different sub-planes of head, heart, abdomen, and spine to aid this formation) is a step to cluster individual anatomical centerlining. This will form a raw connected graph (from top- down or vice versa). Due to (slight) fetal movement or non-tight detection outcomes, or evenfalse detections, these graphs of individual anatomies can be highly nonlinear. With the presence of surrounding anatomical graphs following an anatomical context / ordering, these graphs can be refined / combined following a more realistic (non)linear trajectory representing a skeletonization of a single fetus or multiple fetuses.

[0136] In some cases, even if only one skeleton is initially identified, some detections may fall outside of a threshold distance, e.g., two clusters of head detections (whether from the same sweep or different sweeps) may be too far apart to plausibly belong to the same fetal head. In such cases, the head detections will be split and assigned to different entities (e.g., different fetuses). In other cases, even if two skeletons are initially identified, the connecting / splitting module 1310 may determine that all of the detections for a given anatomy fall within a threshold distance of one another, and may thus be connected as part of a single entity (e.g., a single fetus).

[0137] Thus, merging and splitting forms the multi-sweep graph structures and / or skeletonizations. First, from raw detections, connecting / splitting the dots of individual classes, the system will form individual class-level connected graphs (e.g., clusters of head detections, clusters of heart detections, etc.). The individual anatomical / class graphs are then merged or connected, following an anatomical sequence based on proximity, to form the individual skeletal structures. All of this proceeds repeatedly across multiple sweeps. Then, this splitting or merging of different anatomical graphs or skeletonizations from different sweeps helps in shaping the multi-sweep fetal skeleton(s). In some instances, the multisweep skeletons may be more accurate than the single-sweep skeletons, because they cover more area. In other instances, the single-sweep skeletonizations can be more accurate because of fetal movement during the time it takes to do multiple sweeps.

[0138] At the end, the spatial relationship between anatomical structures (e.g., head, spine, heart, etc.) aids in determining the fetal orientation, and with this orientation information, the system can form the multi-sweep fetal skeleton model (single or multiple) with orientation information. This may for example help in more realistic augmentation of fetal 3D diagrams (adjusting scaling, orientation, sheers, etc. on the fetal 3D templates) knowing the trajectory of actual fetal presence, which can be used for report generation. As a fmal / interim report-out of the derived clinical features from the blind sweep protocol, the system may showcase respective visual representations catering to end-users, e.g. showcasing specific cine-clips depicting cardiac activity. For example, in the case of the fetal presentation feature, the system may display a predefined fetal icon (2D / 3D) overlaying (in predefined size and orientation) on a rigid abdominal grid (2D / 3D). The present disclosureallows for a more realistic-sized fetus with a more accurate orientation with respect to the abdominal grid (considering scale, angle, shear, and even relative twists between different anatomies), bringing a more realistic representation for the end-user.

[0139] In some aspects, after completion of the connecting / splitting step 1310, the skeletonizations may be considered reportable multi-sweep skeletonizations.

[0140] The inter-sweep processing module 1190 then performs entity estimation 1320 (e.g., counting of the fetuses) based on the multi-sweep skeletons or connected anatomical graphs. Skeletonization (partial or full, in individual anatomical level and / or the inter-sweep or intra-sweep connection / refinement) in individual sweep level 1180 (e.g., in a particular direction) will be processed in (e.g. in step 1190 of Figure 11) to split / merge in graph space based on knowledge of anatomical proximity and presence. This is performed in both craniocaudal and lateral sweeps to form the finalized graph(s) / skeleton(s) 1320. Each identified skeleton in the multi-sweep skeletonizations will be considered a separate entity. Then, based on the number of entities (e.g., one vs. two or more), the determination of multiple gestation can be made.

[0141] Outputs of steps 1310 and 1320 can serve as inputs to either or both of the fetal orientation identification step 1220 or the detector module refinement step 1230. Once the entity estimation step 1320 is complete, execution proceeds to the visual representation (display) step 430.

[0142] Figure 14 is a 3D graph 1400 of head detections 1410, heart detections 1420, and abdomen detections 1430, according to aspects of the present disclosure. The detections 1410, 1420, 1430 (enclosed by respective illustrative ellipses 1415, 1425, and 1435) may for example come from a single sweep, and are positioned within a 3D space defined by an azimuthal axis 1440, a lateral axis 1450, and a depth axis 1460. In an example, the lateral axis 1450 and depth axis 1460 correspond to the exes of an ultrasound image frame, whereas the azimuthal axis 1440 corresponds to the time or position at which each image frame was captured during the sweep. A skeletonization process identified a skeletal centerline 1470 for a given entity (e.g., a given fetus), based on the positions or centers of the head detections 1410, heart detections 1420, and abdomen detections 1430.

[0143] In the example shown in Figure 14, the detections all come from a single fetus 1480, (e.g., as determined by the multiple pregnancy / gestation determination module 630 of Figure 6). This representation can be extended to multiple fetuses enabling the geometric mapping of the twin or multiple gestation. The skeletonization process can also be used to detect (or cancel out) movement based on a change in location or orientation of the centerlinewithin a sweep, or in between multiple sweeps. There are touchpoints where the system can retrospect the fetal movement including: (a) intra sweep: at the time of acquiring a particular sweep, the system can appreciate a fetal movement, which might captured by abrupt relative anatomical structural movement in the frame / sweeping planes, and (b) inter-sweep: between sweeps there might be movement of a fetus that already occurred, which can be realized by understanding sweep / frame-to-abdominal mapping, knowing the presence of the fetus in either instance. If the movement is large, that might be confused with multiple fetal entries, and may thus require further retrospection.

[0144] These slight fetal movements between sweeps can be taken care of by registering the skeletons, tackling slight deformation across or within sweeps. However, a larger deviation might be confused for multiple gestations. The use of both orthogonal sweeps will decrease this chance (as will a higher gestational age), whereas identification of heavy fetal movement while sweeping (frame-based) can be accounted for to safeguard the method as boundary conditions.

[0145] It is noted that in the example of Figure 14, the cranial direction of the fetus is roughly aligned with the azimuthal direction 1440 of the vertical sweeps, which should also be aligned with the cranial direction of the mother. Thus, the fetal lie determination module or fetal orientation determination module may determine that this fetus 1480 is in a breech orientation.

[0146] An objective of the skeletonization step is to see the distribution of skeletal structures of the same anatomies across multiple and singleton exams, as this graph formulation may be able to distinguish two or more fetuses. To form the skeletal structure (whether partial or full) of the fetus, connecting different anatomical classes following their position and orientations, the system makes use of different sub planes of head, heart, abdomen, spine to aid this formation.

[0147] While connecting the detection dots in 3D (e.g., graph formation), the system is also performing its pruning and refinement stages based on temporal position / consistency across intra-sweep and inter-sweep anatomies, as detections are not always foolproof.Rather, leveraging this graph formation can help refining the detection outcome by exploiting the temporal (and thus, spatial) relationship between detections. In an example, to make this 3D grid from an individual sweep, the system may make use of inertial measurement unit (IMU) information (e.g., 9-axis position, orientation, and speed information from an IMU located within the probe) to correct inter-alignment of frames (see step 615 of Figure 6),which in turn will help forming more realistic skeletonization by compensating for any non- uniform movement of the probe.

[0148] Figure 15 is an ultrasound image frame 1500 showing a fetal head 1510, according to aspects of the present disclosure. Within a sweep, numerous contiguous frames may show different cross-sections of the head 1510. Depending on the position of the fetus, the head 1510 will likely also show up in at least one vertical sweep and at least one horizontal sweep. By identifying other anatomical features along with the head (e.g., the heart, abdomen, etc.), the ultrasound blind sweep multiple pregnancy detection system can skeletonize the detections and thus determine the number and orientations of fetuses present in the womb.

[0149] Figure 16 is a 3D graph 1600 of head detections 1410 for a single fetus, according to aspects of the present disclosure. The detections 1410 may for example come from a single sweep, and are positioned within a 3D space defined by an azimuthal axis 1440, a lateral axis 1450, and a depth axis 1460. Additional detections may occur in different subplanes. Depending on the position of the fetus, the head 1510 will likely also show up in at least one vertical sweep and at least one horizontal sweep.

[0150] Figure 17 is a pair of ultrasound image frames 1700, 1720 from a single sweep, each showing a different fetal head 1710, 1730, according to aspects of the present disclosure. Because they are in two different positions within the womb, the two heads 1710, 1730 are visible in different portions of the sweep (in this case, frame 115 for head 1710 vs. frame 354 for head 1730). Depending on the positions and orientations of the fetuses, each head 1710, 1730 will likely show up in multiple frames of at least one vertical sweep and multiple frames of at least one horizontal sweep.

[0151] Figure 18 is a 3D graph 1800 of head detections for two fetuses, according to aspects of the present disclosure. The detections are positioned within a 3D space defined by an azimuthal axis 1440, a lateral axis 1450, and a depth axis 1460. In the example shown in Figure 18, detections 1810, 1820, and 1830 come from three imaging sub-planes, but are all assigned to the same fetal head 1710 (for one fetus), due to their proximity to one another. Similarly, detections 1870, 1880, and 1890 all come from different imaging sub-planes, but are not assigned to fetal head 1710 because they fall outside of a threshold distance and / or are attached to other anatomical features not associated with head 1710. Rather, the detections 1870, 1880, and 1890 are assigned to a separate head 1730 (for a second fetus). Additional detections may occur in different sweeps and / or different sub-planes. Depending on the position of the fetus, each head will likely also show up in multiple frames of at least onevertical sweep and multiple frames of at least one horizontal sweep. It is noted that combining two or more graphs can result in an increase in the size of one or more axes (e.g., the azimuthal axis, the lateral axis, and / or the depth axis 1460of the graph) because the axes of the two graphs are being combined. If two distinct head clusters are detected in larger proximity within and between sweeps, it is straightforward to determine that multiple gestations are present.

[0152] However, the false positives of each of these anatomies can create multiple distinct clusters that might be confused for MG, which can be ruled out heuristically using more hyperparameters. Hence, the skeletonization of different individual anatomies and their inter-relationship will help in nullifying these false cases.

[0153] In the case of only head detection within single and across sweeps, and in the absence of other anatomies (which is highly unlikely), the system may have to only rely on head detections and their clustering (e.g., skeletonization based on knowing the view planes of the detections, e.g. axial or others). However, with detection of other anatomical parts, it helps in shaping the anatomical skeletonization of complete or partial fetal structure, refining the MG determination. Moreover, multiple anatomical detections and their contextually connected skeletonization help in identifying fetal orientations.

[0154] As will be readily appreciated by those having ordinary skill in the art after becoming familiar with the teachings herein, the ultrasound blind sweep multiple pregnancy detection system advantageously permits untrained and minimally trained users to perform an ultrasound blind sweep protocol to gather anatomical images of high quality, including automated detection of potential health conditions such as multiple gestation (MG). This may result in higher accuracy and higher clinician trust in the results, while potentially improving health outcomes and / or decreasing the total cost of care. Potential benefits include detection of multiple pregnancies via blind sweeps performed by novice ultrasound users. The solution can be a quick initial check scan for a center with high volume ultrasound turnover to triage patients for a more detailed obstetric scan, and can provide or support referral of the subject diagnosed with multiple gestation for further diagnosis and management to a tertiary care center. Early detection of multiple gestation may be extremely helpful for follow-up and monitoring of the pregnancy.

[0155] The systems, methods, and devices described herein may be applicable in point of care and handheld ultrasound use cases such as with the Philips Lumify system. The ultrasound blind sweep multiple pregnancy detection system can be used for any handheld imaging applications, including but not limited to obstetrics and echocardiography. Theultrasound blind sweep multiple pregnancy detection system could be deployed on handheld mobile ultrasound devices, and on portable or cart-based ultrasound systems. The ultrasound blind sweep multiple pregnancy detection system can be used in a variety of settings including emergency departments, ambulances, accident sites, and homes. The applications could also be expanded to other settings.

[0156] The system is detectable from its functionality and output such as reporting of detected health conditions such as multiple pregnancy. This invention increases the value proposition of ultrasound applications in the obstetrics context, especially for use by minimally trained users.

[0157] Accordingly, the logical operations making up the aspects of the technology described herein are referred to variously as operations, steps, objects, layers, elements, components, models, or modules. Furthermore, it should be understood that these may occur or be performed or arranged in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0158] All directional references e.g., upper, lower, inner, outer, upward, downward, left, right, lateral, front, back, top, bottom, above, below, vertical, horizontal, clockwise, counterclockwise, proximal, and distal are only used for identification purposes to aid the reader’s understanding of the claimed subject matter, and do not create limitations, particularly as to the position, orientation, or use of the ultrasound blind sweep multiple pregnancy detection system. Connection references, e.g., attached, coupled, connected, joined, or “in communication with” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and / or” rather than “exclusive or.” The word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

[0159] The above specification, examples and data provide a complete description of the structure and use of exemplary aspects of the ultrasound blind sweep multiple pregnancy detection system as defined in the claims. Although various aspects of the claimed subject matter have been described above with a certain degree of particularity, or with reference to one or more individual aspects, those skilled in the art could make numerous alterations to the disclosed aspects without departing from the spirit or scope of the claimed subject matter.

[0160] Still other aspects are contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular aspects and not limiting. Changes in detail or structure may be made without departing from the basic elements of the subject matter as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: a processor configured for communication with an ultrasound probe, wherein the processor is configured to: control the ultrasound probe to obtain a plurality of ultrasound image frames during a blind sweep protocol on a patient with a pregnancy; provide the plurality of ultrasound image frames as an input to a deep learning network trained to detect fetal anatomy; generate, as an output of the deep learning network, detections of a plurality of detections of at least one fetal anatomical part within the plurality of ultrasound image frames; place the detections into a 3D graph; determine, using the 3D graph, at least one skeletonization; based on the at least one skeletonization, determine whether the pregnancy comprises a multiple gestation; and provide, to a display in communication with the processor, an output representative of the determination of whether the pregnancy comprises the multiple gestation.

2. The system of claim 1, wherein the processor is further configured to, based on the at least one skeletonization: determine an orientation of at least one fetus; and provide, to a display in communication with the processor, an output representative of the determination of the orientation of the at least one fetus.

3. The system of claim 1, wherein the processor is further configured to, based on the at least one skeletonization, update the deep learning network.

4. The system of claim 1, wherein the plurality of detections of at least one fetal anatomical part includes detections of at least a head, a heart, and an abdomen of at least one fetus.

5. The system of claim 4, wherein the determining the at least one skeletonization involves identifying a centerline between the head, heart, and abdomen of the at least one fetus.

6. The system of claim 4, wherein the processor is further configured to merge or split the at least one skeletonization based on anatomical context.

7. The system of claim 4, wherein the processor is further configured to merge or split the at least one skeletonization based on geometric positioning.

8. The system of claim 1, wherein the deep learning network is a multi-class anatomy detector.

9. The system of claim 1, wherein placing the detections into the 3D graph involves inertial measurement unit (IMU) data associated with movements of the ultrasound probe.

10. The system of claim 1, wherein determining whether the pregnancy comprises a multiple gestation involves counting a number of skeletons formed by the at least one skeletonization.

11. The system of claim 1, further comprising the ultrasound probe.

12. A method, comprising: with a processor configured for communication with an ultrasound probe: controlling the ultrasound probe to obtain a plurality of ultrasound image frames during a blind sweep protocol on a patient with a pregnancy; providing the plurality of ultrasound image frames as an input to a deep learning network trained to detect fetal anatomy; generating, as an output of the deep learning network, a plurality of detections of at least one fetal anatomical part within the plurality of ultrasound image frames; placing the detections into a 3D graph; determining, using the 3D graph, at least one skeletonization;based on the at least one skeletonization, determining whether the pregnancy comprises a multiple gestation; and providing, to a display in communication with the processor, an output representative of the determination of whether the pregnancy comprises the multiple gestation.

13. The method of claim 12, further comprising, based on the at least one skeletonization: determining an orientation of at least one fetus; and providing, to a display in communication with the processor, an output representative of the determination of the orientation of the at least one fetus.

14. The method of claim 12, further comprising, based on the at least one skeletonization, updating a training of the deep learning network.

15. The method of claim 12, wherein the plurality detections of at least one fetal anatomical part includes detections of at least a head, a heart, and an abdomen of at least one fetus.

16. The method of claim 15, wherein the determining the at least one skeletonization involves identifying a centerline through the head, heart, and abdomen of the at least one fetus.

17. The method of claim 15, further comprising merging or splitting the at least one skeletonization based on anatomical context.

18. The method of claim 15, further comprising merging or splitting the at least one skeletonization based on geometric positioning.

19. The method of claim 12, wherein the deep learning network is a multi-class anatomy detector.

20. The method of claim 12, wherein placing the detections into the 3D graph involves inertial measurement unit (IMU) data associated with movements of the ultrasound probe.

Images