Left atrial appendage segmentation and quantification in 3D and 4d cardiac ultrasound

Abstract

To provide systems and methods for real-time three-dimensional left atrial appendage (LAA) quantification.SOLUTION: Deep learning is used to perform LAA segmentation, tracking and live quantification on critical measurement in 4D ultrasound sequences. The collected information may be used for device selection and anomaly detection to assist early-phase disease finding.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to left atrial appendage segmentation and analysis.

Background Art

[0002] The left atrial appendage (LAA) is a small pouch extending from the left atrium of the heart that functions as a decompression chamber when atrial pressure is high. The LAA plays an important role in the thromboembolic risk associated with atrial fibrillation and may also have the potential to induce atrial tachycardia. Atrial fibrillation (A-fib) is a type of abnormal heart rhythm in which the upper heart chambers beat irregularly and rapidly, increasing the risk of thrombus formation within the heart. If a thrombus in the upper left heart chamber (left atrium) breaks out of the heart area, the thrombus may move to the brain and cause a stroke, which is a major cause of death. Studies have shown that the left atrial appendage (LAA) is a site where thrombosis is particularly common in people with A-fib.

[0003] Understanding the morphology and function of the LAA is extremely important for the treatment of A-fib. However, in most cases, due to factors such as the small size of the TAA and its position away from the transducer, it is impossible to clearly detect the LAA by ordinary two-dimensional transthoracic echocardiography (2D·TTE) alone. Real-time three-dimensional echocardiography (RT-3D·TEE) imaging can be used to significantly improve the visualization of complex 3D structures such as the LAA by maintaining spatial and temporal resolution. 3D intracardiac echocardiography (ICE) can also be used to capture images of the LAA. However, existing approaches do not focus on 4D ultrasound sequences such as 4D intracardiac echocardiography, especially live tracking and measurement of the LAA. Live tracking and measurement of the LAA in 4D ultrasound sequences have unique challenges, including high anatomical variability in each cardiac cycle and a large amount of data related to real-time processing.

Summary of the Invention

[0004] Initially, the preferred embodiments described below include methods, systems, instructions, and computer-readable media related to real-time segmentation and quantification of the LAA in 3D and 4D cardiac ultrasound examinations.

[0005] In a first aspect, a real-time three-dimensional left atrial appendage quantification method includes obtaining a patient's real-time three-dimensional ultrasound sequence (a series of ultrasound images), selecting a seed position of the left atrial appendage in a frame of the real-time three-dimensional ultrasound sequence, cropping the frame to a region of interest based on the seed position, generating a three-dimensional left atrial appendage segmentation from the region of interest, extracting one or more landmarks from the three-dimensional left atrial appendage segmentation, slicing the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each including a part of the contour of the left atrial appendage, tracking the contour of the left atrial appendage and one or more landmarks in the plurality of two-dimensional slices and in the real-time three-dimensional ultrasound sequence, generating a real-time three-dimensional left atrial appendage segmentation from the tracked contour, and quantifying at least one metric from the tracked contour and / or the tracked one or more landmarks.

[0006] In one form, selecting the seed position includes stopping the real-time three-dimensional ultrasound sequence and the user clicking on a seed position within the left atrial appendage in the stopped frame of the real-time three-dimensional ultrasound sequence. The real-time three-dimensional ultrasound sequence is unstopped after the seed position is selected.

[0007] In one form, cropping includes calculating an estimated shape and size of the left atrial appendage based on a dataset collecting images of the left atrial appendages of a plurality of patients and centering the region of interest around the seed position based on the estimated shape and size.

[0008] Three-dimensional left atrial appendage segmentation can be generated using a machine-trained network. One or more landmarks may include at least one inlet and neck landmark that define the diameter of the inlet and neck regions of the left atrial appendage. The contour and one or more landmarks can each be tracked in real time using a neural network trained using a cycle consistency loss.

[0009] In one form, at least one metric provides the diameter of the left atrial appendage over time. The diameter of the left atrial appendage can be used to determine and select the characteristics of a device during a left atrial appendage closure procedure that match the characteristics of the patient's left atrial appendage.

[0010] In one form, a real-time three-dimensional ultrasound sequence is provided by an ultrasound system configured for intracardiac echocardiography.

[0011] In a second aspect, a real-time three-dimensional left atrial appendage segmentation system is provided. The system includes an ultrasound imaging system and a processor. The ultrasound imaging system is configured to acquire a 4D ultrasound sequence of a patient. The processor selects a seed position of the left atrial appendage in a frame of the 4D ultrasound sequence, crops the frame to a region of interest based on the seed position, generates a three-dimensional left atrial appendage segmentation from the region of interest, extracts one or more landmarks from the three-dimensional left atrial appendage segmentation, slices the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each containing a portion of the contour of the left atrial appendage, tracks the contour over time in the plurality of two-dimensional slices and tracks one or more landmarks in the 4D ultrasound sequence, and quantifies at least one metric from the tracked contour and / or the tracked one or more landmarks.

[0012] In one form, cropping includes calculating an estimated shape and size of the left atrial appendage based on a dataset that has collected images of the left atrial appendages of a plurality of patients, and centering the region of interest around the seed position based on the estimated shape and size.

[0013] The processor may be configured to generate three-dimensional left atrial appendage segmentation using a machine-trained network. The one or more landmarks include at least one inlet and neck landmark that determines the diameter of the inlet and neck regions of the left atrial appendage. The processor is configured to use neural networks trained using cycle consistency loss to track the contour and one or more landmarks in real time, respectively. At least one metric provides the diameter of the left atrial appendage over time. The diameter of the left atrial appendage is used to determine and select the characteristics of the device during left atrial appendage closure procedures that match the characteristics of the patient's left atrial appendage.

[0014] In a third aspect, a system is provided that includes an ultrasound imaging system, a display, an input device, and a processor. The ultrasound imaging system is configured to acquire a 4D ultrasound sequence of a patient. The display is configured to display the 4D ultrasound sequence. The input device is configured to identify a seed position of the left atrial appendage in a frame of the 4D ultrasound sequence. The processor is configured to crop the frame to a region of interest based on the seed position, generate a three-dimensional left atrial appendage segmentation from the region of interest, slice the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each including a portion of the contour of the left atrial appendage, track the contour in real time in the plurality of two-dimensional slices, and generate a real-time three-dimensional left atrial appendage segmentation from the tracked contour.

[0015] Any one or more of the above-described aspects can be used alone or in combination. These and other aspects, features, and advantages will become apparent from the following detailed description of the preferred embodiments read in conjunction with the accompanying drawings. The invention is defined by the claims, and nothing herein should be construed as limiting the claims. Further aspects and advantages of the invention will be described below in conjunction with the preferred embodiments, but these may be claimed later independently or in combination.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Best Mode for Carrying Out the Invention

[0017] The embodiments described herein provide a system and method for left atrial appendage segmentation and quantification (numericalization) in 3D and 4D cardiac ultrasound procedures. The embodiments provide a system framework that uses deep learning to perform LAA segmentation, tracking, and live quantification for important measurements in 4D ultrasound sequences. The information collected can be used for device selection and anomaly detection to assist in early disease detection. The systems and methods described herein can be applied to both 3D ultrasound volumes and 4D ultrasound sequences.

[0018] Figure 1 shows an example of the left atrium 20 and the surrounding structures of the region of interest. The structures include the left atrial appendage (LAA) 26, the left inferior pulmonary vein (LIPV) 25, the left superior pulmonary vein (LSPV) 28, the right inferior pulmonary vein (RIPV) 22, and the right superior pulmonary vein (RSPV) 24. The left ventricle or other cardiac chambers can be the anatomical structures of interest when performing an ultrasound procedure. The embodiments described herein specifically focus on the LAA, but the methods and systems can be applied to other organs, tissues, or structures that can benefit from real-time segmentation and quantification.

[0019] Everyone has an LAA, but the size and anatomical structure of each patient's LAA vary, and thus the problems that the LAA can cause also vary. As shown in Figure 1, the LAA is located on the side of the pulmonary veins that carry blood from the lungs. The position of the LAA allows blood to pool there without flowing into the left ventricle of the heart. In patients with a normal heart rhythm, the left atrial appendage contracts rhythmically along with the rest of the left atrium. This rhythmic contraction ensures that the blood in the pouch is ejected into the left ventricle when the left atrium empties and is then pumped out to the body. On the other hand, patients with AFib experience insufficient atrial contractions during an episode. This weak contraction, combined with blood stasis, can lead to the formation of thrombi that can form within the LAA. Anticoagulant therapy is typically used as the primary thromboprophylactic treatment for patients with atrial fibrillation, but long-term adherence is poor and there can be a risk of hemorrhagic complications. Left atrial appendage closure / (LAAC) is a minimally invasive cardiac intervention to prevent the formation of thrombi within the LAA by closing the LAA with a device. This technique can significantly reduce the risk of stroke in people with A-fib who are at high risk of developing thrombi. This functions as an alternative solution to anticoagulant therapy and is recommended for people with A-fib who cannot take anticoagulants.

[0020] Understanding the morphology of the LAA is an important step towards the success of LAA intervention. Additional information such as the patient's CT data can also be collected prior to the procedure (which may be a few days before the procedure) to assist in guiding the LAA intervention. However, the structure and characteristics of the patient's LAA can change during the intervention. Real-time segmentation and quantification can solve this problem. Existing solutions in ultrasound examinations only focus on 3D·TEE and do not obtain any movement information of the LAA. The embodiments described herein provide live (active) segmentation and quantification of the LAA. This embodiment provides structure tracking for both LAA landmarks and 2D slices. By integrating and tracking the 2D slices in 3D, the system and method track the 3D·LAA structure over time. The resulting quantification and analysis can be used for device selection, verification, and anomaly detection for early disease detection. By analyzing changes in LAA measurements, anomaly detection can be performed to detect abnormal activity of the LAA, which can be used to facilitate diagnosis and early disease detection.

[0021] Figure 2 shows an ultrasonic system 100 for real-time quantification of 4D ultrasonic sequences. This system includes an image processing system 100, a medical imaging device 130, and optionally a server 140. The server 140 can be configured to perform any of the tasks of the image processing system 100, including processing and / or storing data and models. The server 140 may be a cloud-based platform or may include such a platform. The image processing system 100 includes a processor 110, a memory 120, and a display 115. The image processing system 100 is included in or connected to the medical imaging device 130. The image processing system 100 receives image data from the medical imaging device 130, crops and segments the LAA (or other organ) based on a seed position provided by an operator or automatically, identifies landmarks in the segmented LAA, slices this segmentation into 2D slices, and uses these slices and landmarks to track and quantify the LAA over time. The image processing system 100 can be further configured to generate extended image data that depicts the contour of the LAA in real time in a 4D ultrasonic sequence. The image processing system 100 can also be configured to train or store a machine learning model related to these tasks. The imaging data is acquired in real time from the medical imaging device 130, for example, as a 4D (3D + time) ultrasonic sequence. Additional, different, or fewer components may be provided. For example, a computer network may be included for remotely processing ultrasonic data captured on-site, for example, by the server 140. In another example, a user input device (e.g., a keyboard, button, slider, dial, trackball, mouse, or other device) is provided for user modification or placement of one or more markers. In yet another example, one or more devices or components used in a medical procedure such as left atrial appendage closure (LAAC) may be included.For example, a device that blocks the opening of the left atrial appendage to prevent thrombi formed within the LAA from entering the bloodstream, a device that clamps and closes the base of the LAA, or among other medical devices, a device that uses a band or a loop of suture to close the LAA in particular.

[0022] The medical imaging device 130 can be an ultrasonic system 130 configured to generate live (4D) ultrasonic sequences. Ultrasonic imaging uses sound waves to image internal structures. As techniques, in particular, it includes transthoracic echocardiography (TTE), transesophageal echocardiography (TEE), and intracardiac echocardiography (ICE). TTE is a non-invasive procedure in which a transducer (or probe) is placed on the patient's chest. Images are recorded using ultrasonic data. In the case of TEE, the probe is passed through the patient's esophagus to place it near the patient's heart. The probe may have an ultrasonic transducer at its tip to provide an imaging function. ICE uses a catheter transducer element. The catheter is sent through the femoral vein to the heart. ICE can be used to perform an echocardiogram that uses sound waves to generate detailed images of the heart valves along with detailed images of the size, structure, and function of the heart. Echocardiography can also be used to measure the blood volume of the heart, the velocity and direction of blood flow through the heart. In one embodiment, in the case of a LAAC procedure, a catheter is introduced via femoral vein access and placed in either the right atrium, right ventricular outflow tract, coronary sinus, or left atrium. Real-time and 3D volumetric acquisition makes it possible to reduce the need to acquire images from different anatomical positions with respect to 4D volume ICE technology, and the coronal plane enables a planar evaluation of the LAA anatomical structure, facilitates spatial orientation, and provides circumferential perivalvular and peridevice flow evaluation using color Doppler.

[0023] Processor 110 is a general-purpose processor, a digital signal processor, a graphics processing unit, an application-specific integrated circuit, a field-programmable gate array, an artificial intelligence processor, a digital circuit, an analog circuit, a combination thereof, or any other device currently known or later developed for real-time quantification of 4D ultrasound sequences among other processes described below. Processor 110 is a single device, multiple devices, or a network. In the case of multiple devices, parallel or sequential partitioning of processing may be used. Each device constituting Processor 110 may perform a separate function. In one embodiment, Processor 110 is the control processor of medical imaging device 130 or another processor. In other embodiments, Processor 110 is part of a separate workstation or computer. Processor 110 operates according to instructions stored to execute the various processes described herein. Processor 110 is configured by software, design, firmware, and / or hardware and executes any one or all of the processes of FIGS. 3-6 and any other calculations described herein.

[0024] In one embodiment, the processor 110 is configured to perform segmentation, for example, using a machine-trained model. The processor 110 is configured to crop and segment a live ultrasound sequence. Image segmentation extracts or identifies regions of interest (ROIs) through a semi-automatic or automatic process. Segmentation divides an image into regions based on a specified description, such as segmenting body organs / tissues. The segmented data can be used for various purposes, such as tracking the movement of each organ or feature (here, the LAA). Segmentation can be provided by a segmentation model and the output of the medical imaging device 130. Various models or networks can be trained and used for the segmentation task. The segmented data includes a plurality of voxels. Each of the voxels represents a three-dimensional display element. For example, one voxel represents a certain amount of 3D data, similar to how a pixel is a point or a cluster of points in two-dimensional data. The processor 110 is further configured to slice the 3D segmentation of the LAA into a plurality of 2D slices and process the 2D slices in parallel. The processing of the 2D slices can include contour tracking and landmark identification through subsequent frames of the live ultrasound sequence. The processor 110 is further configured to quantify and analyze metrics regarding the LAA based on the tracked LAA contour and landmarks. Regarding segmentation, landmark detection, and tracking, the processor 110 can apply one or more trained machine learning networks or models trained for each task (segmentation, detection, tracking, etc.).

[0025] One or more machine learning networks or models may include a neural network defined as a plurality of sequential feature units or layers. Sequential is used to refer to the general flow where feature values are output from one layer and input into the next layer. Sequential is used to refer to the general flow where feature values are output from one layer and input into the next layer. Information from the next layer is fed to the next layer and continues until the final output. The layers can be sent only in the forward direction or can be bidirectional, including some feedback to the previous layer. Each node of a layer or unit can be connected to all or only a subset of the nodes of the previous and / or subsequent layers or units. Skip connections can also be used, such as a layer that outputs to other layers as well as the next layer in sequence. Instead of pre-programming features and attempting to associate features with attributes, deep architectures are defined to learn features at various levels of abstraction based on input data. Features are learned to reconstruct lower-level features (i.e., more abstract or compressed-level features). Each node of a unit represents a feature. Separate units are provided to learn different features. Various units or layers can be used, such as convolutional, pooling (e.g., max pooling), deconvolutional, fully connected, or other types of layers. Any number of nodes can be provided within a unit or layer. For example, 100 nodes are provided. Subsequent or later units can have more, fewer, or the same number of nodes.

[0026] Image data, a machine-trained network, training data, calculated metrics, and other data may be stored in the memory 120. The memory 120 may be or include an external storage device, RAM, ROM, a database, and / or local memory (e.g., a solid state drive or a hard drive). The same or another non-transitory computer-readable medium may be used for instructions and other data. The memory 120 may be implemented using a database management system (DBMS) resident in the memory 120 such as a hard disk, RAM, or removable media. Alternatively, the memory 120 is internal to the processor 110 (e.g., a cache). Instructions for performing the processes, methods, and / or techniques described herein are provided in a non-transitory computer-readable storage medium or memory such as a cache, buffer, RAM, removable media, hard drive, or other computer-readable storage medium (e.g., the memory 120). The instructions are executable by the processor 110 or another processor. The computer-readable storage medium includes various types of volatile and non-volatile storage media. The functions, processes, or tasks illustrated or described herein are executed in response to one or more sets of instructions stored in the computer-readable storage medium. These functions, processes, or tasks may be executed by software, hardware, integrated circuits, firmware, microcode, etc. operating alone or in combination separate from the set of instructions, storage medium, processor 110, or processing strategy. In one embodiment, the instructions are stored in a removable media device read by a local or remote system. In other embodiments, the instructions are stored remotely such that they are transferred via a computer network. In other embodiments, the instructions are stored in a given computer, CPU, GPU, or system. Since some of the components and method steps constituting the system shown in the drawings may be implemented in software, the actual connections between system components (or process steps) may vary depending on the programming techniques of this embodiment.

[0027] The display 115 is configured to display the user's model or otherwise provide it to the user. The display 115 is a CRT, LCD, projector, plasma, printer, tablet, smartphone, or other currently known or later developed display device for displaying output.

[0028] Figure 3 shows a method for quantifying a 4D ultrasound sequence in real time. Starting with a live 4D ultrasound sequence, a hybrid 3D-2D segmentation and tracking framework is used to process high-throughput data and enable real-time quantification and analysis of the LAA.

[0029] In process A110, a live ultrasound sequence is acquired. The method is performed by an ultrasound scanner for medical diagnosis. A plane or volume can be acquired. In one embodiment, the transducer of an ICE catheter provides a 3D volume scan. Each scan by the ICE catheter results in 3D volume visualization of the heart. Different volumes are scanned according to the movement of the catheter (e.g., translating or rotating). Each scan generates a frame of data representing the volume at that time. The volume of scan data can be represented by scalar values or display values (e.g., RGB) in polar or Cartesian coordinate formats. The ultrasound data can be B-mode, color flow, or other ultrasound images. The sequence of data frames is derived from ICE imaging.

[0030] In one example, intracardiac echocardiography (ICE) is used to acquire images of heart structures (e.g., the left atrium and pulmonary veins). These images are used to guide cardiologists. This requires accurate delineation of the boundaries of distinct anatomical structures. The context of use is for treating atrial fibrillation under the guidance of ICE imaging. The left atrium can be the most common anatomical structure of interest in this context.

[0031] A live 4D ultrasound sequence can be acquired, for example, by a 4D echo catheter that is small enough to be inserted into a heart chamber. The 4D echo catheter is improved compared to a standard intracardiac echo catheter that can only provide 2D images and enables a physician to view the structures and blood flow during the activity inside the heart. This live view from inside the heart provides useful information to clinicians, especially in areas of the heart that are difficult to appropriately observe using transthoracic echocardiogram or transesophageal echocardiogram. The 4D technology provides more detailed visual details to guide interventional cardiac procedures such as left atrial appendage closure, mitral valve repair, and tricuspid regurgitation treatment. Also, 4D echo does not require the patient to be anesthetized, which is important for patients with sedation sensitivity. FIG. 4 shows an example of a 3D volume including the LAA401. The 4D ultrasound stream can be provided to the operator in color. A graphical interface with two or more windows can be used to render the 3D volume and various 2D slices.

[0032] In process A120, a target frame is selected and a seed is placed / positioned in the LAA within this selected target frame. FIG. 5 shows a user interface that can be used to select the target frame and the seed position 301. The user interface of FIG. 5 includes a target frame that includes three images of a 2D plane 510 and a representation of a 3D volume 520. The target frame includes the LAA 401. The target frame can be any frame, but in practice, the frame in which the LAA 401 is well visualized is the target frame. Such a frame can be selected automatically by the system or manually by the operator. FIG. 6 shows a cursor 601 for selecting or identifying the seed position 301. The drawing of the seed position 301 and the LAA 401 in this frame serves as an initial point for temporal tracking and analysis. The seed position 301 can be automatically placed by searching the image frame for the LAA 401. Alternatively, for example, if the frame and the seed position 301 are manually selected, the user can place the seed position 301 within the LAA 401, for example, by using an input device to place a dot inside the LAA anatomical structure using, for example, the cursor 601. The seed position 301 helps the system quickly locate the approximate position of the LAA 401 within the field of view and helps significantly reduce the computational time consumed by searching the entire image.

[0033] In process A130, the selected target frame is cropped to the region of interest 705 that includes the LAA 401. Then the LAA is segmented by processing the region of interest 705 using the 3D segmentation network 701. FIG. 7 shows an example of cropping to the region of interest 705 and segmentation by the 3D segmentation network 701 that outputs the 3D segmentation 703 of the LAA 401. Different methods may also be used for cropping and segmentation. The region of interest 705 to be cropped can be a region that completely contains the LAA 401 (or another target organ or feature). The pixels representing the LAA 401 are not excluded. However, the region of interest 705 after cropping is smaller than the target frame. By cropping the target frame to the region of interest 705, the computational resources required for subsequent segmentation of the LAA 401 can be reduced compared to the case where the entire image needs to be searched / segmented. The region 705 to be cropped can include a buffer region around the LAA 401 so that the region to be cropped always includes the LAA 401 even if it is supposed to move or change. The size of the region to be cropped can be determined by obtaining a dataset of possible target frames and determining the minimum size required to contain the LAA 401.

[0034] Various methods can be used for segmentation. For example, segmentation can be threshold-based, region-based, shape-based, model-based, neighborhood-based, and / or machine learning-based, among numerous segmentation techniques. The threshold-based method segments image data by creating a binary partition (bisection) based on the image attenuation value, such as determined by the relative attenuation of structures in the image. In region-based segmentation, one pixel in the image is compared with adjacent pixels, and when a predetermined region criterion (e.g., homogeneity) is satisfied, the pixel / voxel is assigned to the same class as one or more of its adjacent regions. Shape-based techniques use either an atlas (reference image)-based approach or a model-based approach to find the boundary of LAA401. The model-based method uses prior shape information, similar to the atlas-based approach, but in order to better adapt to shape variations, the model-based approach can fit either a statistical shape or an appearance model of LAA401 to the image by using an optimization procedure. The neighborhood anatomy-guided method uses the spatial context of adjacent anatomical objects. In the machine learning-based method, boundaries are predicted based on features extracted from the image data.

[0035] In at least one embodiment, a machine learning network (3D segmentation network 701) is used to crop and / or segment the LAA 401 within the region of interest. The seed position 301 can be used to guide the cropping process. Statistical prior information regarding the shape and size of the LAA 401 can be calculated from the patient dataset being collected. Based on this prior information, the region of interest is cropped in 3D, centered around the seed from the previous step. The 3D segmentation network 701 is used to perform a 3D segmentation of the LAA 401. This process can be continued in each of the subsequent frames to generate a 4D segmentation of the LAA 401. However, since 3D processing using neural networks is time-consuming, it poses a challenge for real-time processing. To overcome this processing and time challenge, a technical solution is applied by extracting the anatomical landmarks and 2D projections of the segmentation from the 3D segmentation 703, as detailed below. The processing of the 2D projections can be done in parallel and thus enables real-time 4D segmentation and analysis. This provides an efficient way to process high-throughput data while providing live calculation of the LAA 401 statistics.

[0036] In one embodiment, the processor 110 generates a 3D segmentation 703 from the input of the ICE volume to the machine learning network 701. The 3D segmentation 703 is a labeling by voxels or positions of different anatomical structures. Each anatomical structure represented by each position is labeled. Alternatively, the segmentation 703 forms a 3D mesh for each anatomical structure. Other segmentation results may be provided. The 3D segmentation 703 provides boundaries regarding one or more structures in 3D. The segmentation 703 is one or more structures of interest, such as identifying a subset or all positions of the anatomically relevant structures of the left atrium.

[0037] 3D segmentation 703 uses machine learning network 701. The ICE volume is input into the machine learning network 701 regardless of the presence or absence of other data, and in response, 3D segmentation 703 is output.

[0038] Machine learning for image segmentation can be performed by carefully selecting and extracting features from the input image. Such features can include, for example, pixel gray levels, pixel positions, image moments, information about the neighborhood of pixels, and the like. Then, a vector of image features is supplied to a trained classifier that classifies each pixel of the image into a class. The parameters of the classifier are automatically learned by providing a classifier input image for which the ground truth classification result is known. Next, the output of the model can be compared with the ground truth, and the parameters of the model are adjusted so that the output of the model better matches the ground truth value. This procedure is repeated for a large number of input images, and the learned parameters are generalized for new unseen examples. The process of adjusting the parameters of the model is called training. Deep learning can also be used for segmentation using, for example, neural networks. Deep learning-based image segmentation can be performed using, for example, a convolutional neural network (CNN). A convolutional neural network has a layered structure in which a series of convolutions are performed on the input image. The kernels of the convolutions are learned during training. The convolution results are then combined using a trained statistical model that outputs the segmented image.

[0039] The machine learning network 701 can be an image-to-image network such as a fully convolutional U-net trained to convert ICE volumes into 3D segmentations. For example, trained convolutional units, weights, links, and / or other network characteristics are applied to the ICE volume data and / or derived feature values to extract corresponding features through multiple layers and output a 3D segmentation. The input features are extracted from ICE images arranged in 3D. Other more abstract features can be extracted from the features extracted using the architecture. Depending on the number and / or arrangement of units or layers, other features are extracted from the input. The network includes an encoder (convolutional) network and a decoder (transposed convolutional) network, which form a "U" shape with connections between them to pass features from the encoder to the decoder with the maximum level of compression or abstraction. Skip connections can be provided. Any of the currently known or later-developed U-Net architectures can be used. Other fully convolutional networks can be used. In one embodiment, the network is a U-Net with one or more skip connections. The skip connections pass features from the encoder to the decoder at levels other than the most abstract level of abstraction or resolution (i.e., other than the bottleneck). The skip connections provide more information to the decoding layer. The fully convolutional layer can be at the bottleneck of the network (i.e., between the encoder and the decoder at the most abstract level layer). The fully connected layer can ensure that as much information as possible is encoded. Batch normalization can be added to stabilize the training.

[0040] Any machine learning architecture can be used for segmentation. Similarly, different machine learning architectures can be used for the other tasks described herein. For example, U-Net is used. A convolutional-transposed convolutional network is used. One segment of layers or units applies a convolution to increase abstraction or compression. And the most abstract feature values are output to another segment. Next, another segment of layers or units applies a transposed convolution to reduce abstraction or compression, and as a result, outputs an indicator of class membership by position. The architecture can be a fully convolutional network. Other deep networks can be used.

[0041] The network used here can be defined as a plurality of sequential feature units or layers. Sequential is used to indicate the normal flow where the features output from one layer are input to the next layer. Information from the next layer is sent to the next layer and continues until the final output. The layers can be sent only in the forward direction or can be bidirectional including some feedback to the previous layer. Each node of the layer or unit can be connected to all or only a subset of the nodes of the previous or subsequent layer or unit. Any number of nodes can be provided within the unit or layer. For example, 100 nodes are provided. Subsequent or later units can have more, fewer, or the same number of nodes. The features of the nodes are learned mechanically using any building block. For example, an autoencoder (AE) or restricted Boltzmann machine (RBM) approach is used. The AE linearly transforms the data and then applies a non-linear rectification such as a sigmoid function. The objective function of the AE is the expected mean squared error between the input image and the reconstructed image using the learned features. The AE can be trained using stochastic gradient descent or other approaches to learn the features leading to the best reconstruction. The objective function of the RBM is the energy function. The exact calculation of the likelihood term associated with the RBM is difficult to handle. Therefore, an approximate algorithm such as contrastive-divergence based on k-step Gibbs sampling is used to train the RBM to reconstruct an image from the features. The training of the AE or RBM tends to overfit to the high-dimensional input data. Sparsity or noise removal techniques (e.g., sparse denoising AE (SDAE)) are used to constrain the degrees of freedom of the parameters and the forced learning of the structure of interest in the data. Enforcing sparsity in the hidden layer (i.e., only a few units in the hidden layer are activated at a time) can also regularize the network. In other embodiments, at least one unit is a convolution with ReLU activation or is batch normalization with ReLU activation followed by a convolutional layer (BN+LeakyRU+convolution).Max pooling, upsampling, downsampling, and / or a softmax layer or unit may be used. Each individual unit may be of the same or different type.

[0042] In process A140, 3D segmentation 703 is passed through 3D detection network 801 to extract important landmarks 803 for LAA 401. This includes, for example, inlet and neck landmarks that determine the diameters of the inlet and neck regions of LAA 401. FIG. 8 shows an example of a landmark detection process including 3D detection network 801 and a plurality of landmarks 803. Understanding the sizes of these portions is important for determining the size of the implant device. Any method for landmark detection may be used. Similar to the segmentation task of process A130, a machine learning / deep learning method may be used. Network 801 may identify / detect landmarks based on previously annotated images / volumes. A classifier that identifies / labels voxels or pixels within the input image may also be used. The output of this process is the identification of one or more landmarks within the 3D volume / 3D segmented volume.

[0043] In process A150, the 3D segmentation volume 703 of the 4D ultrasound sequence is sliced into 2D slices 901 and, in process A160, processed as described below. The sliced output is a 2D ultrasound image having a 2D contour 903 in the 2D projection of the LAA 401. For example, when the 3D segmentation 703 is sliced into 2D slices 901, each of the 2D slices 901 will include at least some portion of the LAA segmentation 703. By slicing the 3D data into 2D slices 901, the data dimension is reduced to 2D which is amenable to parallel processing. Since the slices 901 can be processed by parallel computation and the time required for contour tracking in the 2D image is less, the efficiency of the system is improved. By slicing and then processing the data, the present system and method can provide real-time tracking and analysis of the LAA 401 in a 4D ultrasound sequence. Depending on the processing capabilities of the system, any number of slices 901 can be used. As shown in FIG. 9, the 3D segmentation 703 is sliced into a plurality of 2D slices 901 including the LAA contour 903.

[0044] In process A160, the contour 903 and landmark 803 are tracked in a 4D ultrasound sequence. From frame to frame, the contour 903 and landmark 803 of the LAA401 are identified in a plurality of 2D slices 901 or in a 3D volume generated from the 2D slices 901. Two additional neural networks can be trained to track the 2D contour 903 and the anatomical landmark 803 respectively accordingly. In one embodiment, a cycle consistency loss can be used to train a neural network that performs real-time tracking of anatomical landmarks 803 within an ultrasound image. Since the contour 903 can also be discretized as a set of points, the contour 903 can also be processed using a similar network and training mechanism. In this embodiment, the 3D medical image tracking problem is treated as a motion learning process, the purpose of which is to find the best motion action based on the temporal changes of neighboring frames. Next, the motion trajectory is tracked to identify the position of the object of interest. The keyframe features and cycle consistency over time can also be used as two teacher signals for the learning of the trajectory prediction model, requiring much less annotated data compared to a supervised model. Using cycle consistency loss is a semi-supervised learning process that reduces the amount of data annotation required for supervised learning throughout the training process. To improve the efficiency of the training process, a pre-trained deep reinforcement learning strategy can be used. In this case, the artificial agent is trained to learn the anatomical structures within a static frame. This agent can effectively encode complex spatial features. Since the tracking problem is converted into a motion learning problem, feature matching no longer exists. The landmark 803 and contour 903 can still be clearly located and tracked even with high feature ambiguity. Additionally, due to the time constraints for real-time analysis and quantification, the tracking network only needs to explore small patches in the neighboring areas compared to the conventional method that needs to learn the entire frame. In another embodiment, a transformer-based network that uses both spatial context and temporal context can be used to track the landmark 903 and the LAA contour 903.In one embodiment, the processed 2D slices are integrated to generate a live 3D view that includes contours and landmarks.

[0045] As shown in FIG. 10, the contour 903 is tracked for each frame and for each slice over time. The landmark 803 can also be tracked. As shown in FIG. 11, in process A170, the tracked 3D landmark 803 and 2D contour 903 are used to determine the measurement profile of the LAA401 in real time. In one example, by tracking the inlet and neck points in the LAA401, the method can provide the diameter size of the LAA401 over time. This information helps to determine the stretchability of the LAA401 and can ultimately help to select the material used for the device. Using the tracked contours, in each frame, 3D LAA segmentation can be reconstructed in real time as a stack of 3D contours. The 3D volume and its respective volume changes can be identified and quantified in each frame. Since there is a positive correlation between the LAA volume and A-fib, the temporal record of the LAA volume and its changes can help to more reliably assess the potential risk. Other clinically relevant metrics can also be collected using 4D segmentation. In one example, the diameter of the LAA401 can be used to determine and select the characteristics of the device in a left atrial appendage closure procedure that match the characteristics of the patient's left atrial appendage.

[0046] In one embodiment, the measurement profile and other metrics help to select a device that matches the characteristics of the patient's LAA401, resulting in a customized care solution for the patient. Further, as a temporal aspect of the LAA measurement, the system or operator can analyze the pattern of indeterminate changes and detect whether abnormal activity is present in the LAA401. This can assist in diagnosis and early disease detection.

[0047] It should be understood that the elements and features recited in the claims may be combined in other ways to form new claims that also fall within the scope of the present invention. That is, the dependent claims that follow depend only on one independent or dependent claim, but these dependent claims may alternatively depend selectively on preceding or subsequent claims that are independent or dependent, and it should be understood that such new combinations are to be understood as forming part of this specification.

[0048] The present invention has been described by reference to various embodiments, but it is of course possible to make many changes and modifications to the described embodiments. Accordingly, the above description is intended to be construed as illustrative and not restrictive, and it should be understood that all equivalent forms and / or combinations of embodiments are intended to be included within this scope. Regardless of the usage of grammatical terms, individuals of male, female, or other gender identities are included within the terms.

[0049] The following is a list of non-limiting exemplary embodiments disclosed herein.

[0050] Exemplary Embodiment 1: A method for real-time three-dimensional left atrial appendage quantification, comprising obtaining a real-time three-dimensional ultrasound sequence of a patient, selecting a seed position of the left atrial appendage in a frame of the real-time three-dimensional ultrasound sequence, cropping the frame to a region of interest based on the seed position, generating a three-dimensional left atrial appendage segmentation from the region of interest, extracting one or more landmarks from the three-dimensional left atrial appendage segmentation, slicing the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each containing a part of the contour of the left atrial appendage, tracking the contour of the left atrial appendage and one or more landmarks in the plurality of two-dimensional slices and in the real-time three-dimensional ultrasound sequence, generating a real-time three-dimensional left atrial appendage segmentation from the tracked contour, and quantifying at least one metric from the tracked contour and / or the tracked one or more landmarks.

[0051] Exemplary Embodiment 2: The method of Exemplary Embodiment 1 above, wherein selecting a seed position includes stopping the real-time three-dimensional ultrasound sequence and the user clicking on a seed position within the left atrial appendage in a frame of the stopped real-time three-dimensional ultrasound sequence, and the real-time three-dimensional ultrasound sequence is resumed after the seed position is selected.

[0052] Exemplary Embodiment 3: The method according to any one of the above exemplary embodiments, wherein cropping includes calculating an estimated shape and size of the left atrial appendage based on a dataset collecting images of the left atrial appendages of a plurality of patients, and centering the region of interest around the seed position based on the estimated shape and size.

[0053] Exemplary Embodiment 4: The method according to any one of the above exemplary embodiments, wherein the three-dimensional left atrial appendage segmentation is generated using a machine-trained network.

[0054] Exemplary Embodiment 5: A method according to any one of the above exemplary embodiments, wherein one or more landmarks include at least one inlet and neck landmark that determines the diameter of the left atrial appendage inlet and neck region.

[0055] Exemplary Embodiment 6: A method according to any one of the above exemplary embodiments, wherein the contour and one or more landmarks are each tracked in real time using a neural network trained using cycle consistency loss.

[0056] Exemplary Embodiment 7: A method according to any one of the above exemplary embodiments, wherein at least one metric provides the diameter of the left atrial appendage over time.

[0057] Exemplary Embodiment 8: The method of Exemplary Embodiment 7 above, wherein the diameter of the left atrial appendage is used to determine and select the characteristics of the device during a left atrial appendage closure procedure that match the characteristics of the patient's left atrial appendage.

[0058] Exemplary Embodiment 9: A method according to any one of the above exemplary embodiments, wherein the real-time three-dimensional ultrasound sequence is provided by an ultrasound system configured for intracardiac echocardiography.

[0059] Exemplary Embodiment 10: A system for real-time three-dimensional left atrial appendage segmentation, comprising an ultrasonic imaging system configured to acquire a 4D ultrasonic sequence of a patient, and a processor, wherein the processor selects a seed position of the left atrial appendage in a frame of the 4D ultrasonic sequence, crops the frame to a region of interest based on the seed position, generates a three-dimensional left atrial appendage segmentation from the region of interest, extracts one or more landmarks from the three-dimensional left atrial appendage segmentation, slices the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each containing a part of the contour of the left atrial appendage, tracks the contour over time in the plurality of two-dimensional slices and tracks one or more landmarks in the 4D ultrasonic sequence, and quantifies at least one metric from the tracked contour and / or the one or more tracked landmarks.

[0060] Exemplary Embodiment 11: A system according to Exemplary Embodiment 10 above, wherein the cropping includes calculating an estimated shape and size of the left atrial appendage based on a dataset collecting images of the left atrial appendages of a plurality of patients, and centering the region of interest around the seed position based on the estimated shape and size.

[0061] Exemplary Embodiment 12: A system according to any one of the above exemplary embodiments, wherein the processor is configured to generate a three-dimensional left atrial appendage segmentation using a machine-trained network.

[0062] Exemplary Embodiment 13: A system according to any one of the above exemplary embodiments, wherein the one or more landmarks include at least one inlet and neck landmark that determines the diameter of the inlet and neck region of the left atrial appendage.

[0063] Exemplary Embodiment 14: A system according to any one of the above exemplary embodiments, wherein the processor is configured to track a contour and one or more landmarks in real time using a neural network trained using cycle consistency loss for each.

[0064] Exemplary Embodiment 15: A system according to any one of the above exemplary embodiments, wherein at least one metric provides the diameter of the left atrial appendage over time.

[0065] Exemplary Embodiment 16: A system according to the above Exemplary Embodiment 15, wherein the diameter of the left atrial appendage is used to determine and select the characteristics of the device during a left atrial appendage closure procedure that match the characteristics of the patient's left atrial appendage.

[0066] Exemplary Embodiment 17: A system according to any one of the above exemplary embodiments, wherein the ultrasonic imaging system is configured for intracardiac echocardiography.

[0067] Exemplary Embodiment 18: A system including an ultrasonic imaging system, a display, an input device, and a processor, wherein the ultrasonic imaging system is configured to acquire a 4D ultrasonic sequence of a patient, the display is configured to display the 4D ultrasonic sequence, the input device is configured to identify a seed position of the left atrial appendage in a frame of the 4D ultrasonic sequence, the processor is configured to crop the frame to a region of interest based on the seed position, generate a three-dimensional left atrial appendage segmentation from the region of interest, slice the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each including a part of the contour of the left atrial appendage, track the contour in real time in the plurality of two-dimensional slices, and generate a real-time three-dimensional left atrial appendage segmentation from the tracked contour.

[0068] Exemplary Embodiment 19: A system according to any one of the above exemplary embodiments, wherein the processor is further configured to identify one or more landmarks in three-dimensional left atrial appendage segmentation and track the one or more landmarks over time in a 4D ultrasound sequence.

[0069] Exemplary Embodiment 20: A system according to any one of the above exemplary embodiments, wherein the processor is further configured to quantify at least one metric from the one or more landmarks being segmented and / or tracked in real-time three-dimensional left atrial appendage.

Claims

1. A method for real-time three-dimensional left atrial appendage quantification, comprising: acquiring a real-time three-dimensional ultrasound sequence of a patient; selecting a seed position of the left atrial appendage in a frame of the real-time three-dimensional ultrasound sequence; cropping the frame into a region of interest based on the seed position; generating a three-dimensional left atrial appendage segmentation from the region of interest; extracting one or more landmarks from the three-dimensional left atrial appendage segmentation; slicing the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices, each containing a part of the contour of the left atrial appendage; tracking the contour of the left atrial appendage in the plurality of two-dimensional slices and the one or more landmarks in the real-time three-dimensional ultrasound sequence; generating a real-time three-dimensional left atrial appendage segmentation from the tracked contour; quantifying at least one metric from the tracked contour and / or the tracked one or more landmarks.

2. The selecting of the seed position comprises: stopping the real-time three-dimensional ultrasound sequence; the user clicking on the seed position within the left atrial appendage in a frame of the stopped real-time three-dimensional ultrasound sequence, and the real-time three-dimensional ultrasound sequence being resumed after the seed position is selected, according to the method of Claim 1.

3. The cropping comprises: calculating an estimated shape and size of the left atrial appendage based on a dataset collecting images of the left atrial appendages of multiple patients; centering the region of interest around the seed position based on the estimated shape and size, according to the method of Claim 1.

4. The three-dimensional left atrial appendage segmentation is generated using a machine-trained network, according to the method of Claim 1.

5. The one or more landmarks include at least one inlet and neck landmark that determines the diameter of the inlet and neck regions of the left atrial appendage, according to the method of Claim 1.

6. The contour and the one or more landmarks are each tracked in real time using a neural network trained using cycle consistency loss, according to the method of Claim 1.

7. The method of claim 1, wherein the at least one metric provides the diameter of the left atrial appendage over time. **Claim 8** The method of claim 7, wherein the diameter of the left atrial appendage is used to determine and select the characteristics of a device during a left atrial appendage closure procedure that match the characteristics of the patient's left atrial appendage. **Claim 9** The method of claim 1, wherein the real-time three-dimensional ultrasound sequence is provided by an ultrasound system configured for intracardiac echocardiography. **Claim 10** A system for real-time three-dimensional left atrial appendage segmentation, an ultrasound imaging system configured to acquire a 4D ultrasound sequence of a patient, and a processor, wherein the processor selects a seed position of the left atrial appendage in a frame of the 4D ultrasound sequence, crops the frame to a region of interest based on the seed position, generates a three-dimensional left atrial appendage segmentation from the region of interest, extracts one or more landmarks from the three-dimensional left atrial appendage segmentation, slices the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices, each including a portion of the contour of the left atrial appendage, tracks the contour over time in the plurality of two-dimensional slices and tracks the one or more landmarks in the 4D ultrasound sequence, and is configured to quantify at least one metric from the tracked contour and / or the tracked one or more landmarks. **Claim 11** The cropping includes calculating an estimated shape and size of the left atrial appendage based on a dataset that has collected images of the left atrial appendages of a plurality of patients, and centering the region of interest around the seed position based on the estimated shape and size. The system of claim 10. **Claim 12** The system of claim 10, wherein the processor is configured to generate the three-dimensional left atrial appendage segmentation using a machine-trained network. **Claim 13** The system of claim 10, wherein the one or more landmarks include at least one inlet and neck landmark that determines the diameter of the inlet and neck regions of the left atrial appendage. **Claim 14** The system according to claim 10, wherein the processor is configured to track the contour and the one or more landmarks in real time using a neural network trained using cycle consistency loss for each.

15. The system according to claim 10, wherein the at least one metric provides the diameter of the left atrial appendage over time.

16. The system according to claim 15, wherein the diameter of the left atrial appendage is used to determine and select the characteristics of the device during a left atrial appendage closure procedure that match the characteristics of the patient's left atrial appendage.

17. The system according to claim 10, wherein the ultrasonic imaging system is configured for intracardiac echocardiography.

18. A system comprising an ultrasonic imaging system, a display, an input device, and a processor, wherein the ultrasonic imaging system is configured to acquire a 4D ultrasonic sequence of a patient, the display is configured to display the 4D ultrasonic sequence, the input device is configured to identify a seed position of the left atrial appendage in a frame of the 4D ultrasonic sequence, the processor crops the frame to a region of interest based on the seed position, generates a three-dimensional left atrial appendage segmentation from the region of interest, slices the three-dimensional left atrial appendage segmentation into a plurality of two-dimensional slices each containing a part of the contour of the left atrial appendage, tracks the contour in real time in the plurality of two-dimensional slices, and is configured to generate a real-time three-dimensional left atrial appendage segmentation from the tracked contour.

19. The processor is further configured to identify one or more landmarks in the three-dimensional left atrial appendage segmentation and track the one or more landmarks over time in the 4D ultrasonic sequence. The system according to claim 18.

20. The system according to claim 19, wherein the processor is further configured to quantify at least one metric from the real-time three-dimensional left atrial appendage segmentation and / or the one or more landmarks being tracked.

Images