Right atrial pressure echocardiogram estimation using lightweight and open world artificial intelligence system

Abstract

An artificial intelligence (AI) system for estimating inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) in real time from echocardiography studies is provided. The system includes an image retrieval network configured to receive an image from an echocardiography study and perform a quality assessment of the image to generate a selected image. The system also includes a region segmentation network for localizing and segmenting the selected image to obtain an IVC region in the selected image. The system also includes an IVC quantization and RAP estimation network for performing distance calculations to find diameters in IVC regions at different spatial and time points, thereby allowing for a more reliable estimation of RAP values; the system further comprises an open world active learning system, and the open world active learning system comprises a classification engine and a clustering engine.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 458,054 filed on April 7, 2023 and U.S. Provisional Patent Application No. 63 / 350,720 filed on June 9, 2022, which are incorporated herein by reference.

[0003] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0004] This application was made with government support under Grant Nos. LM010018, Z99CL9999, and HL006199 awarded by the National Institutes of Health. The government has certain rights in this application. Background Art

[0005] Echocardiography ("echo") is performed by taking ultrasound images of the heart and its associated anatomical structures using either a dedicated bedside imaging system or a portable imaging system. Using ultrasound to capture cardiac structure and function has several advantages over other imaging modalities, including high temporal resolution, non-invasiveness, low cost, and portability. Images / videos captured during echocardiographic studies are typically analyzed manually by a sonographer or cardiologist who interprets the results to guide diagnosis, treatment options, and prognosis for different cardiovascular diseases. However, this manual analysis is time-consuming and requires a high level of training, which can be expensive or hampered by a lack of expertise in low-resource areas. In addition, manual interpretation can be subjective and lack consistency due to specific factors that lead to poor reproducibility that can affect patient management or treatment outcomes. Therefore, accurately and automatically analyzing echocardiograms offers the potential to mitigate these issues: 1) it can improve efficiency, thereby enabling physicians to have more analysis time; 2) it can provide consistent and reproducible results; and 3) it can enhance clinical interpretation and decision making. Summary of the invention

[0006] In a specific aspect of the present application, an artificial intelligence (AI) system for real-time estimation of inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) from an echocardiographic study is provided. The system includes: an image retrieval network configured to receive images from an echocardiographic study and perform quality assessment of the images to generate a selected image; a region segmentation network configured to locate and segment the selected image to obtain the IVC region in the selected image; an IVC quantification and RAP estimation network configured to perform distance calculations to find the diameter in the IVC region at different spatial and time points; and an open world active learning system including a classification engine and a clustering engine. By performing IVC thickness calculations and collapsibility analysis at different spatial locations and different time points, the AI ​​system can more reliably measure IVC and RAP values.

[0007] In another aspect of the present application, a method for real-time estimation of inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) from an echocardiographic study performed by an artificial intelligence (AI) system is provided. The method includes: receiving images from an echocardiographic study and performing quality assessment of the images to generate a selected image; locating and segmenting the selected image to obtain the IVC region in the selected image; performing distance calculations to find the diameter in the IVC region at different spatial and time points; classifying known views and unknown views in the selected image; and grouping unknown views into one or more clusters for updating the classification capabilities in the AI ​​system. By performing IVC thickness calculations and collapsibility analysis at different spatial locations and different time points, the method is able to more reliably measure IVC and RAP values.

[0008] In yet another aspect of the present application, a non-transitory computer-readable medium for real-time estimation of inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) from an echocardiographic study implemented in an artificial intelligence (AI) system is provided. The non-transitory computer-readable medium includes instructions that, when executed by a computer, configure the computer to perform steps including: receiving an image from an echocardiographic study and performing a quality assessment of the image to produce a selected image; locating and segmenting the selected image to obtain an IVC region in the selected image; performing distance calculations to find diameters in the IVC region at different spatial and temporal points; classifying known views and unknown views in the selected image; and grouping unknown views into one or more clusters for updating classification capabilities in the AI ​​system. By performing IVC thickness calculations and collapsibility analysis at different spatial locations and different time points, the AI ​​system is able to more reliably measure IVC and RAP values. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 shows the diameter of the IVC measured perpendicular to the long axis of the IVC according to an embodiment of the present application;

[0010] Figure 2 An end-to-end and automated AI system 200 for real-time estimation of IVC collapsibility and RAP according to a specific embodiment of the present application is shown;

[0011] Figure 3 shows an IVC segmentation according to a specific embodiment of the present application;

[0012] Figure 4 shows the IVC diameter per frame plotted as a time signal according to a specific embodiment of the present application;

[0013] Figure 5 Correlation and Bland-Altman plots compared to standard manual IVC quantification according to specific embodiments of the present application are shown;

[0014] Figure 6 shows a confusion matrix according to a specific embodiment of the present application;

[0015] Figure 7 An IVC retrieval network according to a specific embodiment of the present application is shown;

[0016] Figure 8 A trilateral attention network (TaNET) according to a specific embodiment of the present application is shown;

[0017] Fig. 9 The steps of path fusion according to a specific embodiment of the present application are shown; and

[0018] Fig.10 The open world learning process according to a specific embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] Accurate estimation of right atrial pressure (RAP) by echocardiography is critical for assessing hemodynamics and intravascular volume status and guiding clinical decision making to provide optimal patient care. Embodiments of the present application provide an echocardiography artificial intelligence (AI) system capable of estimating inferior vena cava (IVC) collapsibility and RAP in real time. The system uses a lightweight and open-world architecture to quickly generate reproducible and interpretable results. The lightweight feature facilitates integration of the system into handheld devices, which can enhance accessibility, and the open-world feature makes the system more robust in detecting new / unknown / unpredictable conditions or conditions in real-world clinical scenarios.

[0020] Embodiments of the present application provide: 1) automated echocardiographic assessment of IVC diameter and collapsibility and RAP estimation; 2) a lightweight system for real-time quantitative echocardiographic assessment; and 3) an open-world system for active echocardiographic interpretation and learning.

[0021] The inferior vena cava (IVC) is a blood vessel responsible for circulating deoxygenated blood from the lower extremities and abdomen back to the right atrium. Several studies [3, 4, 7] have reported that the IVC diameter and its changes with inspiration (also known as IVC collapsibility) can be easily captured using ultrasound imaging equipment and used to determine the patient's fluid status in various scenarios such as acute heart failure (HF) or critically ill patients. IVC collapsibility is measured based on the difference between the largest and smallest diameters. This collapsibility can be used to easily estimate right atrial pressure (RAP) as follows [7]: an IVC diameter of 2.1 cm with >50% collapse supports an estimated RA pressure of 3 mm Hg (range, 0-5 mm Hg), while an IVC diameter of 2.1 cm with <50% collapse supports an estimated RA pressure of 15 mm Hg (range, 10-20 mmHg). RAP is a key indicator of right ventricular diastolic function, volume status, and right heart compliance, and has been used as a predictor of mortality in patients with heart failure and cardiogenic shock.

[0022] Clinically, Figure 1As shown, the IVC diameter is measured perpendicular to the long axis 102 of the IVC within the atrial node of 1.0 cm to 2.0 cm. Then, the collapsibility is visually estimated based on the change in the IVC diameter accompanying inspiration. Specifically, the current practice of estimating the IVC and RAP involves several steps. First, the cardiologist needs to manually select the subcostal long axis view of the IVC and visually assess the view quality. Then, the cardiologist needs to measure the IVC diameter at the correct position and determine the collapsibility. Finally, the RAP is manually estimated by substituting the determined IVC collapsibility into the given formula. This practice: 1) is time-consuming and prone to errors; 2) is inaccurate because the practice relies on manual and visual assessments by cardiologists; and 3) there is subjectivity and inconsistency, which may lead to variable decision making. Since the IVC collapsibility has a strong potential in predicting the fluid status of patients with cardiac disease and non-cardiac disease, there is an urgent need to establish and develop standardized and fully automated methods for real-time IVC collapsibility analysis and RAP estimation.

[0023] Figure 2 An end-to-end and automated AI system 200 for real-time estimation of IVC collapsibility and RAP is shown. Figure 2 As shown, an echocardiographic study 202 is performed, wherein images are captured, and high quality images 204 from the study are input into the system 200 for analysis. Using the high quality images 204, the system 200 performs real-time subcostal IVC view selection and IVC image quality assessment 206, IVC region segmentation 208, and IVC diameter quantification, collapsibility, and RAP estimation 210. Below, these algorithms for performing IVC view retrieval and quality assessment 206, IVC region segmentation 208, and IVC quantification and collapsibility and RAP estimation 210 are described.

[0024] IVC view selection and quality assessment

[0025] The image retrieval network 206 of the system 200 retrieves a specific view with acceptable (moderate to good) quality. The retrieval is performed using a lightweight model with a shared encoder and two heads (see Figure 7 and related discussion below for more details), where a first head is used for view classification to identify IVC views, and a second head is used for quality assessment to assess the quality of the identified views.

[0026] The image retrieval network 206 includes five inverted residual bottleneck (see MobileNetV2-s residual bottleneck [8]) blocks and a final pooling layer. The two heads perform view classification and quality assessment and are configured to run with the encoder. Each head has the following layers: a global average pooling (GAP) layer, a dropout layer, a fully connected (FC) layer, and an open maximum (OpenMax) layer. Each head is tuned together with the encoder by: (1) initializing the encoder with ultrasound-specific weights; and (2) fine-tuning the encoder and its view classification layer. The view classification head detects IVC views from a given ultrasound study, while the quality assessment head labels a given IVC view as good quality or poor quality. Therefore, the model includes three task-specific layers, and the two heads share the same encoder. In addition, both heads can be configured to work simultaneously or in parallel, which further improves the network efficiency of the image retrieval component 206.

[0027] In clinical practice, echocardiographers visually identify ultrasound views and manually exclude low-quality ultrasounds because they can lead to inaccurate measurements. The image retrieval component 206 enables automated real-time IVC view classification and quality assessment in clinical practice. The performance of view classification and quality assessment using the image retrieval component 206 is provided in Tables 1 and 2, respectively. As can be seen from the tables, the image retrieval component 206 can achieve comparable performance, if not better, than the most advanced echocardiography systems, while having lower computational complexity. In addition, in a specific embodiment, the image retrieval component 206 has 55,620 parameters, which is smaller than most prior art networks in the literature; for example, VGG16 has 138MM parameters (528MB), while the residual network (ResNet) 18 has 11MM (44MB).

[0028] Model Accuracy AUC Sensitivity Accuracy F-score Suggested 0.977±0.01 0.985±0.02 0.978±0.07 0.977±0.06 0.974±0.06 VGG 0.953±0.02 0.958±0.04 0.954±0.02 0.953±0.03 0.954±0.02 Residual Network 0.936±0.11 0.951±0.03 0.938±0.10 0.935±0.12 0.936±0.11

[0029] Table 1: Performance of view classification head and state-of-the-art models (mean ± standard deviation (SD))

[0030] Model Accuracy AUC Sensitivity Accuracy F-score Suggested 0.943±0.10 0.962±0.22 0.946±0.09 0.944±0.04 0.945±0.06 VGG 0.933±0.15 0.951±0.43 0.933±0.34 0.932±0.29 0.933±0.30 Residual Network 0.913±0.02 0.933±0.05 0.914±0.03 0.912±0.02 0.912±0.05

[0031] Table 2: Performance of the quality assessment head and state-of-the-art models (mean ± SD)

[0032] These results shown in Tables 1 and 2 demonstrate the advantages of the image retrieval component 206 based on a shared encoder with two heads for view classification and quality assessment. After retrieving high-quality images using the image retrieval component 206, the IVC images with poor quality are automatically excluded from further analysis while sending the ultrasounds with good quality to the IVC region segmentation network 208.

[0033] IVC Region Segmentation

[0034] The IVC region segmentation network 208 uses a novel, lightweight, and technically efficient network called the Trilateral Attention Network (TaNet) to detect and segment the IVC region (see Figure 8 and Fig. 9 and the related discussion below for more details). TaNet of the IVC region segmentation network 208 achieves excellent performance segmentation of IVC in each frame from the image retrieval component 206. In some embodiments, the TaNet network achieves a high accuracy of 98% and an intersection-over-union ratio of 96%, and runs at a high speed of 85 frames per second (FPS). This speed is much faster than the most widely used segmentation networks (e.g., Unet and FCN). Figure 3 IVC segmentation and ground truth annotation of TaNet using the IVC region segmentation network 208 are shown.

[0035] Real-time IVC quantification

[0036] Instead of extracting cardiac biomarkers in specific frames (e.g., end-diastole and end-systole) as is done manually, the IVC quantification and RAP estimation network 210 performs temporal analysis of cardiac indices on all frames, thereby improving reproducibility in clinical cardiology practice and research. Analysis of all frames provides information about temporal changes during respiration over multiple cardiac cycles. This temporal IVC analysis allows for measurement of IVC collapsibility, an important component for estimating RAP.

[0037] Before delineating the IVC boundary and calculating the diameter, the IVC quantification and RAP estimation network 210 performs morphological cleaning to remove any isolated unwanted pixels and keep only the closed regions of interest. Next, the IVC quantification and RAP estimation network 210 uses the Moore-Neighbor tracing algorithm modified by Jacob's stopping criteria to calculate the contours of the clean regions (see the article by Pradhan, Ratika et al., "Contour line tracing algorithm for digital topographic maps," International Journal of Image Processing (IJIP) 4.2 (2010): 156-163, incorporated herein by reference for a description of the algorithm). Next, the IVC quantification and RAP estimation network 210 divides the delineated region into equal segments (or sectors). To calculate the IVC diameter (IVCD), the IVC quantification and RAP estimation network 210 finds the major axis of the subsegment located approximately 2 cm proximal to the RA opening. Then, the IVC quantization and RAP estimation network 210 calculates the Euclidean distance between the endpoints of the major axis. Finally, the IVC quantization and RAP estimation network 210 converts the calculated pixel distance into millimeters (mm).

[0038] After calculating IVCD, the IVC quantization and RAP estimation network 210 constructs an IVCD curve by plotting IVCD values ​​based on the frame. Next, the IVC quantization and RAP estimation network 210 uses a Savitzky-Golay filter to obtain a smooth IVCD curve. Then, the smoothed curve is used to calculate RAP.

[0039] RAP is calculated as follows: (1) the IVC quantization and RAP estimation network 210 calculates the IVC collapsibility based on the difference between the absolute maximum peak and the minimum valley in the IVCD curve; and (2) the IVC quantization and RAP estimation network 210 calculates the RAP value by substituting the IVC diameter and the collapsibility value into equation (1).

[0040]

[0041] Equation (1) is used to calculate RAP using the standard equation configured for the American Society of Echocardiography (ASE). Equation (1) is an exemplary equation for calculating RAP and can be adapted for various site-specific applications. For example, system 200 (see Figure 2) is adjustable so that any site-specific version of the above RAP equation can be used to determine the RAP within the context of the relevant site.

[0042] Figure 4 The calculated IVC diameter in each frame plotted as a time signal is shown. From this curve, the IVC quantification and RAP estimation network 210 can estimate the absolute maximum (highest peak) and absolute minimum (lowest valley) of the measurement. The IVC quantification and RAP estimation network 210 can also estimate the average maximum and average minimum by averaging the peaks and valleys of the curve.

[0043] To assess the agreement between the automatically derived values ​​and the expert estimated values, and thus determine the performance of the IVC quantification and RAP estimation network 210, Pearson correlation coefficient and Bland-Altman analysis were performed. Figure 5 Correlation and Bland-Altman plots of automated IVC for IVC quantification and RAP estimation network 210 compared to standard manual IVC quantification are shown. Figure 5 It can be seen that the IVC quantification extracted by the IVC quantization and RAP estimation network 210 is highly correlated with the value calculated by the human expert.

[0044] To estimate RAP, the IVC quantization and RAP estimation network 210 measures the collapsibility of the IVC. Next, the IVC quantization and RAP estimation network 210 substitutes the absolute IVC value and the percentage of collapsibility into equation (1) to estimate RAP. To assess the performance of the RAP estimation performed by the IVC quantization and RAP estimation network 210, the automatically derived RAP values ​​are compared with the RAP values ​​estimated by the expert. Figure 6 A confusion matrix showing this comparison is shown. Figure 6 From the confusion matrix in , it can be concluded that the IVC quantization and RAP estimation network 210 is able to accurately estimate the RAP value based on the segmented IVC region.

[0045] The automatically calculated IVC can then be combined with other clinical and laboratory biomarkers for diagnostic purposes. This automation can greatly enhance current practice, particularly in low- and medium-resource areas where expertise is scarce and available advanced laboratories and diagnostic resources are limited. Another advantage of this automated technology will enable standardized and personalized diagnosis and prognosis based on IVC and other laboratory and clinical biomarkers. Other benefits include speeding up and streamlining tedious tasks (e.g., visual IVC observation) in clinical practice, potentially saving doctors hours of time and allowing them to focus on innovation and discovery. In addition, the ability of artificial intelligence (AI)-based technologies to simultaneously monitor and analyze several data sources can play an important role in preventive medicine, potentially leading to better patient outcomes. More details on the components of the present application, including IVC retrieval, segmentation, diameter quantification, and RAP estimation, can be found in “Real time Echocardiography Image Analysis and Quantification of Cardiac Indices” by Ghada Zamzmi, Sivarama Krishanan Rajaraman, Li-Yueh Hsu, Vandana Sachdev, and Sameer Antani, Medical Image Analysis 80 (2022) 102438, which is incorporated herein by reference.

[0046] A lightweight system for real-time quantitative echocardiographic assessment

[0047] above Figure 2 The described system 200 is designed to be efficient in terms of space and speed. The efficiency arises due to the efficient design of the classification and segmentation algorithms. The structural components and related functions of the system 200 are described below.

[0048] Lightweight retrieval network

[0049] IVC View Classification and Quality Assessment 206 Figure 7 The IVC retrieval network 700 shown is executed. As described above, the IVC retrieval network 700 uses a single shared encoder 702 and two task-specific heads 704 and 706, each of which has three task-specific layers in the illustrated embodiment. In some embodiments, the two heads 704 and 706 can operate simultaneously to further enhance the efficiency of the IVC retrieval network 700. Table 3 below provides the size and inference time of the IVC retrieval network 700 according to a specific embodiment.

[0050] Model Size ≈Inference IVC Search <20MB ≈0.221 seconds TaNet segmentation <0MB >80 frames per second

[0051] Table 3: Comparison of computational complexity.

[0052] Lightweight Segmentation Network

[0053] For the IVC region segmentation network 208 implemented in the form of TaNet (see Figure 2 ), the experimental results discussed above show that the proposed network achieves very fast performance segmentation IVC (i.e., in some embodiments, TaNet achieves 85 frames per second (FPS) IVC segmentation rate). Therefore, the IVC region segmentation network 208 is a lightweight network.

[0054] The IVC region segmentation network 208, hereinafter referred to as TaNet 208, is used to locate the IVC region. TaNet 208 uses a localization component and three paths to learn rich texture, low-level and environmental features. TaNet 208 is end-to-end fine-tuned to learn localization and segmentation. The joint learning of localization and segmentation within TaNet 208 prevents unnecessary duplication of training individual models in isolation and allows TaNet 208 to focus on specific IVC regions. Using a single network for both regional localization and segmentation improves the efficiency of system 200. Figure 8 A diagram of the efficient TaNet 208 is shown in FIG.

[0055] TaNet for IVC region localization and segmentation

[0056] Convolutional neural networks (CNNs) often operate on the entire image and are limited by the spatial invariance of the input data. Traditional approaches to dealing with these problems involve using separate models for spatial transformation and localization. Jaderberg et al. [6] proposed a more efficient spatial transformation network called STN to apply these transformations (e.g., scaling, translation, attention) to the input image or feature map without the need for additional training supervision. STN is a plug-and-play module that can be inserted into existing CNNs. It is also differentiable, i.e., it computes the derivatives of the transformations within the module, which allows learning loss gradients with respect to the module parameters. In IVC segmentation, the target region occupies a relatively small portion of the entire image (see Figure 1 ). Therefore, considering the entire image for segmentation will increase the noise caused by irrelevant regions. Therefore, TaNet208 uses STN 802 to focus the attention of segmentation on specific regions while suppressing irrelevant regions.

[0057] IVC Segmentation

[0058] Three paths are used to perform segmentation: a spatial or detail path 804, a manual extraction path 806, and a context path 808. Each of these paths extracts a unique set of features. The spatial path 804 is shallow in some embodiments, comprising only three convolutional layers with high channel capacity, and the spatial path extracts rich low-level details at a low computational cost. Specifically, the path has three blocks, each of which contains a 3×3 convolutional layer with a stride of 2, followed by batch normalization and ReLU activation. In some embodiments, the number of filters in the first block, the second block, and the third block are 64, 64, and 128, respectively. In these embodiments, the feature map output by the path is 1 / 8 the size of the input image.

[0059] The hand-crafted path 806, which is a shallow path with only three local binary pattern (LBP) coded convolutional layers, extracts rich texture features from ultrasound images. Mathematical expressions for these LBP coded convolutional kernels are proposed [1]. Each LBP block includes a layer with fixed anchor weights (m) followed by a second layer with learnable convolutional filters of size 1×1. The anchor weights are randomly generated with different sparsity ranges. Similar to propagating gradients through layers with learnable and non-learnable parameters (e.g., ReLU, Max Pooling), the entire path can be trained by back-propagating gradients through the anchor weights as well as the learnable weights. In other words, the anchor weights remain unaffected and only the weights of the learnable filters are updated. Compared to the spatial path 804, the hand-crafted path 806 is designed to extract specific features (e.g., texture, geometry) that are different from the features extracted by the spatial path 804. For example, texture features (e.g., LBP) have a strong ability to distinguish subtle differences in texture and morphology, especially at the boundaries between complex regions that are difficult to separate. Therefore, the hand-crafted path 806 is used to enhance the spatial path 804 and extract rich texture features from different directions without increasing the computational burden, while the spatial path 804 is used to extract general low-level details from the image.

[0060] Finally, the environment path 808 is used for fast downsampling of feature maps to obtain sufficient receptive fields. For the environment path, a lightweight model (e.g., SqueezeNet) is used to quickly downsample the feature map of the input image to obtain sufficient receptive fields and encode high-level environmental information. Subsequently, global average pooling is attached at the end of the lightweight model to provide global environmental information to the maximum receptive field. In segmentation, the network analyzes the feature maps of the input image at different receptive fields. In a convolutional neural network, the receptive field provides an indication of the extent of the range of input data that a neuron or unit within a layer can access, and is defined by the filter size of the layer. The environment path allows images to be analyzed with receptive fields of different sizes, and it starts with the image, followed by downsampling. Specifically, since downsampling can reduce the feature representation and expand the receptive field, a lightweight model is used to downsample the feature representation of the image. Global average pooling provides the maximum receptive field with global environmental information. Finally, the output of the global average pooling is upsampled and combined with the outputs of the other paths, and after extracting features from the three paths, they are combined.

[0061] The above process can be performed by a simple summation of feature representations; however, embedding may degrade performance and complicate network optimization via summation methods because they are different in nature and length. Therefore, in other embodiments, in order to efficiently combine these paths, path fusion 810 is performed. Fig. 9 The steps performed by path fusion 810 are shown. Path fusion 810 fuses features from three paths by first concatenating the outputs of the paths at step 902, and then using batch normalization to balance the different scales of the features. The concatenated features are then combined into a single feature vector at step 904. The feature vector is sent to global pooling at step 906, and then a convolutional layer (1×1) is applied at step 908, a rectified linear unit (ReLU) activation is performed at step 910, a convolutional layer (1×1) is applied at step 912, and finally a sigmoid function is used at step 914 to generate a weight vector. The weight vector is used to reweight the concatenated feature vector.

[0062] Training TaNet for joint IVC localization and segmentation

[0063] TaNet 208 (see Figure 8) is trained in two stages: pre-training and fine-tuning. In the pre-training stage, there are two steps: 1) pre-training the coarse segmentation model (CSM), and 2) pre-training the localization network (L). The coarse segmentation model (CSM) is trained to obtain a rough prediction of the IVC region of interest (ROI). Then, the localization network (L) is trained to estimate the approximate location of the IVC ROI. In a specific embodiment, in order to generate a coarse ROI, the coarse segmentation model (CSM) is trained with a batch size of 32. This can be performed using an Adam optimizer to minimize the loss between the GT mask and the predicted coarse segmentation mask. The output of the coarse segmentation (z) is then used as the input of the localization network (L). In a specific embodiment, the localization network (L) is trained with a batch size of 32 to optimize the smooth L1 loss. The smooth L1 loss commonly used for bounding box regression is less sensitive to outliers. The purpose of the localization network is to minimize the smooth L1 loss between the predicted value and the true value. In the end-to-end fine-tuning stage, we use the Adam optimizer to end-to-end fine-tune the entire network with the pre-trained parameters (stage 1). The optimization goal is to minimize the loss between the IVC prediction and the IVC ground-truth labeling.

[0064] Table 3 above provides the size and inference time of the IVC segmentation network. The small size and high inference speed are attributed to the three shallow paths 804, 806, and 808, each of which has only three layers. It is also attributed to the fact that these paths 804, 806, and 808 extract features simultaneously, which further improves the IVC region segmentation network 200 (see Figure 2 This efficient design of the IVC region segmentation network 208 enables it to be used in resource-constrained and easy-to-operate devices to achieve IVC segmentation and other ultrasound applications.

[0065] IVC Quantification

[0066] The IVC quantization and RAP estimation network 210 performs distance calculations to find the diameter in each frame, which adds a small computational complexity to the system 200. As shown in Table 3, the system 200 has shallow and small classification, segmentation, and quantization components and is lightweight. The lightweight enables the system to perform real-time IVC diameter and RAP estimation in each frame, as well as real-time quantization in other ultrasound applications.

[0067] An open-world system for active class-based echocardiographic interpretation and learning

[0068] Because traditional machine learning models require that samples from a specific set of classes be available during training, these models will fail or perform poorly when faced with new data that was not considered during training. To overcome this challenge, system 200 is designed to operate efficiently in an open-world clinical scenario. Existing systems for automated echocardiography analysis are designed based on the assumption that samples in the testing or deployment phase must belong to the same limited number of classes that appeared in the training phase. This assumption, known as a closed-world scenario, may be too strict for real-world environments that are open and often have unseen examples, which can greatly weaken the robustness of machine learning classifiers. Here, system 200 integrates an open-world active learning approach into our IVC echocardiography system by providing a feedback network. Although the open-world approach is described below for IVC, this approach can be integrated or used with other echocardiography applications.

[0069] The open-world active learning method disclosed herein has two main engines: a classification engine and a clustering engine. The classification engines in our system are a view classification model, a quality assessment model, and a disease diagnosis model. Each of these classification engines has a clustering engine that clusters new or unknown cases based on their similarity or other clustering metrics, and then sends these clusters to human experts for feedback. For example, the view classification engine includes a classifier that is trained to recognize different echocardiographic views including IVC views, but is also trained using OpenMax (described further below) to recognize unknown views. In each run, the view classifier will detect a specific view, or mark it as unknown. The clustering engine then groups the unknown views into clusters (based on their similarity) that will be marked by human experts before passing the newly marked clusters / classes to the classification engine for model updates. In the case of quality assessment, if the classification engine cannot provide a quality mark for a particular image / view with certainty, it marks the image as unknown, groups similar unknown images together, and then sends these images to human experts for feedback. In the final stage of the system, an open-world active learning approach is integrated into a disease classification model that uses automated IVC diameter and collapsibility. This disease classification model classifies known heart diseases into their respective classes and identifies new (unseen) diseases as unknown. Cases with unknown labels are then grouped into different clusters, and each cluster is sent back to the human expert for feedback. The disease classification model will then be updated based on the feedback provided.

[0070] The importance of open-world learning can be demonstrated in several IVC applications. One possible application is to group rare cases of IVC morphology that may not be present in the training data. Examples of these rare cases include: a very large dilated IVC with poor collapsibility due to heart failure; or a very small IVC with complete collapsibility due to dehydration. Another "unknown" cluster could be IVC images that appear collapsed but actually exhibit artifacts where the image appears to move out of the ultrasound beam plane with breathing. Grouping a group of patients with similar IVC collapsibility patterns is a useful application of open-world learning.

[0071] Classification engine, open world active learning classifier

[0072] In a specific embodiment, the method for identifying unknown classes is to set a threshold on the Softmax output, that is, if none of the classes reaches a predetermined threshold, a given input image is marked as unknown. The performance of this method is sensitive to the threshold used. OpenMax uses the concept of failure probability and meta-recognition to predict unknown classes. In order to estimate whether the input is an unknown class or "very different" from a known class, the scores from the penultimate layer (i.e., the fully connected layer) of the CNN are used. Then, inputs that are very different from known classes (in terms of distribution) are rejected. We note that the OWE function can be replaced by one of the three methods described above (Softmax thresholding, a pair of multiple layers, and OpenMax), or can be replaced by any new function that can distinguish new classes with different distributions.

[0073] Clustering engine, clustering-based active learning

[0074] In certain embodiments, a clustering algorithm is used to group similar images of an unknown class into clusters that will be labeled by a human expert. There are several methods that can be used to cluster images of unknown classes. Examples of these methods include K-Medoids, K-Means, and K-Centers. The selected clustering algorithm is used to group unknown images into cluster representatives. Specifically, embedded features of the unknown image are used to generate k clusters of the unknown class. The optimal number of clusters K can be determined empirically (e.g., elbow method) or specified by a human expert. After the unknown images are grouped into different clusters, the cardiologist labels each cluster grouping of the unknown images, rather than labeling all unknown images, which results in a significant reduction in the time and number of manual interactions required. Finally, the newly labeled images (previously unknown) are sent back to the classification stage and used to enhance training and update the open world classifier.

[0075] Fig.10An iterative process 1000 for training and updating an open-world classifier is shown. At step 1002, process 1000 trains an initial open-world classifier to classify known classes while identifying new (unseen) classes as unknown. At step 1004, process 1000 uses the feature embedding of the unknown image (obtained by encoder 702 (see Figure 7 ) extraction) and clustering algorithms to group similar unknown samples into K clusters. At step 1006, process 1000 provides the K clusters of unknown samples to a human expert for labeling. At step 1008, process 1000 adds the newly labeled samples to the initially labeled data, and at step 1010, process 1000 retrains the open-world classifier using the newly labeled data (returning to step 1002). Whenever a new cluster of unknown samples is created, process 1000 is repeated for labeling the clusters and using them to update the open-world classifier.

[0076] To evaluate the performance of the open-world approach, the open-world active learning method was integrated with the view classification model. In a specific embodiment, the OpenMax function was integrated with the clustering algorithm, which improved the performance (F score) of the view classification model from 0.698 to 0.858, where the F score of 0.698 was obtained using the closed-world classifier.

[0077] Using a system such as system 200 (see Figure 2 )'s lightweight, open-world AI system for automated echocardiogram IVC diameter and collapsibility assessment is an innovative and efficient approach that standardizes the estimation of RAP, improves reproducibility, and positively impacts clinical management. System 200 is lightweight and can be easily implemented in any ultrasound imaging system for real-time automated RAP assessment. Additionally, the open-world active learning capabilities of system 200 allow it to be extended to a wider range of echocardiogram interpretations and applications.

[0078] Throughout this disclosure, the following references are cited:

[0079] [1] Rao Muhammad Anwer, Fahad Shahbaz Khan, Joost Van De Weijer, Matthieu Molinier, and Jorma Laaksonen, “Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification,” ISPRS journal of photogrammetry and remote sensing, 138: 74–85, 2018.

[0080] [2] Abhijit Bendale and Terrance E Boult, “Towards open set deep networks”, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563–1572, 2016.

[0081] [3] Morgan Caplan, Arthur Durand, Perrine Bortolotti, Delphine Colling, Julien Goutay, Thibault Duburcq, Elodie Drumez, Anahita Rouze, Saad Nseir, Michael Howsam, et al., “Measurement site of inferior vena cava diameter affects the accuracy with which fluid responsiveness can be predicted in spontaneously breathing patients: a post hoc analysis of two prospective cohorts,” Annals of intensive care, 10(1): 1–10, 2020.

[0082] [4] Salvatore Di Somma, Silvia Navarin, Stefania Giordano, Francesco Spadini, Giuseppe Lippi, Gian-franco Cervellin, Bryan V Dieffenbach, and Alan S Maisel, “The emerging role of biomarkers and bio-impedance in evaluating hydration status in patients with acute heart failure,” Clinical Chemistry and Laboratory Medicine (CCLM), 50(12): 2093–2105, 2012.

[0083] [5] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen, “Recent advances in open set recognition: A survey,” IEEE transactions on pattern analysis and machine intelligence, 43(10): 3614–3631, 2020.

[0084] [6] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al., “Spatial transformer networks”, Advances in neural information processing systems, 28, 2015.

[0085] [7] Roberto M. Lang, Luigi P. Badano, Victor Mor-Avi, Jonathan Afilalo, Anderson Armstrong, Laura Ernande, Frank A. Flachskampf, Elyse Foster, Steven A. Goldstein, Tatiana Kuznetsova, et al., “Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the american society of echocardiography and the european association of cardiovascular imaging,” European Heart Journal-Cardiovascular Imaging, 16(3): 233–271, 2015.

[0086] [8] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, “Mo-BileNetV2: Inverted residuals and linear bottlenecks,” In Proceedings of the IEEE conference on computer vision and pattern recognition, 4510–4520, 2018.

[0087] [9] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang, “Bisenet: Bilateral segmentation network for real-time semantic segmentation,” In Proceedings of the European conference on computer vision (ECCV), 325–341, 2018.

[0088] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0089] In the context of describing the present application (particularly in the context of the following claims), the use of the terms "one" and "a kind of" and "the" and "at least one" and similar references should be interpreted as covering both the singular and the plural, unless otherwise specified herein or clearly contradicted by the context. The use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be interpreted as meaning one item (A or B) selected from the listed items or any combination of two or more of the listed items (A and B), unless otherwise specified herein or clearly contradicted by the context. The terms "comprising", "having", "including" and "containing" should be interpreted as open terms (i.e., meaning "including but not limited to"), unless otherwise specified. The enumeration of the range of values ​​herein is intended only to be used as a convenient method of referring to each individual value falling within the range, unless otherwise specified herein, and each individual value is incorporated into this specification as if it were individually listed herein. All methods described herein can be performed in any suitable order, unless otherwise specified herein or clearly contradicted by the context. The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the present application and does not limit the scope of the present application unless otherwise stated. No language in this specification should be construed as indicating that any non-claimed element is essential to the practice of the present application.

[0090] The preferred embodiments of the present application are described herein, including the best models known to the inventor for implementing the present application. After reading the foregoing description, the changes of those preferred embodiments are obvious to those of ordinary skill in the art. The inventor expects that those skilled in the art will appropriately adopt such changes, and the inventor intends to practice the present application in a manner different from that specifically described herein. Therefore, the present application includes all modifications and equivalences of the subject matter described in the attached claims as allowed by applicable law. In addition, the present application includes any combination of the above-mentioned elements in all possible variations thereof, unless otherwise stated herein or clearly contradictory to the context.

Claims

1. An artificial intelligence (AI) system for real-time estimation of inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) from echocardiographic studies, the system comprising: an image retrieval network configured to receive images from the echocardiographic study and perform a quality assessment of the images to generate a selected image; A region segmentation network, wherein the region segmentation network is configured to locate and segment the selected image to obtain an IVC region in the selected image; an IVC quantification and RAP estimation network configured to perform distance calculations to find diameters in the IVC region at different spatial (different locations) and temporal points (over time); as well as An open-world active learning system includes a classification engine and a clustering engine.

2. The system according to claim 1, wherein: The classification engine is configured to detect known views in the selected image and unknown views in the selected image.

3. The system according to claim 2, wherein: The clustering engine is configured to group the unknown views into one or more clusters for use in updating the classification engine.

4. The system according to claim 3, wherein: The open world active learning system is configured as follows: classifying the known view into a known class, and classifying the unknown view into an unknown class; grouping the unknown views of the unknown class based on feature embeddings of the selected image determined by the image retrieval network; obtaining expert labels for the unknown views of the unknown class; adding the expert markings to the unknown view of the unknown class to determine an updated class; as well as The classification engine of the open-world active learning system is retrained to learn the updated classes.

5. The system according to claim 1, wherein: The region segmentation network includes a spatial transformer network to focus on specific regions of the selected image.

6. The system according to claim 5, wherein: The region segmentation network also includes: a spatial path for extracting low-level details of the specific region of the selected image; A manual extraction path for extracting rich texture features of the specific area of ​​the selected image; and The environment path is used to downsample the feature map to obtain the receptive field.

7. The system according to claim 6, wherein: Using path fusion, the outputs of the spatial path, the manually extracted path, and the environmental path are combined to obtain a weighted feature vector.

8. The system according to claim 7, wherein: The path fusion includes: cascading the outputs of the spatial path, the manually extracted path, and the environmental path to obtain a cascade feature; Combining the concatenated features into a feature vector; Sending the feature vector to global pooling; Performing a 1×1 convolution on the feature vector after global pooling; Performing a rectified linear unit ReLU activation on the feature vector after the 1×1 convolution; Perform a second 1×1 convolution on the feature vector after ReLU activation; Applying the Sigmoid function to generate a weight vector; and Based on the weighting vector, the feature vector is reweighted to obtain the weighted feature vector.

9. The system according to claim 1, wherein: The image retrieval network comprises: Shared encoder; a first task-specific header; and Second task specific header, The first task-specific header and the second task-specific header each have at least three task-specific layers.

10. The system according to claim 1, wherein: The region segmentation network is a lightweight region segmentation network configured to achieve an IVC segmentation rate of 85 frames per second (FPS).

11. A method performed by an artificial intelligence (AI) system for real-time estimation of inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) from echocardiographic studies, the method comprising: receiving images from the echocardiographic study and performing a quality assessment of the images to generate a selected image; Positioning and segmenting the selected image to obtain an IVC region in the selected image; performing distance calculations to find diameters in the IVC region at different points in space and time; classifying known views and unknown views in the selected image; as well as The unknown views are grouped into one or more clusters for updating classification capabilities in the AI ​​system.

12. The method according to claim 11, further comprising: classifying the known view into a known class, and classifying the unknown view into an unknown class; grouping the unknown views of the unknown class based on feature embeddings of the selected image; obtaining expert labels for the unknown views of the unknown class; adding the expert markings to the unknown view of the unknown class to determine an updated class; as well as The AI ​​system is retrained to learn the updated classes.

13. The method according to claim 11, wherein: Positioning and segmenting the selected image to obtain an IVC region in the selected image includes: extracting low-level details of a specific region of the selected image; extracting rich texture features of the specific area of ​​the selected image; and The feature map is downsampled to obtain the receptive field.

14. The method according to claim 13, further comprising: The low-level details of the specific region of the selected image, the rich texture features of the specific region of the selected image, and the receptive field are combined using path fusion to obtain a weighted feature vector.

15. The method according to claim 14, wherein: Combining the low-level details of the specific area of ​​the selected image, the rich texture features of the specific area of ​​the selected image, and the receptive field using the path fusion includes: cascading the low-level details of the specific area of ​​the selected image, the rich texture features of the specific area of ​​the selected image, and the receptive field to obtain a cascade feature; Combining the concatenated features into a feature vector; Sending the feature vector to global pooling; Performing a 1×1 convolution on the feature vector after global pooling; Performing a rectified linear unit ReLU activation on the feature vector after the 1×1 convolution; Perform a second 1×1 convolution on the feature vector after ReLU activation; Applying the Sigmoid function to generate a weight vector; and Based on the weighting vector, the feature vector is reweighted to obtain the weighted feature vector.

16. A non-transitory computer readable medium implemented in an artificial intelligence (AI) system for real-time estimation of inferior vena cava (IVC) collapsibility and right atrial pressure (RAP) from an echocardiographic study, the non-transitory computer readable medium comprising instructions that, when executed by a computer, configure the computer to perform steps comprising: receiving images from said echocardiographic study, and performing a quality assessment of said images to generate a selected image; Positioning and segmenting the selected image to obtain an IVC region in the selected image; performing distance calculations to find diameters in the IVC region at different points in space and time; classifying known views and unknown views in the selected image; as well as The unknown views are grouped into one or more clusters for updating classification capabilities in the AI ​​system.

17. The non-transitory computer readable medium of claim 16, further comprising instructions that, when executed, configure the computer to perform further steps comprising: classifying the known view into a known class, and classifying the unknown view into an unknown class; grouping the unknown views of the unknown class based on feature embeddings of the selected image; obtaining expert labels for the unknown views of the unknown class; adding the expert markings to the unknown view of the unknown class to determine an updated class; as well as The AI ​​system is retrained to learn the updated classes.

18. The method according to claim 16, wherein: Positioning and segmenting the selected image to obtain an IVC region in the selected image includes: extracting low-level details of a specific region of the selected image; extracting rich texture features of the specific area of ​​the selected image; and The feature map is downsampled to obtain the receptive field.

19. The method according to claim 18, wherein: Also included are instructions that, when executed, configure the computer to perform further steps, the steps comprising: combining the low-level details of the specific area of ​​the selected image, the rich texture features of the specific area of ​​the selected image, and the receptive field using path fusion to obtain a weighted feature vector.

20. The method according to claim 19, wherein: Combining the low-level details of the specific area of ​​the selected image, the rich texture features of the specific area of ​​the selected image, and the receptive field using the path fusion includes: cascading the low-level details of the specific region of the selected image, the rich texture features of the specific region of the selected image, and the receptive field to obtain a cascade feature; Combining the concatenated features into a feature vector; Sending the feature vector to global pooling; Performing a 1×1 convolution on the feature vector after global pooling; Performing a rectified linear unit ReLU activation on the feature vector after the 1×1 convolution; Perform a second 1×1 convolution on the feature vector after ReLU activation; Applying the Sigmoid function to generate a weight vector; and The feature vector is reweighted based on the weighting vector to obtain the weighted feature vector.

Images