Video-based image segmentation method and its applications

Abstract

This provides a video-based image segmentation method that overcomes the limitations of single-frame approaches. [Solution] The method includes the steps of acquiring an input video containing multiple image frames, and generating predicted contour points for one or more target contours in one or more of the multiple image frames using a trained machine learning model. The model is trained on a training dataset prepared from multiple training videos containing sequential frames having annotated contour points for one or more target contours. The architecture of this model may include a feature extractor, a feature fusion network, and a contour regressor, the feature extractor of which may be built on a video-swing transformer backbone. The video-based image segmentation method may be used to generate contours for echocardiographic images and may be used to compute clinical indices to assess the structure and function of the heart.

Description

[Technical Field]

[0001] [Related applications] This application claims the benefit of U.S. Provisional Application No. 63 / 744,827, filed on 13 January 2025, entitled “A VIDEO-BASED MULTI-CONTOUR LEARNING METHOD FOR CARDIAC IMAGE SEGMENTATION AND ANALYSIS,” which is incorporated in its entirety by reference.

[0002] The present invention relates to a computer implementation method for video-based image segmentation, particularly for segmentation of echocardiogram videos. [Background technology]

[0003] Image segmentation is a computer vision process that divides a digital image into meaningful parts or segments to simplify analysis, identify objects, and determine boundaries. Traditional image segmentation techniques, such as convolutional neural network (CNN)-based segmentation, work well for tasks like single-image segmentation. Some traditional image segmentation techniques can also be applied to image frames within video. However, these models are typically inherently constrained by limited receptive fields and a lack of temporal consistency due to their model architecture.

[0004] For example, cardiac ultrasound (echocardiography) is widely used in clinical practice to evaluate the structure and function of the heart. However, because the shape of the heart changes continuously throughout the cardiac cycle, relying solely on single-frame segmentation methods often fails to capture temporal consistency and morphological stability. Existing techniques such as single-frame mask segmentation or single-frame contour prediction typically require additional post-processing steps to correct for discontinuities and distortions in the segmentation results.

[0005] When analyzing echocardiographic videos using segmentation techniques, the predicted contour and its changes during the cardiac cycle across different video frames of the echocardiogram can be further used to calculate clinical indicators such as ejection fraction (EF) and global longitudinal strain (GLS) to assess cardiac structure and function. In echocardiography, tracking changes in ventricular volume requires identifying end-systolic (ES) and end-diastolic (ED) phases in each cardiac cycle. However, due to the irregularity of temporal volume changes and signal noise, conventional methods such as peak-finding and maximum / minimum-finding often identify local maximums or minimums rather than true extremes within a single cardiac cycle, or require many predetermined parameters, making it difficult to reliably estimate cardiac cycles and ES / ED frames with these methods. [Overview of the Initiative] [Means for solving the problem]

[0006] This invention aims to overcome the limitations of single-frame approaches by introducing a video-based deep learning model that processes the entire video rather than individual frames. This model can learn and predict contour structures by combining a contour annotation strategy with direct contour point regression, thereby ensuring excellent shape continuity and segmentation accuracy without the need for post-processing. Furthermore, by applying a multi-contour annotation strategy, the model can learn and predict multiple contour structures simultaneously (e.g., left ventricular endocardium, left ventricular epicardium, and left atrial endocardium in an echocardiogram), further improving the model's segmentation accuracy.

[0007] As a video-based deep learning architecture, the present invention treats the entire video as a continuous input rather than individual frames. By utilizing local window self-attention (video transformer), the invention extracts both spatial and temporal information from all frames and captures dynamic changes over time (e.g., throughout the entire heart cycle). This video-based approach significantly improves tolerance to frame-to-frame variations and noise.

[0008] The present invention also provides a method for identifying the extreme positions of peaks and troughs in a waveform signal with fewer required parameters compared to the prior art. By combining this method with predictive contours from echocardiography, it is possible to accurately identify end-systolic (ES) and end-diastolic (ED) frames in each cardiac cycle, thereby easily tracking changes in ventricular volume and strain.

[0009] One aspect of the present disclosure provides a computer implementation method for video-based image segmentation. The method includes (1) acquiring an input video comprising a plurality of image frames, and (2) generating predicted contour points for one or more target contours in one or more of the plurality of image frames by a trained machine learning model. The trained machine learning model is trained on a training dataset prepared from a plurality of training videos, each of which comprises a plurality of consecutive frames having the one or more target contours annotated contour points.

[0010] In one embodiment, the machine learning model includes a feature extractor that applies a local window self-attention mechanism to both the spatial and temporal dimensions to improve the stability of segmentation across the image frames. In a preferred embodiment, the machine learning model may further include a multiscale feature fusion mechanism for generating a fused feature map from the outputs of different stages of the feature extractor, and the machine learning model may further include a contour regressor for generating the predicted contour points of the one or more target contours from the fused feature map. In a preferred embodiment, the feature extractor is built on a video swin transformer (VSwinT) backbone.

[0011] In one embodiment of this method, the annotated contour points of one or more target contours are used as training ground truth during training. In some embodiments, the machine learning model does not require mask-based segmentation or post-processing to generate the predicted contour points.

[0012] One application example of the method shown in the preferred embodiments is video-based image segmentation for echocardiogram (cardiac ultrasound) videos. In these embodiments, the input video is an echocardiogram video.

[0013] If the input video is an echocardiogram video, the one or more target contours generated may include the left ventricular (LV) endocardial line. The one or more target contours may further include the left ventricular (LV) epicardial line and the left atrium (LA) endocardial line.

[0014] Video-based image segmentation methods can be applied to echocardiogram videos with different images. For example, based on the setup and training of the machine learning model, the input video may be an echocardiogram video with an apical two-chamber image (A2C), an echocardiogram video with an apical three-chamber image (A3C), or an echocardiogram video with an apical four-chamber image (A4C).

[0015] The echocardiographic image frame segmentation method may further include phase detection and / or strain calculation. The phase detection step may include (1) calculating the left ventricular endocardial length for each of the plurality of the multiple image frames based on the predicted contour points of the left ventricular (LV) endocardial line, and (2) identifying end-diastolic (ED) frames and end-systolic (ES) frames from the plurality of image frames based on the left ventricular endocardial length. After phase detection, a strain calculation step may also be performed. The strain calculation step may include (1) obtaining one or more ED-ES pairs by pairing each of the identified ED frames with a subsequent ES frame selected from the identified ES frames, (2) calculating a longitudinal strain value for each of the one or more ED-ES pairs, and (3) obtaining a final left ventricular longitudinal strain (LVLS) by averaging the longitudinal strain values ​​of the one or more ED-ES pairs.

[0016] In one embodiment of phase detection, the end-systolic (ED) frame and the end-diastolic (ES) frame are identified by (1) acquiring an endocardial length sequence including the left ventricular endocardial length of the plurality of image frames; (2) acquiring an endocardial change sequence for each of the plurality of image frames including the difference in left ventricular endocardial length between two consecutive image frames; (3) determining an optimized state sequence that maximizes the penaltyed entire path based on the endocardial change sequence; and (4) determining the positions of the end-diastolic (ED) frame and the end-systolic (ES) frame based on the optimized state sequence. In this embodiment, the total state sequence includes state values ​​representing the systolic or diastolic state in each time frame, and each time the state values ​​in the total state sequence change, a penalty value is introduced into the penaltyed entire path. The optimized state sequence may be determined by a dynamic programming algorithm.

[0017] In a preferred embodiment for identifying the ED frame and ES frame, the extended end (ED) frame is the image frame in which the state changes from an extended state to a contracted state, and the contracted end (ES) frame is the image frame in which the state changes from a contracted state to an extended state in the optimized state sequence.

[0018] In a preferred embodiment for identifying ED frames and ES frames, the step of determining the optimized state sequence includes iteratively calculating one or more penalized temporal paths in each of the image frames and identifying one or more suboptimal temporal paths from the one or more penalized temporal paths.

[0019] Another aspect of the present disclosure provides a method for training a machine learning model that connects a feature extractor, a multiscale feature fusion mechanism and a contour regressor to generate contour points of a video-based image. The method includes (1) obtaining a training dataset comprising a plurality of training data derived from a plurality of training videos, each of which comprises a plurality of training data comprising one of the plurality of training videos as a training input, and each of the plurality of training videos comprising a plurality of training image frames labeled with annotated contour points of one or more target contours as a corresponding training ground truth; and (2) feeding the training dataset into the machine learning model and performing parameter fitting to train the machine learning model so that the machine learning model can use an input video comprising a plurality of image frames to generate predicted contour points of the one or more target contours for the plurality of image frames. In this method, the feature extractor applies a local window self-attention mechanism to both the spatial and temporal dimensions of the input video; the multiscale feature fusion mechanism receives the output from different stages of the feature extractor and generates a fused feature map based on the output; and the contour regressor receives the fused feature map and generates the predicted contour points of one or more target contours for the multiple image frames based on the fused feature map. In one embodiment, a mean squared error (MSE) loss function is used to parameterize the machine learning model.

[0020] In one embodiment, the feature extractor is built on a video-swin transformer (VSwinT) backbone. In another embodiment, the feature extractor generates three temporally complete feature tensors for the multiscale feature fusion mechanism to generate the fused feature map.

[0021] One application of this method is to train a machine learning model for echocardiographic image segmentation. In this application, each of the plurality of training videos is an echocardiographic video, and the one or more target contours may include the left ventricular (LV) endocardial line. The one or more target contours may further include the left ventricular (LV) epicardial line and the left atrial (LA) endocardial line.

[0022] Other objects, advantages, and novel features of the present invention will become more apparent from the following detailed description, along with the accompanying drawings. [Brief explanation of the drawing]

[0023] [Figure 1] This document describes the architecture of a video-based image segmentation model. [Figure 2] The operation of two consecutive transformer blocks in this disclosure is shown. [Figure 3A] This shows the contour prediction results of a VSwinT-based segmentation model. The contour points were generated for an echocardiogram with an apical four-chamber view (A4C). [Figure 3B] This shows the contour prediction results of a VSwinT-based segmentation model. The contour points were generated for an echocardiogram with an apical three-chamber view (A3C). [Figure 3C] This shows the contour prediction results of a VSwinT-based segmentation model. The contour points were generated for an echocardiogram with an apical two-chamber view (A2C). [Figure 4] This is a state transition diagram showing the rising (S+) state, the falling (S-) state, and possible transitions over time frames. [Figure 5] An example of ED / ES detection using the dynamic programming method disclosed in this application is shown. [Figure 6A]Figures 6A–6I show scatter plots of LVGLS from 1050 individuals in a hybridization experiment. The x-axis represents strain (%) measured by experts, and the y-axis represents strain (%) generated by automated measurements. Mean strain values ​​are provided as mean ± SD, and Spearman's rank correlation coefficient ρ is reported to four decimal places. The numbers in parentheses indicate the number of videos in the validation dataset (from 1050) for which each hybridization combination can successfully calculate GLS. Figure 6A shows that the hybrid model used for automated measurements is EchoNet-Dynamic. [Figure 6B] Figure 6B shows that the hybrid model used for automated measurement is EchoNet+DP strain. [Figure 6C] Figure 6C shows that the hybrid model used for automated measurement is the EchoNet+Unity strain. [Figure 6D] Figure 6D shows that the hybrid model used for automated measurement is the VSwinT+EchoNet strain. [Figure 6E] Figure 6E shows that the hybrid model used for automated measurement is the VSwinT-DP strain. [Figure 6F] Figure 6F shows that the hybrid model used for automated measurement is VSwinT + Unity strain. [Figure 6G] Figure 6G shows that the hybrid model used for automated measurement is Unity + EchoNet strain. [Figure 6H] Figure 6H shows that the hybrid model used for automated measurement is Unity + DP strain. [Figure 6I] Figure 6I shows that the hybrid model used for automated measurement is Unity-GLS. [Figure 7A]Figures 7A-7I show the Bland-Altman analysis between AI strain values ​​and expert strain values. The x-axis represents the average strain (%) for automated and expert measurements, and the y-axis represents the difference (%) between the strain for automated measurements and the strain for expert measurements. LOA: Limit of agreement, n: The number in parentheses indicates the number of videos in the validation dataset (from 1050) for which each hybridization combination can successfully calculate the GLS. Figure 7A shows that the hybrid model used for automated measurements is EchoNet-Dynamic. [Figure 7B] Figure 7B shows that the hybrid model used for automated measurement is EchoNet+DP strain. [Figure 7C] Figure 7C shows that the hybrid model used for automated measurement is the EchoNet+Unity strain. [Figure 7D] Figure 7D shows that the hybrid model used for automated measurement is the VSwinT+EchoNet strain. [Figure 7E] Figure 7E shows that the hybrid model used for automated measurement is the VSwinT-DP strain. [Figure 7F] Figure 7F shows that the hybrid model used for automated measurement is VSwinT + Unity strain. [Figure 7G] Figure 7G shows that the hybrid model used for automated measurement is Unity + EchoNet strain. [Figure 7H] Figure 7H shows that the hybrid model used for automated measurement is Unity + DP strain. [Figure 7I] Figure 7I shows that the hybrid model used for automated measurement is Unity-GLS. [Figure 8A]This paper presents ROC analysis for predicting LV function using AI strains with unique combinations. ROC curves were used to evaluate the discriminative performance of three unique combinations. AUC: Area under the curve, LVGLS: Left ventricular global longitudinal strain. In 1050 validation videos, expert strain >-16% was defined as LV dysfunction. [Figure 8B] This paper presents ROC analysis for predicting LV function using AI strains with unique combinations. ROC curves were used to evaluate the discriminative performance of three unique combinations. AUC: Area under the curve, LVGLS: Left ventricular global longitudinal strain. Expert strain <-18% was defined as normal LV function. [Figure 9A] This shows the performance of three AI models under poor image quality conditions. It shows the contours generated by the AI ​​models for two selected frames of echocardiogram video. In the case of EchoNet-Dynamic, shown in the left panel of Figure 9A, the model first generates a mask (top left panel) and then converts it into contour lines (bottom left panel). In the case of Unity-GLS, shown in the right panel of Figure 9A, the model first generates an LV boundary heatmap (top right panel) and then converts it into contour lines (bottom right panel). [Figure 9B] This shows the performance of three AI models under poor image quality conditions. It shows the contours generated by the AI ​​models for two selected frames of echocardiogram video. In the case of EchoNet-Dynamic, shown in the left panel of Figure 9B, the model first generates a mask (top left panel) and then converts it into contour lines (bottom left panel). In the case of Unity-GLS, shown in the right panel of Figure 9B, the model first generates an LV boundary heatmap (top right panel) and then converts it into contour lines (bottom right panel). [Figure 9C] This demonstrates the performance of three AI models under poor image quality conditions. It shows the long-axis strain calculated based on the generated contours of each frame. [Figure 10A] This demonstrates the performance of three AI models under the influence of multiple reflection artifacts. It shows the contours generated by the AI ​​models for two selected frames of an echocardiogram video. [Figure 10B] This demonstrates the performance of three AI models under the influence of multiple reflection artifacts. It shows the contours generated by the AI ​​models for two selected frames of an echocardiogram video. [Figure 10C] This demonstrates the performance of three AI models under the influence of multiple reflection artifacts. It shows the longitudinal strain calculated based on the generated contours of each frame. [Figure 11A] This demonstrates the performance of three AI models under conditions where visualization of the LV apex is poor. It shows the contours generated by the AI ​​models for five selected frames of the echocardiogram video. [Figure 11B] This demonstrates the performance of three AI models under conditions where visualization of the LV apex is poor. It shows the contours generated by the AI ​​models for five selected frames of the echocardiogram video. [Figure 11C] This demonstrates the performance of three AI models under conditions where visualization of the LV apex is poor. It shows the contours generated by the AI ​​models for five selected frames of the echocardiogram video. [Figure 11D] This demonstrates the performance of three AI models under conditions where visualization of the LV apex is poor. It shows the contours generated by the AI ​​models for five selected frames of the echocardiogram video. [Figure 11E] This demonstrates the performance of three AI models under conditions where visualization of the LV apex is poor. It shows the contours generated by the AI ​​models for five selected frames of the echocardiogram video. [Figure 11F] This demonstrates the performance of three AI models under conditions where visualization of the LV apex is poor. It shows the longitudinal strain calculated based on the generated contours of each frame. [Figure 12A] Examples of irrational end-diastolic or end-systolic timings resulting from a lack of temporal consistency are shown. Contours generated by the AI ​​model for three selected frames of an echocardiogram video are also shown. [Figure 12B] Examples of irrational end-diastolic or end-systolic timings resulting from a lack of temporal consistency are shown. Contours generated by the AI ​​model for three selected frames of an echocardiogram video are also shown. [Figure 12C] Examples of irrational end-diastolic or end-systolic timings resulting from a lack of temporal consistency are shown. Contours generated by the AI ​​model for three selected frames of an echocardiogram video are also shown. [Figure 12D] Examples of irrational end-of-expansion or end-of-contraction points due to a lack of temporal consistency are shown. Longitudinal strain calculated based on the generated contours of each frame is also shown. [Modes for carrying out the invention]

[0024] Terms used in the following description are intended to be interpreted in their broadest reasonable sense, even when used in connection with a detailed description of a particular embodiment of the Art. Certain terms may be emphasized below, but any term intended to be interpreted in a restricted sense is specifically defined in this [Modes for Carrying Out the Invention] section.

[0025] The embodiments described below may be implemented by programmable circuits programmed or configured by software and / or firmware, by entirely dedicated circuits, or by a combination of these methods. Such dedicated circuits (if any) may take the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), etc.

[0026] 1. Video-based image segmentation One aspect of the present invention relates to a video-based deep learning model that processes image segmentation using information from the entire video, rather than from frames on which the segmentation task is performed.

[0027] Model Architecture Figure 1 shows the architecture of a video-based image segmentation model. The model includes a feature extractor, a feature fusion network, and a contour regressor.

[0028] First, the input video frames are fed into a feature extractor backbone, and multiscale features are extracted from the backbone's intermediate layers. Next, a feature fusion network merges the multiscale features to generate a fused feature map. Finally, the fused feature map is fed into a contour regressor to predict the contour of each frame.

[0029] In the implementation of this application, this model architecture is used to construct a video-based image segmentation model for echocardiography. This segmentation model integrates a video-swin transformer (VSwinT) backbone for spatiotemporal feature extraction and a contour regression network for accurate prediction of 2D contour points of the left ventricular (LV) endocardium from echocardiographic video. This architecture is carefully designed to efficiently process medical video data by leveraging multiscale representation and hierarchical attention mechanisms.

[0030] Feature Extractor Backbone VSwinT backbone acts as a feature extractor, processing echocardiogram video into 4D tensor X∈R T*×H*×W*×C The data is processed as follows, where T* is the number of video frames, H* and W* are the height and width of each video frame, and C is the number of channels in each frame. The VSwinT backbone includes three consecutive stages (stages 1 to 3 shown in Figure 1).

[0031] The input video (4D tensor) has size P. t ×P h ×P w It is divided into non-overlapping 3D patches, where P t =2, P h =8, P w = 8, and these are the patch sizes along the time dimension, height dimension, and width dimension, respectively.

[0032] Embed each patch linearly into a feature vector of dimension d to form a sequence of tokens. Z0 = Linear(X パッチ ) ∈ R T×H×W×d , where

Number

[0033] Vision Transformer (ViT) utilizes global self-attention (SA) which causes a quadratic increase in computational complexity and memory as the number of tokens increases. Due to this scalability issue, video tasks involving multiple frames become inefficient. Video Swin Transformer addresses this limitation by using a local window-based self-attention mechanism, thereby improving the efficiency of video-related applications.

[0034] Each video Swin Transformer block is designed to process video tokens by integrating both spatial and temporal dimensions. Layer normalization (LN) normalizes the input features before both the 3D windowed multi-head self-attention (3D W-MSA) and the feed-forward network (FFN) module with GELU non-linearity. 3D W-MSA performs self-attention within a normal 3D window, focusing on interactions between local tokens. To enable cross-window connections and expand the representation capacity, 3D shifted window MSA (3D SW-MSA) is applied by shifting the window configuration within consecutive blocks. FFN improves the model's ability to learn complex representations, while residual connections after MSA / SW-MSA and FFN facilitate gradient flow and prevent vanishing gradients. This architecture effectively balances computational efficiency and robust spatio-temporal feature modeling, making the video Swin Transformer ideal for tasks such as video classification and segmentation.

[0035] Figure 2 shows the operation of two consecutive transformer blocks. The operation of two consecutive transformer blocks can be expressed by the following formula:

number

[0036] To efficiently manage video data, the video swin transformer employs a shift window partitioning strategy, alternating between normal window configurations and shift window configurations within consecutive transformer blocks. In normal window partitioning, tokens are divided into non-overlapping 3D windows of size P×M×M.

number

number

[0037] Self-aware behavior within each window is defined as follows:

number

[0038] In each stage, adjacent patches are merged to reduce spatial resolution while maintaining temporal length. For example, adjacent 2x2 spatial patches are merged so that the number of tokens is reduced to one-quarter. A linear layer unifies the channel dimension and doubles the token feature dimension. The spatial resolution (H,W) is halved in each stage.

[0039] This hierarchical approach balances efficiency with effective spatiotemporal modeling. The output of stage l is expressed as follows:

number

[0040] At each stage, a stack of transformer blocks is applied. In the implementation example of this invention, each block first performs a 3D windowed multi-head self-attention (3D W-MSA) on a local cuboid of 8x7x7 tokens, and then performs a position-based feedforward network. The next block in a pair repeats the same operation after shifting the attention window by (4,3,3) tokens along the time, height, and width axes, respectively. This alternation between standard window and shifted window allows tokens that were previously separated by window boundaries to pay attention to each other, thereby eliminating spatiotemporal seams and linearly expanding the receptive field with the network depth.

[0041] After the final three stages, a spatial 2x2 patch-merging layer is created, halving the height and width while maintaining the full time length. Thus, the encoder is created from three temporally complete feature tensors F that will later be fused together. 1 = 32 × 28 × 28, F 2 = 32 × 14 × 14, F 3 Output =32×7×7.

[0042] Feature Fusion Network Output from the three VSwinT stages (F 1 ,F 2 ,F 3 ) is gradually fused, inspired by skip connections in a fully convolutional network (FCN) style. Each feature map is first convolved to match the channel, then upsampled and added to the high-resolution map. This is because the 3D W-MSA already introduces local time correlations, and the shift window propagates them across the window boundary. The fused network is then (F 2 ,F 3 ) to F 1 By upsampling to a certain resolution, the three tensors can be combined using a lightweight 2D convolution. The final fused feature map integrates both local details and global context.

[0043] First, the feature maps of the three stages have different resolutions. Minimum feature map F 3 The height and width dimensions are (7,7), and the intermediate feature map F 2 The dimension of is (14,14), and the maximum feature map F 1 The dimension of is (28,28). Each feature map F l This involves spatial refinement and channel unification using 2D convolution. This step ensures that all feature maps have the same channel dimension D (where D=1024 is chosen), followed by the subsequent upscaling and additive process.

number

[0044] The feature map undergoes gradual upscaling and addition. Minimum feature map F 3’ is, F 2 It is upscaled to 2x to match the resolution:

number

number

number

number

number

number

[0045] The resulting fusion feature map F fused It integrates information from all scales.

number

[0046] Contour Regressor Fusion feature map F for each frame t tThis data is input into a contour regression network that predicts the 2D coordinates of 49 contour points representing the left ventricular (LV) endocardium.

[0047] The fused feature map for each frame is a compact vector z t It is pooled globally: z t =GlobalPoolF t ∈R D .

[0048] pooled feature vector z t This is passed directly to a fully connected linear layer to predict the 2D coordinates of 49 contour points, resulting in a total of 49 coordinates (98 outputs in total):Y t =Linearz t ∈R 98 .

[0049] The final prediction forms a set of contours across all frames.

number

[0050] training goals Mean Squared Error (MSE) Loss between Predicted Coordinates and Ground Truth Coordinates

number

number

[0051] This model architecture can be used to train a machine learning model that can generate contour points for one or more structures (e.g., left ventricular endocardium, left ventricular epicardium, and / or left atrial endocardium) using echocardiogram video as input. To achieve this objective, echocardiograms with annotated contour points were used as material for model training.

[0052] Training materials and model training 2,340 anonymized echocardiograms, including 1,282 videos from Taiwanese institutions, 72 videos featuring athletes provided by the International University Sports Federation (FISU) "Check-up Your Heart" program during the 2017 Summer Universiade Taipei, and 986 videos from the publicly available CAMUS dataset, were used as source material for the training dataset. These 2,340 videos featured a diverse distribution of imaging systems, including Philips Medical Systems iE33 (38.12%), EPIQ 7C (16.41%), GE Vivid systems (2.68% total, FISU), and GE Vivid E95 machines (42.14%, CAMUS).

[0053] Similar to CAMUS annotation, a frame-by-frame multitask annotation strategy was performed on 1282 intra-articular videos and 72 FISU videos by cardiologists meeting the Level III-IE competency criteria defined by the American Society of Echocardiography. In addition to annotating the training data, these same cardiologists also manually annotated and measured strain on a validation dataset of another 1050 cases. While CAMUS annotated only systolic periods, this application performed annotation across two full cardiac cycles for each video. This included annotation of the LV endocardium line defining the medial LV boundary for assessing ventricular volume and function, annotation of the LV epicardium line depicting the lateral boundary for measuring myocardial thickness and wall motion, and annotation of the LA endocardium line marking the medial LA boundary for assessing atrial size and function.

[0054] The preprocessing pipeline converts DICOM frames to 8-bit grayscale images and normalizes pixel values ​​to the range of 0 to 255. Each frame is cropped to a square dimension based on the smaller of its width and height, resized to 224x224 pixels, and expanded into three channels before being normalized to the range of 0 to 1. To ensure temporal resolution consistency, the video is divided into 64-frame chunks with 10 overlapping frames. For videos with fewer than 64 frames, the last frame is repeated to meet the required length. The resulting frames are stacked into a tensor of shape [64,224,224,3] and transposed to [3,64,224,224] to match the transformer's input specifications. All chunks are aggregated into a tensor with the final shape [#chunk,3,64,224,224].

[0055] The model outputs 2D coordinates of 49 LV endocardial contour points for each frame. This facilitates detailed tracking of left ventricular shape and function throughout the cardiac cycle and enables accurate contour-based assessment of myocardial action.

[0056] To optimize performance, key training parameters were carefully selected. The loss function used was the mean squared error (MSE) calculated for each coordinate to minimize the regression error. The Adam optimizer was used with a learning rate of 0.0001, momentum of 0.9, and weight decay of 0.0001 to ensure stable convergence during training. Extensive data augmentation techniques were implemented to improve the model's robustness and generalizability. These techniques include brightness adjustment with a variance of 75 and contrast scaling with a coefficient of 0.3. Furthermore, geometric transformations such as shifts of up to 6.25%, scaling of up to 20%, and rotations within a range of ±40° were applied.

[0057] Characteristics of VSwinT-based segmentation models The constructed segmentation model utilizes continuous video frames to model the temporal evolution of the cardiac shape throughout the entire cardiac cycle, ensuring greater shape continuity and stability. By directly regressing contour points during contour generation, it eliminates the risk of fragmented or erroneous boundaries commonly encountered in mask-based systems. By directly generating the final contour coordinates, this method eliminates the need for the shape correction or smoothing steps typical of conventional approaches.

[0058] Furthermore, during model training, a multi-contour annotation approach was implemented, including the LV endocardial line (medial boundary of the left ventricle), LV epicardial line (lateral boundary of the left ventricle), and LA endocardial line (medial boundary of the left atrium). Thus, the model can learn to generate different contours, thereby accelerating clinical analysis by providing multiple structures in a single pass. Moreover, this method simultaneously learns and predicts the endocardial and epicardial contours of the left ventricle, as well as the endocardial contour of the left atrial, thereby improving segmentation accuracy across various cardiac phases and patient groups.

[0059] Figures 3A to 3C show the contour prediction results of a VSwinT-based segmentation model. By selecting different echocardiographic images as training inputs, the model can generate contour points for image frames with the corresponding images. Figure 3A shows the contour points generated for an echocardiogram with an apical four-chamber image (A4C), Figure 3B shows the contour points generated for an echocardiogram with an apical three-chamber image (A3C), and Figure 3C shows the contour points generated for an echocardiogram with an apical two-chamber image (A2C).

[0060] Comparison of VSwinT-based models with other segmentation models Table 1 shows the computational and resource profiles of the VSwinT model and two other frame-based segmentation models, Unity and EchoNet. The Unity model directly predicts keypoints along the LV contour using HRNet, while EchoNet directly estimates the left ventricular (LV) mask from individual frames. Among these, VSwinT excels at processing video input with remarkable efficiency and scalability. Table 1 Comparison of Unity model, EchNet model, and VWinT model [Table 1]

[0061] Table 1 shows that VSwinT processes 64 frames at 224x224 resolution at a total cost of 574.53 GFLOPs, which is equivalent to just 8.98 GFLOPs per frame. This efficiency highlights VSwinT's ability to process large amounts of temporal data without excessive per-frame overhead, making it ideal for video-based tasks. With 121.38 million parameters, VSwinT excels at feature extraction and temporal modeling, providing rich expressiveness suitable for complex video understanding tasks. Despite processing 64 frames simultaneously, VSwinT's GPU memory usage (2289.51 MB) remains proportional to its computational needs, demonstrating effective resource utilization for high-throughput video pipelines.

[0062] Unity's high computational cost (275.77 GFLOPs per frame) reflects its focus on detailed spatial processing (608x608) which limits its scalability. EchoNet is a lightweight alternative designed for resource-constrained, low-resolution tasks, using 7.83 GFLOPs per frame and 162.22 MB of memory. However, its analytical capabilities are limited to a single image frame.

[0063] In summary, VSwinT stands out as the most balanced and efficient model for video applications, combining low computational cost per frame, high feature capacity, and scalability for multi-frame processing. VSwinT offers an attractive option for video tasks requiring large-scale temporal and spatial modeling.

[0064] 2. ED / ES identification The predicted length of the LV endocardial contour in different image frames forms a length-versus-time (frame) sequence with a waveform appearance. End-diastolic (ED) and end-systolic (ES) frames can be detected using computer-implemented methods to identify the extreme positions of the waveform signal.

[0065] In this method, an objective function derived from the waveform signal is used to identify the extreme values. The objective function has a "state" parameter that relates to the different states (e.g., rising and falling) of the waveform signal in each time frame. By selecting an appropriate state parameter that maximizes the objective function and identifying the locations in the waveform signal where the state parameter changes, the extreme values ​​(i.e., turning points) and corresponding signal values ​​of the waveform signal can be identified.

[0066] Objective function The objective function aims to reliably identify state transitions while maximizing the difference in signal values ​​between a identified peak and its adjacent trough. To achieve this objective, a transition penalty is introduced into the objective function to prevent misidentification of local extrema caused by noise during the wave period as true extrema. The objective function resembles the cumulative path that a virtual point has moved during signal recording, except that a penalty value is subtracted from the cumulative path each time a state transition occurs. Because a penalty is imposed each time a transition occurs, the transition penalty prevents misidentification of noise-derived local extrema as transition points.

[0067] In a preferred embodiment, the objective function has the following form:

number

[0068] In this function, V diff (t) = V(t) - V(t-1), which is the difference in signal values ​​between consecutive frames t-1 and t, or the current displacement between time frames t-1 and t. Time frames are counted from the first frame (t=1) to the last frame (t=T). s(t) represents the state in time frame t. If the waveform signal is in an ascending state (i.e., the signal value is rising) in time frame t, s(t) = 1, and if the waveform signal is in a descending state (i.e., the signal value is falling), s(t) = -1. Z(t) is the transition penalty in time frame t. A penalty value σ is applied when switching between ascending and descending states to ensure the algorithm avoids unnecessary state transitions. That is, if s(t) = s(t-1), Z(t) = 0, and if s(t) ≠ s(t-1), Z(t) = σ. The objective function is the penalty-applied individual paths V in all time frames. diff (t)·s(t)-Z(t) can be considered as the total penaltyed paths calculated by summing them up. Also, in order to calculate the individual penaltyed paths at t=1, V(0) is defined as 0 when s(0)=1 and s(0)=-1.

[0069] Penalty value The penalty value (σ) may be any appropriate value to avoid unnecessary state transitions. In preferred embodiments, a statistical measure that measures the spread or variance of the signal values ​​is used. Examples of statistical measures that may be used as the penalty value include, but are not limited to, the interquartile range (IQR), standard deviation (SD), mean absolute deviation (AAD), or median absolute deviation (MAD) of the waveform signal. The definitions and important characteristics of these statistical measures are described below.

[0070] 1. Interquartile range (IQR) The Interpolational Quarter Value (IQR) is defined as the range of the middle 50% of the data, i.e., the difference between the third quartile (Q3) and the first quartile (Q1), and is expressed by the formula IQR = Q3 - Q1. IQR is robust to outliers and focuses on the central portion of the data. However, it ignores the edges of the data.

[0071] 2. Standard deviation (SD) SD is the square root of variance, measuring the mean deviation from the mean using the same units as the data, and is expressed by the formula SD = √Variance. It has the same units as the data and is widely used. However, it is more sensitive to outliers.

[0072] 3. Mean absolute deviation (AAD) AAD refers to the average of the absolute deviations from the mean, and the formula is

number

[0073] 4. Central absolute deviation (MAD) The median absolute deviation (MAD) is a robust measure that is the median of absolute deviations from the median of the data, and is given by the formula MAD = median(|x). i It is expressed as -median(x)|). This is very robust to outliers, but may not be sensitive enough to datasets with fine variability.

[0074] Status determination For a given signal V, the objective function

number

[0075] State series

number

[0076] Here, we define a time path with a penalty as follows:

number

[0077] The penalty-inducing time path is time frame t ’ State value s ’ Using t=1 to t=t ’This is the penalty-inclusive route calculated up to that point. ’ If is the time frame currently in focus, ’ This is the current state value. All possible TP(t ’ ,s ’ ) within the current time frame t ’ Current state value s ’ The maximum TP value with the following characteristics is called the current route, TP Max (t ’ ,s ’ ) or DP(t ’ ,s ’ ) is represented by the current state value s ’ The current path with =1 is called the first current path, and the current state value s ’ The other path with =-1 is called the second current path.

[0078] By definition, the penaltyed temporal path in the current time frame t is equal to the penaltyed temporal path in the previous time frame t-1 (called the previous temporal path) plus the penaltyed individual path in the current time frame t. That is, TP(t-1,s ’ )+V diff (t)·s(t)-Z(t)=TP(t,s). In the formula, s ’ This is called the previous state value, which is the state value at the previous time frame t-1. Previous time frame TP Max (t-1,s ’ The maximum TP value in ) is defined as the previous path, where the previous state value s ’ One path with =1 is called the first previous path, and the previous state value s ’ The other path with = -1 is called the second previous path. In other words, the first previous path is the largest of the previous temporal paths that have an ascending state in the previous time frame t-1, and the second previous path is the largest of the previous temporal paths that have a descending state in the previous time frame t-1. From the above equation, the following recurrence relation is obtained.

number

[0079] This is the TP Max (t, 1) is the first ascending path TP Max (t - 1, 1) + V diff (t) and the second ascending path TP Max (t - 1, -1) + V diff (t) - σ, meaning the larger one of them. Similarly, TP Max (t, -1) is the first descending path TP Max (t - 1, 1) - V diff (t) - σ and the second descending path TP Max (t - 1, -1) - V diff (t), which is the larger one of them. According to the recurrence formula of Equation 3, TP Max (t, 1) and TP Max (t, -1) can be further used to calculate TP Max (t + 1, 1) and TP Max (t + 1, -1).

[0080] Based on the above analysis, it is shown that TP Max (T, s) for all t including TP Max (t, s) can be calculated. Also, since the optimized final path is the maximum among all possible objective values, TP​​​​​​​​​​​​​​​​​​​​​​​​​​​After calculating (t,s), TP Max The previous state value and TP corresponding to (t,1) Max The two previous state values ​​corresponding to (t,-1) can be determined. Starting from the final optimization path and its corresponding state value s(T), all previous state values ​​can be traced back to t=1, and the resulting state sequence can be obtained.

number

[0082] The optimized state sequence may also be obtained by another approach. Based on Equation 3, each current path TP Max (t,s) is the first previous route TP Max (t-1,1) and the second previous route TP Max The previous route TP selected from (t-1,-1) Max (t-1,s ’ It can be seen that this corresponds to ). Therefore, the state sequence may be recorded and updated continuously from the first time frame to the last time frame. In this approach, the calculated state in the previous time frame is recorded as the first previous state sequence and the second previous state sequence, and the first previous state sequence and the second previous state sequence are updated with the newly calculated results in each time frame. In particular, the first previous state sequence corresponds to the first previous path TP Max State sequence corresponding to (t-1,1)

number

number

[0083] The optimized state sequence may be determined as follows: In each time frame, the first previous path TP Max(t-1,1), the first previous state sequence, the second previous path TP Max (t-1,-1), the second previous state sequence, and the current displacement V diff (t) is obtained. Then, based on the first previous path, the second previous path, and the current displacement, the first current path TP is obtained. Max (t,1) and the second current route TP Max Calculate (t,-1). As mentioned above, this is the first ascending path TP Max (t-1,1)+V diff (t) and the second ascending path TP Max (t-1,-1)+V diff The larger of (t)-σ is the first current route TP Max Select (t,1) and the first descent route TP Max (t-1,1)-V diff (t)-σ and the second descending path TP Max (t-1,-1)-V diff The larger of (t) is the second current route TP Max This may be done by selecting (t,-1). Then, the first previous path and the second previous path are replaced with the first current path and the second current path, respectively, for use in the next iteration. Finally, based on the selection to obtain the first current path and the second current path, the first current state sequence corresponding to the first current path and the second current state sequence corresponding to the second current path can be obtained. For example, if the first current path is the first ascending path, the ascending states are added to the first previous state sequence to form the first current state sequence; if the first current path is the second ascending path, the ascending states are added to the second previous state sequence to form the first current state sequence; if the second current path is the first descending path, the descending states are added to the first previous state sequence to form the second current state sequence; and if the second current path is the second descending path, the descending states are added to the second previous state sequence to form the second current state sequence. Then, for use in the next iteration, the first previous state sequence and the second previous state sequence are also replaced with the first current state sequence and the second current state sequence, respectively.

[0084] Repeat the above steps until the last time frame. Then, select the larger of the first current path and the second current path in the last time frame as the final optimized path, and select the entire state sequence corresponding to the final optimized path as the optimized state sequence.

[0085] Dynamic programming The tasks of calculating the objective value and determining the state sequence described above can be performed using dynamic programming (DP) algorithms. As mentioned above,

number

[0086] In the model of the present invention, the ED / ES detection algorithm utilizes dynamic programming to optimally detect the transition between diastolic and systolic phases. Figure 5 shows an example of ED / ES detection using the disclosed dynamic programming method. The curves show the Lagrangian strain (described below) values ​​for different frames and the ED and ES frames tracked by the DP algorithm. The curves are strain signals, and the tracked ED and ES frames are indicated by the symbols ▼ and ▲. This example demonstrates that the model of the present invention can ensure accurate phase detection by maximizing the overall objective function.

[0087] 3. Strain calculation After extracting the left ventricular contour from the model, the left ventricular longitudinal strain (LVLS) can be calculated based on the length of the left ventricular contour in all video frames. This is done using Lagrangian strain equations.

number

[0088] LV segmentation models are typically paired with proprietary strain calculation algorithms for AI-based LVGLS computations. To investigate the interaction between LV contour prediction and strain calculation algorithms, a hybrid experimental setup was designed by combining three segmentation models with three strain calculation algorithms, resulting in nine different configurations. The three strain estimation methods are (1) DP strain (the model of the present invention), (2) EchoNet strain calculation, and (3) Unity strain calculation.

[0089] The DP strain model uses dynamic programming to identify globally optimal ED and ES phases without requiring parameter adjustments, thereby improving robustness against noise.

[0090] EchoNet strain calculation involves estimating the LV endocardial mask using EchoNet, extending it, extracting the LV endocardial contour, obtaining a time series of lengths, identifying local maximum and minimum values ​​using SciPy's find_peaks function (which requires parameter tuning to optimize performance), pairing ED and ES points, and averaging the strain across these pairs.

[0091] For Unity strain calculations, the LV endocardial heatmap is obtained from the Unity model, converted to an LV endocardial contour, a time series of lengths is obtained, local maximum and minimum values ​​are identified using SciPy's find_peaks (which requires parameter tuning), the ED lengths are averaged, the ES lengths are averaged, and these averages are used to estimate the strain.

[0092] 4. Performance evaluation In the validation phase, a comprehensive set of echocardiographic videos is used to evaluate the performance and reliability of the segmentation results and strain estimation of the model of the present invention. Three contour prediction methods and three strain measurement methods are included in the evaluation.

[0093] The three segmentation models for comparison are the VSwinT-based model, the EchNet model, and the Unity model of the present invention, as described above in Section 1. The three strain estimation methods used are the DP-strain model, the EchoNet strain calculation model, and the Unity strain calculation model of the present invention, as described above in Section 3.

[0094] Mixture analysis evaluates the interaction between contour prediction and strain measurement by comparing performance across nine combinations. This approach assesses how segmentation accuracy affects the reliability of strain estimation, ensuring that segmentation models not only function well independently but also contribute effectively to accurate downstream clinical evaluations.

[0095] Evaluation dataset Here, two different test sets are used: (1) the primary evaluation set and (2) the HMC-QU evaluation set.

[0096] The primary evaluation set is a dataset of 1050 apical four-chamber (A4C) echocardiograms with manually measured longitudinal strain measurements and LV endocardial contour labels at end-diastolic and end-systolic as contour and strain annotations. This evaluation set is used to assess segmentation performance and strain estimation accuracy.

[0097] The HMC-QU evaluation set is a dataset of videos containing 162 apical four-chamber (A4C) images from the HMC-QU open dataset (publicly available). This dataset includes multiple pathological conditions and imaging scenarios, enabling evaluation of the clinical applicability of the models. Ground truth strain values ​​are not provided in this evaluation set; therefore, it is used for pairwise comparisons of strain performance between different models.

[0098] Models for performance comparison The VSwinT-based model and DP-strain model of the present invention can be integrated to perform strain calculations directly based on input echocardiograms (the integrated model is abbreviated as VSwinT-DP-strain). EchoNet-Dynamic and Unity-GLS are two other integrated models used for performance comparisons for both contour prediction and strain measurement. These two models are frame-based segmentation models commonly used for baseline comparisons in echocardiographic analysis.

[0099] EchoNet-Dynamic is a deep learning model for evaluating cardiac function in echocardiogram videos. It uses the EchNet model to directly estimate the left ventricular (LV) mask from individual frames, and then extracts the LV contour from the mask using a computer vision algorithm.

[0100] Unity-GLS utilizes HRNet to directly predict keypoints along LV contours using Unity models, providing accurate contour rendering.

[0101] The two models, operating in a single frame without incorporating temporal context, are suitable for evaluating static frames, but have limitations in capturing dynamic heart movements. They were selected to demonstrate that the video-based models of the present invention can perform better than those frame-based models.

[0102] In addition to the three proprietary models (VSwinT-DP strain, EchoNet-Dynamic, and Unity-GLS), six hybrid models combining different LV segmentation models with different strain measurement models were also tested for model performance comparison. The six hybrid models tested were VSwinT+EchoNet strain, VSwinT+Unity strain, EchoNet+DP strain, EchoNet+Unity strain, Unity+DP strain, and Unity+EchoNet strain.

[0103] Process integrity Process integrity assesses each model's ability to flawlessly execute the entire segmentation and strain analysis pipeline. This assessment includes successful contour segmentation and successfully estimated strain values. Successful contour segmentation measures the number of videos in which contour segmentation was successfully completed, while successfully estimated strain values ​​measures the number of strain measurements successfully derived from the segmentation results. This metric demonstrates the robustness of the segmentation and strain estimation processes across the dataset.

[0104] Performance metrics Two performance metrics, a segmentation performance metric and a strain estimation performance metric, are used independently to evaluate the segmentation and strain estimation performance of different models. In particular, the VSwinT-based architecture of the present invention can be trained to generate different contours (e.g., LV endocardium, LV epicardium, and LA endocardium) in echocardiograms with different images (e.g., A4C, A3C, and A2C), but this evaluation will focus primarily on LV endocardium segmentation in A4C due to limitations of other comparison methods.

[0105] Segmentation performance metrics Segmentation performance is evaluated using the following metrics across 2100 annotated frames (1050 videos from the end of systole and end of diastole): (1) Dice coefficient, (2) crossover rate (IoU), (3) Hausdorff distance, and (4) mean Hausdorff distance.

[0106] The Dice coefficient measures the overlap between the predicted contour and the ground truth contour; the intersection ratio (IoU) calculates the ratio of the intersection to the union of the predicted region and the ground truth region; the Hausdorff distance determines the maximum distance between a point on the predicted contour and a point on the ground truth contour; and the mean Hausdorff distance provides the average measurement of the distance between the predicted contour and the ground truth contour. These metrics provide a comprehensive assessment of the contour prediction accuracy of each model.

[0107] The contour segmentation performance of different models is shown in Table 2 below. Table 2 Segmentation performance of the EchoNet-Dynamic model, VSwinT-DP strain model, and Unity-GLS model [Table 2]

[0108] The table above compares the performance of three contour segmentation models—EchoNet-Dynamic, VSwinT-DP Strain, and Unity-GLS—across multiple metrics for evaluating contour accuracy and reliability. The results show that the model of the present invention significantly outperforms the Hausdorff metric while achieving high dice and IoU values ​​comparable to Unity-GLS, indicating better shape retention and outlier reduction. Furthermore, the model of the present invention achieves a perfect success rate and surpasses both EchoNet-Dynamic and Unity-GLS in reliability.

[0109] Strain estimation performance metrics After extracting the LV contour from the model, the left ventricular longitudinal strain (LVLS) can be calculated based on the length of the left ventricular contour in all video frames. (1) Mean squared error (MSE), (2) Root mean squared error (RMSE), (3) Coefficient of determination (R 2 The strain estimation performance is evaluated using four metrics: (4) the correlation coefficient (rrr), (5), and (6) the Pearson correlation coefficient (rrr).

[0110] Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) quantify the deviation between the predicted value and the ground truth strain value, and the coefficient of determination (R) is used to calculate the coefficient of determination. 2 The metric represents the proportion of the variance of the ground truth strain explained by the predicted values, while Pearson's correlation coefficient (rrr) measures the linear correlation between the predicted values ​​and the ground truth strain values, reflecting the consistency of the estimation. These metrics provide a comprehensive assessment of the accuracy and reliability of the model's strain estimation ability.

[0111] The strain estimation performance of different models is shown in Table 3 below. Table 3 Strain estimation performance of the EchoNet-Dynamic model, VSwinT-DP strain model, and Unity-GLS model. [Table 3]

[0112] The table above compares the performance of three models—EchoNet-Dynamic, VSwinT-DP strain, and Unity-GLS—in estimating left ventricular longitudinal strain (LVLS). The results show that the model of the present invention achieved a perfect strain success rate (1050 / 1050) and the highest R than the other models. 2 The report highlights its superior performance in strain estimation, particularly in the correlation coefficient (0.8846) and the lowest MSE (7.3476) and RMSE (2.7106). These metrics demonstrate superior accuracy, consistency, and reliability in strain estimation compared to EchoNet-Dynamic and Unity-GLS.

[0113] Hybridization of segmentation models and strain calculation algorithms In addition to the strain estimation performance of three proprietary models (VSwinT-DP strain, EchoNet-Dynamic, and Unity-GLS), six hybrid models combining different LV segmentation models with different strain measurement models were also tested. A total of nine hybrid strain methods were tested on the test dataset, and the results were compared with expert-derived strain values. The results are plotted as nine scatter plots of AI-derived strain values ​​against expert-derived strain values, as shown in Figures 6A to 6I. Figures 6A to 6I summarize the Spearman rank correlation coefficient ρ values ​​of the nine hybrid strain methods compared with expert strain values. VSwinT-DP strain had the highest correlation with expert-derived strain. The intraclass correlation coefficients (ICCs) between the three unique combinations and expert strain values ​​were as follows: VSwinT+DP strain values: 0.868 (0.807~0.905), p<0.001; Unity-GLS: 0.627 (0.583~0.667), p<0.001; and EchoNet-Dynamic: 0.501 (0.454~0.545), p<0.001.

[0114] Compared to proprietary combinations, the EchoNet-Dynamic contour produces higher absolute strain values ​​(EchoNet+DP strain vs. VSwinT-DP strain, n=1034: -19.18±9.72 vs. -17.51±5.13, p<0.001; EchoNet+Unity strain vs. Unity-GLS, n=979: -23.52±11.12 vs. -17.31±8.86, p<0.001). Interestingly, the EchoNet-Dynamic strain algorithm yields lower absolute strain values ​​(VSwinT-DP strain vs. VSwinT+EchoNet strain, n=1050: -17.53±5.11 vs. -14.95±5.04, p<0.001; Unity-GLS vs. Unity+EchoNet strain, n=1007: -17.46±6.14 vs. -14.43±5.04, p<0.001). On the other hand, in VSwinT-DP strain and Unity-GLS, no significant deviation in strain values ​​occurred compared to the EchoNet-Dynamic model even when either the segmentation contour or the strain calculation algorithm was changed (VSwinT-DP strain vs. Unity+DP strain, n=1050: -17.52±5.11 vs. -17.24±6.83, p=0.1151; VSwinT-DP strain vs. VSwinT+Unity strain, n=1012: -17.585.10 vs. -17.665.19, p=0.0607).

[0115] Figures 7A to 7I summarize the results of the Bland-Altman analysis. All three unique combinations had small mean differences compared to the expert strains, but DTHR+DTHR had the lowest limit of agreement (LOA) (bias, LOA: VSwinT-DP strain: -1.012±5.031, Unity-GLS: -0.894±9.829, EchoNet-Dynamic: -0.491±13.704). The VSwinT contour maintained good agreement regardless of the strain calculation algorithm used (bias, LOA: VSwinT+EchoNet strain 1.561±5.993, VSwinT+Unity strain -1.110±5.487). However, the agreement between EchoNet-Dynamic and Unity-GLS decreases when using other strain algorithms (bias, LOA: EchoNet+DP strain -2.717±14.573, EchoNet+Unity strain -7.099±19.648, Unity+DP strain -0.733±11.814, Unity+EchoNet strain 2.080±8.554).

[0116] ROC analysis to predict LVGLS values ​​that are above / below the normal range Clinically, LVGLS values ​​below -18% are considered normal, while values ​​above -16% indicate LV dysfunction. ROC analysis was performed to evaluate the diagnostic performance of these three unique combinations of AI-based strain methods. The AUC values ​​for predicting LV dysfunction were 0.960 (0.949~0.971) for VSwinT-DP strain, 0.921 (0.904~0.938) for Unity-GLS, and 0.824 (0.797~0.851) for EchoNet-Dynamic (Figure 8A). The AUC values ​​for predicting normal LV function were 0.950 (0.939~0.962) for VSwinT-DP strain, 0.912 (0.894~0.930) for Unity-GLS, and 0.808 (0.781~0.835) for EchoNet-Dynamic (Figure 8B).

[0117] Application of VScwinT-DP strain in A2C, A3C, and mean LVGLS The agreement between LVGLS derived from the VSwinT-DP strain and expert reference values ​​in apical two-chamber (A2C), apical three-chamber (A3C), and averaged three-image LVGLS was evaluated. In A2C, the VSwinT-DP strain achieved a Spearman rank correlation coefficient ρ of 0.8450 and an ICC of 0.833 (0.814~0.851), with a bias of 0.488 and a LOA of ±5.61. In A3C, the model achieved ρ of 0.8585 and an ICC of 0.853 (0.836~0.896), with a slightly higher bias of 1.143 and a LOA of ±5.11. When averaging GLS across A4C, A2C, and A3C, the model achieved the strongest correlation with expert values ​​(ρ=0.9193, ICC=0.913 (0.903~0.923)) and the narrowest LOA (±3.51%), with the smallest bias of 0.21%.

[0118] Actual examples The EchoNet-Dynamic model generates a segmentation mask as output, requiring additional computer vision algorithms to extract LV contours from the mask. Similarly, the Unity-GLS model generates a heatmap corresponding to the contour region, which is then processed using computer vision techniques to derive the LV contours.

[0119] In contrast, the model proposed in this invention directly predicts the precise 2D coordinates of LV contour points, eliminating the need for intermediate transformation steps, thereby ensuring high efficiency and accuracy.

[0120] Figures 9A to 9C show the performance of three AI models under poor image quality conditions. Under poor image quality, EchoNet-Dynamic (left panel) produces an irregular and temporally inconsistent LV segmentation mask, resulting in very large jitter in the strain-versus-time curve and errors in identifying ED and ES time points (left panel of Figure 9C). Unity-GLS successfully calculates LVGLS, but the strain-versus-time curve exhibits significant jitter (blue curve in the right panel of Figure 9C) that cannot be resolved even with a smoothing filter (orange curve in the right panel of Figure 9C) (parts distinguished by color in the original text are identified in the drawings of this application based on shape, positional relationship, and illustrated content. Since the drawings of this application are represented in grayscale, color distinctions are not made. The same applies below).

[0121] Figures 10A and 10C show the performance of three AI models under the influence of multiple reflection artifacts. Due to multiple reflection artifacts, the LV mask is distorted in EchoNet-Dynamic (red mask in the upper left panel of Figures 10A and 10B), and the apex is truncated in Unity-GLS (upper right panel of Figure 10A, faded-out red shadow and yellow stripe at the LV apex). These errors further affect the accuracy of myocardial length estimation, resulting in an incorrect strain-versus-time curve (blue curve in the left and right panels of Figure 10C) that cannot be corrected by the smoothing filter (orange curve in the left and right panels of Figure 10C).

[0122] Figures 11A–11F show the performance of three AI models under conditions of poor visualization of the LV apex. Figures 11A–11E are selected frames to demonstrate the segmentation performance of the AI ​​models. Unity-GLS produces noisy contour lines during LV boundary calculation (yellow stripes and green contours in the right panel of Figures 11C and 11D) due to the low reliability of its LV boundary heatmap. The temporal consistency of EchoNet-Dynamic further deteriorates under conditions of poor apical visualization (red masks and corresponding green contours in the left panel of Figures 11A–11E), resulting in a reduced number of ED / ES points identified by the strain algorithm (Figure 11F). Both EchoNet-Dynamic and Unity-GLS failed to identify the end-diastolic or end-systolic points in the strain-versus-time curve (in Figure 11F, there are errors in the number and position of the red (end-diastolic) and green (end-systolic) points on the left and right strain curve panels). In this image, the blood-muscle boundary of the apical lateral wall is easier to identify in the video than in the still frame. The spatiotemporal consistency of the VSwinT-DP strain effectively solves this problem (center panel in Figures 11A-11F).

[0123] Figures 12A–12D show examples of irrational end-extension or end-contraction points due to a lack of temporal consistency. Due to the threshold limitations for end-extension / end-contraction in the Unity-GLS strain algorithm, the end-extension point could not be identified in this video (missing red dot on the curve on the right side of Figure 12D), making LVGLS calculation impossible. Furthermore, a fragmented mask was observed during segmentation using EchoNet-Dynamic, resulting in a fragmented, irregular LV mask (upper left panel of Figures 12A–12C) and anatomically inconsistent contours (lower left panel of Figures 12A and 12C).

[0124] Advantages of VSwinT and regression-based design The framework of this invention uses a VSwinT backbone combined with a regression head and FCN-style multiscale feature fusion. This design improves both computational efficiency and accuracy and addresses the limitations seen in conventional methods such as EchoNet-Dynamic and Unity-GLS.

[0125] EchoNet-dynamic relies on a DeepLab V3+ architecture that includes a CNN backbone, neck, and segmentation head. While this approach excels at extracting a comprehensive segmentation map of the heart contour, it is computationally intensive and not optimized for direct strain calculations. Similarly, Unity-GLS uses an HRNet architecture with a heatmap head for feature point localization. While robust in many configurations, heatmap prediction occasionally fails, resulting in positional errors (Figures 10A, 11C, and 11D).

[0126] In contrast, the use of a regression head avoids post-processing steps, eliminates heatmap quantization errors, and eliminates the need for manually designed ground truth heatmaps. While previous CNN-based regression approaches struggled with stability and accuracy, VSwinT overcomes these problems. The VSwinT-DP strain of this invention not only reduces computational overhead but also effectively captures the shape and spatiotemporal relationships of the heart. These improvements restore the feasibility of using a regression head in LVGLS calculations.

[0127] Furthermore, the FCN-style multiscale fusion in this architecture balances global context with fine, local details. Unlike the segmentation head of EchoNet-Dynamic, the regression head of the present invention produces compact, parameterized outputs with minimal distortion. This makes the approach of the present invention more efficient and more applicable in clinical settings.

[0128] The importance of spatiotemporal consistency in echocardiography CNN-based neural networks have achieved great success in semantic segmentation of static images. This success has naturally led to the use of 3D spatiotemporal convolution in dynamic cardiac ultrasound. While this approach works well when estimating LVEF with pre-selected high-quality echocardiographic images, it has limitations when applied to LVGLS calculations.

[0129] Unlike conventional object segmentation, cardiac chambers are not distinct objects with clear boundaries. Instead, they are separated by structures such as thin, fast-moving valves or the interventricular septum, which have inconsistent echogenicity. When each frame is segmented independently, the resulting pixel masks tend to have irregular, jagged boundaries and low temporal coherence (red masks in the left panel of Figures 9–12). This lack of temporal consistency leads to instability in the strain-versus-time curve. Researchers often reduce such variability by applying smoothing filters (orange curves in Figures 9C, 10C, 11F, and 12D) and relying on precise criteria for identifying ED and ES time points.

[0130] The performance of CNN-based segmentation is further limited by several operator-related factors, including shortened diagrams and suboptimal acquisition angles; patient-related factors, including translational motion and occasionally occurring out-of-plane motion; and ultrasound-specific artifacts such as reverberation, refraction, side lobes, mirror artifacts, and near-field clutter. These challenges lead to an increased failure rate of segmentation, as shown in Figures 9–12. For example, EchoNet-Dynamic often produces non-uniform contours (Figure 9) and fragmented segmentation masks (Figure 12), sometimes even including masks of other cardiac chambers or extracardiac regions. Unity-GLS also fails when its LV boundary heatmap is unreliable (Figure 10), resulting in noisy contours (Figure 11). These inaccuracies lead to excessive contour lengths that significantly distort strain values. To compensate for this, Unity-GLS applies length-based thresholds in its strain algorithm for ED and ES. However, this approach may result in abandoning strain calculation (Figure 12). This explains why Unity-GLS did not achieve a 100% contour-to-strain conversion rate in the experiment.

[0131] Transformer-based architectures are particularly well-suited to echocardiographic image segmentation, where ambiguity in inter-domain relationships and boundaries is common, because they leverage self-attention to effectively capture long-range dependencies. Compared to convolutional neural networks, they offer stronger global context modeling, enhanced robustness to artifacts, and better integration of local and global features. While traditional transformer-based approaches such as SegFormer and SwinT have shown advantages in echocardiographic segmentation, they overlook temporal dynamics because they were originally designed for static 2D frames. VSwinT addresses this limitation by extending SwinT's shift window mechanism to three dimensions and incorporating the time axis. In this design, global self-attention enforces spatial consistency within each frame, while the 3D shift window introduces temporal consistency across consecutive frames. Using these mechanisms together, VSwinT achieves robust spatiotemporal consistency and is particularly well-suited to echocardiographic video segmentation tasks. This disclosure demonstrates that VSwinT addresses these challenges through superior spatiotemporal consistency. Thus, the strain-versus-time curve eliminates the need for additional smoothing filters, simplifies the design of strain algorithms, and reduces measurement variability.

[0132] The above description of the embodiments is provided so that those skilled in the art can manufacture and use the subject matter. Various modifications to these embodiments will be readily apparent to those skilled in the art. The new principles and subject matter disclosed herein can be applied to other embodiments without the use of innovative skills. The subject matter shown and claimed in the claims is not intended to be limited to the embodiments described herein, but is given the broadest scope that corresponds to the principles and novel features disclosed herein. Additional embodiments are intended to be within the spirit and true scope of the disclosed subject matter. Accordingly, the present invention is intended to include modifications and variations that are within the scope of the appended claims and their equivalents.

Claims

1. A computer implementation method for video-based image segmentation, The steps include: obtaining an input video containing multiple image frames, The process includes the step of generating one or more predicted contour points of a target contour in one or more of the plurality of image frames using a trained machine learning model, A computer implementation method wherein the trained machine learning model is trained on a training dataset prepared from multiple training videos, each of which includes multiple consecutive frames having annotated contour points of one or more target contours.

2. The computer implementation method according to claim 1, wherein the machine learning model includes a feature extractor that applies a local window self-attention mechanism to both the spatial and temporal dimensions in order to improve the stability of segmentation across the image frames.

3. The computer implementation method according to claim 2, wherein the feature extractor is built on a video-swin transformer (VSwinT) backbone.

4. The computer implementation method according to claim 2, further comprising a multiscale feature fusion mechanism for generating a fused feature map based on the outputs of different stages of the feature extractor for the machine learning model.

5. The computer implementation method according to claim 4, further comprising a contour regressor for generating the predicted contour points of one or more target contours based on the machine learning model and the fused feature map.

6. The computer implementation method according to claim 1, wherein the annotated contour points of one or more target contours are used as training ground truth during training.

7. The computer implementation method according to claim 1, wherein the machine learning model does not require mask-based segmentation or post-processing for generating the predicted contour points.

8. The computer implementation method according to claim 1, wherein the input video is an echocardiogram video.

9. The computer implementation method according to claim 8, wherein one or more target contours include the left ventricular (LV) endocardial line.

10. The computer implementation method according to claim 9, wherein the one or more target contours further include a left ventricular (LV) epicardial line and a left atrium (LA) endocardial line.

11. The computer implementation method according to claim 8, wherein the input video is an echocardiogram video having an apical two-chamber view (A2C), an echocardiogram video having an apical three-chamber view (A3C), or an echocardiogram video having an apical four-chamber view (A4C).

12. The trained machine learning model generates predicted contour points in a plurality of consecutive image frames of the input video, and the method For each of the plurality of image frames, the step of calculating the left ventricular endocardial length based on the predicted contour points of the left ventricular (LV) endocardial line; The computer implementation method according to claim 9, further comprising the step of identifying end-diastolic (ED) frames and end-systolic (ES) frames from the plurality of image frames based on the left ventricular endocardial length.

13. Each of the identified ED frames, The steps include obtaining one or more ED-ES pairs by pairing them with subsequent ES frames selected from the identified ES frames, The steps include calculating the long-axis strain value for each of the one or more ED-ES pairs, The method according to claim 12, further comprising the step of averaging the longitudinal strain values ​​of one or more ED-ES pairs to obtain a final left ventricular longitudinal strain (LVLS).

14. The end-contraction (ED) frame and the end-expansion (ES) frame are, The steps include obtaining an endocardial length series including the left ventricular endocardial length of the plurality of image frames, For each of the plurality of image frames, the step of obtaining an endocardial change sequence including the difference in left ventricular endocardial length between two consecutive image frames, Based on the aforementioned endocardial change sequence, the step of determining the optimal state sequence is the total state sequence that maximizes the penalty-bearing total pathway. The steps include determining the positions of the extended end (ED) frame and the contracted end (ES) frame based on the optimized state sequence, The aforementioned state sequence includes state values ​​representing the contraction or expansion state in each time frame, The method according to claim 12, wherein the penalty-equipped entire pathway is calculated based on the endocardial change sequence and the entire state sequence, and a penalty value is introduced into the penalty-equipped entire pathway each time the state value changes in the entire state sequence.

15. The method according to claim 14, wherein the extended end (ED) frame is the image frame in which the state changes from an extended state to a contracted state, and the contracted end (ES) frame is the image frame in which the state changes from a contracted state to an extended state in the optimized state sequence.

16. The method according to claim 14, wherein the optimization state sequence is determined by a dynamic programming algorithm.

17. The method according to claim 14, wherein the step of determining the optimized state sequence includes the step of iteratively calculating one or more penalized temporal paths in each of the image frames and identifying one or more suboptimal temporal paths from the one or more penalized temporal paths.

18. A method for training a machine learning model that connects a feature extractor, a multiscale feature fusion mechanism, and a contour regressor to generate contour points of a video-based image, A step of obtaining a training dataset comprising multiple training data derived from multiple training videos, wherein each of the multiple training data includes one of the multiple training videos as a training input, and each of the multiple training videos includes multiple training image frames labeled with annotated contour points of one or more target contours as the corresponding training ground truth. The steps include: feeding the training dataset into the machine learning model and performing parameter fitting to train the machine learning model so that the machine learning model can generate predicted contour points for one or more target contours for the multiple image frames using an input video that includes multiple image frames; The feature extractor applies a local window self-attention mechanism to both the spatial and temporal dimensions of the input video. The multiscale feature fusion mechanism receives outputs from different stages of the feature extractor and generates a fused feature map based on those outputs. A method comprising: a contour regressor receiving the fused feature map and generating predicted contour points for one or more target contours for the plurality of image frames based on the fused feature map.

19. The method according to claim 18, wherein the feature extractor is built on a video swin transformer (VSwinT) backbone.

20. The method according to claim 19, wherein the feature extractor generates three temporally complete feature tensors for the multiscale feature fusion mechanism to generate the fused feature map.

21. The method according to claim 18, wherein the mean squared error (MSE) loss function is used for parameter fitting of the machine learning model.

22. The method according to claim 18, wherein each of the plurality of training videos is an echocardiogram video.

23. The method according to claim 22, wherein one or more target contours include the left ventricular (LV) endocardial line.

24. The method according to claim 23, wherein the one or more target contours further include a left ventricular (LV) epicardial line and a left atrium (LA) endocardial line.

Images