Heart ultrasound video left ventricle segmentation and ejection fraction calculation method based on DeepLabv3 + and R (2 +1) D network

Abstract

The invention discloses a heart ultrasound video left ventricle segmentation and ejection fraction calculation method based on a DeepLabv3 + and R (2 + 1) D network, and belongs to the field of medical artificial intelligence. The method aims at solving the problems that in the prior art, left ventricle function evaluation time sequence information is insufficient in utilization, the key frame recognition automation degree is low, and the segmentation precision and the overall process automation degree are not high. According to the technical scheme, the method comprises the following steps: preprocessing an original cardiac ultrasound video, including frame extraction, intelligent cutting of a region of interest and data enhancement; performing time sequence modeling on a video sequence by using an R (2 + 1) D network, and automatically identifying key frames of the end of diastole (ED) and the end of systole (ES); performing high-precision pixel-level left ventricle region segmentation on the identified key frame by adopting a DeepLabv3 + network; estimating a left ventricular end diastolic volume (LVEDV) and a left ventricular end systolic volume (LVESV) by combining an improved Simpson method on the basis of a segmentation result and image physical size calibration information; and finally, calculating a left ventricular ejection fraction (LVEF) according to a standard formula, and outputting a structured report and a visual image. Through multi-network collaboration and full-process automation, the accuracy, efficiency, objectivity and consistency of LVEF evaluation are remarkably improved, and the method has a good clinical application prospect.

Description

Technical Field

[0001] The present invention belongs to the fields of medical artificial intelligence, medical image processing and computer vision technology, and particularly relates to a system and method integrating deep temporal modeling and fine spatial segmentation network for fully automatic left ventricle (LV) dynamic segmentation and intelligent identification of key physiological phases (end-diastole and end-systole) in cardiac ultrasound videos, and further accurate calculation of left ventricular ejection fraction (LVEF). Background Art

[0002] Heart disease is one of the leading causes of death and disability worldwide. Due to its non-invasive, real-time, portable, and relatively low-cost nature, echocardiography has become a clinically useful tool for assessing cardiac structure and function, with left ventricular ejection fraction (LVEF) being a core metric. Traditional LVEF assessment relies on manual identification of end-diastole (ED) and end-systole (ES) frames and delineation of the left ventricle, which is time-consuming and highly subjective.

[0003] Deep learning methods such as the DeepLab series of networks have been used for medical image segmentation. Authorized patent CN113689441B uses the DeepLabV3 network for left ventricular segmentation, but it is mainly targeted at static images and has the following shortcomings: First, it lacks effective temporal modeling, resulting in incoherent video segmentation results; second, the identification of key phase frames (ED / ES frames) relies on manual or partial labeling and is not fully automated; third, the decoder structure of DeepLabV3 is relatively simple, and the segmentation accuracy in the fuzzy boundary areas of ultrasound images is limited. The R(2+1)D network can efficiently model video timing information by decomposing 3D convolutions and is suitable for identifying ED / ES frames in the cardiac cycle. DeepLabv3+ improves the precision of spatial segmentation by introducing a more complete encoder-decoder structure.

[0004] Therefore, existing technologies still have shortcomings in combining spatial segmentation accuracy with temporal modeling capabilities, achieving automatic keyframe recognition and end-to-end LVEF calculation. This paper aims to combine the fine segmentation of DeepLabv3+ with the temporal modeling advantages of R(2+1)D to solve the above problems. Summary of the Invention

[0005] The present invention aims to solve the following core problems in the existing technology for automatic left ventricular function assessment using cardiac ultrasound video:

[0006] Problem 1: Insufficient utilization of temporal information and temporal inconsistency in segmentation results. Existing methods primarily perform 2D frame-level segmentation, lacking effective modeling of temporal continuity between video frames and cyclical cardiac motion patterns. This can easily lead to unreasonable jitter or jumps in the left ventricular boundary in the temporal sequence, affecting the accuracy and reliability of dynamic analysis.

[0007] Problem 2: Low automation or insufficient precision in critical phase frame (ED / ES) identification. Accurate calculation of LVEF relies heavily on accurate identification of ED and ES frames. Existing technologies often require manual selection or can only process data with pre-labeled key frames, lacking the ability to fully automatically and accurately identify ED / ES frames from continuous video streams.

[0008] Problem 3: Spatial segmentation accuracy in complex boundary regions needs improvement. Low contrast, speckle noise, and blurred tissue boundaries, characteristic of cardiac ultrasound images, make accurate delineation of the left ventricle (particularly the apex and papillary muscle regions) extremely challenging. Existing segmentation networks (such as DeepLabV3) are limited in their ability to decode and recover detailed information.

[0009] Problem 4: Lack of an end-to-end, fully automated solution from video to LVEF. Current research focuses primarily on segmentation algorithms, with few providing a complete automated process that integrates keyframe recognition, segmentation, volume calculation, and final LVEF output. This makes it difficult to directly meet the needs of efficient clinical assessment.

[0010] To address the above issues, this paper proposes a method for left ventricular segmentation and ejection fraction calculation in cardiac ultrasound video based on DeepLabv3+ and R(2+1)D network. The system processes the following core steps in sequence:

[0011] Step 1: Data preprocessing module

[0012] This module normalizes cardiac ultrasound videos for use in subsequent models. First, the video is decoded to extract image frames and record metadata. Second, a region of interest (ROI) containing the left ventricle is extracted using an adaptive cropping method. The image frames are then resized to a standard input size (e.g., 256 pixels × 256 pixels) and grayscale converted and pixel values ​​normalized. During training, data augmentation is applied to ultrasound image characteristics, including spatial geometric transformations, appearance and noise simulation (e.g., multiplicative speckle noise), and temporal dimension enhancement for the R(2+1)D network.

[0013] Step 2: Intelligent Timing Analysis and Key Frame Recognition Module

[0014] This module automatically locates end-diastolic (ED) and end-systolic (ES) keyframes using an R(2+1)D network. A pretrained R(2+1)D model processes video clips, which can be connected to a temporal modeling head for frame-level classification. The model is fine-tuned using transfer learning on a dataset with precisely labeled ED / ES frames, using cross-entropy. During inference, a sliding window is used to process the video, integrating overlapping frame probabilities. ED / ES frames are then identified through peak detection based on cardiac cycle constraints and physiological rule verification.

[0015] Step 3: High-precision left ventricle segmentation module

[0016] This module uses the DeepLabv3+ network to perform left ventricular segmentation on identified ED / ES frames. The network utilizes an advanced backbone network with an encoder featuring dilated convolutions, an ASPP module, and a refined decoder architecture to fuse multi-scale information and recover boundary details. The model is initialized with ImageNet pre-trained weights and trained on LV segmentation masks, incorporating a boundary-aware loss. Post-processing optimization is performed using a conditional random field (CRF).

[0017] Step 4: Left ventricular volume quantitative calculation module

[0018] This module estimates left ventricular end-diastolic volume (LVEDV) and end-systolic volume (LVESV) using a modified Simpson method based on the LV segmentation mask and physical dimensioning information from ED / ES frames. First, accurate pixel physical dimensions are acquired (e.g., from DICOM metadata). Then, left ventricular geometric parameters are automatically measured, including physical area calculation from the segmentation mask and determination of the left ventricular long axis (LAX) using a robust algorithm. Finally, LVEDV and LVESV are calculated using a modified Simpson biplane method.

[0019] Step 5: LVEF calculation and intelligent result output module

[0020] This module calculates left ventricular ejection fraction (LVEF) using the standard formula based on the calculated LVEDV and LVESV, and performs a validity check. The system then provides multi-dimensional output, including numerical parameters such as LVEDV, LVESV, and LVEF. It also dynamically visualizes segmentation contours and highlights ED / ES keyframes. Advanced features include displaying left ventricular volume-time curves and automatically generating structured diagnostic reports containing key images and parameters.

[0021] This paper deeply integrates R(2+1)D time series analysis with DeepLabv3+ fine segmentation and builds an end-to-end fully automatic LVEF calculation process. Compared with the existing technology, it has the following beneficial effects:

[0022] 1. Collaborative and efficient spatiotemporal feature modeling to improve dynamic segmentation accuracy and consistency.

[0023] 2. Fully automated and highly accurate identification of key physiological phases (ED / ES).

[0024] 3. End-to-end fully automatic intelligent LVEF assessment method to improve clinical efficiency.

[0025] 4. The objectivity, repeatability and standardization of LVEF assessment have been significantly improved.

[0026] 5. Modular design makes expansion and maintenance easy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the overall architecture of the system of the present invention;

[0028] Figure 2 This is a schematic diagram of the R(2+1)D network structure of the present invention;

[0029] Figure 3 This is a schematic diagram of the DeepLabv3+ network structure of the present invention;

[0030] Figure 4 Schematic diagram of the segmentation result of the present invention. DETAILED DESCRIPTION

[0031] Step 1: Data preparation and preprocessing

[0032] This step aims to acquire and process cardiac ultrasound video data to provide standardized input for subsequent model training and testing. This step first acquires cardiac ultrasound video data, then extracts and selects frames from the video, and then performs preprocessing operations.

[0033] Step 1a: Data Source. This example uses the publicly available EchoNet-Dynamic dataset as the primary data source. This dataset contains a large number of cardiac ultrasound videos of the apical four-chamber (A4C) view and provides indexes of ED and ES frames, as well as corresponding left ventricular segmentation masks and LVEF values.

[0034] Step 1b: Frame Extraction and Selection. For each video file, decode it into a continuous sequence of image frames using the OpenCV library. For R(2+1)D network training, extract fixed-length (32-frame) video segments centered around the annotated ED / ES frames. For DeepLabv3+ network training, primarily use ED and ES frames with segmentation masks provided in the dataset, although additional frames from the cardiac cycle may be supplemented to enhance model robustness.

[0035] Step 1c: Preprocessing Operations: This sub-step contains further processing of the extracted frames.

[0036] Step 1c.1: Intelligently locate and crop the region of interest. To reduce irrelevant background interference and focus on the left ventricle, an adaptive cropping method is used to extract the region of interest containing the left ventricle. The cropped region will serve as input for subsequent processing.

[0037] Step 1c.2: Image resizing: All cropped image frames are resized to 256 pixels × 256 pixels using the Lanczos interpolation algorithm.

[0038] Step 1c.3: Normalize pixel values. Linearly map image pixel values ​​from their original range to the [-1, 1] interval. This operation helps accelerate model convergence and improve training stability.

[0039] Step 1c.4: Data augmentation (applied only during training). To expand the amount of training data and improve the generalization ability of the model, various data augmentation techniques are applied. Temporal dimension augmentation for the R(2+1)D model includes: randomly sampling temporal segments from the full video, padding or truncating segments that are insufficient or too long, and slightly adjusting the playback rate of the segments with a certain probability. Spatial dimension augmentation for the DeepLabv3+ model includes: random rotation (angle range ±10 degrees), random scaling (ratio range 0.85 to 1.15 times), random horizontal flipping, random adjustment of image brightness and contrast (adjustment factor within the range of ±0.15), injection of multiplicative speckle noise, and application of elastic deformation.

[0040] Step 1d: Dataset Partitioning: The preprocessed data is divided into specific proportions (70% training set, 15% validation set, and 15% test set) to ensure randomness and repeatability of the partitioning process and avoid data leakage.

[0041] Step 2: Training and reasoning of intelligent timing analysis and keyframe recognition models

[0042] This step aims to train and use the R(2+1)D network model to automatically identify ED and ES frames in the video, providing a temporal reference for subsequent accurate segmentation and LVEF calculation.

[0043] Step 2a: Network training (R(2+1)D). The R(2+1)D-50 model, pre-trained on the Kinetics-600 large-scale video action recognition dataset, was selected as the base architecture. To enhance the ability to capture complex temporal patterns in cardiac ultrasound videos, a bidirectional long short-term memory (LSTM) layer was added to the output features of the R(2+1)D backbone network, followed by a fully connected layer and a softmax activation function. This layer outputs a probability of ED, ES, or "other" for each frame in the sequence. The input consists of a video clip of 32 frames with dimensions of 256 pixels × 256 pixels × 1 channel. During training, the AdamW optimizer was used with an initial learning rate of 5e-5 and a batch size of 8. A cosine annealing learning rate scheduling strategy was used to dynamically adjust the learning rate. To address the class imbalance caused by the small proportion of ED / ES frames in the entire video, a weighted focal loss was used as the loss function. The model is trained for approximately 60 epochs, and performance is monitored on the validation set to prevent overfitting.

[0044] Step 2b: Keyframe Identification Inference. The video to be analyzed is segmented into overlapping segments using a sliding window strategy (window length 32 frames, step size 16 frames). Each segment is independently fed into the trained R(2+1)D model to obtain the ED and ES probabilities for each frame within the segment. For frames predicted multiple times due to window overlap, their final probabilities are obtained by averaging or weighted averaging the predictions. A Gaussian smoothing filter is applied to the frame-by-frame probability sequence of the entire video to remove short-term noise. Subsequently, combined with an estimate of cardiac cycle length, candidate ED and ES frames are preliminarily identified within each identified cardiac cycle by searching for the local maximum peak of the smoothed ED and ES probability sequences. Furthermore, physiological rules are applied for verification and optimization. For example, these include ensuring that the ED frame typically precedes the ES frame within a cycle; that the left ventricular area of ​​the ED frame is the largest within the cycle, while that of the ES frame is the smallest; that the time interval between the ED and ES frames is within a physiologically reasonable range; and that the identification of ED / ES frames exhibits a certain rhythmicity over multiple consecutive cardiac cycles.

[0045] Step 3: Training and inference of a high-precision left ventricle segmentation model

[0046] This step aims to train and use the DeepLabv3+ network model to accurately segment the left ventricle on the ED and ES frames identified in step 2.

[0047] Step 3a: Network training. The DeepLabv3+ architecture is selected, and its backbone network adopts the ResNeSt-50 model pre-trained on ImageNet, which enhances the feature representation capability through the Split-Attention mechanism. The output stride of the encoder is set to 16 to retain sufficient spatial information. The decoder partially fuses low-level features from the backbone network Block1 (corresponding to an output stride of 4) to achieve finer boundary recovery. The input is a single frame image with a size of 256 pixels × 256 pixels × 1 channel. During training, the AdamW optimizer is used, the initial learning rate is set to 1e-4, and the batch size is set to 16. The PolyLR learning rate decay strategy is used. The loss function uses a combined loss, specifically 0.7 times the Lovasz-Softmax Loss plus 0.3 times the Boundary-aware Dice Loss. Its calculation formula is:

[0048] L seg =0.7×L Lovasz +0.3×L BoundaryDice (1)

[0049] Among them, L Dice is the Dice loss, L BCE is the binary cross entropy loss, and α and β are weight coefficients. During inference, the output is a probability map, which is thresholded to obtain a segmentation mask. Conditional Random Field (CRF) is used for post-processing and optimization. The model is trained for approximately 120 epochs.

[0050] Step 3b: Segmentation Inference. The identified ED and ES frames are fed into the trained DeepLabv3+ model, which outputs a probability map of each pixel belonging to the left ventricle. This probability map is converted into a binary segmentation mask by applying the Otsu adaptive thresholding method.

[0051] Step 3c: Advanced Segmentation Post-Processing. To further improve the smoothness and anatomical plausibility of the segmentation results, a fully connected conditional random field (DenseCRF) is optionally applied as a post-processing step. DenseCRF uses the probability map output by DeepLabv3+ as a unary potential and combines it with the color / intensity similarity and spatial proximity between pixels as a binary potential. The final refined segmentation result is obtained through iterative optimization.

[0052] Step 4: Quantitative calculation of left ventricular volume

[0053] This step aims to automatically calculate the left ventricular volume in the ED and ES phases using the clinically recommended modified Simpson method based on the left ventricular segmentation mask obtained in step 3 and the physical size calibration information of the image.

[0054] Step 4a: Accurately obtain pixel spacing. Automatically extract the physical size of the pixel (mm / pixel) from the metadata of the DICOM file.

[0055] Step 4b: Automatic left ventricular long axis (LAX) detection. To automate the Simpson method, the left ventricular long axis must be accurately determined. In this embodiment, a lightweight convolutional neural network (CNN) is trained to regress the precise coordinates of the cardiac apex and the midpoint of the mitral valve annulus. The input is the left ventricular segmentation mask or the original grayscale image, and the output is the coordinates of these two key anatomical landmarks. Connecting these two points yields the left ventricular long axis.

[0056] Step 4c: Volume calculation. The modified Simpson biplane method is preferred. After acquiring or simulating the corresponding A2C view data, the area and long axis are measured on the ED and ES frames in two orthogonal planes, and the volume is calculated using the biplane Simpson formula. This method divides the left ventricle into N disks (usually N = 20) along its long axis. For each view (e.g., Figure 1 Heshi Figure 2 ), measure the minor axis diameter D at the i-th disc level 1,i and D 2,i Assuming that the cross section of each disc is elliptical, its thickness is h = L / N (where L is the length of the long axis used to divide the disc, usually the shorter or the average of the two views). The left ventricular volume V is calculated as follows:

[0057]

[0058] Step 5: Calculation of LVEF and intelligent output of results

[0059] This step aims to calculate the final LVEF value based on the calculated LVEDV and LVESV, and provide a multi-dimensional, user-friendly result display and report. The calculation formula is as follows:

[0060]

[0061] Step 5a: Calculate LVEF. Follow the formula strictly. Check for validity, such as ensuring that LVEDV is greater than LVESV and greater than zero.

[0062] Step 5b: Output the results. The output includes the calculated core numerical parameters such as LVEDV (ml), LVESV (ml), and LVEF (%). On the user interface, the segmented left ventricular contour is superimposed on the original images of the ED and ES frames. If segmentation and volume estimation are performed for all frames within the cycle, a curve showing the left ventricular volume change over time (Volume-Time Curve) can be drawn and displayed. Finally, a structured diagnostic auxiliary report in PDF format can be generated, which includes patient information (if any), examination information, key frame images and segmentation, a quantitative parameter list, and an optional volume curve graph.

[0063] Those skilled in the art will appreciate that the above embodiments are merely preferred examples of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for left ventricular segmentation and ejection fraction calculation in cardiac ultrasound video based on DeepLabv3+ and R(2+1)D network, characterized in that: The following steps are involved: Step 1: performing data preprocessing on the cardiac ultrasound video, wherein the preprocessing includes video frame extraction, region of interest (ROI) intelligent cropping, image size normalization, pixel value normalization, and data enhancement for the training phase; Step 2: Use the R(2+1)D network to perform temporal modeling analysis on the preprocessed video frame sequence, and combine it with the decision logic based on cardiac cycle constraints to automatically identify the end-diastolic frame (ED frame) and the end-systolic frame (ES frame); Step 3: Input the identified ED frame and ES frame into the segmentation model with DeepLabv3+ network as the core to perform pixel-level segmentation of the left ventricular region and obtain the corresponding left ventricular segmentation mask map; Step 4: Based on the physical size calibration information contained in the image metadata of the cardiac ultrasound video and combined with the automatic detection algorithm of the left ventricular long axis (LAX), the modified Simpson method is used to calculate the left ventricular end-diastolic volume (LVEDV) and end-systolic volume (LVESV) corresponding to the ED frame and ES frame; Step 5: Calculate the left ventricular ejection fraction (LVEF) and output the LVEF value, related volume parameters and segmentation image results. Optionally, a structured diagnostic auxiliary report containing key frame images, segmentation contours, quantitative parameters and visual analysis charts can be generated.

2. The method according to claim 1, characterized in that In the second step, the R(2+1)D network is used in conjunction with the timing enhancement module to enhance the recognition accuracy and robustness of key phases in the cardiac cycle.

3. The method according to claim 1, characterized in that In step three, the DeepLabv3+ network model is optimized using a combined loss function including a boundary-aware loss function during training, and after segmentation inference, a conditional random field (CRF) is optionally used to post-process and optimize the segmentation mask image to improve the smoothness and accuracy of the segmentation edges.

4. The method according to claim 1, wherein In step 4, the left ventricular volume is calculated preferably using the modified Simpson biplane method, and under the condition of a single view, the modified Simpson single plane method is used.

5. The method according to claim 1, wherein In step five, the structured diagnostic auxiliary report includes but is not limited to the following: patient identification information (if any), video file identification, original images of ED frames and ES frames and images of their superimposed segmentation contours, calculated LVEDV, LVESV, and LVEF values, and optional left ventricular volume-time change curves and segmentation quality assessment indicators.

Images