Heart echocardiographic function prediction method and device based on deep learning

Abstract

The invention relates to the field of cardiac function prediction, and provides a cardiac echocardiography function prediction method and device based on deep learning in order to improve the accuracy of cardiac echocardiography function prediction, a positioning key frame and a left ventricle segmentation task are brought into an analysis process, key information of an ultrasound image is fully mined, and data availability is enhanced; the key frame is identified firstly and then the left ventricle segmentation is performed on the key frame, so that the data processing mode is more convenient and efficient; a feature fusion thought is introduced into a calculation framework, overall motion features and key part features of the heart are combined, the heart function is evaluated more accurately, and the performance of the model is improved; the design of the variable window and the hierarchical attention is combined with the cardiac function prediction model, the long-distance adaptability of the attention mechanism is fully exerted, the understanding ability of the model to the time-space motion information of the ultrasonic video sequence is enhanced, and the model accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the field of cardiac function prediction, and in particular to a method and device for predicting cardiac function using cardiac ultrasound based on deep learning. Background Art

[0002] Cardiac function prediction algorithms mainly rely on deep learning models (such as CNN, RNN, etc.) to evaluate cardiac function indicators such as left ventricular ejection fraction (LVEF), cardiac diastolic and systolic function, etc. Cardiac ultrasound imaging data is widely used in the field of cardiac function prediction because of its non-invasiveness, low cost, strong real-time performance and high availability. Deep learning cardiac function prediction algorithms based on ultrasound data have shown great potential, especially in the automated extraction of cardiac features.

[0003] CN116130090A discloses a method for measuring ejection fraction, including: obtaining ultrasonic cardiogram sequence frames; inputting each frame of the ultrasonic cardiogram sequence frames into a pre-trained deep neural network model to obtain a left ventricle image sequence frame; calculating the pixel area of ​​each frame of the left ventricle image sequence frame to obtain a left ventricle pixel area change sequence; based on the left ventricle pixel area change sequence, determining the end-diastolic image and the end-systolic image from the left ventricle image sequence frame; and calculating the ejection fraction based on the end-diastolic image and the end-systolic image. This method requires processing each frame of the image, resulting in a large number of processing operations and low efficiency; the processing result only includes the ejection fraction, which is limited in scope of application; and the ejection fraction is calculated based on only the single parameter of the pixel area change, which is not accurate. Summary of the invention

[0004] In order to improve the accuracy of cardiac ultrasound cardiac function prediction, the present application provides a cardiac ultrasound cardiac function prediction method and device based on deep learning.

[0005] The technical solution adopted by the present invention to solve the above problems is:

[0006] The cardiac ultrasound cardiac function prediction method based on deep learning includes:

[0007] Step 1: construct a key frame prediction data set and a key frame prediction model, and use the key frame prediction data set to train the key frame prediction model; the key frames include frames corresponding to the end diastole and the end systole;

[0008] Step 2: construct a key frame left ventricle segmentation data set and a left ventricle segmentation model, and use the key frame left ventricle segmentation data set to train the left ventricle segmentation model;

[0009] Step 3: Input the ultrasound image to be predicted into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y, auxiliary feature vector Feature Y It is the concatenation result of the feature vectors of the last layer of the segmentation model corresponding to the frames at the end of diastole and end of systole;

[0010] Step 4: Construct a cardiac function data set and a cardiac function analysis model, and use the cardiac function data set to train the cardiac function analysis model; Y Input the cardiac function analysis model, which processes the ultrasound image to obtain the feature vector Feature X , the feature vector Feature X and auxiliary feature vector Feature Y Splicing to get the feature vector Feature Z , the feature vector Feature Z Input prediction head, output three cardiac function parameters: LVEF, EDV and ESV values;

[0011] Step 5: Input the ultrasound image to be predicted into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y , the ultrasound image to be predicted and the auxiliary feature vector Feature Y The cardiac function analysis model was input to obtain LVEF, EDV, and ESV values.

[0012] Furthermore, the key frame prediction model consists of five spatiotemporal convolutional layers and two fully connected layers. The spatiotemporal convolutional layers use 3D convolutional units to simultaneously capture the features of the temporal and spatial dimensions, and output timing information containing myocardial movement; the fully connected layers integrate high-level semantic information into key frame numbers.

[0013] Furthermore, when the key frame prediction data set is used to train the key frame prediction model, the loss function adopts MSE.

[0014] Furthermore, the left ventricle segmentation model is built based on the UNet framework, which includes four layers of encoders and four layers of decoders. The data between layers with the same feature dimension are directly connected, and the feature analysis of the image is realized at four scales.

[0015] Furthermore, when the key frame left ventricle segmentation dataset is used to train the left ventricle segmentation model, the loss function adopts Dice.

[0016] The cardiac ultrasound cardiac function prediction device based on deep learning includes:

[0017] Key frame prediction module: obtains key frames in ultrasound images based on the key frame prediction model, and the key frames include frames corresponding to the end of diastole and end of systole;

[0018] Left ventricle segmentation module: obtains the left ventricle segmentation results at the end of diastole and end of systole based on the left ventricle segmentation model;

[0019] Cardiac function analysis module: completes cardiac function analysis based on cardiac function analysis model, ultrasound images, and left ventricular segmentation results at end-diastole and end-systole, and outputs three cardiac function parameters: LVEF, EDV, and ESV values.

[0020] Compared with the prior art, the present invention has the following beneficial effects: the key frame positioning and left ventricle segmentation tasks are incorporated into the analysis process to fully mine the key information of the ultrasound image itself and enhance data availability; the key frame is first identified and then the left ventricle is segmented, making the data processing method more convenient and efficient; the idea of ​​feature fusion is introduced into the calculation framework, and the overall motion characteristics of the heart and the characteristics of key parts are combined to more accurately evaluate the cardiac function and improve the performance of the model; the design of variable windows and hierarchical attention is combined with the cardiac function prediction model to give full play to the long-distance adaptability of the attention mechanism, enhance the model's ability to understand the spatiotemporal motion information of the ultrasound video sequence, and improve the model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the cardiac ultrasound cardiac function prediction method based on deep learning;

[0022] Figure 2 This is a flow chart of a cardiac ultrasound cardiac function prediction method based on deep learning corresponding to the first embodiment;

[0023] Figure 3 Schematic diagram of the cardiac function analysis model structure. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] like Figure 1 As shown, the cardiac ultrasound cardiac function prediction method based on deep learning includes:

[0026] Step 1: Build a key frame prediction data set and a key frame prediction model, and use the key frame prediction data set to train the key frame prediction model.

[0027] Construct a key frame prediction dataset: manually annotate the original cardiac ultrasound data, that is, the ultrasonic motion images of the heart over a period of time, and mark two key frames for each image: ED frame and ES frame, corresponding to the end of diastole and end of systole respectively; use the frame number of the key frame as the label.

[0028] Constructing a key frame prediction model: As an independent module, the key frame prediction model is used to extract ultrasound image motion information, and then determine the location of the end diastole and end systole, and output the prediction results of the key frame in the form of frame numbers. The model consists of five layers of spatiotemporal convolutional layers and two layers of fully connected layers. The convolutional layer uses 3D convolutional units to capture the features of the time dimension and the spatial dimension at the same time, and outputs the timing information containing the myocardial motion. The fully connected layer integrates the high-level semantic information into the key frame number.

[0029] The key frame prediction data set is used to train the key frame prediction model: the model is trained using the constructed key frame prediction data set. Each training sample consists of 3D cardiac ultrasound data with a uniform number of frames and the corresponding ED / ES frame number. The model input is a 3D ultrasound image, and the output is the predicted two key frame numbers, which are used to calculate the loss with the label. The loss function uses MSE.

[0030] The MSE loss function is as follows:

[0031]

[0032] Among them, p ED and p ES are the ED and ES frame numbers predicted by the model, g ED and g ES is the corresponding true label, and n is the number of samples for calculating the loss once.

[0033] Step 2: construct a key frame left ventricle segmentation dataset and a left ventricle segmentation model, and use the key frame left ventricle segmentation dataset to train the left ventricle segmentation model.

[0034] Constructing a keyframe left ventricle segmentation dataset: The keyframe left ventricle segmentation dataset is constructed only for data with keyframe labels, that is, the training samples used by the two models can be consistent. For each 3D ultrasound image data, select the two frames of images img_ED and img_ES corresponding to the keyframe, and then outline the left ventricle area for the two frames of images respectively to obtain the left ventricle segmentation labels seg_ED and seg_ES.

[0035] Constructing a left ventricular segmentation model: The segmentation model is an independent module after key frame prediction, which is used to segment the left ventricular region for a given single-frame image. The segmentation model is built based on the UNet framework, and contains four layers of encoders and four layers of decoders. The data between layers with the same feature dimension are directly connected, and the feature analysis of the image is realized at four scales. The last layer outputs the segmentation result, which is a segmentation map composed of 0 and 1.

[0036] The left ventricle segmentation model is trained using the key frame left ventricle segmentation dataset: The left ventricle segmentation model is trained using the key frame left ventricle segmentation dataset. The key frames in each sample are input separately and sent to the model training as two independent samples. The input of the model is a single ultrasound image containing the heart area, and the output is the corresponding left ventricle segmentation result. The Dice loss is calculated between the predicted segmentation result and the segmentation label, and the objective function is as follows:

[0037]

[0038] Among them, P is the segmentation result predicted by the model, and G is the segmentation label.

[0039] Step 3: Input the ultrasound image to be predicted into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y , auxiliary feature vector Feature Y It is the concatenation result of the feature vectors of the last layer of the segmentation model for the frames corresponding to the end-diastole and the end-systole.

[0040] Step 4: Construct a cardiac function dataset, which contains ultrasound images and corresponding LVEF, EDV, and ESV values. Construct a cardiac function analysis model based on the VST model structure with multi-level attention and variable windows, and use the cardiac function dataset, combined with the key frame prediction model and the left ventricle segmentation model to train the cardiac function analysis model. Input the ultrasound image into the VST backbone network to obtain the feature vector Feature X ; Feature vector Feature X and auxiliary feature vector Feature Y Splicing to get the feature vector Feature Z , Feature Z =concatenate(Feature X ,Feature Y );The feature vector Feature Z The prediction head is input and the predicted values ​​of LVEF, EDV and ESV are output and compared with their true cardiac function parameter values ​​to calculate the MSE loss.

[0041] Step 5: Input the ultrasound image to be predicted into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y , the ultrasound image to be predicted and the auxiliary feature vector Feature Y The cardiac function analysis model was input to obtain LVEF, EDV, and ESV values.

[0042] The present invention also provides a cardiac ultrasound cardiac function prediction device based on deep learning, comprising:

[0043] Key frame prediction module: obtains key frames in ultrasound images based on the key frame prediction model, and the key frames include frames corresponding to the end of diastole and end of systole;

[0044] Left ventricle segmentation module: obtains the left ventricle segmentation results at the end of diastole and end of systole based on the left ventricle segmentation model;

[0045] Cardiac function analysis module: completes cardiac function analysis based on cardiac function analysis model, ultrasound images, and left ventricular segmentation results at end-diastole and end-systole, and outputs three cardiac function parameters: LVEF, EDV, and ESV values.

[0046] Embodiment 1

[0047] In this case, the Video Swin Transformer (VST) model will be used as the basic network structure for cardiac function prediction, the AlexNet-based regression network will be used as the key frame prediction model, and UNet will be used as the segmentation model for left ventricle segmentation. Finally, the key frame prediction model and the segmentation model will be connected in series, and the outputs of the two will be combined with the features of the VST as the basis for the regression of cardiac function indicators. Figure 2 As shown, the specific steps include:

[0048] 1) Build a key frame prediction model. The specific steps are as follows:

[0049] Step 1: Dataset creation. The number of cardiac ultrasound data frames collected in this case is between 120 and 220, including 2 to 5 complete cardiac cycles, with long-axis and short-axis modes, and each mode is extracted separately. Mark the end diastole (ED) and end systole (ES) of each sample. For example, sample a is a long-axis cardiac image with a total of 169 frames, including 83 ED frames and 118 ES frames. Note that the marked ED and ES frames need to be in the same cardiac cycle. Unify the video image size to 112*112, select the cardiac cycle containing the key frames, and randomly truncate it to 50 frames, and adjust the marked serial numbers accordingly. For example, the sample dimension of the input network is 112*112*50, and its key frames are marked as 13, 48.

[0050] Step 2: Build a key frame prediction model. The prediction model uses the AlexNet framework. The network structure includes five 3D convolutional layers and three fully connected layers. The last layer outputs two prediction values. The size of the 3D convolution is [3,3,2], corresponding to the three dimensions of the image [height, width, time]. The fully connected layer integrates high-dimensional features and projects the change relationship between frames as the position information of the key frames.

[0051] Step 3: Train the key frame prediction model. The model uses random parameter initialization, uses long-axis or short-axis cardiac ultrasound video as input, and two predicted values ​​as output. The model uses the Adam optimizer, and the learning rate is set to 0.001. In order to make the model output close to the actual annotation, the annotation value is reduced by 100 times when calculating the loss. For example, the actual key frame numbers of the sample are 13 and 48, which are converted to 0.13 and 0.48 when calculating the loss in the model.

[0052] The loss function of the model is:

[0053]

[0054] Where n is the sample size for calculating the loss once. In this case, n is set to 2.

[0055] 2) Construct the left ventricle segmentation model. The specific steps are as follows:

[0056] Step 1: Construct a segmentation dataset. Select ultrasound images with keyframe labels as the dataset, extract the ED and ES frames according to the frame numbers, and obtain two 2D heart images. Use annotation software such as 3DSlicer to annotate the left ventricle of each image. The annotation range includes the left ventricular area within the myocardium, and the annotated pixel values ​​are 0 and 1, 0 represents the background, and 1 represents the left ventricular area. Each ultrasound image obtains two keyframe images and their corresponding annotations, both of which are 112*112 in size. The segmentation dataset does not distinguish between keyframes, that is, each ultrasound image finally obtains two segmentation samples.

[0057] Step 2: Build a left ventricle segmentation model. The left ventricle segmentation model uses the UNet framework. The network structure includes four scale encoder-decoder layers. The encoder uses maximum pooling to reduce the feature size, and the decoder uses upsampling to restore the feature size. The feature vectors are directly connected between each layer with the same feature size to enhance the model's understanding of the low-level resolution information of the image.

[0058] Step 3: Train the left ventricle segmentation model. The model input is a 2D single-frame image of the long axis or short axis, and the prediction result with a dimension of 112*112*2 is used as the output. The model uses the Adam optimizer, and the learning rate is set to 0.001. Since UNet performs four downsamplings and the input size is small, the image is resized to 256*256 in the data preprocessing stage, and the prediction results and annotations are also enlarged to the same size accordingly.

[0059] The loss function of the model is calculated based on the Dice value:

[0060]

[0061] Among them, P is the segmentation result predicted by the model, G is the segmentation label, and the size is 256*256.

[0062] 3) Obtain auxiliary information. The specific steps are as follows:

[0063] The key frame prediction model is used to detect the key frame number of the original 3D ultrasound image, and the predicted ED and ES frames are extracted and input into the left ventricle segmentation model in sequence to obtain the feature vector of each image at the last layer of the segmentation model. Finally, the vectors of the two key frames are concatenated as auxiliary information to be fused with the main frame features.

[0064] 4) Analyze cardiac function in combination with auxiliary information. The specific steps are as follows:

[0065] Construct and train a cardiac function analysis model. The backbone of the cardiac function analysis model uses the VideoSwinTransformer framework. The network structure is divided into four layers, and the number of blocks in each layer is [2, 2, 6, 2]. The model structure is as follows: Figure 3 As shown in the figure, the attention block combines the multi-head self-attention mechanism and the variable window mechanism. The multi-head module sets the number of groups G. i =C i / C', where C i is the number of input channels of the i-th layer, and C' is the number of channels output by each attention branch. The prediction head of the cardiac function analysis model is composed of three fully connected layers, and the size of the output signal is 3, representing the predicted cardiac function parameters LVEF (ejection fraction), EDV (left ventricular end-diastolic volume) and ESV (left ventricular end-systolic volume). The 3D ultrasound image is input into the VST model, and after the calculation of the variable window and hierarchical attention, the feature vector of the last layer is obtained, which represents the cardiac motion relationship that accommodates the spatiotemporal dimension. The feature vector is connected with the auxiliary information and input into the prediction head together, and the three cardiac function parameter values ​​are output. The loss function of the model is calculated using MSE.

[0066] The ultrasound image to be predicted is input into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y , the ultrasound image to be predicted and the auxiliary feature vector Feature Y Input the cardiac function analysis model to obtain LVEF, EDV and ESV values.

[0067] Embodiment 2

[0068] The cardiac ultrasound cardiac function prediction device based on deep learning includes:

[0069] The key frame prediction module uses the 3D convolutional AlexNet structure, which is suitable for short-axis and long-axis cardiac motion image analysis and feature learning of the complete cardiac motion cycle. Its methods include at least: spatiotemporal coding of cardiac regions, end-diastolic and end-systolic positioning, key frame sequence number prediction regression and MSE optimization.

[0070] The left ventricle segmentation module uses a multi-scale skip-layer UNet structure to identify the left ventricle region at multiple scales of feature levels, and achieves left heart segmentation for a single ultrasound image. It is suitable for short-axis and long-axis left ventricle identification and positioning. Its method includes at least: cardiac region feature encoding, multi-scale feature skip-layer connection, left ventricle region segmentation, and Dice optimization.

[0071] The cardiac function analysis module uses a multi-level attention and variable window VST model structure as the cardiac function analysis model, and combines the key frame prediction module and the left ventricle segmentation module to independently extract the key information used for cardiac function evaluation as an auxiliary guiding condition for the cardiac function analysis module, and realizes the prediction of cardiac function parameters based on 3D ultrasound images and their key motion information, which is suitable for cardiac function evaluation of short-axis and long-axis cardiac ultrasound. Its methods include at least: variable window attention mechanism, multi-scale transformer architecture, spatiotemporal joint encoding of 3D ultrasound images, feature fusion, and MSE regression optimization.

[0072] Specifically, in the cardiac function analysis module, the cardiac function prediction process for 3D ultrasound images is as follows:

[0073] Input the ultrasound image into the VST backbone network to obtain the feature vector Feature X ;

[0074] The feature vector Feature X and auxiliary feature vector Feature Y Splicing to get the feature vector Feature Z , Feature Z =concatenate(Feature X ,Feature Y );Feature Y It is the concatenation result of the feature vectors of the last layer of the segmentation model corresponding to the frames at the end of diastole and end of systole;

[0075] Feature Z Input the prediction head and output three cardiac function parameters.

Claims

1. A cardiac ultrasound cardiac function prediction method based on deep learning, characterized in that: include: Step 1: construct a key frame prediction data set and a key frame prediction model, and use the key frame prediction data set to train the key frame prediction model; the key frames include frames corresponding to the end diastole and the end systole; Step 2: construct a key frame left ventricle segmentation data set and a left ventricle segmentation model, and use the key frame left ventricle segmentation data set to train the left ventricle segmentation model; Step 3: Input the ultrasound image to be predicted into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y , auxiliary feature vector Feature Y It is the concatenation result of the feature vectors of the last layer of the segmentation model corresponding to the frames at the end of diastole and end of systole; Step 4: Construct a cardiac function data set and a cardiac function analysis model, and use the cardiac function data set to train the cardiac function analysis model; Y Input the cardiac function analysis model, which processes the ultrasound image to obtain the feature vector Feature X , the feature vector Feature X and auxiliary feature vector Feature Y Splicing to get the feature vector Feature Z , the feature vector Feature Z Input prediction head, output three cardiac function parameters: LVEF, EDV and ESV values; Step 5: Input the ultrasound image to be predicted into the key frame prediction model and then into the left ventricle segmentation model to obtain the auxiliary feature vector Feature Y , the ultrasound image to be predicted and the auxiliary feature vector Feature Y The cardiac function analysis model was input to obtain LVEF, EDV, and ESV values.

2. The method for predicting cardiac function using cardiac ultrasound based on deep learning according to claim 1, characterized in that: The key frame prediction model consists of five spatiotemporal convolutional layers and two fully connected layers. The spatiotemporal convolutional layers use 3D convolutional units to simultaneously capture the features of the temporal and spatial dimensions, and output timing information containing myocardial movement; the fully connected layers integrate high-level semantic information into key frame numbers.

3. The method for predicting cardiac function using cardiac ultrasound based on deep learning according to claim 2, characterized in that: When the key frame prediction data set is used to train the key frame prediction model, the loss function uses MSE.

4. The method for predicting cardiac function using cardiac ultrasound based on deep learning according to claim 1, characterized in that: The left ventricle segmentation model is built based on the UNet framework, which includes four layers of encoders and four layers of decoders. The data between layers with the same feature dimension are directly connected to realize feature analysis of images at four scales.

5. The method for predicting cardiac function using cardiac ultrasound based on deep learning according to claim 1, characterized in that: When the key frame left ventricle segmentation dataset is used to train the left ventricle segmentation model, the loss function is Dice.

6. A cardiac ultrasound cardiac function prediction device based on deep learning, characterized in that: include: Key frame prediction module: obtains key frames in ultrasound images based on the key frame prediction model, and the key frames include frames corresponding to the end of diastole and end of systole; Left ventricle segmentation module: obtains the left ventricle segmentation results at the end of diastole and end of systole based on the left ventricle segmentation model; Cardiac function analysis module: completes cardiac function analysis based on cardiac function analysis model, ultrasound images, and left ventricular segmentation results at end-diastole and end-systole, and outputs three cardiac function parameters: LVEF, EDV, and ESV values.

Images