Left ventricular ejection fraction prediction method and system based on echocardiography

Abstract

The present application relates to a method for predicting left ventricular ejection fraction based on echocardiography. The present application includes: obtaining a real-time echocardiogram of a target patient, preprocessing it, obtaining a real-time echocardiogram video, inputting a frame sequence in each real-time echocardiogram video into a preset classification model, obtaining a section classification result, using a real-time echocardiogram video corresponding to a frame sequence of an A4C section and an A2C section as a key video, inputting the key video into a preset left ventricular endocardium segmentation model, obtaining a left ventricular endocardium segmentation result sequence, inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model, obtaining an A4C section LVEF and an A2C section LVEF, respectively, and calculating the LVEF based on the A4C section LVEF and the A2C section LVEF. Compared with the Simpson method, the method has a simpler prediction process and is more interpretable than direct prediction based on video.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a left ventricular ejection fraction prediction method based on echocardiography. Background Art

[0002] Left ventricular ejection fraction (LVEF) is one of the important indicators for judging the basic condition of cardiac function and the type of heart failure. At present, echocardiography is often used in clinical practice to measure left ventricular ejection fraction (LVEF). The commonly used methods are: Teichholz method, uniplane Simpson method and biplane Simpson method. Among them, the biplane Simpson method has fewer geometric assumptions about the morphology of the left ventricle, so it is suitable for different types of patients including patients with left ventricular morphological changes. It is a widely recognized LVEF measurement method.

[0003] like Figure 1 As shown, the biplane Simpson method includes the following steps: step S101, obtaining the apical four-chamber (A4C) and apical two-chamber (A2C) sections in the two-dimensional echocardiogram; step S102, obtaining the end-diastolic frame of the A4C section, the end-systolic frame of the A4C section, the end-diastolic frame of the A2C section, and the end-systolic frame of the A2C section from the A4C and A2C sections; step S103, outlining the left ventricular endocardial contour on the end-diastolic frame of the A4C section and the end-diastolic frame of the A2C section, respectively, and determining the key points; step S104, determining the apex of the heart according to the endocardial contour and the key points. , mitral valve ring, divide the line from the apex to the mitral valve ring plane into 20 equal parts, and the left ventricle can be divided into 20 elliptical cylinders of equal height; step S105, obtain the major axis, minor axis length, and height of the elliptical cylinder, calculate the volume of the elliptical cylinder, sum the volumes of the 20 elliptical cylinders, and obtain the end-diastolic frame volume; step S106, repeat steps S103-S105, and obtain the end-systolic frame volume based on the end-systolic frame of the A4C section and the end-systolic frame of the A2C section; step S107, calculate the left ventricular ejection fraction based on the end-diastolic volume and the end-systolic volume. The dual-plane Simpson method requires key frame screening of the A4C and A2C sections, which has the problems of complex process and long measurement time. In addition, due to the low quality of echocardiographic images, the LVEF results have large intra-observer and inter-observer differences.

[0004] To solve the above problems, technicians in this field often adopt methods directly based on video prediction to circumvent them. However, this method has the problem of poor interpretability. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present disclosure provide a method and system for predicting left ventricular ejection fraction based on echocardiography.

[0006] A first aspect of the disclosed embodiments provides a method for predicting left ventricular ejection fraction based on echocardiography, characterized by comprising:

[0007] Obtain real-time echocardiograms of target patients;

[0008] Preprocessing the real-time echocardiogram to obtain a preprocessed real-time echocardiogram, wherein the preprocessed real-time echocardiogram includes a plurality of real-time echocardiogram videos, and each real-time echocardiogram video includes a plurality of frame sequences;

[0009] Input one or more frame sequences in each real-time echocardiography video into a preset classification model to perform section classification and obtain a section classification result, where the section classification result is an A4C section or an A2C section or others;

[0010] The real-time echocardiographic video corresponding to the frame sequence of the section classification results of A4C section and A2C section is used as a key video, and the key video is input into a preset left ventricular endocardium segmentation model to obtain a left ventricular endocardium segmentation result sequence;

[0011] Inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain A4C section LVEF and A2C section LVEF respectively;

[0012] The LVEF was calculated based on the LVEF of the A4C section and the LVEF of the A2C section.

[0013] Optionally, the preset left ventricular endocardium segmentation model includes an encoder, a pooling layer, a first spatial attention layer, a first channel attention layer, a decoder, and a first convolutional layer, wherein the real-time echocardiogram video corresponding to the frame sequence of the section classification results as the A4C section and the A2C section is used as a key video, and the key video is input into the preset left ventricular endocardium segmentation model to obtain a left ventricular endocardium segmentation result sequence, including:

[0014] Step S301, encoding the kth frame of the key video by the encoder to obtain a key frame vector;

[0015] Step S302: performing pooling processing on the key frame vector through the pooling layer to obtain a first feature vector;

[0016] Step S303: Process the first feature vector through a first spatial attention layer to obtain a first spatial feature vector for representing spatial features;

[0017] Step S304: Process the first feature vector through a first channel attention layer to obtain a first channel feature vector for characterizing channel features;

[0018] Step S305: Merge the first spatial feature vector and the first channel feature vector into the second feature vector;

[0019] Step S306: Process the second feature vector through a decoder to obtain the third feature vector;

[0020] Step S307, processing the third feature vector through the first convolutional layer to obtain a first left ventricular endocardium segmentation result;

[0021] Step S308, repeat steps S301-S307, process each frame sequence of the key video, obtain the first left ventricular endocardium segmentation result corresponding to each frame in the frame sequence, construct the first left ventricular endocardium segmentation result sequence corresponding to the frame sequence based on the first left ventricular endocardium segmentation result, and construct the left ventricular endocardium segmentation result sequence based on the first left ventricular endocardium segmentation result sequence.

[0022] Optionally, the step of inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF, respectively, comprises the following steps:

[0023] The left ventricular endocardium segmentation result sequence includes a plurality of first left ventricular endocardium segmentation result sequences, and the first left ventricular endocardium segmentation result sequence is divided into a first left ventricular endocardium segmentation result sequence of an A4C section and a first left ventricular endocardium segmentation result sequence of an A2C section according to the frame sequence section classification result;

[0024] Input the first left ventricular endocardium segmentation result sequence of the A4C section into the preset LVEF prediction model to obtain i first LVEF values, calculate the average of all the first LVEF values ​​to obtain the A4C section LVEF, where i is the number of the first left ventricular endocardium segmentation result sequence of the A4C section;

[0025] The first left ventricular endocardium segmentation result sequence of the A2C section is input into the preset LVEF prediction model to obtain j second LVEF values, and all the second LVEF values ​​are averaged to obtain the A2C section LVEF, where j is the number of the first left ventricular endocardium segmentation result sequence of the A2C section.

[0026] Optionally, the preset classification model includes multiple 3D encoders, feature fusion layers, and fully connected layers, and the step of inputting one or more frame sequences in each real-time echocardiography video into the preset classification model to perform section classification, and obtaining the section classification result includes the following steps:

[0027] Dividing the one or more frame sequences into first, second, and third frame sequences;

[0028] Input the first, second and third frame sequences into the 3D encoder respectively to obtain the first frame sequence feature, the second frame sequence feature and the third frame sequence feature;

[0029] Input the first, second and third frame sequence features into the feature fusion layer for fusion, and output the fused frame sequence features;

[0030] The fused frame sequence features are input into the fully connected layer and the slice classification results are output.

[0031] Optionally, the preprocessing includes the following steps:

[0032] Screening the real-time echocardiogram to obtain all two-dimensional grayscale echocardiogram (dcm) videos;

[0033] Converting the two-dimensional grayscale echocardiogram (DCM) video into a target format to obtain a first echocardiogram video;

[0034] Cutting out an ultrasound image sector region from the first ultrasound cardiography video to obtain a second ultrasound cardiography video;

[0035] dividing the second echocardiographic video into one or more frame sequences as a real-time echocardiographic video;

[0036] The collection of all real-time echocardiogram videos is used as the preprocessed real-time echocardiogram.

[0037] Optionally, the method further comprises super-resolution processing of the real-time echocardiography video, the steps being as follows:

[0038] Step S601: input the real-time echocardiogram video into a preset super-resolution model, wherein the preset super-resolution model includes: a third spatial attention layer, a spatial feature extraction layer, a temporal feature extraction layer, and a feature fusion layer;

[0039] Step S602: input the nth frame of the mth frame sequence in the real-time echocardiogram video into the third spatial attention layer to obtain the first spatial attention feature F n , n≥3;

[0040] Step S603: The first spatial attention feature F n Input spatial feature extraction layer to obtain the first intermediate attention feature F n ′;

[0041] Step S604: Using the same method, obtain the first intermediate attention feature F corresponding to the n-1th frame n-1 ′, the first intermediate attention feature F corresponding to the n-2th frame n-2 ′, F n-1 ′、F n-2′ Input feature fusion layer to get the second intermediate attention feature F s ;

[0042] Step S605: The first intermediate attention feature F n ′ and the second intermediate attention feature F s Adding them to obtain a first supplementary frame, adding the first supplementary frame to the nth frame to obtain a first intermediate frame;

[0043] Step S606, repeat the above steps to supplement each frame in the m-th frame sequence to obtain a first intermediate frame sequence;

[0044] Step S607: Input the mth frame sequence into the temporal attention layer to obtain the temporal attention feature Km;

[0045] Step S608: Add the temporal attention feature Km to the first intermediate frame sequence to obtain a predicted frame sequence;

[0046] Step S609, repeating steps S601-S608, acquiring all frame sequences, and taking all predicted frame sequences as real-time echocardiogram video after super-resolution processing.

[0047] Optionally, before obtaining the echocardiogram of the target patient, the method further comprises:

[0048] Obtain historical echocardiograms of multiple patients;

[0049] Preprocessing the historical echocardiograms of the multiple patients to obtain preprocessed historical echocardiograms, wherein the preprocessed historical echocardiograms include multiple historical echocardiogram videos, and each historical echocardiogram video includes multiple historical echocardiogram frame sequences;

[0050] Performing slice classification and labeling on the historical echocardiogram frame sequence, wherein the slice classification and labeling result is divided into A4C slice or A2C slice or others;

[0051] Input historical echocardiogram frame sequences and slice classification annotation results into the classification model framework, and output the trained classification model and slice classification results;

[0052] For the historical echocardiographic frame sequence whose section classification and annotation results are A4C section or A2C section, left ventricular endocardium annotation is performed to obtain left ventricular endocardium annotation results;

[0053] Based on the left ventricular endocardium annotation result, multiple key points are generated on the left ventricular endocardium contour to obtain the key point annotation result;

[0054] The historical echocardiographic frame sequence of the A4C section and the A2C section, the left ventricular endocardium annotation results, and the key point annotation results are input into the left ventricular endocardium segmentation model framework, and the trained left ventricular endocardium segmentation model is output.

[0055] Optionally, the method further includes pre-training, and the pre-training includes the following steps:

[0056] The historical echocardiographic frame sequences with all sections classified and labeled as A4C and A2C sections were divided into frame-by-frame images, and then 40-60% of all the images were randomly masked to save the image copies;

[0057] The image copy is input into an encoder in the left ventricular endocardium segmentation model framework for self-supervised learning to obtain a trained encoder of the left ventricular endocardium segmentation model.

[0058] A second aspect of the disclosed embodiment provides a left ventricular ejection fraction prediction device based on echocardiography, comprising:

[0059] An acquisition module for acquiring a real-time echocardiogram of a target patient;

[0060] A preprocessing module, used for preprocessing the real-time echocardiogram to obtain a preprocessed real-time echocardiogram, wherein the preprocessed real-time echocardiogram includes a plurality of real-time echocardiogram videos, and each real-time echocardiogram video includes a plurality of frame sequences;

[0061] A section classification module is used to input one or more frame sequences of each real-time echocardiography video into a preset classification model to perform section classification and obtain a section classification result, where the section classification result is an A4C section or an A2C section or others;

[0062] A left ventricular endocardium segmentation module is used to use the real-time echocardiographic video corresponding to the frame sequence of the section classification results A4C section and A2C section as a key video, input the key video into a preset left ventricular endocardium segmentation model, and obtain a left ventricular endocardium segmentation result sequence;

[0063] An LVEF prediction module, used for inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF respectively;

[0064] The calculation module is used to calculate the LVEF based on the A4C section LVEF and the A2C section LVEF.

[0065] A third aspect of the disclosed embodiments provides a heart failure identification device, comprising the above-mentioned left ventricular ejection fraction prediction device based on echocardiography, and also comprising an LVEF classification model for classifying patients with heart failure into HFrEF, HFmEF or HFpEF according to the LVEF value.

[0066] A fourth aspect of an embodiment of the present disclosure provides a computer device, comprising: a memory, a processor, and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in any one of claims 1-8.

[0067] A fifth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0068] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0069] The left ventricular ejection fraction prediction method and system based on echocardiography classifies the sections of the frame sequence of echocardiography and segments the left ventricular endocardium, obtains the left ventricular endocardium segmentation result sequence, obtains the LVEF values ​​corresponding to the endocardium segmentation result sequences of A4C and A2C sections based on the left ventricular endocardium decomposition result sequence, and then averages them to obtain the A4C section LVEF and the A2C section LVEF; this method only segments the endocardium, and does not need to further obtain key frames (i.e., end-systolic frames and end-diastolic frames) from the A4C and A2C sections to calculate the end-diastolic volume and end-systolic volume, which simplifies the process compared with the biplane Simpson method. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 It is a schematic diagram of the process of the dual-plane Simpson method in the prior art;

[0073] Figure 2 is a flow chart of an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0074] Figure 3 It is a schematic flow chart of obtaining a sequence of left ventricular endocardial segmentation results in an optional left ventricular ejection fraction prediction method based on echocardiography in an embodiment of the present invention;

[0075] Figure 4 It is a schematic diagram of the working principle of a classification model in an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0076] Figure 5 It is a schematic diagram of a process for obtaining LVEF values ​​of A4C and A2C sections in an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0077] Figure 6 It is a schematic diagram of the process of super-resolution processing in an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram of a model training process in an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0079] Figure 8 It is a schematic diagram of the training process of the left ventricular endocardium segmentation model in an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0080] Fig. 9 It is a schematic diagram of the encoder training process of the left ventricular endocardium segmentation model in an optional left ventricular ejection fraction prediction method based on echocardiography according to an embodiment of the present invention;

[0081] Fig.10 is a schematic diagram of an optional left ventricular ejection fraction prediction device based on echocardiography according to an embodiment of the present invention;

[0082] Fig.11 is a schematic diagram of an optional heart failure identification device according to an embodiment of the present invention;

[0083] Fig.12 A schematic diagram of the structure of an optional computer device according to an embodiment of the present invention; DETAILED DESCRIPTION

[0084] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0085] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0086] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0087] Figure 2 1 is a flowchart of a method for predicting left ventricular ejection fraction based on echocardiography provided by an embodiment of the present disclosure. The method can be executed by a left ventricular ejection fraction determination system based on echocardiography. Figure 2 As shown, the left ventricular ejection fraction prediction method based on echocardiography provided in this embodiment includes the following steps:

[0088] A method for predicting left ventricular ejection fraction based on echocardiography, comprising:

[0089] Step S201, obtaining a real-time echocardiogram of a target patient;

[0090] Specifically, the real-time echocardiogram of the target patient can be obtained through electronic devices, such as ultrasound machines and special probes, application systems, servers and other devices for collecting echocardiograms. For the convenience of explanation, this application adopts a left ventricular ejection fraction prediction system based on echocardiogram. The target patients are patients over 18 years old with clear transthoracic echocardiogram images. The echocardiograms described in this application are all transthoracic echocardiograms. At the same time, patients with the following conditions need to be excluded: (1) previous history of cardiac surgery, such as coronary artery bypass grafting, valve replacement; (2) those with extremely poor echocardiogram image quality.

[0091] Step S202, preprocessing the real-time echocardiogram to obtain a preprocessed real-time echocardiogram, wherein the preprocessed real-time echocardiogram includes a plurality of real-time echocardiogram videos, and each real-time echocardiogram video includes a plurality of frame sequences;

[0092] The real-time echocardiogram is divided into DCM images, DCM videos, and structured reports according to the content, and is divided into two-dimensional grayscale ultrasound, color Doppler ultrasound, and spectral Doppler ultrasound according to the ultrasound type. Because the content and ultrasound type of the real-time echocardiogram are different, in order to reduce the difficulty of subsequent calculations, the real-time echocardiogram of the target patient needs to be preprocessed to obtain the content and ultrasound type of the real-time echocardiogram that meet the prediction needs.

[0093] The specific pre-processing method will be described in detail in the subsequent embodiments, so it will not be described in detail here.

[0094] Step S203, inputting one or more frame sequences in each real-time echocardiography video into a preset classification model, performing section classification, and obtaining a section classification result, where the section classification result is an A4C section or an A2C section or others;

[0095] Specifically, if Figure 4 As shown, the preset classification model includes multiple 3D encoders, which are used to extract features from one or more frame sequences in each real-time echocardiography video, and can capture the time series information in the real-time echocardiography video, so as to better understand the video content, and have stronger robustness than the image classifier. In the echocardiogram obtained in the prior art, a small part of the frame image quality may be poor, and it is difficult to judge the image quality using the traditional image model. When extracting features from the frame sequence with poor image quality, the extracted features are not accurate enough, resulting in inaccurate section classification results. Our technology uses a classification model with a 3D encoder, which is not easily affected by a small number of frames with poor quality.

[0096] In this embodiment, the classification model may also include other types of encoders. As long as other types of encoders can extract features from one or more frame sequences in each real-time echocardiography video and capture the time series information in the real-time echocardiography video, 3D encoders are preferred in this embodiment.

[0097] The specific method of using the classification model to obtain the section classification results will be described in detail in the subsequent embodiments, so it will not be described in detail here.

[0098] Step S204, taking the real-time echocardiographic video corresponding to the frame sequence of the section classification results A4C section and A2C section as the key video, inputting the key video into a preset left ventricular endocardium segmentation model, and obtaining a left ventricular endocardium segmentation result sequence;

[0099] Specifically, the classification model performs section classification on the frame sequence and outputs the section classification result. If the section classification result obtained by the classification model is not A4C or A2C, it means that the frame sequence does not include the A4C section or the A2C section. The real-time echocardiography video corresponding to the frame sequence is not used as a key video and is directly abandoned.

[0100] If the section classification result obtained by classification is A4C or A2C, it means that the frame sequence includes the A4C section or the A2C section. At this time, the real-time echocardiographic video corresponding to the frame sequence whose section classification results are the A4C section and the A2C section is used as the key video, and the key video is input into the preset left ventricular endocardium segmentation model, and the left ventricular endocardium is segmented for the key video to obtain the left ventricular endocardium segmentation result sequence corresponding to the key video;

[0101] At present, before using the biplane Simpson method to perform endocardial segmentation on echocardiography videos, it is often necessary to obtain key frames first. Existing technologies mostly use image-based left ventricular endocardial segmentation models to segment all frames of the video, and then find key frames through the maximum and minimum values ​​of the segmented area. Due to the problem of labeling workload, such segmentation models are often only trained on key frames, not on non-key frames, which may affect its left ventricular endocardial segmentation performance on non-key frames, resulting in errors in key frame selection tasks. If the endocardial segmentation is annotated on non-key frames to improve the accuracy of the segmentation model, not only is there no guide to clearly define the endocardial boundary on non-key frames, but it also increases the workload of labeling. In a small number of technical solutions, there is a special key frame detection model for obtaining key frames, which requires separate training of the key frame detection model, which is complex and labor-intensive.

[0102] Our technical solution does not select key frames. Instead, we directly use the real-time echocardiography video corresponding to the frame sequence with the section classification result of A4C section or A2C section as the key video, and perform left ventricular endocardium segmentation on all frames in the key video. We no longer search for key frames through the maximum and minimum values ​​of the segmentation area. Compared with the biplane Simpson method, this simplifies the process.

[0103] In traditional video prediction models of left ventricular ejection fraction (LVEF), the entire cardiac ultrasound video is usually used as input. However, although this method can provide a certain degree of accuracy, it does not pay attention to the model's focus on local structural changes, especially the dynamic changes in the left ventricular region. Our technical solution selects the real-time echocardiography video corresponding to the frame sequence of the A4C section or A2C section as the key video, processes the key video, and focuses on identifying changes in the left ventricular region. Since the calculation of LVEF directly depends on the contraction and relaxation process of the left ventricle, only paying attention to changes in the left ventricular region helps to improve the interpretability of the model, and focuses the model's learning focus on myocardial activity directly related to LVEF, reducing noise interference in non-left ventricular regions.

[0104] The specific method of using the left ventricular endocardium segmentation model to obtain a left ventricular endocardium segmentation result sequence corresponding to the key video will be described in detail in subsequent embodiments, so it will not be described in detail here.

[0105] Step S205, inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain A4C section LVEF and A2C section LVEF respectively;

[0106] The specific method of using the LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF will be described in detail in subsequent embodiments, so it will not be described in detail here.

[0107] Step S206, calculating the left ventricular ejection fraction based on the LVEF of the A4C section and the LVEF of the A2C section.

[0108] A method for calculating left ventricular ejection fraction based on A4C LVEF and A2C LVEF.

[0109] The left ventricular ejection fraction prediction method and system based on echocardiography disclosed in the embodiments of the present application can accurately determine the left ventricular endocardium segmentation result and achieve the same accuracy as the left ventricular ejection fraction calculated by the biplane Simpson method.

[0110] In a possible implementation, the preset left ventricular endocardium segmentation model includes an encoder, a pooling layer, a first spatial attention layer, a first channel attention layer, a decoder, and a first convolutional layer, wherein: Figure 3 As shown, the real-time echocardiographic video corresponding to the frame sequence of the section classification results A4C section and A2C section is used as the key video, and the key video is input into the preset left ventricular endocardium segmentation model to obtain the left ventricular endocardium segmentation result sequence, including:

[0111] Step S301, encoding the kth frame of the key video by the encoder to obtain a key frame vector;

[0112] In this embodiment, the encoder is an encoder of a U-Net model, or may be other encoders, which are used to encode the kth frame of the key video and convert the kth frame into a key frame vector to facilitate further processing of the left ventricular endocardium segmentation model.

[0113] Step S302: performing pooling processing on the key frame vector through the pooling layer to obtain a first feature vector;

[0114] The pooling layer is a pooling layer of a U-Net model, and may also be a spatial pyramid pooling layer containing a dilated convolution, which is used to perform pooling processing on the key frame vector to obtain a first eigenvector.

[0115] Step S303: Process the first feature vector through a first spatial attention layer to obtain a first spatial feature vector for representing spatial features;

[0116] Step S304: Process the first feature vector through a first channel attention layer to obtain a first channel feature vector for characterizing channel features;

[0117] Step S305: Merge the first spatial feature vector and the first channel feature vector into the second feature vector;

[0118] The first channel attention layer and the first spatial attention layer are used to process the first eigenvector to obtain the first channel eigenvector and the first spatial eigenvector, and then fuse the first channel eigenvector and the first spatial eigenvector to obtain the second eigenvector used to characterize the left ventricular endocardial features; using the first channel attention layer and the first spatial attention layer to process the first eigenvector respectively can better mine and identify the channel features and spatial features in the image, so that the left ventricular endocardium segmentation can utilize common features to improve performance.

[0119] The attention layer may also be in other forms, such as only a spatial attention layer or only a channel attention layer. There are multiple attention layers. In this embodiment, one channel attention layer and one spatial attention layer are preferred.

[0120] Step S306: Process the second feature vector through a decoder to obtain the third feature vector;

[0121] The decoder is a U-Net decoder, or may be other decoders, and the decoder is used to decode the weighted second eigenvector to obtain a third eigenvector.

[0122] Step S307, processing the third feature vector through the first convolutional layer to obtain a left ventricular endocardium segmentation result;

[0123] The first convolution layer is a two-dimensional convolution layer, which is used to process the third eigenvector to obtain a first left ventricular endocardium segmentation result. The first endocardium segmentation result is two binary images, the black part is the background, and the white part is the left ventricle.

[0124] Step S308, repeat steps S301-S307, process each frame sequence of the key video, obtain the first left ventricular endocardium segmentation result corresponding to each frame in the frame sequence, construct the first left ventricular endocardium segmentation result sequence corresponding to the frame sequence based on the first left ventricular endocardium segmentation result, and construct the left ventricular endocardium segmentation result sequence based on the first left ventricular endocardium segmentation result sequence.

[0125] In an embodiment of the present application, the real-time echocardiography video corresponding to the frame sequence whose section classification result is A4C section or A2C section is directly used as the key video, and the left ventricular endocardium is segmented for all frames in the key video. The key frame is no longer searched through the maximum and minimum values ​​of the segmentation area. Compared with the biplane Simpson method, the process is simplified, and the accuracy of the left ventricular endocardium segmentation result is comparable to that of the prior art.

[0126] In a possible implementation, the left ventricular endocardium segmentation result sequence is input into a preset LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF, respectively. Figure 5 As shown, the following steps are included:

[0127] Step S501, the left ventricular endocardium segmentation result sequence includes a plurality of first left ventricular endocardium segmentation result sequences, and the first left ventricular endocardium segmentation result sequence is divided into a first left ventricular endocardium segmentation result sequence of an A4C section and a first left ventricular endocardium segmentation result sequence of an A2C section according to the frame sequence section classification result;

[0128] Step S502, inputting the first left ventricular endocardium segmentation result sequence of the A4C section into a preset LVEF prediction model to obtain i first LVEF values, averaging all the first LVEF values ​​to obtain the A4C section LVEF, where i is the number of the first left ventricular endocardium segmentation result sequence of the A4C section;

[0129] Step S503: input the first left ventricular endocardium segmentation result sequence of the A2C section into a preset LVEF prediction model to obtain j second LVEF values, average all the second LVEF values ​​to obtain the A2C section LVEF, where j is the number of the first left ventricular endocardium segmentation result sequence of the A2C section.

[0130] In an embodiment of the present application, the left ventricular endocardium segmentation sequence is divided into the first left ventricular endocardium segmentation result sequence of the A4C section and the first left ventricular endocardium segmentation result sequence of the A2C section, and then the first LVEF value of the A4C section and the second LVEF value of the A2C section are obtained, and then averaged, which is a new and feasible LVEF prediction method.

[0131] In a possible implementation, the preset classification model includes multiple 3D encoders, feature fusion layers, and fully connected layers, such as Figure 4 As shown, the step of inputting one or more frame sequences in each real-time echocardiogram video into a preset classification model to perform section classification and obtain the section classification result includes the following steps:

[0132] Dividing the one or more frame sequences into first, second, and third frame sequences;

[0133] The first, second and third frame sequences do not overlap each other;

[0134] Input the first, second and third frame sequences into the 3D encoder respectively, and extract the first frame sequence features, the second frame sequence features and the third frame sequence features;

[0135] Input the first, second and third frame sequence features into the feature fusion layer for fusion, and output the fused frame sequence features;

[0136] The fused frame sequence features are input into the fully connected layer and the slice classification results are output.

[0137] Specifically, before using the preset classification model to process the frame sequence, the frame sequence needs to be divided into the first, second, and third frame sequences. In order to allow the classification model to obtain as much content in the frame sequence as possible, the first, second, and third frame sequences do not overlap. The first, second, and third frame sequences are respectively input into the 3D encoder. The 3D encoder can capture the time series information in the frame sequence, so as to better understand the real-time echocardiography video content, and has clinical interpretability than the image classifier. The extracted first frame sequence features, second frame sequence features, and third frame sequence features are fused through the feature fusion layer, and the fused frame sequence features are output, which can include the overall features of the frame sequence as much as possible and capture the time series information. The fused frame sequence features are processed by the fully connected layer to obtain the section classification results.

[0138] In a possible implementation, the preprocessing comprises the following steps:

[0139] Screening the real-time echocardiogram to obtain all two-dimensional grayscale echocardiogram (dcm) videos;

[0140] The real-time echocardiogram is divided into DCM images, DCM videos, and structured reports according to the content, and is divided into two-dimensional grayscale ultrasound, color Doppler ultrasound, and spectral Doppler ultrasound according to the ultrasound type. Because the content and ultrasound type of the real-time echocardiogram are different, in order to reduce the difficulty of subsequent calculations, the real-time echocardiogram of the target patient needs to be preprocessed. In this embodiment, all two-dimensional grayscale echocardiogram DCM videos are screened out from the real-time echocardiogram.

[0141] Converting the two-dimensional grayscale echocardiogram (DCM) video into a target format to obtain a first echocardiogram video;

[0142] In this embodiment, the format is avi.

[0143] Cutting out an ultrasound image sector region from the first ultrasound cardiography video to obtain a second ultrasound cardiography video;

[0144] The real-time echocardiography video is cropped to crop redundant information, retain the ultrasound image sector area, and obtain the cropped real-time echocardiography video as the second echocardiography video.

[0145] dividing the second echocardiographic video into one or more frame sequences as a real-time echocardiographic video;

[0146] The second echocardiogram video is divided into frame sequences of 64 frames in length (e.g., 1 to 64 frames as one segment, 2 to 65 frames as one segment), and the portion less than 64 frames is discarded. Generally speaking, the second echocardiogram video is more than 200 frames in length, so the real-time echocardiogram video includes multiple frame sequences.

[0147] The collection of all real-time echocardiogram videos is used as the preprocessed real-time echocardiogram.

[0148] The set of all real-time echocardiography videos is taken as the preprocessed real-time echocardiogram, that is, the preprocessed real-time echocardiogram includes a plurality of real-time echocardiography videos, and each real-time echocardiography video includes a plurality of frame sequences.

[0149] In this embodiment, after converting the two-dimensional grayscale real-time echocardiogram DCM video format, the time taken by the left ventricular ejection fraction prediction device based on echocardiogram to read the real-time echocardiogram video can be shortened; the format of the real-time echocardiogram video can be other video formats besides avi, such as WMV and MPEG. In this embodiment, it is preferred to convert the two-dimensional grayscale real-time echocardiogram DCM video to AVI, because the AVI format has the best compatibility under the Windows platform.

[0150] The cropping of real-time echocardiography video specifically includes: cropping the smallest area containing the complete ultrasound probe acquisition content, that is, the ultrasound image fan-shaped area, according to the real-time echocardiography video, removing the acquisition parameters, patient information, etc. outside the ultrasound image fan-shaped area to reduce interference from other areas and improve the processing speed of the left ventricular ejection fraction prediction system based on echocardiography.

[0151] In a possible embodiment, the method further comprises super-resolution processing of the real-time echocardiographic video. Figure 6 As shown, the steps are as follows:

[0152] Step S601: input the real-time echocardiogram video into a preset super-resolution model, wherein the preset super-resolution model includes: a third spatial attention layer, a spatial feature extraction layer, a temporal feature extraction layer, and a feature fusion layer;

[0153] Step S602: input the nth frame of the mth frame sequence in the real-time echocardiogram video into the third spatial attention layer to obtain the first spatial attention feature F n , n≥3;

[0154] Step S603: The first spatial attention feature F n Input spatial feature extraction layer to obtain the first intermediate attention feature F n ′;

[0155] Step S604: Use the same method to obtain the first intermediate attention feature F corresponding to the n-1th frame n-1 ′, the first intermediate attention feature F corresponding to the n-2th frame n-2 ′, F n-1 ′、F n-2 ′ Input feature fusion layer to get the second intermediate attention feature F s ;

[0156] Step S605: The first intermediate attention feature F n ′ and the second intermediate attention feature F s Adding them to obtain a first supplementary frame, adding the first supplementary frame to the nth frame to obtain a first intermediate frame;

[0157] Step S606, repeat the above steps to supplement each frame in the m-th frame sequence to obtain a first intermediate frame sequence;

[0158] Step S607: Input the mth frame sequence into the temporal attention layer to obtain the temporal attention feature Km;

[0159] Step S608: Add the temporal attention feature Km to the first intermediate frame sequence to obtain a predicted frame sequence;

[0160] Step S609, repeating steps S601-S608, acquiring all frame sequences, and taking all predicted frame sequences as real-time echocardiogram video after super-resolution processing.

[0161] Due to the limitations of ultrasound technology itself, echocardiography is easily affected by noise, artifacts, motion blur and other factors, and its spatial resolution is lower than other imaging technologies. Therefore, our technical solution provides a super-resolution model, which can remove the interference of these factors to a certain extent and improve image quality, which is not only helpful for doctors to analyze images, but also very helpful for endocardial segmentation models.

[0162] In a possible implementation, Figure 7 As shown in the method, before obtaining the echocardiogram of the target patient, the method further includes:

[0163] Step S701, obtaining historical echocardiograms of multiple patients;

[0164] Specifically, historical echocardiograms of multiple patients can be obtained through electronic devices, application systems, servers and other devices. For the convenience of explanation, this application adopts a left ventricular ejection fraction determination system based on echocardiogram, and the collection conditions are the same as those for obtaining real-time echocardiograms of target patients.

[0165] Step S702: preprocessing the historical echocardiograms of the multiple patients to obtain preprocessed historical echocardiograms, wherein the preprocessed historical echocardiograms include multiple historical echocardiogram videos, and each historical echocardiogram video includes multiple frame sequences;

[0166] The preprocessing method was the same as that of real-time echocardiography.

[0167] Step S703, performing section classification and labeling on all frame sequences included in each historical echocardiogram video in the preprocessed historical echocardiogram, wherein the section classification and labeling result is divided into A4C section or A2C section or others;

[0168] Step S704, inputting the preprocessed historical echocardiogram frame sequence and the section classification annotation results into the classification model framework, and outputting the trained classification model and section classification results;

[0169] Specifically, the preprocessed historical echocardiogram frame sequence is randomly divided into a training set, a validation set and a test set in a ratio of 14:3:3, the training set and the section classification annotation results are input into the classification model framework, the classification model is trained, the trained classification model is obtained, the trained classification model is fine-tuned using the validation set, the model training is completed, the fine-tuned classification model is tested based on the test set, and the model effect is evaluated. If the model evaluation result does not meet the preset requirements, the model parameters are adjusted, and the model pre-training and the model fine-tuning are repeated until the model evaluation result meets the preset requirements; the preset requirements are set as needed, for example, the average accuracy of the section classification model should exceed 90%, the error of the key frame selection model should be less than 5 frames, the left ventricular endocardial segmentation accuracy should reach dice greater than 0.92, and the left ventricular key point pixel error should be less than 3 pixels.

[0170] The preset classification model includes multiple 3D encoders, which are used to extract features from one or more frame sequences in each real-time echocardiography video, and can capture the time series information in the real-time echocardiography video, so as to better understand the video content, and have stronger robustness than the image classifier. In the echocardiogram obtained in the prior art, a small part of the frame image quality may be poor. It is difficult to judge the image quality using the traditional image model, and feature extraction will be performed on the frame sequence with poor image quality. The extracted features are not accurate enough, resulting in inaccurate section classification results. Our technology uses a classification model with a 3D encoder, which is not easily affected by a small number of frames with poor quality.

[0171] Step S705, labeling the left ventricular endocardium of the historical echocardiogram frame sequence whose section classification and labeling results are A4C section or A2C section, and obtaining the left ventricular endocardium labeling result;

[0172] Specifically, the left ventricular endocardium annotation result is to mark whether each frame image in the historical echocardiogram frame sequence of the A4C section or the A2C section is the left ventricular endocardium region, for example, black is the background and white is the left ventricular endocardium region.

[0173] Step S706, based on the left ventricular endocardium labeling result, generate multiple key points on the left ventricular endocardium contour and obtain the key point labeling result;

[0174] In each frame of the historical echocardiogram frame sequence where the section classification and labeling result is A4C section or A2C section, there is a left ventricular endocardial contour. A number of key points are equidistantly extracted from the left ventricular endocardial contour. The key points can be generated by a program. This is a prior art. The key points are generated using the existing findContours function, and their coordinates are recorded in sequence. In this embodiment, the specific sequence is: let the coordinates of the several points be x i ,y i (i=1,2,3,…), first according to y i The distribution threshold t of y i For all points greater than t, follow x i The coordinates are arranged in order from small to large, that is, (x1, y1, x2, y2, ...). Next, for y i All points less than t are also "according to x i The coordinates are sorted from small to large, and the results are concatenated, that is, (x1, y1), (x2, y2), …, (x k+1 , y k+1 , ) (x k+2 , y k+2 , )…), record it as the key point labeling result;

[0175] Step S707, input the historical echocardiogram frame sequence with the section classification and annotation results as A4C section or A2C section, the left ventricular endocardium annotation results, and the key point annotation results into the left ventricular endocardium segmentation model framework, and output the trained left ventricular endocardium segmentation model.

[0176] The left ventricular endocardium segmentation model framework includes an encoder, a pooling layer, a first spatial attention layer, a first channel attention layer, a decoder, a first convolutional layer, a second spatial attention layer, a second channel attention layer, and a second convolutional layer. The historical echocardiogram frame sequence, the left ventricular endocardium annotation result, and the key point annotation result of the section classification annotation result as A4C section or A2C section are input into the left ventricular endocardium segmentation model framework, and the trained left ventricular endocardium segmentation model is output, such as Figure 8 As shown, specifically including:

[0177] The historical echocardiogram corresponding to the historical echocardiogram frame sequence with the section classification and annotation result as A4C section or A2C section is used as the historical key video.

[0178] Encoding the kth frame of the historical key video by the encoder to obtain a historical key frame vector;

[0179] Performing pooling processing on the historical key frame vectors through the pooling layer to obtain the fourth eigenvector;

[0180] Processing the fourth eigenvector through the first spatial attention layer to obtain a second spatial eigenvector for characterizing spatial features;

[0181] Processing the fourth feature vector through the first channel attention layer to obtain a second channel feature vector for characterizing channel features;

[0182] Processing the fourth eigenvector through the first spatial attention layer to obtain a third spatial eigenvector for characterizing spatial features;

[0183] Processing the fourth feature vector through the first channel attention layer to obtain a third channel feature vector for characterizing channel features;

[0184] Merging the second spatial feature vector and the second channel feature vector into the fifth feature vector;

[0185] Merging the third spatial feature vector and the third channel feature vector into the sixth feature vector;

[0186] Processing the fifth eigenvector and the sixth eigenvector through a decoder to obtain the seventh and eighth eigenvectors;

[0187] Processing the seventh eigenvector through the first convolutional layer to obtain a left ventricular endocardium segmentation result;

[0188] Processing the eighth eigenvector through a second convolutional layer to obtain a left ventricular key point heat map;

[0189] Processing the seventh eigenvector through the first convolutional layer to obtain a left ventricular endocardium segmentation result;

[0190] The left ventricular endocardium segmentation results and the left ventricular key point heat map are compared with the true value, the loss function is calculated, and the model is optimized based on the loss function to obtain the trained left ventricular endocardium segmentation model.

[0191] In this embodiment, the advantages of the left ventricular endocardium segmentation model are: the auxiliary task of key point detection is introduced, so that the left ventricular endocardium segmentation model can detect several key points on the left ventricular endocardium contour while completing the left ventricular endocardium segmentation, further improving the model's ability to capture the left ventricular geometry and enhancing the accuracy of the segmentation results.

[0192] In a possible implementation, the method further includes pre-training, such as Fig. 9 As shown, the pre-training includes the following steps:

[0193] Step S801, dividing the historical echocardiogram frame sequence with all sections classified and labeled as A4C sections or A2C sections into frame-by-frame pictures, and then randomly masking 40-60% of all pictures to save picture copies;

[0194] Step S802: input the image copy into the encoder in the left ventricular endocardium segmentation model for self-supervised learning to obtain a trained encoder of the left ventricular endocardium segmentation model.

[0195] In this embodiment, pre-training can improve the generalization ability of the model, reduce the risk of overfitting, and accelerate the training and convergence speed of the model.

[0196] like Fig. 9 As shown, the second aspect of the embodiment of the present disclosure provides a left ventricular ejection fraction prediction device based on echocardiography, comprising:

[0197] An acquisition module for acquiring a real-time echocardiogram of a target patient;

[0198] A preprocessing module, used for preprocessing the real-time echocardiogram to obtain a preprocessed real-time echocardiogram, wherein the preprocessed real-time echocardiogram includes a plurality of real-time echocardiogram videos, and each real-time echocardiogram video includes a plurality of frame sequences;

[0199] A section classification module is used to input one or more frame sequences of each real-time echocardiography video into a preset classification model to perform section classification and obtain a section classification result, where the section classification result is an A4C section or an A2C section or others;

[0200] A left ventricular endocardium segmentation module is used to use the real-time echocardiographic video corresponding to the frame sequence of the section classification result being the A4C section or the A2C section as a key video, input all frames of the key video into a preset left ventricular endocardium segmentation model, and obtain a left ventricular endocardium segmentation result sequence;

[0201] An LVEF prediction module, used for inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF respectively;

[0202] The calculation module is used to calculate the left ventricular ejection fraction based on the LVEF of the A4C section and the LVEF of the A2C section.

[0203] The system can perform fully automatic LVEF calculation quickly, accurately and repeatably based on artificial intelligence.

[0204] A third aspect of the disclosed embodiment provides a heart failure identification device, such as Fig.11 As shown, the left ventricular ejection fraction prediction device based on echocardiography also includes an LVEF classification module 1001, which is used to classify patients with heart failure into HFrEF, HFmEF or HFpEF according to the value of LVEF. For example, patients are divided into three categories according to the value of LVEF: LVEF<40%, 40%<=LVEF<50%, and LVEF>=50%, wherein LVEF<40% is HFrEF, 40%<=LVEF<50% is HFmEF, and LVEF>=50% is HFpEF. This heart failure identification device divides patients suspected of having heart failure through other examinations into three categories: HFrEF, HFmEF and HFpEF, which can help doctors diagnose and treat heart failure patients more accurately and improve treatment effects and prognosis. This also helps researchers better understand the differences in the causes, pathophysiological mechanisms, clinical manifestations, treatment plans, etc. of different types of heart failure.

[0205] A fourth aspect of an embodiment of the present disclosure provides a computer device, comprising: a memory, a processor, and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the described method.

[0206] Fig.12 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0207] like Fig.12As shown, the computer device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0208] Specifically, the processor 1101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0209] The memory 1102 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 1102 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 1102 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 1102 may be inside or outside the integrated gateway device. In a specific embodiment, the memory 1102 is a non-volatile solid-state memory. In a specific embodiment, the memory 1102 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or flash memory, or a combination of two or more of these.

[0210] The processor 1101 reads and executes the computer program instructions stored in the memory 1102 to perform the steps of the echocardiography-based left ventricular ejection fraction prediction method provided in the embodiment of the present disclosure.

[0211] In one example, the computer device may further include a transceiver 1103 and a bus 1104. Fig.12 As shown, the processor 1101, the memory 1102 and the transceiver 1103 are connected via a bus 1104 and communicate with each other.

[0212] The bus 1104 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 1104 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0213] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the steps of the provided echocardiography-based left ventricular ejection fraction prediction method.

[0214] The above-mentioned storage medium may, for example, include a memory 1102 of computer program instructions, and the above-mentioned instructions may be executed by the processor 1101 to complete the left ventricular ejection fraction prediction method based on echocardiography provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact DiscROM, CD-ROM), a tape, a floppy disk, and an optical data storage device, etc. The above-mentioned computer program may be written in any combination of one or more programming languages ​​to perform the program code of the operation of the embodiment of the present disclosure, and the programming language includes an object-oriented programming language, such as Java, C++, etc., and also includes a conventional procedural programming language, such as "C" language or a similar programming language. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on the remote computing device, or completely on the remote computing device or server.

[0215] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0216] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for predicting left ventricular ejection fraction based on echocardiography, characterized in that: include: Obtain real-time echocardiograms of target patients; Preprocessing the real-time echocardiogram to obtain a preprocessed real-time echocardiogram, wherein the preprocessed real-time echocardiogram includes a plurality of real-time echocardiogram videos, and each real-time echocardiogram video includes a plurality of frame sequences; Input one or more frame sequences in each real-time echocardiography video into a preset classification model to perform section classification and obtain a section classification result, where the section classification result is an A4C section or an A2C section or others; The real-time echocardiographic video corresponding to the frame sequence of the section classification results of A4C section and A2C section is used as a key video, and the key video is input into a preset left ventricular endocardium segmentation model to obtain a left ventricular endocardium segmentation result sequence; Inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain A4C section LVEF and A2C section LVEF respectively; The LVEF was calculated based on the LVEF of the A4C section and the LVEF of the A2C section.

2. The method according to claim 1, characterized in that The preset left ventricular endocardium segmentation model includes an encoder, a pooling layer, a first spatial attention layer, a first channel attention layer, a decoder, and a first convolutional layer, wherein the real-time echocardiogram video corresponding to the frame sequence of the A4C section and the A2C section of the section classification result is used as a key video, and the key video is input into the preset left ventricular endocardium segmentation model to obtain a left ventricular endocardium segmentation result sequence, including: Step S301, encoding the kth frame of the key video by the encoder to obtain a key frame vector; Step S302: performing pooling processing on the key frame vector through the pooling layer to obtain a first feature vector; Step S303: Process the first feature vector through a first spatial attention layer to obtain a first spatial feature vector for representing spatial features; Step S304: Process the first feature vector through a first channel attention layer to obtain a first channel feature vector for characterizing channel features; Step S305, fusing the first spatial feature vector and the first channel feature vector into a second feature vector; Step S306: Process the second feature vector through a decoder to obtain a third feature vector; Step S307, processing the third eigenvector through the first convolutional layer to obtain a first left ventricular endocardium segmentation result; Step S308, repeat steps S301-S307, process each frame sequence of the key video, obtain the first left ventricular endocardium segmentation result corresponding to each frame in the frame sequence, construct the first left ventricular endocardium segmentation result sequence corresponding to the frame sequence based on the first left ventricular endocardium segmentation result, and construct the left ventricular endocardium segmentation result sequence based on the first left ventricular endocardium segmentation result sequence.

3. The method according to claim 2, characterized in that The step of inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF respectively comprises the following steps: The left ventricular endocardium segmentation result sequence includes a plurality of first left ventricular endocardium segmentation result sequences, and the first left ventricular endocardium segmentation result sequence is divided into a first left ventricular endocardium segmentation result sequence of an A4C section and a first left ventricular endocardium segmentation result sequence of an A2C section according to the frame sequence section classification result; Input the first left ventricular endocardium segmentation result sequence of the A4C section into the preset LVEF prediction model to obtain i first LVEF values, calculate the average of all the first LVEF values ​​to obtain the A4C section LVEF, where i is the number of the first left ventricular endocardium segmentation result sequence of the A4C section; The first left ventricular endocardium segmentation result sequence of the A2C section is input into the preset LVEF prediction model to obtain j second LVEF values, and all the second LVEF values ​​are averaged to obtain the A2C section LVEF, where j is the number of the first left ventricular endocardium segmentation result sequence of the A2C section.

4. The method according to claim 3, characterized in that The preset classification model includes multiple 3D encoders, feature fusion layers, and fully connected layers. The step of inputting one or more frame sequences in each real-time echocardiography video into the preset classification model to perform section classification and obtain the section classification result includes the following steps: Dividing the one or more frame sequences into first, second, and third frame sequences; Input the first, second and third frame sequences into the 3D encoder respectively to obtain the first frame sequence feature, the second frame sequence feature and the third frame sequence feature; Input the first, second and third frame sequence features into the feature fusion layer for fusion, and output the fused frame sequence features; The fused frame sequence features are input into the fully connected layer and the slice classification results are output.

5. The method according to claim 4, characterized in that The pre-processing comprises the following steps: Screening the real-time echocardiogram to obtain all two-dimensional grayscale echocardiogram (dcm) videos; Converting the two-dimensional grayscale echocardiogram (DCM) video into a target format to obtain a first echocardiogram video; Cutting out an ultrasound image sector region from the first ultrasound cardiography video to obtain a second ultrasound cardiography video; dividing the second echocardiographic video into one or more frame sequences as a real-time echocardiographic video; The collection of all real-time echocardiogram videos is used as the preprocessed real-time echocardiogram.

6. The method according to claim 1, characterized in that It also includes super-resolution processing of real-time echocardiographic video, the steps are as follows: Step S601: input the real-time echocardiogram video into a preset super-resolution model, wherein the preset super-resolution model includes: a third spatial attention layer, a spatial feature extraction layer, a temporal feature extraction layer, and a feature fusion layer; Step S602: input the nth frame of the mth frame sequence in the real-time echocardiogram video into the third spatial attention layer to obtain the first spatial attention feature F n ,n≥3; Step S603: The first spatial attention feature F n Input spatial feature extraction layer to obtain the first intermediate attention feature F n ′; Step S604: Use the same method to obtain the first intermediate attention feature F corresponding to the n-1th frame n-1 ′, the first intermediate attention feature F corresponding to the n-2th frame n-2 ′, F n-1 ′、F n-2 ′ Input feature fusion layer to get the second intermediate attention feature F s ; Step S605: The first intermediate attention feature F n ′ and the second intermediate attention feature F s Adding them to obtain a first supplementary frame, adding the first supplementary frame to the nth frame to obtain a first intermediate frame; Step S606, repeat the above steps to supplement each frame in the m-th frame sequence to obtain a first intermediate frame sequence; Step S607: Input the mth frame sequence into the temporal attention layer to obtain the temporal attention feature Km; Step S608: Add the temporal attention feature Km to the first intermediate frame sequence to obtain a predicted frame sequence; Step S609, repeating steps S601-S608, acquiring all predicted frame sequences, and using all predicted frame sequences as real-time echocardiography videos after super-resolution processing.

7. The method according to claim 1, characterized in that Before acquiring an echocardiogram of the target patient, the method further includes: Obtain historical echocardiograms of multiple patients; Preprocessing the historical echocardiograms of the multiple patients to obtain preprocessed historical echocardiograms, wherein the preprocessed historical echocardiograms include multiple historical echocardiogram videos, and each historical echocardiogram video includes multiple historical echocardiogram frame sequences; Performing slice classification and labeling on the historical echocardiogram frame sequence, wherein the slice classification and labeling result is divided into A4C slice or A2C slice or others; Input historical echocardiogram frame sequences and slice classification annotation results into the classification model framework, and output the trained classification model and slice classification results; For the historical echocardiographic frame sequence whose section classification and annotation results are A4C section or A2C section, left ventricular endocardium annotation is performed to obtain left ventricular endocardium annotation results; Based on the left ventricular endocardium annotation result, multiple key points are generated on the left ventricular endocardium contour to obtain the key point annotation result; The historical echocardiographic frame sequence of the A4C section and the A2C section, the left ventricular endocardium annotation results, and the key point annotation results are input into the left ventricular endocardium segmentation model framework, and the trained left ventricular endocardium segmentation model is output.

8. The method according to claim 7, characterized in that It also includes pre-training, and the pre-training includes the following steps: The historical echocardiographic frame sequences with all sections classified and labeled as A4C and A2C sections are divided into frame-by-frame images, and then 40-60% of all the images are randomly masked to save the image copies; The image copy is input into an encoder in the left ventricular endocardium segmentation model framework for self-supervised learning to obtain a trained encoder of the left ventricular endocardium segmentation model.

9. A left ventricular ejection fraction prediction device based on echocardiography, comprising: An acquisition module for acquiring a real-time echocardiogram of a target patient; A preprocessing module, used for preprocessing the real-time echocardiogram to obtain a preprocessed real-time echocardiogram, wherein the preprocessed real-time echocardiogram includes a plurality of real-time echocardiogram videos, and each real-time echocardiogram video includes a plurality of frame sequences; A section classification module is used to input one or more frame sequences of each real-time echocardiography video into a preset classification model to perform section classification and obtain a section classification result, where the section classification result is an A4C section or an A2C section or others; A left ventricular endocardium segmentation module is used to use the real-time echocardiographic video corresponding to the frame sequence of the section classification results A4C section and A2C section as a key video, input the key video into a preset left ventricular endocardium segmentation model, and obtain a left ventricular endocardium segmentation result sequence; An LVEF prediction module, used for inputting the left ventricular endocardium segmentation result sequence into a preset LVEF prediction model to obtain the A4C section LVEF and the A2C section LVEF respectively; The calculation module is used to calculate the LVEF based on the A4C section LVEF and the A2C section LVEF.

10. A heart failure identification device, comprising the left ventricular ejection fraction prediction device based on echocardiography as described in claim 9, and also comprising an LVEF classification model for classifying patients with heart failure into HFrEF, HFmEF or HFpEF according to the value of LVEF.

11. A computer device, characterized in that: include: A memory, a processor, and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images