A cardiac phase identification method based on dynamic echocardiography

Abstract

The present invention discloses a cardiac phase recognition method based on dynamic echocardiography, comprising: generating data labels that satisfy a Gaussian probability distribution as targets for learning a cardiac phase recognition model; obtaining a dynamic echocardiography video sequence decoded from a Dicom file, and performing preprocessing and data enhancement on the video sequence; sampling the data-enhanced video sequence using a sliding window, thereby sampling the video sequence into a plurality of overlapping video segments; constructing a cardiac phase recognition model; training the cardiac phase recognition model; and using the trained cardiac phase recognition model to recognize the dynamic echocardiography video to be recognized, obtaining the probability that each frame in the video is a critical cardiac phase (ED) or (ES), and averaging multiple probability values ​​obtained from repeatedly sampled frames as the final probability for the frame. The present invention improves the accuracy and robustness of the model in detecting cardiac phase frames.

Description

Technical Field

[0001] The present invention relates to an echocardiogram phase recognition technology, and in particular to a cardiac phase recognition method based on dynamic echocardiography. Background Art

[0002] Echocardiography (ECHO) is a non-invasive and safe medical imaging modality that can accurately identify cardiac structure and is widely used in the diagnosis and treatment of patients with heart disease. During echocardiographic examinations, cardiac phase detection, namely the detection of the end-diastole (ED) and end-systole (ES) frames, is a crucial task. Quantifying cardiac size and evaluating cardiac function require this detection. The ED frame is defined as the first frame after mitral valve (MV) closure, or the freeze frame with the largest left ventricular diameter or volume during the cardiac cycle, or the frame corresponding to the R wave peak in the electrocardiogram (ECG). The ES frame is defined as the first frame after aortic valve (AV) closure, or the freeze frame with the smallest left ventricular diameter or volume during the cardiac cycle, or the frame corresponding to the end of the T wave in the ECG. Relying solely on ultrasound physicians to identify key frames using the R wave and T wave in ECG or visually observing the size of the left ventricle is labor-intensive, time-consuming, highly dependent on operator experience, and has poor repeatability. Taking these factors into consideration, it is necessary to explore an automatic and more accurate method to detect the cardiac phase of echocardiography. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a cardiac phase recognition method based on dynamic echocardiography, which can be applied to common sections of echocardiography and has strong applicability.

[0004] Technical solution: A cardiac phase identification method based on dynamic echocardiography of the present invention comprises the following steps:

[0005] S1. Automatically generate data labels representing the probability of cardiac key phases for each frame in the echocardiographic video. Given the frame index of the cardiac key phase of each video, generate data labels that satisfy the Gaussian probability distribution around the cardiac key phases ED and ES frame indexes as the target of the cardiac phase recognition model learning.

[0006] S2, obtain the dynamic echocardiography video sequence after decoding the Dicom file, and preprocess the video sequence to remove invalid information on the image; then perform data enhancement;

[0007] S3, using a sliding window to sample the data-enhanced video sequence, thereby sampling the video sequence into several overlapping video segments;

[0008] S4. Construct a cardiac phase recognition model. The cardiac phase recognition model includes a spatial feature extractor, a spatiotemporal feature aggregation module, and a regression head. The spatial feature extractor is CSPVoVNet based on CNN. The core of the CSPVoVNet network is to improve the traditional CNN model by introducing channel compensation technology and Cross Stage Partial technology. CSPVoVNet performs channel compensation by splicing in the feature channel dimension, and then reduces the dimension of the spliced ​​feature map through a 1*1 convolutional layer. The spatiotemporal feature aggregation module includes a window self-attention structure and a moving window self-attention structure. The window self-attention structure is used to calculate the attention within an independent window, and the moving window self-attention structure is used to calculate the attention between different windows.

[0009] The spatial feature extractor is used to encode the spatial features of each frame image in the video segment to obtain the spatial feature sequence of the video segment; the spatiotemporal feature aggregation module is used to perform time-series attention calculation on the spatial feature sequence to obtain the spatiotemporal feature sequence; the regression head is used to map the spatiotemporal feature sequence into the probability that each frame image in the video segment is ED or ES;

[0010] S5. Training the cardiac phase recognition model in step S4 using the multiple overlapping video segments obtained in step S3; the method comprising: inputting the video segments obtained in step S3 into the cardiac phase recognition model, calculating the probability of each frame being a critical cardiac phase ED or ES, and then calculating the mean square error with the data labels satisfying the Gaussian probability distribution generated in step S1, performing gradient backpropagation, and updating the cardiac phase recognition model parameters using a stochastic gradient descent algorithm, thereby training the cardiac phase recognition model;

[0011] S6. Use the trained cardiac phase recognition model to identify the dynamic echocardiography video to be identified, obtain the probability that each frame in the video is the critical cardiac phase ED or ES, and average the multiple probability values ​​obtained from repeatedly sampled frames as the final probability of the frame.

[0012] Furthermore, preprocessing the video sequence in step S2 includes:

[0013] Perform threshold segmentation on the image to segment the foreground of the image;

[0014] Perform an open-then-close morphological operation on the image to remove small noise points in the image;

[0015] Perform Canny edge detection on the image to obtain the edge of the image;

[0016] Perform Hough line detection on the image to obtain the two straight lines that make up the fan-shaped area;

[0017] After calculating the intersection of the two straight lines, the radius of the sector area can be calculated, so that the sector area in the echocardiogram can be located;

[0018] All pixels outside the fan-shaped area of ​​each frame in the dynamic echocardiogram video are set to 0, retaining only the effective information of the echocardiogram;

[0019] Furthermore, data enhancement includes spatial dimension enhancement and temporal dimension enhancement; spatial dimension enhancement includes: random scaling, random center cropping, and random rotation; temporal dimension enhancement is to randomly set the sampling step size of the sliding window sampling of the entire video sequence to 1, 2, or 4.

[0020] Furthermore, the core structure of CSPVoVNet Block in step S4 includes two stages. The first stage is: if the input tensor is Then use a 3*3 convolutional layer to extract features and get the feature map Then extract features through a 3*3 convolution layer to obtain the feature map Then the feature map T and As channel compensation, with Do channel splicing and get Will Input into a 1*1 convolution layer with 3C input channels and C output channels to perform feature dimensionality reduction and obtain T1; the second stage is: T1 is passed through two 3*3 convolution layers to obtain and After that, the feature maps T, T1 and As channel compensation and Perform channel splicing to obtain Then The input is sent to a 1*1 convolutional layer with 4C input channels and C output channels for feature dimensionality reduction, and T2 is obtained, which is the output spatial feature sequence.

[0021] Furthermore, in step S4, the window self-attention structure and the moving window self-attention structure appear in pairs. The window self-attention structure includes a layer normalization LN structure, a 3D W-MSA structure and an MLP structure, and the moving window self-attention structure includes an LN structure, a 3D SW-MSA structure and an MLP structure. In the window self-attention structure, the input spatial feature sequence will first pass through the LN structure and then be input into the 3D W-MSA structure. The spatial window feature sequence after calculating the attention is added to the spatial feature sequence to form a residual connection. The spatial window feature sequence after the residual connection is then sequentially input into the LN module and the MLP module to further increase the nonlinearity of the feature map, and then added to the spatial window feature sequence after the residual connection to form a residual connection. The output feature sequence of the window self-attention structure is input into the moving window self-attention structure. The process of the moving window self-attention structure is consistent with that of the window self-attention structure.

[0022] Furthermore, in step S4, attention calculation in the temporal direction is performed on the spatial feature sequence to obtain the spatiotemporal feature sequence, including:

[0023] After dividing the spatial feature sequence into several windows, it is input into the window self-attention structure to calculate the self-attention. The formula is as follows:

[0024]

[0025] Among them, Attn window The window attention matrix calculated for each window, Softmax() is the mapping function, Q represents the Query matrix, K represents the Key matrix, V represents the Value matrix, d k Represents the characteristic dimension of the Key matrix;

[0026] The window of the spatial feature sequence is shifted and then input into the moving window self-attention structure to calculate the self-attention. The calculation process is consistent with that of the window self-attention structure, and the feature obtained is the spatiotemporal feature sequence.

[0027] Furthermore, in step S4, the temporal and spatial feature sequences are mapped into the probability that each frame image in the video segment is ED and ES, including:

[0028] The following formula is performed on the spatiotemporal feature sequence to calculate the probability that each frame of the video segment is a key cardiac phase;

[0029] Prob=Softmax(Wf out +b)

[0030] Among them, f out It is the spatiotemporal feature sequence calculated by the spatiotemporal feature aggregation module. fout Mapping matrix, f out Bias vector, The probability that each frame in the 16 input images is a key phase.

[0031] Based on the same inventive concept, the present invention provides a cardiac phase identification system based on dynamic echocardiography, comprising:

[0032] A data label generation module is used to automatically generate a data label representing the probability of a key cardiac phase for each frame in the echocardiography video. After given the frame index of the key cardiac phase of each video, a data label satisfying a Gaussian probability distribution is generated around the frame indexes of the key cardiac phases ED and ES as the target of the cardiac phase recognition model learning;

[0033] The data processing module is used to obtain the dynamic echocardiography video sequence after decoding the Dicom file, preprocess the video sequence, remove invalid information on the image, and then perform data enhancement;

[0034] A sampling module is used to sample the data-enhanced video sequence using a sliding window, thereby sampling the video sequence into a number of overlapping video segments;

[0035] The model building module is used to build a cardiac phase recognition model. The cardiac phase recognition model includes a spatial feature extractor, a spatiotemporal feature aggregation module, and a regression head. The spatial feature extractor is the CNN-based CSPVoVNet. The core of the CSPVoVNet network is to improve the traditional CNN model by introducing channel compensation technology and Cross Stage Partial technology. CSPVoVNet performs channel compensation by splicing in the feature channel dimension, and then reduces the dimension of the spliced ​​feature map through a 1*1 convolutional layer. The spatiotemporal feature aggregation module includes a window self-attention structure and a moving window self-attention structure. The window self-attention structure is used to calculate the attention within an independent window, and the moving window self-attention structure is used to calculate the attention between different windows.

[0036] The spatial feature extractor is used to encode the spatial features of each frame image in the video segment to obtain the spatial feature sequence of the video segment; the spatiotemporal feature aggregation module is used to perform time-series attention calculation on the spatial feature sequence to obtain the spatiotemporal feature sequence; the regression head is used to map the spatiotemporal feature sequence into the probability that each frame image in the video segment is ED or ES;

[0037] A model training module is used to train a cardiac phase recognition model using the obtained multiple overlapping video segments. The module includes: inputting the obtained video segments into the cardiac phase recognition model, calculating the probability of each frame being the critical cardiac phase ED or ES, calculating the mean square error with the generated data labels that satisfy the Gaussian probability distribution, performing gradient backpropagation, and updating the cardiac phase recognition model parameters using a stochastic gradient descent algorithm, thereby achieving training of the cardiac phase recognition model.

[0038] The recognition module is used to use the trained cardiac phase recognition model to identify the dynamic echocardiography video to be identified, obtain the probability that each frame in the video is the critical cardiac phase ED or ES, and average the multiple probability values ​​obtained from repeatedly sampled frames as the final probability of the frame.

[0039] Based on the same inventive concept, an electronic device of the present invention includes a memory and a processor, wherein:

[0040] a memory for storing computer programs capable of running on the processor;

[0041] The processor is configured to execute the steps of the above-mentioned cardiac phase identification method based on dynamic echocardiography when running the computer program.

[0042] Based on the same inventive concept, the present invention provides a storage medium having a computer program stored thereon, which, when executed by at least one processor, implements the steps of the above-mentioned cardiac phase identification method based on dynamic echocardiography.

[0043] Beneficial effects: Compared with the prior art, the present invention has the following significant technical effects:

[0044] (1) The present invention first obtains the video sequence after video decoding and preprocesses each frame to remove the text and electrocardiogram information outside the echocardiogram sector area, which can greatly reduce the interference of irrelevant information on the model, thereby improving the accuracy of the model;

[0045] (2) The sliding window-based overlapping video segment sampling method proposed in this paper first uses a CNN model to extract spatial features for each frame, and then uses the spatial features of the video segment to perform temporal modeling. This design can effectively reduce the amount of computation while ensuring effective modeling of spatial and temporal features, and can meet the needs of real-time online inference of video streams.

[0046] (3) The sliding window-based probability post-processing method proposed in the present invention can effectively normalize the probability that each frame of the video is a key cardiac phase, thereby avoiding the problem of the probability curve being unable to distinguish the maximum value when inferring an echocardiographic video containing multiple cardiac cycles due to segmented sampling, so that the technical solution can meet the needs of inferring a video containing multiple cardiac cycles.

[0047] (4) The method of the present invention does not require left ventricular segmentation nor rely on electrocardiograms. Instead, it combines a convolutional neural network with a spatiotemporal feature aggregation module to automatically and accurately identify ED and ES frames across multiple cardiac cycles from a two-dimensional echocardiographic sequence of arbitrary length. This method can be applied to common echocardiographic sections and has strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall operation flow of the present invention;

[0049] Figure 2 This is an example of generating data labels indicating that each frame in the video is ED and ES;

[0050] Figure 3 This is a schematic diagram of the structure of CSPVoVNet Block, the core component of the convolutional neural network model used;

[0051] Figure 4 This is a structural diagram of the Video Swin Transformer Block, which is a component of the spatiotemporal feature aggregation module. The left half is the window self-attention structure, and the right half is the moving window self-attention structure;

[0052] Figure 5 It is a schematic diagram of the sliding window post-processing process;

[0053] Figure 6 This is an example of visualizing the prediction results. DETAILED DESCRIPTION

[0054] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] The present invention provides a cardiac phase recognition method based on dynamic echocardiography. The method is applied to a dynamic echocardiography video with multiple cardiac cycles, which can automatically identify the end diastole (ED) and end systole (SD) frames in the video and generate their specific frame indexes. Figure 1 As shown, the specific steps include:

[0056] S1. Automatically generate data labels representing the probability of key cardiac phases for each frame in the echocardiogram video. The results are as follows: Figure 2 Given the frame index of the cardiac key phase in a video, a Gaussian distribution probability density function is generated with the frame index as μ and σ as 10. The formula is shown in Equation (1), where x is the frame index of any frame, μ is the frame index of the ED or ES frame, p(x) is the probability of each frame being an ED or ES frame, and σ is a constant.

[0057]

[0058] The probability of the ED frame of the current cardiac cycle and the ED frame of the next adjacent cardiac cycle is set to 1.0, and the probability of the frames between the two ED frames is interpolated based on the Gaussian probability density function to obtain a probability curve representing that each frame of the cardiac cycle is in the ED phase. The ES probability curve generation rule for this cardiac cycle is the same as that for ED. Assuming that the ED frame index of the current video marker is 20 and the ES frame index is 43, according to the automatic generation rule of the data label described previously, the target probability curves of the key cardiac phases ED and ES are generated respectively, which are data labels, such as Figure 2 As shown in Figure 2, the target of cardiac phase recognition model learning is the Ground Truth.

[0059] S2. Obtain the dynamic echocardiography video sequence after decoding the Dicom file, and preprocess the video sequence to remove invalid information such as the electrocardiogram and scale on the image; then perform data enhancement;

[0060] The preprocessing steps include:

[0061] Perform threshold segmentation on the image to segment the foreground of the image;

[0062] Perform an open-then-close morphological operation on the image to remove small noise points in the image;

[0063] Perform Canny edge detection on the image to obtain the edge of the image;

[0064] Perform Hough line detection on the image to obtain the two straight lines that make up the fan-shaped area;

[0065] After calculating the intersection of the two straight lines, the radius of the sector area can be calculated, so that the sector area in the echocardiogram can be located;

[0066] All pixels outside the fan-shaped area of ​​each frame in the dynamic echocardiogram video are set to 0, and only the effective information of the echocardiogram is retained.

[0067] Data augmentation is performed on echocardiography video sequences, including spatial and temporal dimension augmentation. Spatial dimension augmentation includes random scaling, random center cropping, and random rotation; temporal dimension augmentation involves randomly setting the sampling step size of the sliding window sampling of the entire video sequence to 1, 2, or 4.

[0068] S3, using the same sliding window with a certain length and step size as in step S2 to sample the preprocessed video sequence, thereby sampling the video sequence into a plurality of overlapping video segments;

[0069] S4. Construct a cardiac phase recognition model. The cardiac phase recognition model includes a spatial feature extractor, a spatiotemporal feature aggregation module, and a regression head. The spatial feature extractor is CSPVoVNet based on CNN. The core of the CSPVoVNet network is to improve the traditional CNN model by introducing channel compensation technology and Cross Stage Partial technology. CSPVoVNet performs channel compensation by splicing in the feature channel dimension, and then reduces the dimension of the spliced ​​feature map through a 1*1 convolutional layer. The spatiotemporal feature aggregation module includes a window self-attention structure and a moving window self-attention structure. The window self-attention structure is used to calculate the attention within an independent window, and the moving window self-attention structure is used to calculate the attention between different windows.

[0070] The spatial feature extractor is used to encode the spatial features of each frame image in the video segment to obtain the spatial feature sequence of the video segment; the spatiotemporal feature aggregation module is used to perform time-series attention calculation on the spatial feature sequence to obtain the spatiotemporal feature sequence; the regression head is used to map the spatiotemporal feature sequence into the probability that each frame image in the video segment is ED or ES;

[0071] The CSPVoVNet network is used to encode the spatial features of each frame of the video segment to obtain the spatial feature sequence of the video segment; specifically, the following steps are involved:

[0072] For each frame of the video segment, a 2D convolutional neural network (CNN) CSPVoVNet is used to extract spatial features, and the spatial feature sequence of the video segment can be obtained.

[0073] The core structure of CSPVoVNet, CSPVoVNet Block, includes channel compensation technology and Cross Stage Partial technology. The specific operation steps can be seen in Figure 3 .

[0074] The core structure of CSPVoVNet CSPVoVNet Block consists of two stages. The first stage is: if the input tensor is Then use a 3*3 convolutional layer to extract features and get the feature map Then extract features through a 3*3 convolution layer to obtain the feature map Then the feature map t and As channel compensation, with Do channel splicing and get Will Input into a 1*1 convolution layer with 3C input channels and c output channels to perform feature dimensionality reduction and obtain T1; the second stage is: T1 is passed through two 3*3 convolution layers to obtain and After that, the feature maps T, T1 and As channel compensation and Perform channel splicing to obtain Then The input is sent to a 1*1 convolutional layer with 4C input channels and C output channels for feature dimensionality reduction, and T2 is obtained, which is the output spatial feature sequence.

[0075] Perform temporal attention calculation on the spatial feature sequence to obtain the spatiotemporal feature sequence; extracting spatial features and temporal features separately can reduce the amount of calculation. The steps include:

[0076] After dividing the spatial feature sequence into several windows, it is input into the window self-attention structure to calculate the self-attention. The formula is as follows.

[0077]

[0078] Among them, Attn window The window attention matrix calculated for each window, is a mapping function used to map the calculation results in the brackets to the range of 0 to 1; Q represents the Query matrix, K represents the Key matrix, V represents the Value matrix, d k Represents the characteristic dimension of the Key matrix;

[0079] The window of the spatial feature sequence is shifted and then input into the moving window self-attention structure to calculate the self-attention. The calculation process is consistent with the self-attention calculation method in the window self-attention structure, and the obtained feature is the spatiotemporal feature sequence.

[0080] The probability of each frame in the video segment being a critical cardiac phase is output through the regression head structure for the spatiotemporal feature sequence;

[0081] The following formula is used to operate the spatiotemporal feature sequence to calculate the probability that each frame of the video segment is a key cardiac phase. outIt is the spatiotemporal feature sequence calculated by the spatiotemporal feature aggregation module. f out Mapping matrix, f out Bias vector, The predicted probability of ED and ES (i.e., key phase) for each of the 16 input images.

[0082] Prob=Softmax(Wf out +b) (3)

[0083] After calculating the predicted probability of ED and ES for each frame, the mean squared error (MSE) is calculated with the previously described data labels, and gradient backpropagation is performed. The parameters of the cardiac phase recognition model are updated using the stochastic gradient descent algorithm, thereby achieving training of the cardiac phase recognition model.

[0084] S6. Using the trained cardiac phase recognition model to identify the dynamic echocardiogram video to be identified, the probability of each frame in the video being the critical cardiac phase ED or ES is obtained. The multiple probabilities obtained from repeatedly sampled frames in the video are averaged to obtain the final probability of each frame being the critical cardiac phase, thereby achieving cardiac phase identification. The steps include:

[0085] When the sampling step size is smaller than the video segment length, some frame images will be repeatedly sampled, that is, multiple phase probability values ​​will be calculated. Therefore, the multiple probability values ​​are averaged as the final probability of the frame.

[0086] The specific implementation plan includes: automatic generation of data labels and data enhancement methods, the structure of the cardiac phase recognition model, and the sliding window probability post-processing method.

[0087] Example 1:

[0088] The automatic generation of data labels and data enhancement methods include the following: Figure 2 Given the frame index of the cardiac key phase in a video, a Gaussian distribution probability density function is generated with the frame index as μ and σ as 10. The formula is shown in Equation (1), where x is the frame index of any frame, μ is the frame index of the ED or ES frame, p(x) is the probability of each frame being an ED or ES frame, and σ is a constant.

[0089] The probability of the ED frame of the current cardiac cycle and the ED frame of the next adjacent cardiac cycle is set to 1.0, and the probability of the frames between the two ED frames is interpolated based on the Gaussian probability density function to obtain the probability curve representing that each frame of the cardiac cycle is in the ED phase. The ES probability curve generation rule for this cardiac cycle is the same as that for ED. Assuming that the ED frame index of the current video marker is 20 and the ES frame index is 43, according to the automatic generation rule of the data label described previously, the target probability curves of ED and ES are generated respectively, which are the data labels, such as Figure 2 As shown in Figure 3, the target of cardiac phase recognition model learning is GroundTruth.

[0090] Data augmentation methods are categorized into spatial and temporal dimension augmentation. Spatial augmentation includes random scaling, random center cropping, and random rotation. Temporal augmentation involves randomly setting the sampling step size of the sliding window described in S2 to 1, 2, or 4, sampling ultrasound sequences at equal intervals of 16 frames to enrich the dataset. If the last video segment sampled by the sliding window is less than 16 frames long, the last frame of the video is used to pad the segment until the length reaches 16 frames.

[0091] Example 2:

[0092] The cardiac phase recognition model consists of three parts: a spatial feature extractor, a spatiotemporal feature aggregation module, and a regression head. Its construction is mainly divided into the following steps:

[0093] In the first step, a spatial feature extractor based on a convolutional neural network (CNN) is used to independently extract the spatial features of each frame image in a 16-frame video segment obtained by random step sampling in the temporal dimension enhancement method mentioned in the above data enhancement method. That is, the input of the spatial feature extractor structure is an RGB image with a length of 16 frames and a width and height of 320 pixels, and the output is a spatial feature sequence of 16 frames, a width and height of 10, and a feature dimension of 2048. The spatial feature extractor used in the present invention is a CNN-based CSPVoVNet, and its core component structure CSPVoVNet Block is as follows: Figure 3As shown. The core of the CSPVoVNet network is to improve the traditional CNN model by introducing channel compensation and Cross Stage Partial (CSP) technology, so that it can achieve similar or higher accuracy than similar CNN models with less computation and fewer memory accesses. Compared with the commonly used CNN model ResNet, CSPVoVNet performs channel compensation by concatenating the feature channel dimension, which is more conducive to the CSPVoVNet model to learn feature information of multiple different receptive fields, and then reduces the dimension of the concatenated feature map through a 1*1 convolution layer, which can effectively compress the features, organize the features under different receptive fields, and remove redundant features. Compared with the DenseNet model, which also uses splicing as a channel compensation method, DenseNet's channel compensation is too frequent, resulting in too many redundant features, introducing a large amount of computation, and frequent channel compensation will inevitably cause frequent memory access, so it is not friendly to end-side deployment. CSPVoVNet splices channels through one-shot fusion, rather than splicing the channel dimension with the previous feature map after each convolutional layer structure, which greatly reduces the amount of computation and feature redundancy.

[0094] For CSPVoVNet Block, the reasoning process is divided into two stages. The first stage is: if the input tensor is Then use a 3*3 convolutional layer to extract features and get the feature map Then extract features through a 3*3 convolution layer to obtain the feature map Then the feature map T and As channel compensation, with Do channel splicing and get Will Input into a 1*1 convolution layer with 3C input channels and C output channels to perform feature dimensionality reduction and obtain T1; the second stage is: T1 is passed through two 3*3 convolution layers to obtain and After that, the feature maps T, T1 and As channel compensation and Perform channel splicing to obtain Then The input is sent to a 1*1 convolutional layer with 4C input channels and C output channels for feature dimensionality reduction, and T2 is obtained, which is the output spatial feature sequence.

[0095] In the second step, the present invention innovatively adds a spatiotemporal feature aggregation module after the convolutional neural network CSPVoVNet for modeling and fusing temporal features and spatial features. Different from the proposed cardiac phase recognition method, modeling temporal features can greatly improve the accuracy of cardiac phase recognition. The input of the spatiotemporal feature aggregation module is the spatial feature sequence of 16 frames of images obtained in the first step. The size of the spatial feature map of each frame of image is 2048*10*10, where 10*10 is the number of pixels of width and height of the feature map, and 2048 is the feature dimension of the image. The output is a feature map that aggregates temporal features and spatial features, with a size of 16*2048*10*10. The spatiotemporal feature aggregation module consists of a Video Swin Transformer Block (hereinafter referred to as VST Block), which is used to model 16 independent spatial feature maps in the temporal dimension, so that the spatial feature map can become a spatiotemporal feature sequence that carries both temporal features and spatial features. The main principle of VST Block is to use the self-attention mechanism to model the temporal information of the ultrasound sequence, thereby modeling the temporal correlation information between spatial feature sequences. The calculation is highly parallel, avoiding the problems of gradient vanishing or gradient exploding.

[0096] VST Block consists of a window self-attention structure ( Figure 4 Left half) and a moving window self-attention structure ( Figure 4 The right half of the image is composed of two layers. Generally, the window self-attention structure and the moving window self-attention structure should appear in pairs. The reason is that the former models the attention within an independent window. Although it greatly reduces the computational effort, the receptive field of the feature map is still local. In order to model the global receptive field, the moving window self-attention structure is needed to implement attention calculations between different windows, thereby expanding the local receptive field to approximately the global receptive field.

[0097] The window self-attention structure consists of a layer normalization (LN) structure, a 3D W-MSA structure, and an MLP structure. The moving window self-attention structure consists of a LN structure, a 3D SW-MSA structure, and an MLP structure. The operation flow is as follows: Figure 4 shown.

[0098] In the windowed self-attention architecture, the input spatial feature sequence first passes through the LN architecture before being input into the 3D W-MSA architecture. The 3D W-MSA architecture first partitions the spatial feature sequence into windows of size 16*2048*10*10. The spatial feature sequence is then divided into 16 independent windows of size 4*8*8, where 4 represents each window encompassing a feature sequence of 4 frames in length, and 8*8 represents a window width and height of 8 pixels. To ensure that the window size is divisible by the width and height of the original feature map, the input feature map is padded with zeros. The padded feature map has a width and height of 16*16, while the feature dimension remains at 2048. A larger window size is used for partitioning because, compared to typical VST block-based models, the input feature map size is already downsampled to only 10 pixels in length and width. Even with larger window sizes, this does not incur excessive computational overhead, opening up unlimited possibilities for real-time online operation and on-device deployment of cardiac phase recognition models. After the spatial feature sequence is partitioned into windows, the self-attention is calculated independently for each window. The calculation process of the independent self-attention calculation is shown in formula (2), where The attention feature map output for an independent window, Q, K, are Query, Key and Value matrices respectively, M 2 T is the size of the independent window, d is the feature dimension, Q, K, V are obtained by formula (4), formula (5) and formula (6) respectively. D , + K , are three transformation matrices with different weights, d in ,d out The feature dimensions of input and output are both set to 2048. is a feature map of an independent window. The spatial window feature sequence after attention calculation is added to the spatial feature sequence to form a residual connection. The spatial window feature sequence after residual connection is then input into the LN module and the MLP module in sequence to further increase the nonlinearity of the feature map. It is then added to the spatial window feature sequence after residual connection to form a residual connection.

[0099]

[0100]

[0101]

[0102] The output feature sequence of the window self-attention structure is input into the moving window self-attention structure. Its process is basically the same as that of the window self-attention module, but the self-attention calculation structure is replaced by the 3D W-MSA structure. This is because the window self-attention module calculates attention independently for each window, and its receptive field is limited to the window. Therefore, the window is translated to expand its receptive field to the adjacent window, which can achieve feature fusion and communication between different windows, thereby expanding the local receptive field to a near-global receptive field.

[0103] Step 3. The regression head structure needs to map the spatiotemporal features extracted in the second step into the probability that each frame in the video segment is ED or ES. The regression head structure consists of two fully connected layers with an input channel of 2048 and an output channel of 16. The output of the regression head structure is the probability of each frame in each video segment being predicted as an ED or ES frame obtained by each sliding window described in S2. The calculation process is shown in formula (3). Where, f out It is the spatiotemporal feature sequence calculated by the spatiotemporal feature aggregation module. f out Mapping matrix, f out Bias vector, The predicted probability of ED and ES for each of the 16 input images.

[0104] After calculating the probability of ED and ES for each frame, the mean squared error (MSE) is calculated with the previously described ground truth, i.e., the probability curve of ED and ES, as shown in the formula, where Toss represents the calculated error, n represents the total number of samples in the frame image, and y i is the model prediction probability of the current video segment, is the data label of the current video segment. After calculating the error, the gradient is backpropagated and the parameters of the cardiac phase recognition model are updated using the stochastic gradient descent algorithm, thereby achieving training of the cardiac phase recognition model.

[0105]

[0106] Example 3:

[0107] After calculating the error and optimizing the parameters of the cardiac phase recognition model, a sliding window probability post-processing is performed. The sliding window probability post-processing method mainly includes: the present invention innovatively uses a sliding window-based probability post-processing structure to accurately calculate the frame indexes of the heart ES and ED in a complete ultrasound video containing multiple cardiac cycles. The process is as follows: Figure 5As shown in Figure 2, a complete echocardiographic video is sampled with a sliding window in the S2 stage. Depending on the sliding window length and the sampling step size, some frames will be sampled multiple times, so multiple predicted probability values ​​will be obtained. Therefore, taking the average of multiple probability values ​​as the final predicted phase probability value can make the phase prediction more accurate. The probability curve can be visualized as shown in Figure 2. Figure 6 As shown in FIG, the corresponding frame index at the maximum value is the frame index of the key cardiac phase predicted by the network. The probabilistic post-processing method proposed in the present invention can find each ED and ES frame index in a dynamic echocardiographic video containing multiple cardiac cycles in a relatively simple manner.

Claims

1. A cardiac phase identification method based on dynamic echocardiography, characterized in that: The following steps are involved: S1. Automatically generate data labels representing the probability of cardiac key phases for each frame in the echocardiographic video. Given the frame index of the cardiac key phase of each video, generate data labels that satisfy the Gaussian probability distribution around the cardiac key phases ED and ES frame indexes as the target of the cardiac phase recognition model learning. S2, obtain the dynamic echocardiography video sequence after decoding the Dicom file, and preprocess the video sequence to remove invalid information on the image; then perform data enhancement; S3, using a sliding window to sample the data-enhanced video sequence, thereby sampling the video sequence into several overlapping video segments; S4. Construct a cardiac phase recognition model. The cardiac phase recognition model includes a spatial feature extractor, a spatiotemporal feature aggregation module, and a regression head. The spatial feature extractor is CSPVoVNet based on CNN. The core of the CSPVoVNet network is to improve the traditional CNN model by introducing channel compensation technology and Cross Stage Partial technology. CSPVoVNet performs channel compensation by splicing in the feature channel dimension, and then reduces the dimension of the spliced ​​feature map through a 1*1 convolutional layer. The spatiotemporal feature aggregation module includes a window self-attention structure and a moving window self-attention structure. The window self-attention structure is used to calculate the attention within an independent window, and the moving window self-attention structure is used to calculate the attention between different windows. The spatial feature extractor is used to encode the spatial features of each frame of the video segment to obtain the spatial feature sequence of the video segment; The spatiotemporal feature aggregation module is used to perform temporal attention calculation on the spatial feature sequence to obtain the spatiotemporal feature sequence; the regression head is used to map the spatiotemporal feature sequence into the probability that each frame image in the video segment is ED and ES; S5. Training the cardiac phase recognition model in step S4 using the multiple overlapping video segments obtained in step S3; the method comprising: inputting the video segments obtained in step S3 into the cardiac phase recognition model, calculating the probability of each frame being a critical cardiac phase ED or ES, and then calculating the mean square error with the data labels satisfying the Gaussian probability distribution generated in step S1, performing gradient backpropagation, and updating the cardiac phase recognition model parameters using a stochastic gradient descent algorithm, thereby training the cardiac phase recognition model; S6. Use the trained cardiac phase recognition model to identify the dynamic echocardiography video to be identified, obtain the probability that each frame in the video is the critical cardiac phase ED or ES, and average the multiple probability values ​​obtained from repeatedly sampled frames as the final probability of the frame.

2. The cardiac phase identification method based on dynamic echocardiography according to claim 1, characterized in that: The pre-processing of the video sequence in step S2 includes: Perform threshold segmentation on the image to segment the foreground of the image; Perform an open-then-close morphological operation on the image to remove small noise points in the image; Perform Canny edge detection on the image to obtain the edge of the image; Perform Hough line detection on the image to obtain the two straight lines that make up the fan-shaped area; After calculating the intersection of the two straight lines, the radius of the sector area can be calculated, so that the sector area in the echocardiogram can be located; All pixels outside the fan-shaped area of ​​each frame in the dynamic echocardiogram video are set to 0, and only the effective information of the echocardiogram is retained.

3. The cardiac phase identification method based on dynamic echocardiography according to claim 1, characterized in that: Data enhancement includes spatial dimension enhancement and temporal dimension enhancement; The spatial dimension enhancement includes: random scaling, random center cropping, and random rotation; Temporal dimension enhancement is to randomly set the sampling step size of the sliding window sampling of the entire video sequence to 1, 2 or 4.

4. The cardiac phase identification method based on dynamic echocardiography according to claim 1, characterized in that: The core structure of CSPVoVNet in step S4, CSPVoVNet Block, consists of two stages. The first stage is: if the input tensor is Then use a 3*3 convolutional layer to extract features and get feature map T1 1 ; Then extract features through a 3*3 convolution layer to obtain the feature map Then the feature map T and As channel compensation, with Do channel splicing and get Will Input into a 1*1 convolution layer with 3C input channels and C output channels to perform feature dimensionality reduction and obtain T1; the second stage is: T1 is passed through two 3*3 convolution layers to obtain and After that, the feature maps T, T1 and As channel compensation and Perform channel splicing to obtain Then The input is sent to a 1*1 convolutional layer with 4C input channels and C output channels for feature dimensionality reduction, and T2 is obtained, which is the output spatial feature sequence.

5. The cardiac phase identification method based on dynamic echocardiography according to claim 1, characterized in that: In step S4, the window self-attention structure and the moving window self-attention structure appear in pairs. The window self-attention structure includes a layer normalization LN structure, a 3D W-MSA structure, and an MLP structure. The moving window self-attention structure includes an LN structure, a 3D SW-MSA structure, and an MLP structure. In the window self-attention structure, the input spatial feature sequence will first pass through the LN structure and then be input into the 3D W-MSA structure. The spatial window feature sequence after attention calculation is added to the spatial feature sequence to form a residual connection. The spatial window feature sequence after the residual connection is then sequentially input into the LN module and the MLP module to further increase the nonlinearity of the feature map, and then added to the spatial window feature sequence after the residual connection to form a residual connection. The output feature sequence of the window self-attention structure is input into the moving window self-attention structure. The process of the moving window self-attention structure is consistent with that of the window self-attention structure.

6. The cardiac phase identification method based on dynamic echocardiography according to claim 1, characterized in that: In step S4, attention calculation in the temporal direction is performed on the spatial feature sequence to obtain the spatiotemporal feature sequence, including: After dividing the spatial feature sequence into several windows, it is input into the window self-attention structure to calculate the self-attention. The formula is as follows: Among them, Attn window The window attention matrix calculated for each window, Softmax() is the mapping function, Q represents the Query matrix, K represents the Key matrix, V represents the Value matrix, d k Represents the characteristic dimension of the Key matrix; The window of the spatial feature sequence is shifted and then input into the moving window self-attention structure to calculate the self-attention. The calculation process is consistent with that of the window self-attention structure, and the feature obtained is the spatiotemporal feature sequence.

7. The cardiac phase identification method based on dynamic echocardiography according to claim 1, characterized in that: In step S4, the spatiotemporal feature sequence is mapped into the probability that each frame image in the video segment is ED and ES, including: The following formula is performed on the spatiotemporal feature sequence to calculate the probability that each frame of the video segment is a key cardiac phase; Prob=Softmax(Wf out +b) Among them, f out It is the spatiotemporal feature sequence calculated by the spatiotemporal feature aggregation module. f out Mapping matrix, f out Bias vector, The probability that each frame in the 16 input images is a key phase.

8. A cardiac phase recognition system based on dynamic echocardiography, characterized in that: include: A data label generation module is used to automatically generate a data label representing the probability of a key cardiac phase for each frame in the echocardiography video. After given the frame index of the key cardiac phase of each video, a data label satisfying a Gaussian probability distribution is generated around the frame indexes of the key cardiac phases ED and ES as the target of the cardiac phase recognition model learning; The data processing module is used to obtain the dynamic echocardiography video sequence after decoding the Dicom file, preprocess the video sequence, remove invalid information on the image, and then perform data enhancement; A sampling module is used to sample the data-enhanced video sequence using a sliding window, thereby sampling the video sequence into a number of overlapping video segments; The model building module is used to build a cardiac phase recognition model. The cardiac phase recognition model includes a spatial feature extractor, a spatiotemporal feature aggregation module, and a regression head. The spatial feature extractor is the CNN-based CSPVoVNet. The core of the CSPVoVNet network is to improve the traditional CNN model by introducing channel compensation technology and Cross Stage Partial technology. CSPVoVNet performs channel compensation by splicing in the feature channel dimension, and then reduces the dimension of the spliced ​​feature map through a 1*1 convolutional layer. The spatiotemporal feature aggregation module includes a window self-attention structure and a moving window self-attention structure. The window self-attention structure is used to calculate the attention within an independent window, and the moving window self-attention structure is used to calculate the attention between different windows. The spatial feature extractor is used to encode the spatial features of each frame of the video segment to obtain the spatial feature sequence of the video segment; The spatiotemporal feature aggregation module is used to perform temporal attention calculation on the spatial feature sequence to obtain the spatiotemporal feature sequence; the regression head is used to map the spatiotemporal feature sequence into the probability that each frame image in the video segment is ED and ES; A model training module is used to train a cardiac phase recognition model using the obtained multiple overlapping video segments. The module includes: inputting the obtained video segments into the cardiac phase recognition model, calculating the probability of each frame being the critical cardiac phase ED or ES, calculating the mean square error with the generated data labels that satisfy the Gaussian probability distribution, performing gradient backpropagation, and updating the cardiac phase recognition model parameters using a stochastic gradient descent algorithm, thereby achieving training of the cardiac phase recognition model. The recognition module is used to use the trained cardiac phase recognition model to identify the dynamic echocardiography video to be identified, obtain the probability that each frame in the video is the critical cardiac phase ED or ES, and average the multiple probability values ​​obtained from repeatedly sampled frames as the final probability of the frame.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: a memory for storing computer programs capable of running on the processor; A processor is configured to execute the steps of a cardiac phase identification method based on dynamic echocardiography as described in any one of claims 1 to 7 when running the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the cardiac phase identification method based on dynamic echocardiography as described in any one of claims 1 to 7.

Images