Heart failure auxiliary diagnosis and treatment system and method based on multi-sequence ultrasound image deep learning

Abstract

This invention relates to the field of medical image analysis technology, and discloses a heart failure auxiliary diagnosis and treatment system and method based on deep learning of multi-sequence ultrasound images. The system includes: a data acquisition module for acquiring multi-sequence cardiac ultrasound images of the subject; a multimodal feature extraction module, comprising: a spatiotemporal motion feature extraction sub-network, a hemodynamic feature extraction sub-network, a tissue Doppler feature extraction sub-network, and a myocardial strain feature extraction sub-network; a missing modality feature generation module, configured to generate approximate feature vectors of the missing modalities through a pre-trained generative network when any cardiac ultrasound sequence is missing or of lower quality than a preset standard; a multimodal feature fusion module that receives all acquired or generated modal feature vectors and outputs multimodal fusion features; and an auxiliary diagnosis and treatment output module that outputs the HFpEF diagnostic probability and major adverse cardiovascular event risk score. This improves the comprehensiveness and accuracy of HFpEF auxiliary diagnosis.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a heart failure auxiliary diagnosis and treatment system and method based on deep learning of multi-sequence ultrasound images. Background Technology

[0002] Diagnosing heart failure with preserved ejection fraction (HFpEF) is a challenge in cardiovascular clinical practice. Echocardiography is the preferred non-invasive imaging method for assessing cardiac structure and function. The diagnosis of HFpEF highly depends on the comprehensive assessment of multiple echocardiographic parameters, including early diastolic velocity e' of the mitral annulus, the ratio of early diastolic blood flow velocity at the mitral orifice to the mitral annulus velocity E / e', left atrial volume index, and peak tricuspid regurgitation velocity.

[0003] However, the pathophysiological mechanisms of HFpEF involve multiple interrelated dimensions, including decreased left ventricular active diastolic function, decreased myocardial compliance, increased left atrial pressure, and abnormal ventricular-arterial coupling. Multi-sequence echocardiography—including two-dimensional grayscale (B-mode) dynamic sequences, color Doppler sequences, tissue Doppler (TDI) sequences, and speckle tracking imaging (STI) parametric maps—reflects the functional state of the heart from different dimensions, such as myocardial structural motion, hemodynamics, myocardial velocity, and myocardial deformation. Single-sequence ultrasound parameters can only reveal one aspect of the aforementioned pathophysiological changes, making it difficult to comprehensively capture the multidimensional characteristics of HFpEF.

[0004] In terms of existing technologies, research has explored ultrasound image analysis techniques based on deep learning. For example, Chinese patent application CN120616602A discloses an ejection fraction prediction system based on echocardiographic video, which uses a 3D convolutional neural network to extract spatiotemporal features from B-mode grayscale sequences and combines a graph convolutional network to model the relationships between cardiac chambers. Chinese patent application CN120894309A discloses a morphological recognition system for cardiomyopathy, which classifies B-mode video frames through spatial attention and temporal fusion modules.

[0005] However, the aforementioned existing technical solutions all only utilize a single B-mode grayscale sequence and do not involve the fusion and utilization of other key ultrasound sequences such as color Doppler, tissue Doppler, or speckle tracking imaging, thus failing to fully explore the complementary diagnostic information of multi-sequence ultrasound data.

[0006] Furthermore, in real-world clinical workflows, ultrasound sequence acquisition is often incomplete or missing. For example, tissue Doppler signal quality may be poor in obese patients or those with severe lung air interference; the tracking quality of speckle tracking imaging may be significantly reduced in patients with arrhythmias; and some primary healthcare facilities may lack TDI or STI modules. Existing technologies generally assume complete input data, and model performance will significantly decline when faced with missing sequences, lacking effective robust handling mechanisms.

[0007] In summary, existing technologies lack a HFpEF-assisted diagnosis and treatment solution that can systematically and deeply integrate multiple different dimensions of cardiac ultrasound sequences, has robust handling capabilities for missing clinical data, and can provide a visualized presentation of decision-making basis. Summary of the Invention

[0008] The present invention aims to address the technical problems in the prior art, namely the lack of systematic and in-depth fusion of multiple different dimensions of cardiac ultrasound sequences and the lack of effective and robust mechanisms for handling common clinical ultrasound sequence omissions.

[0009] In a first aspect, the present invention provides a heart failure-assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images, comprising: The data acquisition module is configured to acquire multi-sequence echocardiogram images of the subject; the multi-sequence echocardiogram images include at least a two-dimensional grayscale B-mode dynamic sequence, a color Doppler sequence, a tissue Doppler TDI sequence, and a strain parameter map derived from speckle tracking imaging STI. The multimodal feature extraction module includes: The spatiotemporal motion feature extraction subnetwork is configured to receive B-mode dynamic sequences and extract feature vectors representing the spatiotemporal motion patterns of the myocardium; A hemodynamic feature extraction subnetwork is configured to receive color Doppler sequences and extract feature vectors characterizing hemodynamic features; The Doppler feature extraction subnetwork is configured to receive TDI sequences and extract feature vectors characterizing the velocity of the mitral annulus. The myocardial strain feature extraction subnetwork is configured to receive STI strain parameter maps and extract feature vectors characterizing the spatial heterogeneity of myocardial deformation. The missing modality feature generation module is configured to generate approximate feature vectors of the missing modalities by using the feature vectors of the remaining valid modalities as conditional inputs when any cardiac ultrasound sequence is missing or of lower quality than a preset standard, through a pre-trained generative network. The multimodal feature fusion module is configured to receive all acquired or generated modal feature vectors, adaptively fuse the feature vectors of each modality based on a cross-attention mechanism, and output multimodal fused features. The auxiliary diagnosis and treatment output module is configured to output the HFpEF diagnostic probability and the risk score of major adverse cardiovascular events based on the multimodal fusion features.

[0010] Furthermore, the generative network in the missing modality feature generation module is a conditional generative adversarial network, wherein: The generator takes the feature vectors of the valid modes as conditional input and generates approximate feature vectors of the missing modes. The discriminator is used to distinguish between the true feature vector and the approximate feature vector generated by the generator; The generator and the discriminator are optimized through adversarial training, so that the approximate feature vector generated by the generator approximates the distribution of the true feature vector in the feature space.

[0011] Furthermore, the multimodal feature fusion module includes a hierarchical feature fusion submodule, which is configured as follows: Map each modality feature vector to the query vector, key vector, and value vector, respectively; The attention weight matrix is ​​obtained by calculating the similarity between the query vector and the key vector; The attention weight matrix and the value vector are weighted and summed to complete the information interaction and adaptive fusion between modalities.

[0012] Furthermore: The data acquisition module is also configured to acquire structured clinical data of the subjects; wherein the structured clinical data includes at least one of age, sex, body mass index, NYHA functional class, natriuretic peptide level, history of atrial fibrillation, history of hypertension and history of diabetes. The multimodal feature extraction module further includes a clinical feature embedding subnetwork, which is configured to encode the structured clinical data into clinical feature vectors through a multilayer perceptron. The hierarchical feature fusion submodule is also configured to perform cross-attention calculation on the clinical feature vector and each image feature vector together, so that the image features and clinical features are fused across modalities.

[0013] Furthermore: The spatiotemporal motion feature extraction subnetwork adopts a Video Swin Transformer architecture or a three-dimensional convolutional neural network architecture. The tissue Doppler feature extraction subnetwork adopts an architecture combining a one-dimensional convolutional neural network and a Transformer; in this architecture, the convolutional neural network is configured to extract local waveform features from the TDI spectrogram, and the Transformer is configured to capture the long-range temporal dependencies between these local waveform features. The myocardial strain feature extraction subnetwork adopts a ResNet or Vision Transformer architecture and is configured to extract the spatial distribution pattern of strain peak values ​​in each segment of the left ventricle and the synchronicity features of strain peak time from the STI strain parameter map. The hemodynamic feature extraction subnetwork adopts the Swing Transformer or three-dimensional ResNet architecture and is configured to extract the spatiotemporal pattern features of mitral valve orifice blood flow velocity, pulmonary vein blood flow spectrum and valvular regurgitation jet from color Doppler sequences.

[0014] Furthermore, it also includes an automatic cardiac structure segmentation and region of interest extraction module, which is deployed with a U-Net or Attention U-Net segmentation network. It is configured to receive each frame of the B-mode dynamic sequence and output pixel-level segmentation masks of the left ventricle, left atrium, right ventricle and myocardial boundaries frame by frame.

[0015] Furthermore, the auxiliary diagnosis and treatment output module includes an interpretability analysis unit, which is configured as follows: A spatiotemporal heatmap is generated using the Grad-CAM algorithm, and a highlighted area is superimposed on the B-mode dynamic sequence to show the ventricular wall segments and cardiac phases that the model focuses on during decision-making. The SHAP analysis tool is used to calculate the marginal contribution of each input feature to the prediction result, and the output features are sorted by their absolute contribution value from largest to smallest.

[0016] Furthermore, the auxiliary diagnosis and treatment output module includes a diagnostic probability output unit and a risk score output unit, and the auxiliary diagnosis and treatment output module is further configured as follows: The probability of occurrence output by the risk scoring output unit is compared with a preset risk stratification threshold. Based on the comparison results, patients are automatically stratified into high-risk, medium-risk, or low-risk groups, and a structured ultrasound-assisted diagnostic report is generated.

[0017] Furthermore, the preset risk stratification threshold is determined on the validation set using the Youden index.

[0018] Another aspect of the present invention provides a method for assisting the diagnosis and treatment of heart failure based on deep learning of multi-sequence ultrasound images, comprising the following steps: Acquire multiple sequence cardiac ultrasound images of the subject, the multiple sequence cardiac ultrasound images including at least: two-dimensional grayscale B-mode dynamic sequence, color Doppler sequence, tissue Doppler TDI sequence, and strain parameter map derived from speckle tracking imaging STI; The B-mode dynamic sequence is input into the spatiotemporal motion feature extraction subnetwork to extract feature vectors representing the spatiotemporal motion pattern of the myocardium. The color Doppler sequence is input into the hemodynamic feature extraction subnetwork to extract feature vectors characterizing hemodynamic features. The TDI sequence is input into the tissue Doppler feature extraction subnetwork to extract feature vectors characterizing the velocity of the mitral valve annulus. Input the STI strain parameter map into the myocardial strain feature extraction subnetwork to extract feature vectors that characterize the spatial heterogeneity of myocardial deformation; When any one or more cardiac ultrasound sequences are missing or of lower quality than a preset standard, the feature vectors of the remaining valid modalities are used as conditional inputs to generate approximate feature vectors of the missing modalities through a pre-trained generative network. All acquired or generated modal feature vectors are input into the multimodal feature fusion module, which adaptively fuses the modal feature vectors based on the cross-attention mechanism and outputs the multimodal fused features. The multimodal fusion feature is input into the auxiliary diagnosis output module, which outputs the HFpEF diagnostic probability and the risk score of major adverse cardiovascular events.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: This invention systematically integrates four different dimensions of cardiac ultrasound sequences—B-mode dynamic sequence, color Doppler sequence, tissue Doppler TDI sequence, and speckle tracking imaging STI strain parameter map—to extract complementary pathophysiological features from four dimensions: myocardial structural motion, hemodynamics, myocardial motion velocity, and myocardial deformation. This overcomes the shortcomings of existing technologies that rely solely on a single ultrasound sequence, which cannot comprehensively reflect the multidimensional pathological changes in HFpEF.

[0020] Based on this, when any ultrasound sequence is missing or of poor quality, the missing modality feature generation module generates an approximate feature vector of the missing modality through a pre-trained generative network, using the feature vectors of the remaining effective modalities as conditions, thus ensuring the system's diagnostic robustness under conditions of incomplete real-world clinical data.

[0021] The multimodal feature fusion module adaptively calculates the interaction weights between feature vectors of each modality based on the cross-attention mechanism, thereby achieving precise complementary fusion of multidimensional pathophysiological information.

[0022] The auxiliary diagnosis and treatment output module directly outputs the HFpEF diagnostic probability and / or major adverse cardiovascular event risk score based on multimodal fusion features, forming a workflow from raw multi-sequence ultrasound images to clinical auxiliary decision-making conclusions, effectively improving the comprehensiveness, accuracy and applicability of HFpEF auxiliary diagnosis in complex clinical scenarios. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the system architecture disclosed in this invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention provides a heart failure auxiliary diagnosis and treatment system based on deep learning of multi-sequence ultrasound images. It aims to build an end-to-end deep learning system to achieve, for the first time, the systematic and deep fusion of four different dimensions of cardiac ultrasound sequences, which are relied upon for the clinical diagnosis of heart failure with preserved ejection fraction: two-dimensional grayscale B-mode dynamic sequence, color Doppler sequence, tissue Doppler TDI sequence, and strain parameter map derived from speckle tracking imaging STI.

[0027] Those skilled in the art will know that two-dimensional grayscale B-mode dynamic sequences mainly reflect macroscopic changes in cardiac chamber structure and ventricular wall motion; color Doppler sequences mainly reflect hemodynamic characteristics within cardiac chambers and great vessels; tissue Doppler TDI sequences mainly reflect the velocity characteristics of myocardial motion in the mitral valve annulus; and strain parameter maps derived from speckle tracking imaging STI mainly reflect the spatial heterogeneity characteristics of myocardial fiber deformation.

[0028] This invention addresses the data characteristics of each sequence by configuring a specially designed sub-network for feature extraction, transforming the four dimensions of pathophysiological information into a unified feature vector. Building upon this, the system introduces a missing modality feature generation module. In cases where a clinically common ultrasound sequence is missing or of poor quality, the system uses the feature vectors of the remaining valid modalities as input conditions to generate approximate feature vectors for the missing modality through a pre-trained generative network, thus ensuring the system's diagnostic robustness under conditions of incomplete real-world clinical data. A multimodal feature fusion module adaptively calculates the interaction weights between the feature vectors of each modality based on a cross-attention mechanism, achieving precise complementary fusion of multi-dimensional pathophysiological information. An auxiliary diagnostic output module directly outputs the HFpEF diagnostic probability and major adverse cardiovascular event risk score based on the multimodal fusion features, providing visualized decision-making support. The entire system forms an end-to-end processing pipeline from raw multi-sequence ultrasound images to clinical auxiliary decision-making conclusions.

[0029] The system mainly includes a data acquisition module, an automatic cardiac structure segmentation and region of interest extraction module, a multimodal feature extraction module, a missing modality feature generation module, a multimodal feature fusion module, and an auxiliary diagnosis and treatment output module. These modules are connected sequentially according to the logical order of data processing, forming a complete processing pipeline from raw ultrasound image input to clinical auxiliary decision-making output.

[0030] The following combination Figure 1 The system provided by the present invention will be described in detail.

[0031] First, the data acquisition module will be explained in detail.

[0032] In this embodiment, the data acquisition module is responsible for acquiring multiple sequences of cardiac ultrasound images of the subject. During a complete echocardiogram examination, the subject undergoes a standardized scan covering multiple sections to acquire multiple sets of dynamic image sequences, including standard sections such as apical four-chamber, apical two-chamber, and apical three-chamber.

[0033] In some specific implementations, the ultrasound images acquired by the data acquisition module include at least a two-dimensional grayscale B-mode dynamic sequence, a color Doppler sequence, a tissue Doppler TDI sequence, and a strain parameter map derived from speckle tracking imaging (STI).

[0034] Two-dimensional grayscale B-mode dynamic sequences record grayscale images of the heart for at least three consecutive cardiac cycles, acquired at a frame rate of 50–80 frames per second. B-mode dynamic sequences contain rich information about the myocardium and cardiac chamber boundaries, visually demonstrating the motion trajectory of the ventricular wall, the periodic changes in ventricular wall thickness, and the dynamic changes in cardiac chamber volume. During acquisition, the operator scans using standard sections such as the apical four-chamber view to ensure clear visibility of the endocardial line and avoid short-sectioning of the left ventricle.

[0035] Color Doppler sequences and B-mode sequences are acquired synchronously within the same cardiac cycle, displaying the direction and velocity of blood flow within the heart chambers and great vessels through color coding. Color Doppler sequences can visually demonstrate the forward blood flow at the mitral valve orifice, the pulmonary vein return blood flow, and the presence and extent of the tricuspid regurgitation jet, providing crucial hemodynamic information for assessing cardiac diastolic function and filling pressure.

[0036] Tissue Doppler TDI sequences are also acquired within the same cardiac cycle. Through specific filtering, only low-velocity motion signals of the myocardial tissue are displayed, used to measure the longitudinal motion velocity of the mitral valve annulus. TDI sequences typically place sampling volumes on the septal and lateral wall sides of the mitral valve annulus, recording the myocardial motion velocity waveforms during systole and diastole, including the peak systolic velocity st. ' Early diastolic peak velocity e ' and peak diastolic velocity a ' .

[0037] The strain parameter map derived from speckle tracking imaging (STI) is calculated by tracking the myocardial speckle pattern in B-mode images frame by frame to calculate the longitudinal, circumferential, and radial strain parameters of each myocardial segment. STI technology can generate overall longitudinal strain values ​​and a bullseye map containing 17 myocardial segments, visually displaying the spatial distribution of strain peak values ​​and strain peak time in each segment, which can be used to quantitatively assess the synchronicity and heterogeneity of myocardial deformation.

[0038] In some specific implementations, in addition to the four ultrasound sequences mentioned above, the data acquisition module is also configured to acquire the subject's structured clinical data, including age, sex, body mass index, NYHA functional class, natriuretic peptide levels, estimated glomerular filtration rate, history of atrial fibrillation, history of hypertension, and history of diabetes. This clinical data provides important reference information for subsequent fusion analysis.

[0039] Next, the automatic segmentation and region of interest extraction module for the heart structure will be described in detail.

[0040] In this embodiment, the automatic heart structure segmentation and region of interest extraction module is deployed with an improved U-Net or Attention U-Net segmentation network.

[0041] As those skilled in the art will know, U-Net is a classic encoder-decoder segmentation network. Its encoder extracts multi-scale semantic features of the image through layer-by-layer downsampling, while the decoder restores spatial resolution through layer-by-layer upsampling. Skip connections are used to concatenate and fuse the detailed features of each encoder layer with the semantic features of the corresponding decoder layer, thus achieving accurate pixel-level classification while maintaining spatial precision. Attention U-Net, based on the standard U-Net, introduces an attention gating mechanism into the skip connections. This attention gating mechanism learns a gating signal and adaptively adjusts the weights of the encoder skip connection features at each step of decoder upsampling based on the semantic features of the current layer. Specifically, the attention gating mechanism assigns higher weights to spatial regions in the encoder features relevant to the current decoding task and lower weights to background and irrelevant regions, thereby improving attention to the cardiac cavity boundary region and segmentation accuracy.

[0042] The aforementioned segmentation network was pre-trained on a large, publicly available cardiac ultrasound segmentation dataset containing thousands of cardiac ultrasound images from different ultrasound devices and operators, along with their corresponding pixel-level annotations. After pre-training, the network was fine-tuned on HFpEF ultrasound data collected at our center. During fine-tuning, a small learning rate and HFpEF case-specific annotation data were used to enhance the accuracy of identifying cardiac structural features in HFpEF patients while maintaining general segmentation capabilities. The deployed segmentation network receives each frame of a B-mode dynamic sequence and outputs pixel-level segmentation masks for the left ventricle, left atrium, right ventricle, and myocardial boundaries frame by frame, providing precise region-of-interest localization for subsequent feature extraction.

[0043] Secondly, the multimodal feature extraction module will be described in detail.

[0044] In this embodiment, the multimodal feature extraction module is one of the core components of the entire system. It includes four parallel feature extraction subnetworks: a spatiotemporal motion feature extraction subnetwork, a hemodynamic feature extraction subnetwork, a tissue Doppler feature extraction subnetwork, and a myocardial strain feature extraction subnetwork. Each subnetwork performs specialized feature extraction for the data characteristics of four ultrasound sequences.

[0045] Specifically: (1) The spatiotemporal motion feature extraction subnetwork receives the segmented B-mode dynamic sequence and extracts the feature vector representing the spatiotemporal motion pattern of the myocardium.

[0046] In some specific implementations, the spatiotemporal motion feature extraction subnetwork adopts the Video Swin Transformer architecture. The Video Swin Transformer is a video understanding architecture based on a self-attention mechanism. Its core idea is to divide video data into several three-dimensional blocks in the spatial and temporal dimensions, and efficiently calculate spatial attention within a local window through a shift-window multi-head self-attention mechanism, while capturing motion dependencies in the temporal dimension through cross-window connections.

[0047] The data processing procedure of the spatiotemporal motion feature extraction subnetwork is as follows: First, the input B-mode dynamic sequence is spatially divided into non-overlapping image blocks, each covering a rectangular region of the original image. Simultaneously, in the temporal dimension, a fixed number of consecutive frames are grouped together to form a three-dimensional input block. Each three-dimensional input block is mapped to a fixed-dimensional embedding vector through a linear embedding layer, and spatial and temporal location codes are added to preserve spatiotemporal location information.

[0048] Subsequently, the embedded vectors are fed into multiple stacked Video Swin Transformer layers for processing. Each layer contains a 3D window multi-head self-attention module and a shifted 3D window multi-head self-attention module. In the 3D window multi-head self-attention module, the feature map is uniformly divided into non-overlapping 3D windows in both spatial and temporal dimensions. Self-attention computation is confined to within each window, thereby reducing the computational complexity from the quadratic level of the entire image to the quadratic level of a window. In the shifted 3D window multi-head self-attention module, the window partitioning is shifted by half a window size in both spatial and temporal dimensions, allowing features originally located in different windows to interact across windows, thus achieving global receptive field coverage without significantly increasing computational cost.

[0049] Through layer-by-layer abstraction using multiple Video Swin Transformer layers, the network gradually extracts hierarchical representations ranging from low-level texture features to high-level semantic features. Finally, a global average pooling operation aggregates the feature vectors from all spatiotemporal locations into a single feature vector of fixed length. This feature vector characterizes the spatiotemporal motion patterns of the myocardium during contraction and relaxation across multiple cardiac cycles, including the amplitude, velocity, and coordination of ventricular wall motion, as well as the dynamic changes in cardiac chamber volume.

[0050] In other specific implementations, the spatiotemporal motion feature extraction subnetwork can also employ a three-dimensional convolutional neural network architecture. The three-dimensional convolutional neural network performs convolution operations simultaneously in both spatial and temporal dimensions using three-dimensional convolutional kernels. Each kernel slides in the spatial width, spatial height, and temporal depth directions of the input video clip, calculating a weighted summation within the local spatiotemporal neighborhood. By stacking multiple layers of three-dimensional convolutions and three-dimensional pooling operations, the network can capture multi-scale spatiotemporal features, ranging from short-term local motion to long-term dynamic patterns, layer by layer.

[0051] (2) The hemodynamic feature extraction subnetwork receives the color Doppler sequence and extracts the feature vectors that characterize the hemodynamic features.

[0052] In some specific implementations, the hemodynamic feature extraction subnetwork adopts a Swing Transformer or a three-dimensional ResNet architecture.

[0053] As those skilled in the art will know, color Doppler sequences and B-mode sequences correspond one-to-one in spatial location, but the value of each pixel no longer represents the echo intensity of the tissue, but rather the magnitude and direction of blood flow velocity at that location. Therefore, color Doppler sequences contain rich hemodynamic information.

[0054] In this embodiment, the Swin Transformer architecture processes features similarly to the aforementioned spatiotemporal motion feature extraction sub-network, using a shift-window self-attention mechanism to extract features in the spatiotemporal dimension of the color Doppler sequence. The difference lies in that the input channels and pre-trained weights of the hemodynamic feature extraction sub-network are adapted to the characteristics of the color Doppler data.

[0055] The 3D ResNet architecture constructs a deep 3D convolutional network through residual connections. Each residual block consists of multiple 3D convolutional layers, and skip connections are introduced to directly add the block's input to the block's output, effectively alleviating the gradient vanishing problem in deep network training. 3D ResNet performs 3D convolutional operations on each frame of the color Doppler sequence and its adjacent frames, extracting multi-level features from local blood flow patterns to global hemodynamic states layer by layer.

[0056] After feature extraction, the feature vector output by the hemodynamic feature extraction subnetwork contains at least one of the following hemodynamic information: peak velocities of the E and A waves of mitral valve orifice blood flow, the ratio of the E wave to the A wave, the deceleration time of the E wave, the velocity ratio of the pulmonary vein systolic S wave to the diastolic D wave, and the spatial range and peak velocity of the tricuspid regurgitation jet. These hemodynamic parameters are the core basis for assessing left ventricular diastolic function, left atrial filling pressure, and pulmonary circulation pressure.

[0057] (3) Organize a Doppler feature extraction subnetwork to receive TDI sequences and extract feature vectors that characterize the mitral annulus motion velocity from them.

[0058] In some specific implementations, this sub-network employs an architecture that combines a one-dimensional convolutional neural network with a Transformer.

[0059] Those skilled in the art will understand that the core information of a TDI sequence is concentrated in the time-velocity spectrogram of the mitral valve annulus. Unlike the three-dimensional spatiotemporal data structure of B-mode dynamic sequences and color Doppler sequences, the essential characteristic of a TDI sequence is a one-dimensional time-series signal at one or more location points. Therefore, an architecture combining a one-dimensional convolutional neural network and a Transformer is more suitable.

[0060] In this embodiment, the data processing procedure for the organization Doppler feature extraction subnetwork is as follows: First, velocity-time curves for the mitral valve annulus septum and lateral wall sides are extracted from the TDI sequence to form one-dimensional time-series data. A one-dimensional convolutional neural network receives this one-dimensional time-series data and performs convolution operations along the time axis through multiple one-dimensional convolutional layers. Each one-dimensional convolutional kernel slides along the time dimension, calculating a weighted sum of velocity values ​​within a local time window. Shallow one-dimensional convolutional layers extract local morphological features of the velocity waveform, such as the ascent slope, peak sharpness, and descent rate; deeper one-dimensional convolutional layers extract composite pattern features of the velocity waveform, such as the systolic phase (s). ' Wave and early diastolic e ' The relative amplitude relationship and temporal relationship of waves.

[0061] The local waveform feature sequence output by the one-dimensional convolutional neural network is then fed into the Transformer module. The Transformer module uses a multi-head self-attention mechanism to calculate the dependency between any two time points in the sequence, thereby capturing the long-range temporal patterns of myocardial velocity across multiple cardiac cycles. For example, under normal conditions, early diastolic e ' The peak velocity of the wave is greater than that of the late diastolic a wave. ' Peak velocity of the wave; while in HFpEF patients with impaired diastolic function, e ' The peak velocity of the wave is significantly reduced, e ' With a ' The ratio reverses. The Transformer module can effectively capture this temporal pattern change across cardiac cycles.

[0062] (4) The myocardial strain feature extraction subnetwork receives the STI strain parameter map and extracts feature vectors that characterize the spatial heterogeneity of myocardial deformation.

[0063] In some specific implementations, the myocardial strain feature extraction subnetwork adopts a ResNet or VisionTransformer architecture.

[0064] As those skilled in the art will know, STI strain parameter maps are typically presented in the form of a bullseye diagram, arranging the longitudinal strain peaks of the seventeen myocardial segments of the left ventricle in polar coordinates, with each color representing a different range of strain values. Unlike the original ultrasound image, the STI strain parameter map is a parametric image generated through calculation, and its pixel values ​​directly reflect the quantitative parameters of myocardial deformation.

[0065] In some specific implementations, the ResNet architecture extracts features from the input strain parameter map through multiple residual blocks. Each residual block consists of two convolutional layers and a skip connection, which directly passes the block's input to its output. As the network depth increases, shallow convolutions extract local features such as segment boundaries and color distribution, mid-level convolutions extract transition features between strain patterns in different regions, and deep convolutions extract spatial coordination patterns between strain values ​​in different segments of the left ventricle.

[0066] In other specific implementations, the Vision Transformer architecture divides the strain parameter map into fixed-size image blocks. Each image block is mapped to an embedding vector through linear projection, and positional encoding is added to preserve the spatial location information of each segment. The sequence of embedding vectors is fed into multiple stacked Transformer layers, and the correlation strength between any two segment embedding vectors is calculated through a multi-head self-attention mechanism, thereby capturing the spatial dependence and overall distribution pattern of strain values ​​between different myocardial segments.

[0067] After feature extraction, the feature vector output by the myocardial strain feature extraction subnetwork contains at least one of the following information: the absolute value of the overall longitudinal strain (reflecting the overall deformation capacity of the myocardium), the spatial distribution pattern of the peak strain in each segment (reflecting regional differences in deformation), and the synchronicity characteristics of the peak strain time in each segment (reflecting the coordination of myocardial contraction and relaxation). In patients with HFpEF, the absolute value of the overall longitudinal strain may still be within the normal range or only slightly reduced, but the spatial heterogeneity of segmental strain and the asynchrony of peak strain time are often abnormal. These features are of great value for the early identification of HFpEF.

[0068] In some specific implementations, the multimodal feature extraction module further includes a clinical feature embedding sub-network. The clinical feature embedding sub-network receives structured clinical data and encodes it into clinical feature vectors using a multilayer perceptron. The structured clinical data includes at least one of the following: age, sex, body mass index, NYHA functional class, natriuretic peptide level, history of atrial fibrillation, history of hypertension, and history of diabetes.

[0069] The layer perceptron consists of multiple stacked fully connected layers. Each fully connected layer multiplies the input vector by a weight matrix, adds a bias term, and then transforms it using a non-linear activation function. The first fully connected layer receives the standardized clinical data vector and maps it to the first hidden dimension. Subsequent fully connected layers perform feature transformation and dimension adjustment layer by layer, ultimately outputting a fixed-dimensional clinical feature vector. The non-linear activation function enables the network to learn complex non-linear interactions between clinical variables, such as the synergistic effect between natriuretic peptide levels and renal function, or between atrial fibrillation history and age.

[0070] Furthermore, the missing modality feature generation module is explained in detail.

[0071] Those skilled in the art will recognize that in clinical practice, ultrasound sequence acquisition is often incomplete or missing. For example, in obese patients or those with severe lung air interference, tissue Doppler signals may be of poor quality, making it impossible to obtain reliable ultrasound data. ' Values; for patients with arrhythmia or tachycardia, the tracking quality of speckle tracking imaging may be significantly reduced; some primary healthcare institutions' ultrasound equipment may not have TDI or STI functional modules.

[0072] To address this practical problem, the missing modality feature generation module of the present invention is configured to generate an approximate feature vector of the missing modality by using the feature vector of the remaining valid modalities as conditional input when any cardiac ultrasound sequence is missing or of lower quality than a preset standard, through a pre-trained generative network.

[0073] In some specific implementations, the generative network in the missing modality feature generation module adopts a conditional generative adversarial network.

[0074] The adversarial network consists of two networks: a generator and a discriminator, which are optimized through adversarial training. The generator aims to produce approximate feature vectors that closely resemble the true feature vectors, making them indistinguishable from the true feature vectors by the discriminator. The discriminator, on the other hand, aims to distinguish the true feature vectors from the approximate feature vectors generated by the generator as accurately as possible. Through adversarial training, the two networks mutually reinforce each other, ultimately enabling the generator to produce approximate feature vectors that are highly similar to the distribution of the true feature vectors in the feature space.

[0075] Specifically, the training process of a conditional generative adversarial network (GAN) is as follows: First, a large-scale training dataset containing all four ultrasound sequences is collected. During training, one or more sequences are randomly selected as pseudo-missing modalities, and the remaining sequences are considered valid modalities. The generator receives the feature vectors of the valid modalities as conditional inputs and outputs approximate feature vectors of the missing modalities. The discriminator receives a feature vector and its corresponding conditional feature vector and determines whether the feature vector comes from real data or was generated by the generator. The generator's loss function consists of two parts: adversarial loss and feature reconstruction loss. The adversarial loss measures the degree to which the discriminator misclassifies generated samples as real samples; the feature reconstruction loss measures the difference between the generated approximate feature vector and the real feature vector, typically using mean squared error or cosine similarity. By simultaneously optimizing the parameters of the generator and discriminator through backpropagation, after multiple rounds of iterative training, the generator can stably generate approximate feature vectors that are highly close to the feature vectors of the real missing modalities, even with only a subset of valid modalities.

[0076] In some specific implementations, the generative network in the missing modality feature generation module employs a variational autoencoder (VAE). The VAE consists of an encoder and a decoder. The encoder receives the feature vectors of the valid modalities as input, encodes them into the mean and variance vectors of the latent variable distribution, and samples the latent variables from this distribution. The decoder receives these latent variables as input and reconstructs an approximate feature vector for the missing modality. The training loss function of the VAE includes a reconstruction loss and a KL divergence regularization term—the reconstruction loss measures the difference between generated samples and real samples; the KL divergence regularization term constrains the latent variable distribution to approximate a standard normal distribution, thereby ensuring the continuity and smoothness of the latent space. After training, the decoder can generate the feature vectors of the missing modalities from the latent variables encoded by the valid modalities.

[0077] In this embodiment, when the missing modality feature generation module detects a missing ultrasound sequence or a quality below a preset standard (e.g., the sampling volume signal intensity of a TDI sequence is below a preset threshold, or the STI tracking quality score is below a preset threshold), the module automatically triggers the generation process. The module concatenates or sums the feature vectors of the acquired valid modalities to form a conditional input vector, and then calls a pre-trained generator or decoder to generate an approximate feature vector for the missing modality using the conditional input vector. The generated approximate feature vector is then added to the subsequent multimodal feature fusion module, enabling the system to perform a complete diagnostic reasoning process even with incomplete data.

[0078] In addition, the multimodal feature fusion module is described in detail.

[0079] In this embodiment, the multimodal feature fusion module receives all acquired or generated modal feature vectors from upstream, including spatiotemporal motion feature vectors, hemodynamic feature vectors, tissue motion velocity feature vectors, myocardial deformation feature vectors, and optional clinical feature vectors. Based on the cross-attention mechanism, it adaptively fuses the various modal feature vectors and outputs multimodal fusion features.

[0080] In some specific implementations, the multimodal feature fusion module includes a hierarchical feature fusion submodule. The hierarchical feature fusion submodule realizes intermodal information interaction and adaptive fusion based on the multi-head cross-attention mechanism in the Transformer architecture.

[0081] Specifically, the data processing flow of the hierarchical feature fusion submodule is as follows: The first step is modal feature vector concatenation. All received modal feature vectors are concatenated along their feature dimensions to form a joint feature representation containing all modal information. Assuming there are K modalities, and each modality's feature vector has a dimension of d, then the concatenated joint feature representation will have a dimension of K*d.

[0082] The second step involves mapping the query vector, key vector, and value vector. The joint feature representation is mapped to the query vector, key vector, and value vector using three different linear projection matrices. These linear projection matrices are learnable parameter matrices, optimized through backpropagation during model training. The query vector expresses what information the current modality wants to obtain from other modalities; the key vector expresses what information each modality can provide; and the value vector represents the actual information content carried by each modality.

[0083] The third step is the calculation of cross-attention weights. The attention weight matrix is ​​obtained by calculating the dot product similarity between the query vector and the key vectors. The dot product similarity measures the degree of matching between the query vector and each key vector; a higher similarity indicates that the information of the corresponding modality is more important to the current query. The dot product similarity is scaled by dividing by the square root of the key vector dimension to prevent the dot product result from being too large and causing the gradient vanishing problem of the softmax function. Then, the scaled similarity is converted into normalized attention weights using the softmax function, with all weights summed to one.

[0084] The fourth step is attention-weighted fusion. The attention weight matrix and the value vector are weighted and summed to obtain the fused feature representation. In this process, modal information that is highly relevant to the current task is given higher weights and enhanced in the fused feature, while modal information that is not highly relevant to the current task is given lower weights and suppressed in the fused feature, thereby achieving adaptive intermodal information complementarity and redundancy suppression.

[0085] It should be noted that the above cross-attention calculation process can be executed in parallel multiple times, i.e., a multi-head attention mechanism is employed. In the multi-head attention mechanism, the query vector, key vector, and value vector are mapped to multiple subspaces through multiple sets of different linear projection matrices. Attention calculations are performed independently within each subspace. Finally, the attention outputs of all subspaces are concatenated and mapped back to the original dimensions through an output projection matrix. The multi-head attention mechanism enables the model to capture different types of intermodal interactions in different representation subspaces.

[0086] The hierarchical feature fusion submodule incorporates the clinical feature vector along with each imaging feature vector into the aforementioned cross-attention calculation process. This means that clinical features (such as natriuretic peptide levels, estimated glomerular filtration rate, and age) and imaging features (such as B-mode spatiotemporal motion features and TDI velocity features) are interactively fused within a unified attention framework. For example, in some specific clinical scenarios, when calculating attention weights, the clinical feature of significantly elevated natriuretic peptides may be correlated with E / e. ' The elevated imaging features reinforce each other, pointing together towards a diagnosis of diastolic dysfunction.

[0087] The multimodal fusion feature output after processing by the hierarchical feature fusion submodule is a fixed-dimensional feature vector. This feature vector integrates complementary information from four ultrasound sequence dimensions and an optional clinical dimension. At the same time, it suppresses redundant information within each modality through an attention mechanism, providing a high-quality comprehensive input for downstream diagnostic output.

[0088] Finally, the auxiliary diagnosis and treatment output module is described in detail.

[0089] In this embodiment, the auxiliary diagnosis and treatment output module receives the multimodal fusion features output by the multimodal feature fusion module, and outputs the HFpEF diagnostic probability and major adverse cardiovascular event risk score based on the fusion features.

[0090] In some specific implementations, the auxiliary diagnosis and treatment output module includes a diagnostic probability output unit and a risk score output unit.

[0091] The diagnostic probability output unit is configured to input multimodal fusion features into a fully connected layer, which then maps these features to a value between 0 and 1 using a Sigmoid activation function, outputting the probability of HFpEF. The fully connected layer multiplies the multimodal fusion feature vector with the weight matrix and adds a bias term to obtain a scalar value. The Sigmoid activation function maps this scalar value to a value between 0 and 1, exhibiting an S-shaped curve, and the output value can be interpreted as the probability of HFpEF. In practical use, when the output probability of HFpEF is greater than or equal to 0.5, the system tends to determine that the patient has HFpEF; when the probability of HFpEF is less than 0.5, the system tends to determine that the patient does not have HFpEF. The specific classification threshold can be adjusted according to the different needs for sensitivity and specificity in clinical application scenarios. For example, in screening scenarios, the threshold can be appropriately lowered to improve sensitivity, while in diagnostic scenarios, the threshold can be appropriately increased to improve specificity.

[0092] The risk scoring output unit is configured to input multimodal fusion features into the risk prediction fully connected layer and output the probability of occurrence of major adverse cardiovascular events. Major adverse cardiovascular events include at least one of heart failure rehospitalization, cardiovascular death, and acute decompensation. The risk prediction fully connected layer is configured in parallel with the diagnostic fully connected layer, receiving the same multimodal fusion features as input, but using independently trained weight parameters, and outputting the predicted probability of major adverse cardiovascular events occurring within one or two years.

[0093] In some specific implementations, the auxiliary diagnostic output module is further configured to compare the occurrence probability output by the risk scoring output unit with a preset risk stratification threshold, and automatically stratify patients into high-risk, medium-risk, or low-risk groups based on the comparison results. The preset risk stratification threshold is determined on the validation set using the Youden index. The Youden index is defined as sensitivity plus specificity minus one, and the probability value that maximizes the Youden index is selected as the cutoff threshold between high and low risk. After stratification, the system generates a structured ultrasound-assisted diagnostic report, which includes the patient's basic information, a summary of key parameters for each ultrasound sequence, the diagnostic probability of HFpEF, the risk stratification results, and the stratification basis.

[0094] In some other embodiments, the auxiliary diagnostic output module also includes an interpretability analysis unit. This interpretability analysis unit generates a spatiotemporal heatmap using the Grad-CAM algorithm.

[0095] The Grad-CAM algorithm works as follows: First, it calculates the gradient of the output corresponding to the HFpEF category in the multimodal fusion features relative to the feature map of the last layer of the spatiotemporal motion feature extraction subnetwork. Then, it performs global average pooling on these gradients to obtain the importance weight of each feature channel. Finally, it weights and sums the feature maps of each channel according to their corresponding weights, and filters out negative value regions using the ReLU activation function to obtain a spatiotemporal heatmap. This heatmap is superimposed onto the original B-mode dynamic sequence, highlighting the ventricular wall segments and cardiac phases that the model focuses on when making HFpEF diagnostic decisions. For example, in some specific clinical scenarios, in early diastole, the model may focus on the motion state of the mitral valve annulus on the interventricular septum side; in late diastole, the model may focus on changes in the left atrial region. By observing the distribution pattern of the heatmap, clinicians can intuitively understand the diagnostic basis of the model and assess its consistency with their own clinical judgment.

[0096] The interpretability analysis unit also calculates the marginal contribution of each input feature to the prediction results using the SHAP analysis tool. The SHAP method, based on the game-theoretic framework of Shapley values, quantifies the importance of each feature by calculating the weighted average of its marginal contribution to the model's prediction results across all possible feature combinations. For a specific patient, the SHAP analysis tool outputs a list of feature contributions, where each feature corresponds to a SHAP value—positive values ​​indicate that the feature pushes the prediction results towards the disease direction, negative values ​​indicate that the feature pushes the prediction results towards the normal direction, and the absolute value indicates the strength of the feature's contribution. The system outputs feature importance sorted from largest to smallest by the absolute value of the SHAP value, listing the clinical variables or ultrasound parameters with the largest contributions. For example, for a patient diagnosed with HFpEF, the feature importance ranking might show: E / e ' The ratio is the most important positive contribution feature, e ' Peak velocity was the second most important positive contributing feature, and natriuretic peptide level was the third most important positive contributing feature.

[0097] This invention also protects a method for assisting the diagnosis and treatment of heart failure based on deep learning of multi-sequence ultrasound images, comprising the following steps: S100. Acquire multiple sequence echocardiogram images of the subject. The multiple sequence echocardiogram images include at least: a two-dimensional grayscale B-mode dynamic sequence, a color Doppler sequence, a tissue Doppler TDI sequence, and a strain parameter map derived from speckle tracking imaging STI.

[0098] S201. Input the B-mode dynamic sequence into the spatiotemporal motion feature extraction subnetwork to extract feature vectors representing the spatiotemporal motion pattern of the myocardium.

[0099] S202. Input the color Doppler sequence into the hemodynamic feature extraction subnetwork to extract feature vectors representing hemodynamic features.

[0100] S203. Input the TDI sequence into the tissue Doppler feature extraction subnetwork to extract feature vectors that characterize the mitral valve annulus motion velocity.

[0101] S204. Input the STI strain parameter map into the myocardial strain feature extraction subnetwork to extract feature vectors that characterize the spatial heterogeneity of myocardial deformation.

[0102] S300: When any one or more cardiac ultrasound sequences are missing or of lower quality than a preset standard, the feature vectors of the remaining valid modalities are used as input conditions to generate approximate feature vectors of the missing modalities through a pre-trained generative network.

[0103] S400: Input all acquired or generated modal feature vectors into the multimodal feature fusion module, adaptively fuse the modal feature vectors based on the cross-attention mechanism, and output the multimodal fused features.

[0104] S500: Input the multimodal fusion features into the auxiliary diagnosis output module to output the HFpEF diagnostic probability and the risk score of major adverse cardiovascular events.

[0105] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A heart failure auxiliary diagnosis and treatment system based on deep learning of multi-sequence ultrasound images, characterized in that, include: The data acquisition module is configured to acquire multiple sequences of echocardiogram images of the subject; The multi-sequence cardiac ultrasound images include at least a two-dimensional grayscale B-mode dynamic sequence, a color Doppler sequence, a tissue Doppler TDI sequence, and a strain parameter map derived from speckle tracking imaging STI. The multimodal feature extraction module includes: The spatiotemporal motion feature extraction subnetwork is configured to receive B-mode dynamic sequences and extract feature vectors representing the spatiotemporal motion patterns of the myocardium; A hemodynamic feature extraction subnetwork is configured to receive color Doppler sequences and extract feature vectors characterizing hemodynamic features; The Doppler feature extraction subnetwork is configured to receive TDI sequences and extract feature vectors characterizing the velocity of the mitral valve annulus. The myocardial strain feature extraction subnetwork is configured to receive STI strain parameter maps and extract feature vectors characterizing the spatial heterogeneity of myocardial deformation. The missing modality feature generation module is configured to generate approximate feature vectors of the missing modalities by using the feature vectors of the remaining valid modalities as conditional inputs when any cardiac ultrasound sequence is missing or of lower quality than a preset standard, through a pre-trained generative network. The multimodal feature fusion module is configured to receive all acquired or generated modal feature vectors, adaptively fuse the feature vectors of each modality based on a cross-attention mechanism, and output multimodal fused features. The auxiliary diagnosis and treatment output module is configured to output the HFpEF diagnostic probability and the risk score of major adverse cardiovascular events based on the multimodal fusion features.

2. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 1, characterized in that, The generator network in the missing modality feature generation module is a conditional generative adversarial network, wherein: The generator takes the feature vectors of the valid modes as conditional input and generates approximate feature vectors of the missing modes. The discriminator is used to distinguish between the true feature vector and the approximate feature vector generated by the generator; The generator and the discriminator are optimized through adversarial training, so that the approximate feature vector generated by the generator approximates the distribution of the true feature vector in the feature space.

3. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 1 or 2, characterized in that, The multimodal feature fusion module includes a hierarchical feature fusion submodule, which is configured as follows: Map each modality feature vector to the query vector, key vector, and value vector, respectively; The attention weight matrix is ​​obtained by calculating the similarity between the query vector and the key vector; The attention weight matrix and the value vector are weighted and summed to complete the information interaction and adaptive fusion between modalities.

4. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 3, characterized in that: The data acquisition module is also configured to acquire structured clinical data of the subjects; wherein the structured clinical data includes at least one of age, sex, body mass index, NYHA functional class, natriuretic peptide level, history of atrial fibrillation, history of hypertension and history of diabetes. The multimodal feature extraction module further includes a clinical feature embedding subnetwork, which is configured to encode the structured clinical data into clinical feature vectors through a multilayer perceptron. The hierarchical feature fusion submodule is also configured to perform cross-attention calculation on the clinical feature vector and each image feature vector together, so that the image features and clinical features are fused across modalities.

5. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 1, characterized in that: The spatiotemporal motion feature extraction subnetwork adopts a Video Swin Transformer architecture or a three-dimensional convolutional neural network architecture. The tissue Doppler feature extraction subnetwork adopts an architecture combining a one-dimensional convolutional neural network and a Transformer; in this architecture, the convolutional neural network is configured to extract local waveform features from the TDI spectrogram, and the Transformer is configured to capture the long-range temporal dependencies between these local waveform features. The myocardial strain feature extraction subnetwork adopts a ResNet or Vision Transformer architecture and is configured to extract the spatial distribution pattern of strain peak values ​​in each segment of the left ventricle and the synchronicity features of strain peak time from the STI strain parameter map. The hemodynamic feature extraction subnetwork adopts the Swing Transformer or three-dimensional ResNet architecture and is configured to extract the spatiotemporal pattern features of mitral valve orifice blood flow velocity, pulmonary vein blood flow spectrum and valvular regurgitation jet from color Doppler sequences.

6. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 1, characterized in that, It also includes a module for automatic segmentation and region of interest extraction of cardiac structures. The module is equipped with a U-Net or Attention U-Net segmentation network and is configured to receive each frame of the B-mode dynamic sequence and output pixel-level segmentation masks of the left ventricle, left atrium, right ventricle and myocardial boundaries frame by frame.

7. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 1, characterized in that, The auxiliary diagnosis and treatment output module includes an interpretability analysis unit, which is configured as follows: A spatiotemporal heatmap is generated using the Grad-CAM algorithm, and a highlighted area is superimposed on the B-mode dynamic sequence to show the ventricular wall segments and cardiac phases that the model focuses on during decision-making. The SHAP analysis tool is used to calculate the marginal contribution of each input feature to the prediction result, and the output features are sorted by their absolute contribution value from largest to smallest.

8. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 1, characterized in that, The auxiliary diagnosis and treatment output module includes a diagnostic probability output unit and a risk score output unit, and the auxiliary diagnosis and treatment output module is further configured as follows: The probability of occurrence output by the risk scoring output unit is compared with a preset risk stratification threshold. Based on the comparison results, patients are automatically stratified into high-risk, medium-risk, or low-risk groups, and a structured ultrasound-assisted diagnostic report is generated.

9. The heart failure assisted diagnosis and treatment system based on deep learning of multi-sequence ultrasound images according to claim 8, characterized in that, The preset risk stratification threshold is determined on the validation set using the Youden index.

10. A method for assisting the diagnosis and treatment of heart failure based on deep learning of multi-sequence ultrasound images, characterized in that, Includes the following steps: Acquire multiple sequence cardiac ultrasound images of the subject, the multiple sequence cardiac ultrasound images including at least: two-dimensional grayscale B-mode dynamic sequence, color Doppler sequence, tissue Doppler TDI sequence, and strain parameter map derived from speckle tracking imaging STI; The B-mode dynamic sequence is input into the spatiotemporal motion feature extraction subnetwork to extract feature vectors representing the spatiotemporal motion pattern of the myocardium. The color Doppler sequence is input into the hemodynamic feature extraction subnetwork to extract feature vectors characterizing hemodynamic features. The TDI sequence is input into the tissue Doppler feature extraction subnetwork to extract feature vectors characterizing the velocity of the mitral valve annulus. Input the STI strain parameter map into the myocardial strain feature extraction subnetwork to extract feature vectors that characterize the spatial heterogeneity of myocardial deformation; When any one or more cardiac ultrasound sequences are missing or of lower quality than a preset standard, the feature vectors of the remaining valid modalities are used as conditional inputs to generate approximate feature vectors of the missing modalities through a pre-trained generative network. All acquired or generated modal feature vectors are input into the multimodal feature fusion module, which adaptively fuses the modal feature vectors based on the cross-attention mechanism and outputs the multimodal fused features. The multimodal fusion feature is input into the auxiliary diagnosis output module, which outputs the HFpEF diagnostic probability and the risk score of major adverse cardiovascular events.

Images