A feature fusion method for intelligent identification of pulmonary valve stenosis echocardiogram
By fusing features from echocardiographic videos and Doppler spectra using deep learning neural networks, the problems of utilizing dynamic temporal information and fusion of heterogeneous data in pulmonary valve stenosis were solved, enabling efficient and accurate identification and visualization analysis of pulmonary valve stenosis, and improving the automation and interpretability of ultrasound examinations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-15
AI Technical Summary
Existing deep learning-based ultrasound image analysis methods suffer from insufficient utilization of dynamic temporal information, difficulty in fusing heterogeneous data features, and insufficient ability to extract key feature coupling when dealing with pulmonary valve stenosis, leading to decreased feature recognition stability.
Deep learning neural networks are used to jointly extract, heterogeneously align, and deeply fuse the spatiotemporal features of echocardiography videos and the hemodynamic features of Doppler spectra. The Transformer architecture and convolutional neural network are used for automatic identification of standard cross sections and selection of target spectral images. The attention mechanism is combined for feature mapping and fusion.
It achieves high-quality automatic data preprocessing, improves the accuracy and robustness of pulmonary valve stenosis identification, and enhances the interpretability of model output through visualized attention heatmaps, thereby improving the efficiency of ultrasound examination and the standardization of analysis results.
Smart Images
Figure CN121661455B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image analysis and artificial intelligence technology, and in particular relates to a feature fusion method for intelligent echocardiographic recognition of pulmonary valve stenosis. Background Technology
[0002] The main pathological features of pulmonary valve stenosis (PS) are thickening, adhesion, or annular dysplasia of the pulmonary valve leaflets, leading to increased right ventricular outflow tract resistance. In imaging evaluation, echocardiography, especially combined with analysis of the parasternal pulmonary artery section and Doppler flow spectrum, is a key non-invasive technique for obtaining relevant physiological and structural information. The data acquired by this technique mainly includes two modalities: one is two-dimensional ultrasound dynamic video (2D video) for observing valve morphology and motion, and the other is Doppler spectrum images reflecting changes in blood flow velocity and pressure gradient.
[0003] However, existing deep learning-based ultrasound image analysis methods still face the following technical bottlenecks when processing the aforementioned multimodal data:
[0004] Insufficient utilization of dynamic temporal information: Existing technologies mostly focus on feature extraction from static single-frame images. Ultrasound videos contain extremely strong temporal correlations, and simple static feature extraction is difficult to capture subtle motion models between frames, resulting in incomplete feature representation.
[0005] The fusion of features from heterogeneous data is challenging: echocardiography videos present as high-dimensional spatiotemporal sequence data, primarily characterizing the cardiac anatomical topology and its instantaneous motion; while Doppler spectra are time-frequency domain mappings, mainly characterizing the distribution of hemodynamic parameters. These two types of data differ significantly in data dimensionality, feature space distribution, and semantic level (heterogeneity). Existing techniques often process single modalities in isolation or perform only simple shallow stitching, making it difficult to achieve effective alignment and complementarity between the two in deep feature spaces.
[0006] Insufficient ability to extract the coupling of key features: The characterization of pulmonary valve stenosis involves morphological features in the video modality and hydrodynamic features in the spectral modality, which have a significant pathological correlation. Existing technical solutions mostly use independent channel processing and have failed to establish an effective multidimensional feature mapping model to characterize this correlation. Due to the lack of modeling the strong coupling relationship between anatomical abnormalities and blood flow changes, existing methods struggle to achieve feature cross-verification and enhancement when processing complex case data, leading to decreased stability in feature recognition. Summary of the Invention
[0007] To address the aforementioned technical problems, this invention provides a feature fusion method for intelligent echocardiographic recognition of pulmonary valve stenosis, designs an application scenario for echocardiography of pulmonary valve stenosis, and utilizes a deep learning neural network to jointly extract, heterogeneously align, and deeply fuse the spatiotemporal features of echocardiographic video with the hemodynamic features of the Doppler spectrum.
[0008] This invention proposes a feature fusion method for intelligent echocardiographic recognition of pulmonary valve stenosis, comprising:
[0009] Echocardiographic videos of the pulmonary valve are acquired, and standard sections are automatically identified to obtain keyframe sequences corresponding to the target anatomical structures.
[0010] Obtain Doppler spectral images of the pulmonary valve, and obtain target spectral images through target filtering processing;
[0011] Based on the keyframe sequence and the target spectral image, fused features are obtained through cross-modal joint feature extraction and feature fusion;
[0012] Based on the fusion features, the identification result of pulmonary valve stenosis is obtained, and the corresponding key region attention heatmap is generated and displayed on the original Doppler spectral image.
[0013] Optionally, acquire echocardiographic video of the pulmonary valve, perform automatic standard section recognition, and obtain keyframe sequences corresponding to the target anatomical structure, specifically including:
[0014] Based on pulmonary valve echocardiographic video, a continuous frame sequence is obtained through temporal sequence analysis;
[0015] Based on the continuous frame sequence, a video section recognition model based on the Transformer architecture is used for classification processing to obtain the confidence score of each frame belonging to a predetermined standard section.
[0016] The confidence score is compared with a preset threshold, and frames with a confidence score greater than or equal to the preset threshold are selected to obtain a key frame sequence; wherein, the predetermined standard section is the parasternal pulmonary artery section.
[0017] Optionally, a Doppler spectral image of the pulmonary valve is acquired, and a target spectral image is obtained through target screening processing, specifically including:
[0018] The acquired Doppler spectrum image of the pulmonary valve is input into the convolutional neural network model;
[0019] The convolutional neural network model is used to classify spectral images to identify and filter out the target spectral images for pulmonary valve hemodynamic analysis.
[0020] Optionally, based on the keyframe sequence and the target spectral image, fused features are obtained through cross-modal joint feature extraction and feature fusion, specifically including:
[0021] Based on the keyframe sequence, time-space feature modeling is performed to obtain video time-space feature vectors;
[0022] Based on the target spectral image, feature extraction is performed using a convolutional neural network to obtain the spectral image feature vector;
[0023] Map the video spatiotemporal feature vector and the spectral image feature vector to the same dimension of semantic space;
[0024] By using attention-based interaction, the mapped features are fused to obtain fused features.
[0025] Optionally, mapping the video spatiotemporal feature vector and the spectral image feature vector to a semantic space of the same dimension specifically includes:
[0026] Based on the video spatiotemporal feature vector and the spectral image feature vector, the data are processed through a linear projection layer to obtain dimension-aligned video mapping features and spectral mapping features.
[0027] Optionally, the mapped features are fused through attention-based interactions, specifically including:
[0028] The mapped spectral image feature vector is used as the query vector, and the mapped video spatiotemporal feature vector is used as the key vector and value vector.
[0029] Calculate the relevance weight matrix between the query vector and the key vector, and use the relevance weight matrix to weight the value vector to obtain attention-weighted video features;
[0030] The attention-weighted video features and the mapped spectral image feature vector are weighted and summed to obtain the fused features.
[0031] Optionally, the weighted summation is achieved through a learnable gating parameter, which adaptively adjusts the contribution ratio of the attention-weighted video features and the mapped spectral image feature vector in the fused features.
[0032] Optionally, based on the fusion features, the identification result of pulmonary valve stenosis is obtained, and a corresponding key region attention heatmap is generated and displayed on the original Doppler spectral image, specifically including:
[0033] Based on the attention weight matrix generated during the multimodal feature fusion stage, a two-dimensional attention map is obtained through upsampling and normalization.
[0034] Based on the two-dimensional attention map, it is converted into a heatmap through pseudo-color mapping, and the heatmap is superimposed on the corresponding original Doppler spectrum image.
[0035] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.
[0036] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.
[0037] Compared with the prior art, the present invention has the following advantages and technical effects:
[0038] This invention achieves automatic and accurate location and extraction of key video frame sequences and spectral images related to pulmonary valve analysis from raw ultrasound data through automatic identification of standard sections and automatic screening of target spectra. This replaces the tedious search and screening work that relies on human experience, providing a high-quality and standardized data foundation for subsequent analysis, effectively reducing the burden of manual preprocessing and improving the automation level of the process.
[0039] This invention constructs a nonlinear mapping and complementarity mechanism between spatiotemporal features of ultrasound video and features of Doppler spectral images through cross-modal joint feature extraction and deep fusion. This method solves the fusion challenge caused by dimensional and semantic differences in heterogeneous data, enabling effective alignment and synergistic enhancement of cardiac anatomical motion information and hemodynamic information in a deep feature space. This significantly improves the accuracy of the model in identifying and analyzing pulmonary valve stenosis and its robustness under different data qualities.
[0040] This invention generates and overlays attention heatmaps of key regions, transforming the attention weights of cross-modal feature interactions within the model into a visualized image region enhancement effect. This method intuitively maps the model's decision-making basis back to the original image, providing visual cues for key analytical regions (such as abnormal valve motion areas and abnormal blood flow spectrum areas), greatly improving the interpretability of the model output.
[0041] This invention, through the systematic integration of the aforementioned technical means, constructs an end-to-end intelligent analysis workflow, from automatic preprocessing of raw data to deep fusion of multimodal information, and then to simultaneous output of results and visualization. This workflow can serve as an efficient auxiliary tool, providing objective and consistent quantitative analysis support for ultrasound examinations, and helping to improve the efficiency of the overall workflow and the standardization of analysis results. Attached Figure Description
[0042] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0043] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;
[0044] Figure 2 This is a schematic diagram of the feature fusion process according to an embodiment of the present invention. Detailed Implementation
[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0046] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0047] Example 1
[0048] like Figure 1 As shown, this embodiment provides a feature fusion method for intelligent echocardiographic recognition of pulmonary valve stenosis, including:
[0049] Echocardiographic videos of the pulmonary valve are acquired, and standard sections are automatically identified to obtain keyframe sequences corresponding to the target anatomical structures.
[0050] Obtain Doppler spectral images of the pulmonary valve, and obtain target spectral images through target filtering processing;
[0051] Based on the keyframe sequence and the target spectral image, fused features are obtained through cross-modal joint feature extraction and feature fusion;
[0052] Based on the fusion features, the identification result of pulmonary valve stenosis is obtained, and the corresponding key region attention heatmap is generated and displayed on the original Doppler spectral image.
[0053] Feasible methods include acquiring echocardiographic videos of the pulmonary valve, automatically identifying standard sections, and obtaining keyframe sequences corresponding to the target anatomical structures, specifically including:
[0054] Based on the pulmonary valve echocardiogram video, a continuous frame sequence is obtained through temporal analysis. Based on the continuous frame sequence, a video section recognition model based on the Transformer architecture is used for classification processing to obtain the confidence score of each frame belonging to a predetermined standard section. The confidence score is compared with a preset threshold, and frames with a confidence score greater than or equal to the preset threshold are selected to obtain the key frame sequence. The predetermined standard section is the parasternal pulmonary artery section.
[0055] In this embodiment, the initial echocardiographic video input is parsed into consecutive frames. Spatial segmentation is applied to extract key spatiotemporal feature information from these frames. The processed frame sequence is then fed into a Transformer model for deep classification to distinguish parasternal pulmonary artery sections. Simultaneously, a CNN processes the Doppler spectral images through a dedicated "spectral classification branch" to identify target Doppler spectral images associated with the PS. This dual-branch strategy provides a reliable data foundation for downstream analysis.
[0056] Feasible methods include acquiring Doppler spectral images of the pulmonary valve, and obtaining target spectral images through target screening processing, specifically including:
[0057] The acquired Doppler spectral images of the pulmonary valve are input into a convolutional neural network (CNN) model; the CNN model is used to classify the spectral images to identify and filter out target spectral images for pulmonary valve hemodynamic analysis.
[0058] This embodiment achieves automatic and accurate location and extraction of key video frame sequences and spectral images related to pulmonary valve analysis from raw ultrasound data through automatic identification of standard sections and automatic screening of target spectra. This replaces the tedious search and screening work that relies on human experience, providing a high-quality and standardized data foundation for subsequent analysis, effectively reducing the burden of manual preprocessing and improving the automation level of the process.
[0059] Implementably, based on the keyframe sequence and the target spectral image, fused features are obtained through cross-modal joint feature extraction and feature fusion, specifically including:
[0060] Based on the keyframe sequence, time-space feature modeling is performed to obtain video time-space feature vectors; based on the target spectral image, feature extraction is performed using a convolutional neural network to obtain spectral image feature vectors; the video time-space feature vectors and the spectral image feature vectors are mapped to the same-dimensional semantic space; through attention-based interaction, the mapped features are fused to obtain fused features.
[0061] Furthermore, mapping the video spatiotemporal feature vector and the spectral image feature vector to a semantic space of the same dimension specifically includes:
[0062] Based on the video spatiotemporal feature vector and the spectral image feature vector, the data are processed through a linear projection layer to obtain dimension-aligned video mapping features and spectral mapping features.
[0063] Furthermore, the mapped features are fused through attention-based interactions, specifically including:
[0064] Using the mapped spectral image feature vector as the query vector, and the mapped video spatiotemporal feature vector as the key vector and value vector; calculating the correlation weight matrix between the query vector and the key vector, and using the correlation weight matrix to weight the value vector to obtain attention-weighted video features; and weighting the attention-weighted video features and the mapped spectral image feature vector to obtain fused features.
[0065] Furthermore, the weighted summation is achieved through a learnable gating parameter, which adaptively adjusts the contribution ratio of the attention-weighted video features and the mapped spectral image feature vector in the fused features.
[0066] As a specific implementation, this embodiment constructs a multimodal joint classification model consisting of a video branch, an image branch, and a feature fusion module. Specifically, the target Doppler spectrum image is input into the CNN branch, while the selected parasternal pulmonary artery video frame is input into the video branch.
[0067] The video branch uses a TimeSformer network to model the temporal-spatial features of keyframe video sequences and outputs a video feature vector. The image branch uses a CNN network to extract features from the spectral image and outputs a spectral feature vector. Subsequently, the features extracted from the two branches are concatenated and passed through a fully connected layer before being input into the classifier for decision-making.
[0068] To address common optimization challenges of Transformer models and accelerate model convergence, this embodiment implements a weight-sharing mechanism across two stages within the video branch. This method not only accelerates training but also enhances the model's generalization ability.
[0069] Furthermore, for the video branch, this embodiment selects the TimeSformer architecture as the core backbone because it has proven its ability to efficiently model time series and extract dynamic features from video data. For the image branch, this embodiment uses the ResNeXt50 architecture as the basic CNN model. Both models are initialized using pre-trained weights: the TimeSformer model is pre-trained on the Kinetics-400 dataset, and the ResNeXt50 model is pre-trained on the ImageNet dataset, thereby leveraging the rich semantic knowledge of large-scale datasets to improve model performance and convergence speed.
[0070] Furthermore, the model training process includes:
[0071] During the data augmentation phase, the datasets for both branches underwent initial preprocessing to ensure consistency and standardization. For the CNN branch, data augmentation techniques including random image rotation, horizontal flipping, and adding Gaussian noise were employed to enhance model robustness and mitigate overfitting. For the video branch, random frame sampling and horizontal flipping were used to augment the training data and enrich the temporal diversity of the video sequences.
[0072] The training process is divided into two phases. In the initial phase, following the traditional deep learning paradigm, the two branches are trained independently. In the second phase, an iterative update strategy is adopted. First, the weights of the video branch are frozen, which are inherited from the slice classification model trained in the first phase. Then, gradient descent optimization is applied only to the CNN (image) branch. Subsequently, after reducing the learning rate, the two branches are trained simultaneously to jointly fine-tune the multimodal model.
[0073] The Adam optimizer was used for model optimization, with an initial learning rate of 1e-4, which was reduced to 1e-5 during the second stage of joint training. The cross-entropy loss function was used, and the area under the receiver operating characteristic (AUC) curve was employed as the primary metric for evaluating classification performance.
[0074] Furthermore, such as Figure 2 As shown, the process of fusing features obtained from the two branches includes:
[0075] Feature preprocessing and mapping:
[0076] Video feature vectorization: The feature tensor output by the video branch is denoted as... ,in For time step, For feature dimensions;
[0077] Spectral feature vectorization: The feature tensor output by the image branch (CNN) is denoted as... ;
[0078] Dimension alignment: achieved through a linear projection layer. and Mapping to the same semantic space ensures that the two have the basis for interaction.
[0079] Cross-modal attention unit:
[0080] Query generation: Spectral features As a query vector, the video's spatiotemporal features As key vectors and value vectors;
[0081] Attention calculation: This involves calculating the correlation matrix between spectral features and features in each frame of the video. This matrix represents the video features that best match the spectral features. This attention is then applied... Then you can get it directly. :
[0082] ;
[0083] Feature adaptive fusion: setting a learnable gating parameter The system adaptively adjusts the contribution ratio of video features and spectral features in the final classification decision. The final fused features... Represented as:
[0084] ;
[0085] Classification output: The fused features are input into the fully connected layer and the classifier, and the output is the intelligent recognition probability or category result of PS.
[0086] Furthermore, to improve training efficiency and generalization performance, this embodiment introduces a weight sharing mechanism in the time series model during the section recognition stage and the multimodal classification stage.
[0087] This embodiment constructs a nonlinear mapping and complementarity mechanism between spatiotemporal features of ultrasound video and features of Doppler spectral images through cross-modal joint feature extraction and deep fusion. This method solves the fusion challenge caused by dimensional and semantic differences in heterogeneous data, enabling effective alignment and synergistic enhancement of cardiac anatomical motion information and hemodynamic information in a deep feature space. This significantly improves the accuracy of the model in identifying and analyzing pulmonary valve stenosis and its robustness under different data qualities.
[0088] Implementably, based on the fusion features, the identification result of pulmonary valve stenosis is obtained, and a corresponding key region attention heatmap is generated and displayed on the original Doppler spectral image, specifically including:
[0089] The cross-attention weight matrix generated during the multimodal feature fusion stage is used to analyze the degree of attention of Doppler spectral features to different spatiotemporal regions in an ultrasound video frame. The weight matrix is upsampled and normalized to generate an attention map, which is then converted into a heatmap through pseudo-color mapping. The heatmap is then overlaid on the original ultrasound image or video for display.
[0090] By setting joint display control parameters, the display intensity and spatial focusing range of the heat map can be adjusted to achieve precise highlighting of key anatomical structures or blood flow areas in the PS under the guidance of hemodynamic characteristics (spectrum).
[0091] This embodiment generates and overlays attention heatmaps of key regions, transforming the attention weights of cross-modal feature interactions within the model into a visualized image region enhancement effect. This method intuitively maps the model's decision-making basis back to the original image, providing visual cues for key analytical regions (such as abnormal valve motion areas and abnormal blood flow spectrum areas), greatly improving the interpretability of the model output.
[0092] This embodiment, through the systematic integration of the aforementioned technical means, constructs an end-to-end intelligent analysis workflow, from automatic preprocessing of raw data to deep fusion of multimodal information, and then to simultaneous output of results and visualization. This workflow can serve as an efficient auxiliary tool, providing objective and consistent quantitative analysis support for ultrasound examinations, and helping to improve the efficiency of the overall workflow and the standardization of analysis results.
[0093] As an additional implementation method, echocardiographic data from children with pulmonary valve stenosis and normal controls were collected, and the data were anonymized and standardized. A multimodal deep learning model was trained using the described method, and its performance was validated on an independent test set. Experimental results show that the method achieves a high AUC value in the PS recognition task and can stably output attention heatmaps highly consistent with the pulmonary valve and blood flow acceleration areas.
[0094] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.
[0095] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.
[0096] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A feature fusion method for intelligent echocardiographic recognition of pulmonary valve stenosis, characterized in that, include: Echocardiographic videos of the pulmonary valve are acquired, and standard sections are automatically identified to obtain keyframe sequences corresponding to the target anatomical structures. Obtain Doppler spectral images of the pulmonary valve, and obtain target spectral images through target filtering processing; Based on the keyframe sequence and the target spectral image, fused features are obtained through cross-modal joint feature extraction and feature fusion; Based on the fusion features, the identification result of pulmonary valve stenosis is obtained, and the corresponding key region attention heatmap is generated and displayed on the original Doppler spectrum image; Based on the keyframe sequence and the target spectral image, fused features are obtained through cross-modal joint feature extraction and feature fusion, specifically including: Based on the keyframe sequence, time-space feature modeling is performed to obtain video time-space feature vectors; Based on the target spectral image, feature extraction is performed using a convolutional neural network to obtain the spectral image feature vector; Map the video spatiotemporal feature vector and the spectral image feature vector to the same dimension of semantic space; By using attention-based interaction, the mapped features are fused to obtain fused features; Mapping the video spatiotemporal feature vector and the spectral image feature vector to a semantic space of the same dimension specifically includes: Based on the video spatiotemporal feature vector and the spectral image feature vector, the data are processed through a linear projection layer to obtain dimension-aligned video mapping features and spectral mapping features. The mapped features are fused through attention-based interactions, specifically including: The mapped spectral image feature vector is used as the query vector, and the mapped video spatiotemporal feature vector is used as the key vector and value vector. Calculate the relevance weight matrix between the query vector and the key vector, and use the relevance weight matrix to weight the value vector to obtain attention-weighted video features; The attention-weighted video features and the mapped spectral image feature vector are weighted and summed to obtain the fused features.
2. The method according to claim 1, characterized in that, Acquire echocardiographic videos of the pulmonary valve, perform automatic standard section recognition, and obtain keyframe sequences corresponding to the target anatomical structures, specifically including: Based on pulmonary valve echocardiographic video, a continuous frame sequence is obtained through temporal sequence analysis; Based on the continuous frame sequence, a video section recognition model based on the Transformer architecture is used for classification processing to obtain the confidence score of each frame belonging to a predetermined standard section. The confidence score is compared with a preset threshold, and frames with confidence scores greater than or equal to the preset threshold are selected to obtain a key frame sequence.
3. The method according to claim 1, characterized in that, Obtain Doppler spectral images of the pulmonary valve, and then perform target filtering to obtain target spectral images, specifically including: The acquired Doppler spectrum image of the pulmonary valve is input into the convolutional neural network model; The convolutional neural network model is used to classify spectral images to identify and filter out the target spectral images for pulmonary valve hemodynamic analysis.
4. The method according to claim 1, characterized in that, The weighted summation is achieved through a learnable gating parameter, which adaptively adjusts the contribution ratio of the attention-weighted video features and the mapped spectral image feature vector in the fused features.
5. The method according to claim 1, characterized in that, Based on the fusion features, the identification result of pulmonary valve stenosis is obtained, and the corresponding key region attention heatmap is generated and displayed on the original Doppler spectral image, specifically including: Based on the attention weight matrix generated during the multimodal feature fusion stage, a two-dimensional attention map is obtained through upsampling and normalization. Based on the two-dimensional attention map, it is converted into a heatmap through pseudo-color mapping, and the heatmap is superimposed on the corresponding original Doppler spectrum image.
6. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-5.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-5.