Method for detecting coronary heart disease based on multi-modal fusion of heart sound signals and clinical data

Abstract

The present application relates to the technical field of medical signal processing and intelligent auxiliary diagnosis, in particular to a method for detecting coronary heart disease based on multi-modal fusion of heart sound signals and clinical data, which first collects heart sound signals and corresponding clinical data of a subject, and carries out filtering, denoising, normalization and segmentation processing on the heart sound signals to obtain a plurality of heart sound segments; then extracts heart sound spectrum graph features and heart sound professional statistical features, and uses a deep learning model to perform feature learning on the above features, and pre-processes the clinical data and obtains clinical deep features through a feature extraction network; on this basis, the heart sound features and the clinical features are jointly modeled through a cross-modal fusion mechanism, and a gating fusion strategy is introduced to generate multi-modal fusion features; finally, a classifier is used to discriminate the fusion features to obtain the coronary heart disease detection result.

Description

Technical Field

[0001] This invention relates to the field of medical signal processing and intelligent assisted diagnosis technology, and more specifically, to a method for jointly detecting coronary heart disease based on multimodal fusion of heart sound signals and clinical data. Background Technology

[0002] Coronary artery disease (CAD) is a common cardiovascular disease caused by narrowing or blockage of the coronary arteries due to atherosclerosis, leading to myocardial ischemia, hypoxia, and even necrosis. It is one of the diseases with the highest morbidity and mortality rates globally. With an aging population and the increase in risk factors such as hypertension, hyperlipidemia, and diabetes, the incidence of CAD is rising year by year. Early detection and timely intervention are crucial for reducing CAD-related complications and mortality. Therefore, how to achieve rapid and effective CAD screening in routine physical examinations or primary healthcare settings has become an important research focus in the field of medical engineering.

[0003] Current diagnostic methods for coronary heart disease primarily rely on medical examinations such as electrocardiograms (ECG), echocardiography, coronary CT angiography, and coronary angiography. While coronary angiography is considered the "gold standard" for diagnosing coronary heart disease, it is an invasive procedure, costly, and requires sophisticated equipment and medical resources, making it unsuitable for large-scale screening. Coronary CT angiography, although non-invasive, still presents challenges due to its high cost, radiation exposure, and reliance on specialized equipment. Therefore, large-scale initial screening for coronary heart disease using these methods is often difficult to implement in routine health monitoring or at primary healthcare facilities.

[0004] Heart sounds, as important physiological signals reflecting the mechanical activity of the heart, contain a variety of information, including the opening and closing of heart valves, hemodynamic changes, and vibrations of the heart chamber structures. When abnormalities occur in the heart's structure or blood flow, they often manifest as murmurs, changes in frequency distribution, or alterations in waveform characteristics in the heart sound signal. Therefore, using heart sound signals for auxiliary detection of heart diseases has advantages such as being non-invasive, low-cost, and easy to acquire, and has gradually become an important direction in intelligent healthcare and digital auscultation research in recent years. However, in practical applications, relying solely on heart sound signals for disease differentiation still has certain limitations. On the one hand, heart sound signals are easily affected by factors such as environmental noise, acquisition location, and individual differences, leading to insufficient characteristic stability; on the other hand, some patients with coronary heart disease do not show obvious changes in heart sounds in the early stages, making it difficult to obtain sufficiently reliable diagnostic criteria by simply relying on heart sound signal analysis.

[0005] Meanwhile, numerous clinical studies have shown that the occurrence and development of coronary heart disease (CHD) are closely related to various clinical indicators such as age, blood pressure, blood lipid levels, and past medical history. If these clinical information can be jointly analyzed with the acoustic features of heart sound signals, it may be possible to more comprehensively reflect the cardiovascular status of the examinee, thereby improving the accuracy of the test results. However, current methods for joint modeling of heart sound signals and clinical data are still relatively limited. Most methods still rely on single-modal data or simply splice multi-source data, failing to fully explore the correlations between different information sources. Therefore, how to construct a CHD detection method that can simultaneously utilize heart sound signal features and clinical indicator information and perform joint analysis through multimodal fusion has become an urgent technical problem to be solved.

[0006] Therefore, we urgently need to design a method for detecting coronary heart disease based on multimodal fusion of heart sound signals and clinical data to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to solve the technical problems mentioned in the background section and to provide a method for joint detection of coronary heart disease based on multimodal fusion of heart sound signals and clinical data.

[0008] The above-mentioned objective of the present invention is achieved through the following technical solution: A method for detecting coronary heart disease based on multimodal fusion of heart sound signals and clinical data is characterized by the following steps: S1. Collect the raw heart sound signals and corresponding clinical data of the subject, wherein the clinical data includes age, gender, systolic blood pressure, diastolic blood pressure, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol and medical history information; S2. Perform signal preprocessing on the raw heart sound signal, including bandpass filtering, noise reduction, signal normalization, and signal segmentation, to obtain standardized heart sound signal segments. S3. Perform time-frequency conversion on the preprocessed heart sound signal segments to construct a heart sound Mel spectrogram, and extract features from the Mel spectrogram using a convolutional neural network to obtain the heart sound spectrum feature vector; S4. Extract heart sound professional features from the preprocessed heart sound signal segments, including time domain statistical features, frequency domain energy features and zero-crossing rate features, and use a multilayer perceptron to perform deep feature extraction on the heart sound professional features to obtain a heart sound professional feature vector. S5. Perform data preprocessing on the clinical data, including numerical feature standardization and categorical feature encoding, and extract clinical feature representations using a clinical feature extraction network based on the Transformer structure. S6. Construct a multimodal fusion model by inputting the cardiac audio spectrogram feature vector, cardiac audio professional feature vector, and clinical feature representation into the cross-modal fusion module for feature fusion; S7. Use the fused multimodal features as input to the classifier to determine the risk of coronary heart disease and output the coronary heart disease detection results.

[0009] As a preferred technical solution of the present invention, the bandpass filtering in step S2 is performed using a Butterworth filter with a low cutoff frequency of 20Hz and a high cutoff frequency of 1000Hz, in order to remove low-frequency baseline drift and high-frequency noise.

[0010] As a preferred technical solution of the present invention, the denoising process in step S2 adopts the wavelet denoising method, which uses Daubechies wavelet to perform multi-scale decomposition of the heart sound signal, and processes the wavelet coefficients through a soft threshold function, and then reconstructs the denoised heart sound signal.

[0011] As a preferred technical solution of the present invention, the signal segmentation process in step S2 divides the heart sound signal into sliding windows according to a preset time window. The window length is 5 seconds and the window overlap rate is 50%, thereby obtaining multiple heart sound signal segments.

[0012] As a preferred technical solution of the present invention, the Mel spectrum in step S3 is calculated by short-time Fourier transform with a sampling rate of 2000Hz, an FFT window length of 512, a window step size of 128, and the spectrum is mapped to the Mel scale to form the Mel spectrum.

[0013] As a preferred technical solution of the present invention, the convolutional neural network in step S3 includes an initial convolutional layer, a residual convolutional module, and an adaptive average pooling layer. The residual convolutional module includes a batch normalization layer, a ReLU activation function, and an SE attention module to enhance the ability to express key spectral features.

[0014] As a preferred technical solution of the present invention, the professional features of heart sounds in step S4 include: heart sound time domain statistical features, heart sound audio domain energy features, signal mean, signal standard deviation, signal skewness, signal kurtosis, and signal zero-crossing rate.

[0015] As a preferred technical solution of the present invention, the clinical feature extraction network in step S5 is an FT-Transformer model, which includes a feature embedding layer, a Transformer encoder and a feedforward neural network, used to learn the nonlinear correlation between clinical features.

[0016] As a preferred technical solution of the present invention, the cross-modal fusion module in step S6 adopts a cross-modal attention mechanism, using cardiac sound spectral features as query vectors and clinical features as keys and values ​​for attention calculation, thereby realizing the association modeling between cardiac sound features and clinical information.

[0017] As a preferred technical solution of the present invention, the classifier in step S7 is a fully connected neural network classifier. The classifier outputs the probability result of coronary heart disease through the Softmax function and gives the coronary heart disease detection conclusion according to the preset threshold.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Compared to existing coronary heart disease detection methods that rely solely on single heart sound signal analysis or clinical indicators, this invention simultaneously incorporates time-frequency structure information from heart sound signals, professional statistical information on heart sounds, and clinical background information of the examinee to jointly model the pathological features related to coronary heart disease. Since the heart sound spectrum branch can characterize the changes in heart sound signals across different time and frequency dimensions, the professional heart sound feature branch can supplement the reflection of waveform statistical regularities and frequency band energy distribution, and the clinical feature branch can reflect individual differences and risk factor backgrounds of the examinee, this invention overcomes the problems of insufficient information and one-sided characterization in single modalities, making the detection results more comprehensive. It is particularly suitable for scenarios where early abnormal signs of coronary heart disease are weak and single indicators are atypical, thereby improving the accuracy and reliability of auxiliary detection of coronary heart disease.

[0019] 2. In terms of technical implementation, this invention filters, denoises, normalizes, and segments the heart sound signal, then extracts the deep features of the spectrogram and the professional features of the heart sound, and combines them with clinical data for cross-modal fusion and gated weighting, forming a complete and clear data processing link. Compared with the existing technology of simply splicing multi-source data directly into a classifier, this invention can establish a correspondence between heart sound features and clinical information at the feature level, so that clinical data not only exists as supplementary information, but can also participate in the interpretation and enhancement of abnormal heart sound patterns. At the same time, by suppressing redundant information and noise interference through the gated fusion mechanism, the robustness and stability of the model under different acquisition environments and individual conditions can be enhanced, and the false detection rate and false negative rate can be reduced.

[0020] 3. This invention has strong engineering feasibility and clinical application value. The data sources it relies on are all relatively conventional. Heart sound signals can be collected through portable devices such as electronic stethoscopes, and clinical data can be directly obtained from routine physical examinations, outpatient consultations, or electronic medical records. Therefore, the overall implementation cost is low, making it easy to deploy and use in primary healthcare institutions, physical examination centers, remote health monitoring platforms, and intelligent auxiliary screening systems. Through this invention, without increasing the burden of invasive examinations, rapid, automated, and intelligent auxiliary identification of coronary heart disease risk can be performed on examinees, providing a basis for whether to further conduct coronary CTA, coronary angiography, and other examinations. Thus, it has good clinical auxiliary decision-making significance and practical application prospects. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method for jointly detecting coronary heart disease based on multimodal fusion of heart sound signals and clinical data proposed in this invention; Figure 2 This is a detailed structural diagram of the joint detection model of heart sound signals and clinical data based on multimodal fusion according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with embodiments and appendices. Figures 1-2 The present invention will be further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] Example 1: This example uses coronary heart disease auxiliary detection as an application scenario, performing joint analysis of the subject's heart sound signals and clinical data. Unlike detection methods that rely solely on a single heart sound signal or a single clinical indicator, this invention unifies the time-frequency information, statistical information, and clinical indicator information in the heart sound into a unified model, and outputs the coronary heart disease discrimination results through multimodal fusion, thereby improving the accuracy, stability, and practical clinical applicability of the detection results.

[0024] In practice, the first step is to establish a data collection process for each subject. For each subject, raw heart sound signals from the precordial region are collected, along with corresponding clinical data. Heart sound signals can be acquired using an electronic stethoscope, a digital heart sound acquisition device, or other equipment with heart sound acquisition capabilities. During the collection process, the subject should be kept as quiet as possible to minimize body movement noise and environmental interference. The clinical data includes at least age, gender, systolic blood pressure, diastolic blood pressure, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, and past medical history. After collection, the heart sound signals and clinical data are linked according to the same subject identity, forming a complete set of input data required for subsequent combined testing.

[0025] After the initial data acquisition, the heart sound signals were preprocessed. Since the original heart sound signals are often contaminated with low-frequency baseline drift, respiratory interference, friction noise, and some high-frequency environmental noise during acquisition, bandpass filtering was first applied to retain the main frequency components related to the heart sounds and remove interference signals that significantly deviate from the effective frequency range of the heart sounds. After bandpass filtering, the filtered signal was then denoised, preferably using wavelet denoising to reduce the impact of random noise while preserving the main waveform characteristics of the heart sounds. Subsequently, to reduce amplitude fluctuations among different subjects due to differences in acquisition equipment gain, individual chest wall conditions, and acquisition intensity, the denoised heart sound signals were further normalized to ensure that the processed signals were on a uniform amplitude scale.

[0026] After the heart sound signal is filtered, denoised, and normalized, it is segmented. In this embodiment, a sliding time window method is used to divide the continuous heart sound signal into multiple heart sound segments of equal length. The window length and overlap ratio can be preset according to actual needs, such as using a fixed duration window with partial overlap to improve the coverage of short-term pathological features. After segmentation, the original long-term heart sound signal of each subject is converted into several short-term heart sound segments. These segments not only retain local heart sound structure information but also facilitate unified input and batch processing for subsequent models.

[0027] After obtaining the heart sound fragment, the heart sound feature extraction stage begins. The heart sound feature extraction of this invention includes two parallel paths, corresponding to heart sound spectrum feature extraction and heart sound professional feature extraction, respectively. The two paths are based on the same heart sound fragment, thereby ensuring that the two types of features have consistent sources and accurate correspondence.

[0028] The first path is the cardiac audio spectrogram feature extraction path. For each cardiac sound segment, a time-frequency transformation is first performed to convert the original time-domain waveform into a time-frequency representation that reflects the frequency change over time. To better conform to the laws of auditory perception and highlight the energy distribution characteristics of different frequency bands in the cardiac sound, this embodiment further maps the time-frequency representation to a Mel spectrogram. In this way, cardiac sound segments can be converted into two-dimensional time-frequency images, making the noise, abnormal vibration components, and spectral structure changes in the cardiac sound more intuitive. Then, the Mel spectrogram is input into a convolutional neural network for deep feature extraction. The convolutional neural network preferably includes an initial convolutional layer, several residual feature extraction modules, and pooling layers. The residual modules can employ batch normalization, activation functions, and attention mechanisms to enhance the representation ability of key frequency bands. After processing through this path, the deep features of the cardiac audio spectrogram corresponding to each cardiac sound segment can be obtained, which are used to represent the abnormal patterns of the segment at the time-frequency level.

[0029] The second path is the heart sound professional feature extraction path. This path does not treat heart sound segments as images, but directly extracts feature information with clear physical and statistical significance from the heart sound segments themselves. In this embodiment, the extracted heart sound professional features include at least time-domain statistical features, frequency-domain energy features, and zero-crossing rate features. Among them, time-domain statistical features can include mean, standard deviation, skewness, kurtosis, etc., to reflect the amplitude distribution and dispersion of heart sound waveforms; frequency-domain energy features are used to describe the energy distribution in different frequency bands to characterize the difference in frequency structure between normal heart sounds and pathological murmurs; zero-crossing rate features are used to reflect the speed of signal waveform changes and the degree of oscillation. After these professional features are extracted, they are input into a multilayer perceptron for further mapping to obtain more compact, abstract, and suitable heart sound professional depth features for fusion with other modalities. Thus, for each heart sound segment, the system has obtained two complementary heart sound representations: one is the deep time-frequency features from the spectrogram branch, and the other is the statistical and energy features from the professional feature branch.

[0030] Simultaneous with the heart sound signal processing is the clinical data processing. Since clinical data typically contains both numerical and categorical variables, preprocessing is performed first. Numerical clinical features such as age, blood pressure, and blood lipids are standardized to reduce the impact of different units and value ranges on model training. Categorical clinical features such as gender and medical history are encoded to convert them into numerical forms that the model can directly recognize. After preprocessing, the clinical data is input into the clinical feature extraction network. In this embodiment, a Transformer structure based on feature embedding is preferably used, leveraging its strong ability to model relationships between multiple variables to learn higher-order associations and combination patterns between different clinical indicators. After passing through this network, a clinical depth feature representation corresponding to each subject is obtained. Since this clinical depth feature is generated at the overall subject level, it will be paired one-to-one with each heart sound segment of that subject for subsequent joint analysis.

[0031] After obtaining the deep features of cardiac audio spectrograms, professional cardiac sound features, and clinical features, the process proceeds to the multimodal fusion model stage. To more clearly illustrate the structure of the multimodal fusion model constructed in this invention, Figure 2 A detailed structural diagram of the joint detection model of heart sound signals and clinical data based on multimodal fusion of the present invention is shown.

[0032] like Figure 2As shown, the coronary artery disease detection model of this invention mainly includes a heart sound Mel-ray spectrogram feature extraction branch, a heart sound professional feature extraction branch, and a clinical feature extraction branch. The heart sound Mel-ray spectrogram feature extraction branch is used for deep feature extraction from the heart sound Mel-ray spectrogram; the heart sound professional feature extraction branch is used for feature learning of heart sound professional features such as statistical features and frequency band energy features; and the clinical feature extraction branch is used for encoding and feature representation of the subject's clinical indicator information. The features obtained from the three branches are then input into a cross-modal attention mechanism module, where the feature correlation between different modalities is modeled to achieve collaborative fusion between heart sound information and clinical information. The fused features are further input into a post-fusion processing module for comprehensive processing, and finally, a classifier outputs the coronary artery disease prediction result.

[0033] Because the three types of features have different sources, dimensions, and expressive focuses, simply splicing them together often fails to fully reflect the correlation between heart sounds and clinical findings. Therefore, this invention introduces a cross-modal fusion mechanism. First, the deep features of the heart sound spectrogram and the deep features of the clinical features are mapped to a unified feature space, giving them a comparable and interactive representational basis. Then, using a cross-modal attention mechanism, based on the heart sound spectrogram features, the model is guided to focus on clinical information with a higher degree of matching, thereby generating cross-modal features enhanced with clinical information. The purpose of this process is to avoid clinical data existing merely as independent supplementary information, but rather to enable it to play a guiding and corrective role in the recognition of abnormal heart sound patterns.

[0034] After cross-modal attention enhancement, the deep features of cardiac phonation spectrograms, the professional deep features of cardiac sounds, and the cross-modal enhancement features are combined to form multimodal joint features. To further suppress noise features and highlight key modalities, a gating fusion mechanism is introduced in this embodiment. Through the gating structure, different importance weights are automatically assigned to features of different dimensions and modalities, thereby reducing the interference of redundant and unstable information on the final discrimination result. After gating fusion, the final fused feature corresponding to each cardiac sound segment is obtained. This feature simultaneously includes cardiac sound time-frequency information, cardiac sound statistical information, and clinical background information, which can more comprehensively reflect whether the examinee has pathological manifestations related to coronary heart disease.

[0035] After obtaining the final fused features, they are input into a classifier for segment-level prediction. The classifier preferably uses a fully connected neural network to output the probability of each heart sound segment being positive or negative for coronary artery disease (CAD). Since the same subject is often segmented into multiple heart sound segments, the prediction result of a single segment may be affected by local noise or segment differences. To improve the stability of subject-level determination, this invention aggregates the prediction results of multiple segments for the same subject, preferably using an average aggregation method, to obtain the overall CAD prediction probability for that subject. Finally, this overall prediction probability is compared with a preset threshold. When the prediction probability is higher than the threshold, a positive CAD detection result is output; when the prediction probability is lower than the threshold, a negative CAD detection result is output. This combination of segment-level discrimination and subject-level aggregation effectively reduces the risk of misjudgment caused by a single abnormal or noisy segment.

[0036] During the model training phase, a training sample set consisting of multiple subjects is first constructed. Each sample contains paired heart sound signals, clinical data, and true labels for coronary artery disease. Then, following the steps described above, heart sound preprocessing, segmentation, two-branch heart sound feature extraction, clinical feature extraction, multimodal fusion, and classification output are completed. During training, based on the difference between the model output and the true labels, the model parameters are iteratively optimized using a loss function. The parameters in the heart sound spectrogram branch, heart sound specialty feature branch, clinical feature extraction branch, cross-modal fusion module, gated fusion module, and classifier are jointly updated until the model converges. After training, a joint discriminant model that can be used for auxiliary detection of coronary artery disease is obtained.

[0037] In practical applications, for new subjects, only their heart sound signals and clinical data need to be collected, and reasoning can be performed according to the same processing flow as in the training phase to automatically output the coronary heart disease detection results. The entire implementation process is carried out in the following order: "data acquisition - heart sound preprocessing - heart sound segmentation - heart sound dual-branch feature extraction - clinical data preprocessing and feature extraction - cross-modal fusion - gated fusion - segment-level classification - subject-level result aggregation - output detection results". The steps are clearly connected, the data flow is complete, and a closed-loop coronary heart disease joint detection process can be formed.

[0038] In summary, this embodiment overcomes the problems of single heart sound detection being easily affected by noise and single clinical indicators failing to accurately reflect the disease state by unifying the time-frequency representation, heart sound statistics and energy characteristics, and clinical indicator information in the heart sound signal. This makes the present invention more robust and more accurate in the auxiliary detection of coronary heart disease, while also having good engineering feasibility and clinical promotion value.

[0039] Example 2: In this embodiment, heart sound signals and clinical data are simultaneously collected for each subject, and these two are used as paired inputs for the same subject. Let the first... If the original heart sound signal of the subject was dated January 19, 2000, then: ; in, Indicates the first The number of examinees on the 1st The original heart sound signal amplitude at each sampling point; Indicates the examinee's serial number; Indicates the sampling point number; Indicates the first The total number of sampling points for the raw heart sound signals of each subject. Let the corresponding raw clinical data vector be: ; in, Indicates the first Original clinical data vectors of the subjects; Indicates the first The first subject of the examination The values ​​of each clinical variable; Indicates the clinical variable number; This represents the total number of clinical variables. These clinical variables include age, sex, systolic blood pressure, diastolic blood pressure, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, and medical history.

[0040] First, the original heart sound signal is bandpass filtered to obtain the filtered heart sound signal: ; in, Indicates the first Heart sound signals of the subjects after bandpass filtering; This represents the bandpass filter operator. In this embodiment, the bandpass filter frequency range is set to 20Hz to 1000Hz to filter out low-frequency baseline drift and high-frequency environmental noise.

[0041] Then, wavelet denoising is performed on the filtered signal. conduct Layer discrete wavelet decomposition yields the th Layer detail factor And perform soft thresholding on it: ; in, Indicates the first The number of examinees on the 1st Detail coefficients after soft thresholding following layer decomposition; Indicates the first The number of examinees on the 1st The first layer obtained by layer decomposition Detail coefficients; Indicates the wavelet decomposition layer number; Indicates the coefficient sequence number; Indicates the first The number of examinees on the 1st Wavelet threshold of layer; sgn Represents a symbolic function; This indicates the operation of taking the absolute value.

[0042] The denoised heart sound signal was then obtained through wavelet reconstruction. ; in, Indicates the first Heart sound signals of the subjects after wavelet denoising; This represents the inverse wavelet transform operator; The meaning is the same as the previous formula.

[0043] To eliminate amplitude differences under different acquisition conditions, the denoised heart sound signal is standardized: ; in, Indicates the first Standardized heart sound signals of the subjects; Indicates the first Heart sound signals of the subjects after wavelet denoising; Indicates the first Denoising heart sound signals of the subjects The mean; Indicates the first Denoising heart sound signals of the subjects Standard deviation; This represents a very small positive number that prevents the denominator from being zero.

[0044] The standardized heart sound signal was segmented using a sliding window method to obtain heart sound segments: ; in, Indicates the first The first subject of the examination A heart sound fragment; Indicates the sequence number of the heart sound segment; Indicates the sliding step size between adjacent segments; Indicates the length of a single heart sound segment; Indicates the first The standardized heart sound signals of the subjects; the items in square brackets represent the continuous sampling points that constitute this heart sound segment.

[0045] For each heart sound segment Performing a short-time Fourier transform yields the complex spectrum: ; in, Indicates the first The first subject of the examination The heart sound fragment in the first Frame, First Short-time Fourier transform coefficients at each frequency index; Indicates frequency index; Indicates the time frame index; Indicates the sampling point number within the window; Indicates the Fourier transform window length; Indicates the frame shift step size; The window function is in the first position. The values ​​at each point; It represents the imaginary unit.

[0046] Calculate the power spectrum based on the complex spectrum: ; in, Indicates the first The first subject of the examination The heart sound fragment in the first Frame, First Power spectrum values ​​at each frequency index; This represents the short-time Fourier transform coefficients at the corresponding positions.

[0047] Then, the power spectrum is mapped to a Mel spectrum using a Mel filter bank: ; in, Indicates the first The first subject of the examination The heart sound fragment in the first The first Mel filter, the first Mel spectrum value at the frame; Indicates the Mel filter number; Indicates the first The Mel filter for the first Filter weights for each frequency index; Indicates the total number of frequency indices; This represents the power spectrum value.

[0048] To compress the dynamic range, a logarithmic transformation is performed on the Mel spectrogram: ; in, Indicates the first The first subject of the examination Log-Mel spectrum values ​​corresponding to each heart sound segment; Indicates Mel spectrogram values; Represents the natural logarithm operation; This indicates that the logarithmic input should be a very small positive number to prevent it from being zero.

[0049] The log-Melogram is input into a convolutional neural network to extract deep features from the cardiac audio spectrogram: ; in, Indicates the first The first subject of the examination The deep feature vector of the cardiac phonograph corresponding to each cardiac phonograph segment; This represents the feature extraction mapping of a convolutional neural network; Indicates the first The first subject of the examination The overall log-Mel spectrogram corresponding to each heart sound segment.

[0050] At the same time, from the same heart sound fragment Extract the professional features of heart sounds to construct a professional feature vector: ; in, Indicates the first The first subject of the examination The heart sound feature vector corresponding to each heart sound segment; Indicates the first Each professional characteristic component; This represents the total number of specific heart sound characteristics. This represents the sequence number of the professional feature components. The segment mean is defined as follows: ; in, Indicates the first The first subject of the examination Mean characteristics of each heart sound segment; Indicates the length of a single heart sound segment; Indicates the first The first subject of the examination The first heart sound segment The amplitude of each sampling point.

[0051] The standard deviation of a segment is defined as: ; in, Indicates the first The first subject of the examination The standard deviation characteristics of individual heart sound segments; Indicates the first The amplitude of each sampling point; This indicates the mean characteristics of the heart sound segment; This indicates the length of a single heart sound segment.

[0052] Fragment skewness is defined as: ; in, Indicates the first The first subject of the examination Skewness characteristics of individual heart sound segments; Indicates the first The amplitude of each sampling point; This indicates the mean characteristics of the heart sound segment; This indicates the standard deviation characteristic of the heart sound segment; This represents a very small positive number that prevents the denominator from being zero.

[0053] Fragment kurtosis is defined as: ; in, Indicates the first The first subject of the examination Kuness characteristics of a heart sound segment; Indicates the first The amplitude of each sampling point; This indicates the mean characteristics of the heart sound segment; This indicates the standard deviation characteristic of the heart sound segment; This represents a very small positive number that prevents the denominator from being zero.

[0054] The zero-crossing rate of a fragment is defined as: ; in, Indicates the first The first subject of the examination Zero-crossing rate characteristics of individual heart sound segments; This indicates an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise. and These represent the amplitudes of two adjacent sampling points, respectively. This indicates the length of a single heart sound segment.

[0055] The frequency band energy characteristic is defined as: ; in, Indicates the first Energy characteristics corresponding to each frequency band; Indicates the frequency band number; Indicates the first A set of frequency indices corresponding to each frequency band; Indicates the total number of time frames; Indicates the first The first subject of the examination The heart sound fragment in the first The frequency index, the first Power spectral density value at the frame; This indicates the total number of frequency bands.

[0056] Inputting the heart sound-specific feature vector into a multilayer perceptron yields heart sound-specific deep features: ; in, Indicates the first The first subject of the examination The professional deep feature vector of heart sound corresponding to each heart sound segment; Represents a multilayer perceptron mapping; This represents the heart sound feature vector of this heart sound segment.

[0057] On the other hand, the original clinical data vector is preprocessed to obtain the clinical input vector: ; in, Indicates the first Clinical input vectors of each subject after preprocessing; This represents the clinical data preprocessing mapping, including the standardization of numerical variables and the coding of categorical variables; This represents the original clinical data vector.

[0058] The preprocessed clinical input vector is mapped to a clinical feature embedding sequence: ; in, Indicates the first Clinical characteristics of the subjects were embedded in the sequence; Indicates the first Embedding vectors corresponding to each clinical variable; This represents the total number of clinical variables.

[0059] The clinical feature embedding sequence is input into the FT-Transformer network to obtain clinical deep features: ; in, Indicates the first Clinical depth feature vectors of each subject; This represents the FT-Transformer feature extraction mapping; This represents the embedded sequence of clinical features.

[0060] In the self-attention calculation of FT-Transformer, the query matrix, key matrix, and value matrix are defined as follows: ; in, Indicates the first Query matrix of examinees; Indicates the first Key matrix of subjects; Indicates the first A matrix of values ​​for the subjects; , , These represent the query mapping weight matrix, the key mapping weight matrix, and the value mapping weight matrix, respectively. This represents the embedded sequence of clinical features.

[0061] The output of self-attention is: ; in, Indicates the first The self-attention output matrix of the subjects; softmax(·) represents the normalized exponential function; This represents the relevance matrix obtained by multiplying the query matrix by the transpose of the key matrix; Indicates the clinical embedding dimension; Represents a value matrix; This indicates the matrix transpose.

[0062] Since clinical data is subject-level information, while heart sound signals are segment-level information, for the same subject, their clinical depth characteristics should be analyzed separately. With the Deep features of cardiac spectral maps corresponding to each heart sound segment and the professional depth characteristics of heart sounds To merge. First, [the following will be done]. and Projected into the same fusion space: ; in, Indicates the first The first subject of the examination Feature vectors of projected cardiac sound spectrum corresponding to each cardiac sound segment; Indicates the first The projected clinical feature vectors corresponding to each subject; Represents the feature projection matrix of the cardiac audio spectrogram; Represents the projection matrix of clinical features; This represents the depth feature vector of the cardiac audio spectrogram; This represents the clinical depth feature vector.

[0063] By calculating the enhancement weights of clinical information on cardiac audio spectrogram features through cross-modal attention, then: ; in, Indicates the first The first subject of the examination Cross-modal attention weights corresponding to each heart sound segment; This indicates the correlation between the characteristics of the projected cardiac audio spectrogram and the clinical characteristics after projection. Indicates the fusion spatial dimension; softmax This indicates that the normalized exponential function yields cross-modal enhancement features: ; in, Indicates the first The first subject of the examination Cross-modal enhanced feature vectors corresponding to each heart sound segment; Represents cross-modal attention weights; This represents the clinical feature vector after projection.

[0064] The projected cardiac audio spectrogram features, cardiac audio professional depth features, and cross-modal enhancement features are concatenated to obtain multimodal joint features, namely: ; in, Indicates the first The first subject of the examination Multimodal joint feature vectors corresponding to each heart sound segment; This represents the feature vector of the projected cardiac audio spectrum; Represents the professional-level deep feature vector of heart sounds; This indicates a cross-modal augmentation feature vector; the symbol "" indicates a vector concatenation operation.

[0065] To suppress redundant information and highlight key modes, a gating fusion mechanism is introduced, and the gating weights are calculated: ; in, Indicates the first The first subject of the examination The gating weight vector corresponding to each heart sound segment; This represents the Sigmoid activation function; This represents the gate layer weight matrix; Represents the gate layer bias vector; This represents the joint feature vector of multimodal features.

[0066] The final fused features are obtained based on the gating weights: ; in, Indicates the first The first subject of the examination The final fused feature vector corresponding to each heart sound segment; Represents the gating weight vector; Represents the joint feature vector of multimodal features; This indicates element-wise multiplication.

[0067] The final fused features are input into the classifier to obtain the fragment-level coronary artery disease prediction probability: ; in, Indicates the first The first subject of the examination The classification probability vector corresponding to each heart sound segment; This represents the output layer weight matrix; This represents the output layer bias vector; Let represent the final fused feature vector; softmax(·) represents the normalization exponential function. ,but For the first The first subject of the examination The probability that a heart sound segment is diagnosed as not having coronary artery disease. The probability of being diagnosed with coronary heart disease.

[0068] For multiple heart sound segments from the same subject, the segment-level coronary artery disease probabilities are averaged and aggregated to obtain the subject-level final predicted probability: ; in, Indicates the first The final predicted probability of each examinee being diagnosed with coronary heart disease; Indicates the first The total number of valid heart sound segments from each subject; Indicates the first The first subject of the examination The probability of a heart sound segment being diagnosed as coronary heart disease.

[0069] Set the classification threshold to The final judgment result for the examinee is: ; in, Indicates the first The final classification results of the examinees; when When it is determined to be a positive result for coronary heart disease; when The time indicates a negative result for coronary heart disease; Indicates the final predicted probability at the subject level; This represents the classification threshold.

[0070] During the model training phase, let the... The true label of the examinee is Then, the binary cross-entropy loss function is used for parameter optimization: ; in, This represents the model training loss function; This indicates the total number of subjects tested in the training set; Indicates the first The real labels of the test subjects, when A score of 1 indicates a true positive result for coronary heart disease. The time indicates that the actual result is negative for coronary heart disease; Indicates the first The final predicted probability of each subject; Represents the natural logarithm operation.

[0071] By minimizing the loss function By jointly updating the trainable parameters in the convolutional neural network, multilayer perceptron, FT-Transformer, cross-modal attention module, gating fusion module, and classifier, joint modeling of heart sound time-frequency information, heart sound professional features, and clinical data can be achieved, thereby improving the accuracy and stability of coronary heart disease detection.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for jointly detecting coronary heart disease based on multimodal fusion of heart sound signals and clinical data, characterized in that, Includes the following steps: S1. Collect the raw heart sound signals and corresponding clinical data of the subject, wherein the clinical data includes age, gender, systolic blood pressure, diastolic blood pressure, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol and medical history information; S2. Perform signal preprocessing on the raw heart sound signal, including bandpass filtering, noise reduction, signal normalization, and signal segmentation, to obtain standardized heart sound signal segments. S3. Perform time-frequency conversion on the preprocessed heart sound signal segments to construct a heart sound Mel spectrogram, and extract features from the Mel spectrogram using a convolutional neural network to obtain the heart sound spectrum feature vector; S4. Extract heart sound professional features from the preprocessed heart sound signal segments, including time domain statistical features, frequency domain energy features and zero-crossing rate features, and use a multilayer perceptron to perform deep feature extraction on the heart sound professional features to obtain a heart sound professional feature vector. S5. Perform data preprocessing on the clinical data, including numerical feature standardization and categorical feature encoding, and extract clinical feature representations using a clinical feature extraction network based on the Transformer structure. S6. Construct a multimodal fusion model by inputting the cardiac audio spectrogram feature vector, cardiac audio professional feature vector, and clinical feature representation into the cross-modal fusion module for feature fusion; S7. Use the fused multimodal features as input to the classifier to determine the risk of coronary heart disease and output the coronary heart disease detection results.

2. The method according to claim 1, characterized in that, The bandpass filtering in step S2 is performed using a Butterworth filter with a low cutoff frequency of 20Hz and a high cutoff frequency of 1000Hz to remove low-frequency baseline drift and high-frequency noise.

3. The method according to claim 1, characterized in that, The denoising process in step S2 uses wavelet denoising, which uses Daubechies wavelet to decompose the heart sound signal into multiple scales and processes the wavelet coefficients through a soft threshold function. The denoised heart sound signal is then reconstructed.

4. The method according to claim 1, characterized in that, In step S2, the signal segmentation process divides the heart sound signal into sliding windows according to a preset time window. The window length is 5 seconds and the window overlap rate is 50%, thereby obtaining multiple heart sound signal segments.

5. The method according to claim 1, characterized in that, The Mel spectrogram in step S3 is calculated by short-time Fourier transform with a sampling rate of 2000 Hz, an FFT window length of 512, and a window step size of 128. The spectrum is then mapped to the Mel scale to form the Mel spectrogram.

6. The method according to claim 1, characterized in that, The convolutional neural network in step S3 includes an initial convolutional layer, a residual convolutional module, and an adaptive average pooling layer. The residual convolutional module includes a batch normalization layer, a ReLU activation function, and an SE attention module to enhance the ability to express key spectral features.

7. The method according to claim 1, characterized in that, The heart sound characteristics in step S4 include: heart sound time-domain statistical characteristics, heart sound audio-domain energy characteristics, signal mean, signal standard deviation, signal skewness, signal kurtosis, and signal zero-crossing rate.

8. The method according to claim 1, characterized in that, The clinical feature extraction network in step S5 is an FT-Transformer model, which includes a feature embedding layer, a Transformer encoder, and a feedforward neural network, used to learn the nonlinear relationships between clinical features.

9. The method according to claim 1, characterized in that, The cross-modal fusion module in step S6 adopts a cross-modal attention mechanism, using cardiac sound spectral features as query vectors and clinical features as keys and values ​​for attention calculation, thereby realizing the association modeling between cardiac sound features and clinical information.

10. The method according to claim 1, characterized in that, The classifier in step S7 is a fully connected neural network classifier. The classifier outputs the probability result of coronary heart disease through the Softmax function and gives the coronary heart disease detection conclusion according to the preset threshold.

Images