Heart sound-electrocardio multi-modal fusion detection method based on federated learning

Abstract

The invention discloses a heart sound-electrocardio multi-modal fusion detection method based on federal learning. According to the method, collaborative fusion of multi-modal information of heart sound and electrocardio data is achieved through a cross-modal collaborative attention mechanism, the comprehensive performance of federal learning is effectively improved, and the detection accuracy of a federal model is remarkably improved; meanwhile, by introducing a weight distribution mechanism, the adaptability of the model to the imbalance of each modal data is enhanced; a differential privacy protection technology is introduced to enhance the safe aggregation of model parameters; and the method has better robustness and privacy protection in practical application.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence medical technology, and in particular to a heart sound-electrocardiogram multimodal fusion detection method based on federated learning. Background Art

[0002] As the incidence of cardiovascular diseases increases year by year, cardiovascular health monitoring and early screening have become important research directions in the medical field. Figure 1 As shown in Figure 1, phonocardiogram (PCG) and electrocardiograph (ECG) are two widely used physiological signals that can provide rich information about the heart's state. In medical testing, heart sounds reflect the mechanical activity of the heart, while electrocardiograms reflect the electrical activity of the heart. Figure 1 As shown in the figure, one cardiac cycle can produce four heart sounds: the first heart sound (S1), the second heart sound (S2), the third heart sound (S3) and the fourth heart sound (S4), among which S1 is mainly composed of the mitral valve component (M1) and the tricuspid valve component (T1), and S2 is mainly produced by the closure of the aortic valve (A2) and the closure of the pulmonary valve (P2). One cardiac cycle describes the electrical activity process of the heart during one contraction and relaxation, among which, P wave: atrial depolarization, reflecting the electrical activity during atrial contraction; Q wave: the first negative wave in the QRS complex, reflecting the early stage of ventricular depolarization; R wave: the positive wave in the QRS complex, which is the main waveform of ventricular depolarization; S wave: the negative wave in the QRS complex, the final stage of ventricular depolarization; T wave: the waveform representing ventricular repolarization, reflecting that the ventricle has returned to a resting state. There is potential complementarity between the phonocardiogram and the electrocardiogram. Therefore, multimodal data fusion based on heart sounds and electrocardiograms can provide more comprehensive and accurate information for early screening and detection of cardiovascular diseases.

[0003] Currently, multimodal data fusion technology has been widely used in the medical field, especially for automated analysis using artificial intelligence (AI) algorithms such as deep learning. However, due to the sensitivity of medical data and the need to protect patient privacy, traditional centralized machine learning methods face great challenges in application. Centralized learning usually requires that data from different medical institutions be centralized in a central server for processing and model learning, which not only increases the risk of data security and patient privacy leakage, but is also vulnerable to cyber attacks.

[0004] Federated Learning (FL), as an emerging distributed machine learning technology, can train AI medical models in a distributed manner without centrally collecting data. By storing medical data locally and exchanging only the parameters of the learning model, federated learning effectively solves data security and patient privacy issues. In medical scenarios, federated learning can share model knowledge between multiple medical centers without transmitting original patient data, so it is a very promising AI medical technology. However, most existing FL studies focus on single-modal data (such as ECG) as model input. In actual medical applications, patients usually need to undergo multiple examinations and use multi-modal data to describe cardiovascular conditions (such as heart sounds + ECG).

[0005] At present, some studies have begun to explore multimodal cardiovascular disease detection methods based on federated learning. For example, by modeling the ECG data and heart sound data of different centers separately, the results of multiple models are integrated to improve the detection performance. However, these methods mainly rely on single modality or independent modality learning, which makes it difficult to fully explore the synergistic relationship between heart sound and ECG, limiting the accuracy and robustness of the medical model. In addition, while the existing federated learning scheme can protect patient privacy to a certain extent while achieving multi-center collaborative modeling, it still faces problems such as the modal distribution of medical data, limited equipment resources, and insufficient privacy protection, especially in medical scenarios. These problems affect the actual medical application effect of federated learning. In addition, medical centers and hospitals have accumulated a large amount of multimodal data, including different physiological examination data, during patient consultation, examination and treatment. These multimodal data usually contain personal sensitive information and require strict privacy protection measures. The practical application of multimodal learning in the field of medical health has important prospects, but data silos and modal heterogeneity have brought challenges to federated medical care.

[0006] In summary, the medical application of federated learning using multimodal data streams has the following disadvantages:

[0007] 1) Data modality distribution problem: In multi-center medical institutions, the distribution of heart sound and ECG data is diverse, and the modality heterogeneity between institutions makes federated learning more complex. Currently, there is a lack of definition and classification of the structured framework of multimodal federated medical care.

[0008] 2) Insufficient collaboration between modalities: Existing cardiovascular disease detection methods based on federated learning usually only model a single modality or independently model multiple modalities. They are unable to effectively utilize the potential synergistic relationship between heart sounds and ECG signals, resulting in insufficient accuracy of AI model-based detection.

[0009] 3) Data privacy and security risks: Although federated learning reduces data transmission by sharing model parameters, it may still face the risk of model reverse reasoning attacks and privacy leakage. Existing methods still need to improve their privacy protection mechanisms. Summary of the invention

[0010] In view of this, the present invention provides a heart sound-electrocardiogram multimodal fusion detection method based on federated learning, which effectively integrates the multimodal data of heart sound and electrocardiogram, fully explores the synergistic relationship between the modalities, and improves the detection accuracy of the local model.

[0011] The heart sound-electrocardiogram multimodal fusion detection method based on federated learning of the present invention comprises:

[0012] S1, each medical institution builds a local model based on local data and performs local model self-training; wherein the local model also includes a cross-modal collaborative attention module, which fuses the features of local modal data to obtain fused features, and the fused features are the output of the local model; if the medical institution only has single modal data, the feature input of other modal data in the cross-modal collaborative attention module in its local model is set to 1;

[0013] The cross-modal collaborative attention module performs the following operations on the features of each modality of input data:

[0014]

[0015] Where tanh is the hyperbolic tangent function; the superscript T indicates transpose; is the heart sound modality data feature, n p is the number of characteristic vectors of heart sound modality data, d p is the dimension of the feature vector of heart sound modality data; is the ECG modality data feature, n e is the number of eigenvectors of ECG modality data, d e is the dimension of the feature vector of ECG modality data; is the feature fusion matrix of heart sound modality data and electrocardiogram modality data; A PCG and A ECG The attention score matrices representing the phonocardiographic modality and the electrocardiographic modality respectively; and is the parameter to be trained, h c represents the dimension of the attention space; They are the attention weights of the normalized heart sound modality and electrocardiogram modality data features respectively;

[0016] The fusion feature is

[0017] S2, uses multimodal federated learning to jointly train the local models of all medical institutions;

[0018] S3 uses the local model trained by multimodal federated learning for detection.

[0019] Preferably, in S1, each modal data is first preprocessed, and then feature extraction and multimodal feature fusion of each modal data are performed based on the local model;

[0020] Among them, for the heart sound signal, its preprocessing is: first, a bandpass filter is used to remove the low-frequency and high-frequency noise in the heart sound signal, then continuous wavelet transform is used to further remove the mixed background noise, and then the denoised heart sound signal is amplitude normalized and standardized;

[0021] For ECG signals, the preprocessing is as follows: first, high-frequency myoelectric noise is removed by low-pass filtering, then baseline drift is eliminated by high-pass filtering, and then background noise is further removed by wavelet transform method.

[0022] Preferably, in S1, the local model uses a CNN+RNN network, a CNN+LSTM network or a Transformer network to extract the temporal and spatial features of each modal data.

[0023] Preferably, in S2, multimodal federated learning uses a federated averaging algorithm for federated aggregation.

[0024] Preferably, each local model participating in multimodal federated learning adds random noise to the local model parameters before uploading them to the federated server.

[0025] Preferably, the random noise is Laplace noise or Gaussian noise.

[0026] Preferably, according to the modality of the local data involved in federated learning, federated healthcare is divided into two categories: homogeneous multimodal federated healthcare and heterogeneous multimodal federated healthcare;

[0027] In homogeneous multimodal federated medical care, each medical institution in the federated medical care has the same modality set M = {m 1 ,m 2 ,...,m M}; Data set x of institution j j for:

[0028]

[0029] in, represents the data set of modality m in the jth medical institution, M is the global modality set, which represents the unified modality category owned by all medical institutions; Dj is the data sample size of the jth medical institution; J is the total number of medical institutions in the federal medical system;

[0030] In heterogeneous multimodal federated healthcare, the modality sets owned by various medical institutions in the federated healthcare are not all the same, that is, the data modality of each institution is a global modality set M = {m 1 ,m 2 ,...,m M}, and there is incomplete overlap between the subsets; the data set x of institution j j for:

[0031]

[0032] in, represents the data set of modality m in the jth medical institution, M j represents the mode set of the jth mechanism, and all mechanisms are not exactly the same; Formula (4) is obtained by M j Representing heterogeneous modal situations, allowing different institutions to have different amounts and types of modal data;

[0033] Homogeneous multimodal federated medical care and heterogeneous multimodal federated medical care were tested using the above method.

[0034] Beneficial effects:

[0035] (1) The present invention realizes the collaborative fusion of multimodal information of heart sound and ECG data through a cross-modal collaborative attention mechanism, effectively improving the comprehensive performance of federated learning and significantly improving the detection accuracy of the federated model. At the same time, by introducing a weight distribution mechanism, the model's adaptability to the imbalance of data in each modality is enhanced, and it has better robustness in practical applications.

[0036] (2) The present invention ensures full protection of patient data throughout the entire multimodal federated learning process by combining secure multi-party computing or differential privacy technology.

[0037] (3) This paper provides a formal definition of the MFLH model and defines two major types of model structures: homogeneous multimodal federated healthcare and heterogeneous multimodal federated healthcare. These architectures are different from traditional FL and provide a structured framework and benchmark for the design and development of MFLH research. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 These are examples of ECG and phonocardiograms. (a) is a phonocardiogram; (b) is an ECG.

[0039] Figure 2 The architecture of a homogeneous multimodal federated healthcare model.

[0040] Figure 3 Model architecture for heterogeneous multimodal federated healthcare.

[0041] Figure 4 This is a structural diagram of the heart sound-electrocardiogram multimodal fusion method based on federated learning. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0043] The present invention provides a heart sound-electrocardiogram multimodal fusion detection method based on federated learning. Aiming at the diversity of multimodal medical data states, multimodal federated medical care is firstly divided into two categories, namely homogeneous multimodal federated medical care and heterogeneous multimodal federated medical care, according to whether all federated participants have the same modality combination, and the definition and formal expression of the MFLH structured model are proposed; then, through the cross-modal collaborative attention mechanism, multimodal collaborative fusion based on heart sound and electrocardiogram data is realized, thereby improving the detection accuracy of the local model.

[0044] The present invention defines multimodal federated healthcare as a federated learning process involving multiple medical institutions, which includes at least two medical data modalities. Taking the multimodal classification task as an example, its definition is as follows:

[0045] Definition 1 (Multimodal federated medical model): Assume that there are J medical institutions in federated learning. For each medical institution j, its data consists of a multimodal set That is, the data mode of each institution is the global mode set M = {m 1 ,m 2 ,...,m M}, represents the data set of the mth modality of medical institution j; the federated medical learning process is expressed as follows:

[0046]

[0047] Among them, F j represents the joint loss of medical institution j for all its modality data and is defined as follows:

[0048]

[0049] Among them, ω represents the parameter set of the federated learning model, represents the loss function of mode m of medical institution j. α,…δ,…γ are the weighting coefficients of the mode, which are used to balance the loss of each mode.

[0050] According to the diversity of local modality data in federated medical learning, the present invention defines two major types of model structures: homogeneous multimodal federated medical and heterogeneous multimodal federated medical.

[0051] Definition 2 (Homogeneous Multimodal Federation Medical Care): All medical institutions in the federation medical care have the same modality set M = {m 1 ,m 2 ,...,m M}; The data set xj of institution j is:

[0052]

[0053] in, represents the data set of modality m in the jth medical institution, M is the global modality set, which represents the unified modality category owned by all medical institutions; D j is the data sample size of the jth medical institution; J is the total number of medical institutions in the federal medical system;

[0054] Figure 2 For the homogeneous multimodal federated healthcare model architecture, all federated healthcare institutions have the same input modality categories and feature space, but different sample spaces.

[0055] Definition 3 (Heterogeneous Multimodal Federation Healthcare): The modal sets owned by the medical institutions in the federation healthcare are not all the same, that is, the data modality of each institution is a global modal set M = {m 1 ,m 2 ,...,m M}, and there is incomplete overlap between the subsets; the data set x of institution j j for:

[0056]

[0057] in, represents the data set of modality m in the jth medical institution, M j Represents the set of modes owned by the j-th organization, which are not exactly the same for all organizations.

[0058] Figure 3 It is a heterogeneous multimodal federated medical model architecture. The heterogeneity of modalities is reflected in the differences in the number of modalities, modal combinations and feature spaces among different medical institutions.

[0059] In view of the homogeneous multimodal federated medical care and heterogeneous multimodal federated medical care defined above, taking the example of each medical center having heart sound data and / or electrocardiogram data, the present invention provides a federated learning method for heart sound-electrocardiogram multimodal collaboration. According to the data characteristics of different medical centers, local models are constructed respectively. For medical institutions that have both heart sound and electrocardiogram data, the cross-modal co-attention mechanism is used to fuse the complementary information of different modalities, further promote information interaction and feature enhancement between modalities, thereby improving the federated model's comprehensive understanding of multimodal data and achieving efficient and accurate auxiliary detection of cardiovascular diseases. The structure diagram of the federated learning of multimodal collaborative fusion of the present invention is shown in the figure: Figure 4 As shown, the specific steps include:

[0060] Step 1: Each medical center builds a local model and performs local model self-training

[0061] In the federated learning system, each medical center has data on any modality of heart sounds and electrocardiograms. Since these data cannot be stored centrally, local modeling and self-training of local models are required in each medical center. The specific steps of local modeling are as follows:

[0062] Step 1.1, data preprocessing

[0063] The original data usually contains a variety of interference factors, such as equipment noise, environmental noise and physiological background signals (such as breathing sounds and baseline drift). These noises will significantly reduce the clarity of the signal and affect the accuracy of subsequent feature extraction and model analysis. In addition, the signal strength and characteristics of different individuals may vary significantly. Through preprocessing steps such as normalization and smoothing, the consistency and stability of the data can be improved, providing a reliable foundation for feature extraction, pattern recognition and algorithm training. Therefore, the present invention first preprocesses the original heart sound and electrocardiogram data, including removing noise, correcting baseline drift, unifying signal amplitude, etc., to improve data consistency and enhance the recognizability of target signal features, providing a reliable foundation for subsequent feature extraction, model learning and disease detection.

[0064] The preprocessing of the heart sound signal is as follows: first, a bandpass filter is used to remove low-frequency and high-frequency noise in the heart sound signal, and then continuous wavelet transform is used to further remove the mixed background noise. Then, the denoised heart sound signal is amplitude normalized and standardized to ensure the consistency of the signal amplitude and the stability of the distribution, and finally the preprocessed heart sound signal is obtained.

[0065] The preprocessing of ECG signals is as follows: first, high-frequency myoelectric noise is removed by low-pass filtering, then baseline drift is eliminated by high-pass filtering to achieve baseline correction of the signal. Subsequently, the background noise is further removed by wavelet transform method to ensure the smoothness and clarity of the signal, and finally the preprocessed ECG signal is obtained.

[0066] Step 1.2: Build a local model

[0067] This method uses models suitable for processing time series signals, such as a model combining convolutional neural network (CNN) and recurrent neural network (RNN) (CRNN), CNN+LSTM network, Transformer network, etc. to extract high-dimensional features of heart sounds / ECG signals, that is, multi-dimensional feature representation of heart sounds and ECG signals, including features in time domain, frequency domain, and time-frequency domain, etc. Among them, CNN is used to extract local features, RNN (or LSTM) is used to model time series dependencies, and the ability to extract complex heart sounds and ECG signal features is improved by using multiple layers of convolutional layers.

[0068] For medical centers that have both heart sound and ECG data, the local model further uses a cross-modal collaborative attention mechanism to fuse features from different modalities, thereby improving the expressiveness of the features.

[0069] Specifically, in the local model, the present invention calculates the feature correlation between different modalities through a cross-modal attention mechanism for heart sound and ECG data, and generates an attention score matrix. This matrix represents the similarity or correlation between heart sound features and ECG features. The features of the heart sound modality and the ECG modality are weighted and fused using the attention score matrix. In this way, the information with high correlation between the two modalities can be highlighted, and the irrelevant parts can be weakened, thereby achieving feature enhancement.

[0070] Assuming that the CRNN network is used, the feature sets extracted from different modalities of PCG and ECG are x PCG and x ECG , then define and Where n p ,n e represents the number of eigenvectors, d p ,d e Represents the dimension of the feature vector. In order to exploit the complementarity and redundancy between different modalities, given features PCG and ECG, the following operations are performed through the collaborative attention mechanism:

[0071]

[0072] in, is the feature fusion matrix of PCG and ECG. The tanh function (hyperbolic tangent function) is used to calculate the correlation between the features after linear transformation and introduce nonlinear mapping, that is, nonlinear activation is performed on the linear transformation results to ensure that the output is in the range of [-1,1]. The superscript T represents transposition. h is the parameter to be trained. c is the dimension of the attention space. PCG and A ECG Represents the attention score matrix of PCG and ECG modalities. The normalized attention weights are obtained through the Softmax activation function Used to indicate the relative importance of different modality features in the final fusion.

[0073] Perform weighted fusion of different modal features to obtain fusion features

[0074] The fused representation obtained through the collaborative attention mechanism contains the complementary information of the two modalities of heart sound and electrocardiogram. Ultimately, these fused feature representations will be used for federated learning.

[0075] The present invention aims at the problem of uneven distribution of cardiac sound and ECG data modalities in multi-center medical institutions. The present invention designs an unbalanced modal data processing strategy. By introducing a modal weight allocation mechanism into the cross-modal collaborative attention mechanism, a PCG ,a ECG is the weighted coefficient of the modality, which is used to balance the loss of each modality. This allows the model to dynamically adjust its reliance on data of different modalities, ensuring that a high-quality model can be obtained even in the case of data imbalance. At the same time, the weight is also adaptively and dynamically adjusted in each subsequent round of federated learning, thereby effectively addressing the challenge of data imbalance, ensuring that the model is more dependent on important modalities, and improving the learning performance of the model in multimodal scenarios.

[0076] Of course, for medical centers that only have ECG data or heart sound data, their local models can also include a cross-modal collaborative attention module, and the feature inputs of other modal data in their cross-modal collaborative attention module are set to 1.

[0077] Step 1.3: Local model self-training

[0078] Each medical center uses a local model to train heart sound and ECG data locally. Optimization algorithms such as Adam and SGD can be used to update local model parameters. The training process does not involve data sharing with other medical centers, and only the local model is updated.

[0079] Step 2: Multimodal Federated Learning

[0080] The federated server conducts multimodal aggregation learning on local models from different, cross-modal medical centers to obtain the global model parameters of the local models, thereby enhancing the feature learning capabilities of the local models, especially the information interaction and collaboration capabilities across medical centers.

[0081] During multiple rounds of iterations of multimodal federated learning, each medical center only shares the parameters of its local model to protect patient privacy.

[0082] This embodiment uses the Federated Averaging (FedAvg) algorithm for global aggregation. The objective function expression in the multimodal federated learning model is as follows:

[0083]

[0084] Among them, υ and θ represent the parameters of the attention module and the classifier respectively. j represents the expected loss of the jth institution on local multimodal data,

[0085]

[0086] in, Represents the model prediction error in predicting the true label.

[0087] To further improve data security and privacy protection, the present invention also combines differential privacy technology to ensure effective data protection during the model parameter exchange process and prevent privacy leakage through model reasoning attacks. Differential privacy is a data privacy protection technology that adds noise to the output data of the local model to ensure that the statistical characteristics of its output are very little dependent on a single data, thereby protecting the privacy of the user. The following are the specific implementation ideas and steps of differential privacy in federated learning.

[0088] S1. Determine the privacy budget

[0089] Privacy Budget, usually expressed as parameter ε, is used to control the trade-off between privacy and accuracy in the model. A smaller ε means stronger privacy protection, but may affect the accuracy of the model. Since medical data is highly sensitive and requires higher privacy protection, this patent selects the privacy budget domain as ε∈[0.1,1] in the federal medical environment. Experiments are conducted based on the initial selection of ε=0.1 to evaluate the model accuracy. When ε=0.5, a balance is achieved between privacy protection and model accuracy.

[0090] S2, local model update

[0091] Each medical institution trains the model based on its local data and updates the local model parameters. Assume that the local model parameters of each institution are θ j , perform several rounds of gradient descent on local data and update the model parameters θ j :

[0092]

[0093] Among them, η represents the learning rate, is the gradient of the loss function of the j-th local mechanism model.

[0094] S3, adding noise

[0095] In order to achieve differential privacy, each local organization adds random noise before uploading model parameters or gradients to the server. The noise distribution used in this invention is Laplace noise, and the intensity of the noise is related to the privacy budget. Laplace noise is added:

[0096]

[0097] Among them, Δf is the sensitivity of the function, which indicates the maximum impact of changing a single data point on the function output. Each medical institution will add the parameters after noise Upload to the federation server. The server receives the parameters of all institutions and aggregates them using the Federated Averaging (FedAvg) algorithm:

[0098]

[0099] Where J represents the number of federal medical institutions, θ (t+1 ) are the parameters of the global model of federated learning. The server updates the global model based on the aggregated model parameters. The updated global model parameters θ (t+1 ) is sent to all participating local agencies, and the local agencies use the aggregated parameters for the next round of local training. The federated learning process goes through multiple rounds of iterations, repeatedly repeating the steps of "local training - adding noise - aggregation - updating" until the desired training goal is achieved.

[0100] Through the above steps, applying differential privacy technology in federated learning can protect data privacy while reducing the probability of attackers inferring a single data point through noise injection, thereby enhancing the security and privacy protection of federated learning. This method is particularly suitable for modeling and analysis of sensitive data such as medical data.

[0101] Step 3: Each medical institution uses a local model trained by multimodal federated learning to conduct testing.

[0102] Example 1

[0103] In order to demonstrate the effects of this invention in federated learning medical modeling on federated multimodal fusion, processing of unbalanced modal data, and privacy protection, we present a typical experimental example to illustrate the detection effect and model performance in practical applications. The following is a specific example:

[0104] We assume that multiple medical institutions participate in the construction of the federated learning model, and each medical institution has data of different quantities and modalities, such as heart sound (PCG) and electrocardiogram (ECG) data. Our goal is to detect cardiac abnormalities through the federated learning model and evaluate the effectiveness of the multimodal fusion mechanism and modality imbalance processing strategy proposed in this invention. The experimental settings are as follows:

[0105] a) Dataset

[0106] A multimodal dataset publicly available from PhysioNet is used. This dataset is a binary classification task (sample labels are normal and abnormal) and contains heart sounds and ECG data from the same group of patients. The data recording time for each patient is usually between 30 seconds and 120 seconds, and the data acquisition sampling rate is 2000Hz (for ECG) and 1000Hz (for PCG) to ensure high-resolution capture of heart sounds and ECG signals. This dataset is suitable for the development of federated learning models and for multimodal data analysis of cardiac abnormalities, which helps to improve the accuracy of multimodal data analysis and effectively protect patient privacy. In the example analysis, the data preprocessing and feature extraction steps of the present invention are followed to ensure the reliability of the signal and the effectiveness of the final model.

[0107] b) Model training

[0108] Each institution uses a model based on a combination of convolutional neural networks (CNN) and recurrent neural networks (RNN) (CRNN) as feature extraction and model training for local data, and uses a cross-modal collaborative attention mechanism to weightedly fuse the multimodal features of the local institution, while addressing the problem of inconsistent importance of data in different modalities. Multiple rounds of local training and global parameter aggregation are performed through a federated learning framework. Model parameters are not directly exchanged between different institutions, and only weighted aggregation is performed on the server. The model iteration process uses a differential privacy mechanism to protect the local model parameter updates of each institution. Laplace noise is added each time the parameters are uploaded to ensure that even if the server is attacked, the sensitive data of a single patient cannot be inferred. The privacy budget is set to ε = 0.5.

[0109] c) Detection tasks

[0110] Detect cardiac abnormalities (such as atrial fibrillation, heart murmur, etc.) by integrating heart sound and ECG data to improve the model's ability to detect cardiac abnormalities. Accuracy, precision, recall and F1-score are used as evaluation indicators.

[0111] d) Example results

[0112] i. Not using attention mechanism and differential privacy protection: In the traditional federated learning model, modality imbalance and privacy protection issues are not dealt with. The model is prone to accuracy degradation under imbalanced data, with an accuracy rate of about 75%.

[0113] ii. Introducing the adaptive attention mechanism: By introducing the adaptive attention mechanism, the model can dynamically adjust the degree of dependence on different modal features, especially when some institutions only have single-modal data, the attention mechanism helps improve the ability to integrate features. The accuracy rate is increased to about 82%, and the recall rate is also significantly improved.

[0114] iii. Introducing differential privacy protection: After the introduction of differential privacy, although noise is added to the model, due to the reasonable setting of the privacy budget (such as ε=0.5), the accuracy of the model only drops by about 1% to 81%, but the privacy protection capability of the data is greatly improved.

[0115] Through the detection of the federated medical multimodal model, the fusion of heart sound and ECG data significantly improves the detection ability of atrial fibrillation and other heart murmurs, especially when the modal information is complementary (such as heart sound signals reflect valve problems, while ECG signals reflect abnormal electrical activity). Through the cross-modal attention mechanism, the model can better utilize multimodal features and effectively improve the detection performance of the model on unbalanced multimodal medical data. In the task of detecting cardiac abnormalities, the model accuracy rate increased from 75% to 82%, and privacy protection measures ensured the data security of various medical institutions.

[0116] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A heart sound-electrocardiogram multimodal fusion detection method based on federated learning, characterized in that: include: S1, each medical institution builds a local model based on local data and performs local model self-training; wherein the local model also includes a cross-modal collaborative attention module, which fuses the features of local modal data to obtain fused features, and the fused features are the output of the local model; if the medical institution only has single modal data, the feature input of other modal data in the cross-modal collaborative attention module in its local model is set to 1; The cross-modal collaborative attention module performs the following operations on the features of each modality of input data: Where tanh is the hyperbolic tangent function; the superscript T indicates transpose; is the heart sound modality data feature, n p is the number of characteristic vectors of heart sound modality data, d p is the dimension of the feature vector of heart sound modality data; is the ECG modality data feature, n e is the number of eigenvectors of ECG modality data, d e is the dimension of the feature vector of ECG modality data; A is the feature fusion matrix of heart sound modality data and electrocardiogram modality data; PCG and A ECG The attention score matrices representing the phonocardiographic modality and the electrocardiographic modality respectively; and is the parameter to be trained, h c represents the dimension of the attention space; They are the attention weights of the normalized heart sound modality and electrocardiogram modality data features respectively; The fusion feature is S2, uses multimodal federated learning to jointly train the local models of all medical institutions; S3 uses the local model trained by multimodal federated learning for detection.

2. The method according to claim 1, characterized in that In S1, each modal data is first preprocessed, and then feature extraction and multimodal feature fusion of each modal data are performed based on the local model; Among them, for the heart sound signal, its preprocessing is: first, a bandpass filter is used to remove the low-frequency and high-frequency noise in the heart sound signal, then continuous wavelet transform is used to further remove the mixed background noise, and then the amplitude of the denoised heart sound signal is normalized and standardized; For ECG signals, the preprocessing is as follows: first, high-frequency myoelectric noise is removed by low-pass filtering, then baseline drift is eliminated by high-pass filtering, and then background noise is further removed by wavelet transform method.

3. The method according to claim 1 or 2, characterized in that In S1, the local model uses a CNN+RNN network, a CNN+LSTM network or a Transformer network to extract the temporal and spatial features of each modal data.

4. The method according to claim 1, characterized in that In S2, multimodal federated learning uses a federated averaging algorithm for federated aggregation.

5. The method according to claim 1, characterized in that Each local model participating in multimodal federated learning adds random noise to the local model parameters before uploading them to the federated server.

6. The method according to claim 5, characterized in that The random noise is Laplace noise or Gaussian noise.

7. The method according to claim 1, characterized in that According to the modality of local data involved in federated learning, federated medical care is divided into two categories: homogeneous multimodal federated medical care and heterogeneous multimodal federated medical care. In homogeneous multimodal federated medical care, each medical institution in the federated medical care has the same modality set M = {m1,m2,...,m M }; Data set x of institution j j for: in, represents the data set of modality m in the jth medical institution, M is the global modality set, which represents the unified modality category owned by all medical institutions; D j is the data sample size of the jth medical institution; J is the total number of medical institutions in the federal medical system; In heterogeneous multimodal federated healthcare, the modal sets owned by various medical institutions in the federated healthcare are not all the same, that is, the data modality of each institution is a global modal set M = {m1,m2,...,m M }, and there is incomplete overlap between the subsets; the data set x of institution j j for: in, represents the data set of modality m in the jth medical institution, M j represents the set of modes possessed by the jth organization, which are not exactly the same for all organizations; Homogeneous multimodal federated medical care and heterogeneous multimodal federated medical care are detected using the method described in any one of claims 1 to 6.

Images