Multi-mode cerebral apoplexy intelligent early warning method and system

Abstract

The invention provides a multi-modal intelligent early warning method and system for cerebral apoplexy, and belongs to the technical field of artificial intelligence, and the method comprises the steps: distributing a first global multi-modal fusion model constructed based on a cross-modal attention mechanism to a plurality of target nodes; acquiring local model parameters returned by the plurality of target nodes, and calculating a weighted average value to update the parameters of the first global multi-modal fusion model to obtain a second global multi-modal fusion model; the local model parameters are obtained by performing local training on the first global multi-modal fusion model by each target node based on local medical data; and distributing the second global multi-modal fusion model to a plurality of target nodes to adjust parameters of the second global multi-modal fusion model based on the local medical data of each target node to obtain a multi-modal cerebral apoplexy intelligent early warning model, thereby performing cerebral apoplexy intelligent early warning. According to the invention, the problem that the existing scheme cannot ensure the privacy of medical data and the accuracy of cerebral apoplexy early warning can be solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal intelligent early warning method and system for stroke. Background Technology

[0002] The diagnosis of brain diseases relies on multi-source medical data such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and clinical scales. Existing stroke prediction models face serious privacy challenges in the process of medical data processing, especially when modeling across institutions, which can easily expose raw medical data.

[0003] To ensure the privacy of medical data and prevent its leakage, existing solutions can also choose to model on the hospital's local computer. However, the fusion modeling of EEG signals (time-series data) and image data (spatial data) requires tensor operations of up to 10^5 dimensions. However, primary healthcare institutions face practical constraints such as insufficient computing power, unstable or delayed networks. In particular, real-time early warning of high-frequency data such as epileptic seizures requires millisecond-level response requirements. On encrypted models, the number of parameters usually exceeds the device's memory capacity, which will increase the error rate of EEG signal recognition under encrypted schemes.

[0004] In summary, existing intelligent stroke early warning methods cannot guarantee both the privacy of medical data and the accuracy of stroke early warning. Summary of the Invention

[0005] In view of this, it is necessary to provide a multimodal intelligent stroke early warning method and system to solve the technical problem that existing solutions cannot guarantee both the privacy of medical data and the accuracy of stroke early warning.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a multimodal intelligent early warning method for stroke, comprising: Distribute the first global multimodal fusion model, constructed based on a cross-modal attention mechanism, to multiple target nodes; The system obtains local model parameters returned by multiple target nodes, calculates a weighted average of the local model parameters, and updates the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain a second global multimodal fusion model. The local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data. A second global multimodal fusion model is distributed to multiple target nodes. The parameters of the second global multimodal fusion model are adjusted based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model. Stroke intelligent early warning is then performed based on the multimodal stroke intelligent early warning model and the local medical data to be tested from the target nodes.

[0007] In one possible implementation, the first global multimodal fusion model is trained locally based on local medical data, including: Local medical data is cleaned, aligned, and standardized to obtain preprocessed data. The preprocessed data is input into a multimodal feature extractor for feature extraction to obtain local medical data features; The first global multimodal fusion model is trained locally based on the local medical data features. The local medical data includes any one of the following: brain imaging data, proteomics data, electroencephalogram (EEG) data, and electronic medical record data.

[0008] In one possible implementation, the multimodal feature extractor includes any one of the following: the MA-3Dnet model, the NOTEARS model, a temporal feature extraction model, and a structured table multidimensional encoder; the temporal feature extraction model is composed of a GAF model and a CNN model. The MA-3Dnet model is used for feature extraction from brain imaging data. The NOTEARS model is used for feature extraction from proteomics data; The time-series feature extraction model is used to extract features from electroencephalogram (EEG) data. The structured table multidimensional encoder is used to extract features from electronic medical record data.

[0009] In one possible implementation, the first global multimodal fusion model is trained locally based on the local medical data features, including: Before training the first global multimodal fusion model locally based on the local medical data features, noise is injected into the local first global multimodal fusion model for training. In the later stages of local training of the first global multimodal fusion model based on the local medical data features, the noise injected during the earlier training process is gradually increased. The enhanced noise satisfies the privacy budget; and during the local training of the first global multimodal fusion model, gradient pruning is used to limit the gradient magnitude of the local first global multimodal fusion model.

[0010] In one possible implementation, calculating the weighted average of the local model parameters includes: Based on the quality of local medical data at each target node and the model performance obtained after local training of the first global multimodal fusion model, the weights of the local model parameters returned by each target node are dynamically adjusted. Based on the weights of the local model parameters returned by each target node, and the local model parameters returned by each target node, a weighted average of the local model parameters returned by multiple target nodes is determined.

[0011] In one possible implementation, before distributing the second global multimodal fusion model to multiple target nodes, the following is included: The obtained second global multimodal fusion model is subjected to dual verification of privacy budget audit and performance validation; If the results of the double verification determine that the second global multimodal fusion model has exceeded the privacy budget or degraded model performance, the noise intensity injected during the local training of the first global multimodal fusion model is readjusted, or the weights of the local model parameters returned by each target node are readjusted, in order to obtain the second global multimodal fusion model again.

[0012] In one possible implementation, the multimodal intelligent stroke early warning model is used for: The input local medical data to be tested is standardized and dimensionality reduced to obtain an intermediate representation; Construct the QKV mechanism matrix corresponding to the intermediate representation; Based on the QKV mechanism matrix, calculate the cross-modal attention score; Based on the cross-modal attention score and the values ​​in the QKV mechanism matrix, multiple aggregated and fused features are calculated. The multiple aggregated and fused features are then combined to obtain the combined feature; Intelligent early warning of stroke based on the splicing features.

[0013] In one possible implementation, intelligent stroke early warning based on the splicing features includes: The probability of stroke risk is determined based on the splicing features; If the probability of stroke risk is greater than a preset probability threshold, an early warning signal is sent to the HIS system of the target node to provide intelligent early warning of stroke.

[0014] In one possible implementation, the multimodal stroke intelligent early warning model is also used for: Determine the contribution of each modal feature of the local medical data to the prediction results; Based on the contribution, heatmap positioning is performed in the local medical data to be tested to identify lesion areas, and causal path analysis is performed based on the local medical data to be tested to obtain key regulatory paths.

[0015] Secondly, the present invention also provides a multimodal intelligent early warning system for stroke, comprising: The first distribution module is used to distribute the first global multimodal fusion model based on the cross-modal attention mechanism to multiple target nodes; The model update module is used to obtain local model parameters returned by multiple target nodes, calculate a weighted average of the local model parameters, and update the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain the second global multimodal fusion model; the local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data; The second distribution module is used to distribute the second global multimodal fusion model to multiple target nodes, adjust the parameters of the second global multimodal fusion model based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model, and perform stroke intelligent early warning based on the multimodal stroke intelligent early warning model and the local medical data to be tested of the target nodes.

[0016] The beneficial effects of the above implementation are as follows: The multimodal stroke intelligent early warning method and system provided by this invention can be executed by the server. The server distributes a first global multimodal fusion model based on a cross-modal attention mechanism to multiple target nodes; obtains local model parameters returned by multiple target nodes, calculates a weighted average of the local model parameters, and updates the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain a second global multimodal fusion model; the local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data; wherein, the local medical data of different target nodes is multimodal heterogeneous data, when updating the first global multimodal fusion model on the server, it is not necessary for each target node to return local medical data, only the trained local model parameters are needed, thus avoiding leakage of local medical data and ensuring data privacy and security. In addition, the first global multimodal fusion model constructed by this invention is based on a multi-head cross-modal attention mechanism, which can fuse the features of multimodal data, thereby significantly improving model performance and cross-institutional collaboration efficiency.

[0017] After the first global multimodal fusion model is updated on the server side, that is, after the federated learning between the server side and the target nodes is completed, the second global multimodal fusion model is distributed to multiple target nodes. The parameters of the second global multimodal fusion model are adjusted based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model. Stroke intelligent early warning is then performed based on the multimodal stroke intelligent early warning model and the local medical data to be tested from the target nodes.

[0018] Each target node only needs to complete local training using its own local medical data, without needing to combine data from other target nodes for training. This reduces the amount of data during local training and lowers the local computing power requirements of the nodes. At the same time, by combining multimodal data and using federated learning, the model training effect is improved, thereby improving the accuracy of the model in stroke early warning. Therefore, this invention can solve the technical problem that existing solutions cannot guarantee both the privacy of medical data and the accuracy of stroke early warning. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of an embodiment of the multimodal intelligent early warning method for stroke provided by the present invention; Figure 2 A flowchart illustrating another embodiment of the multimodal intelligent stroke early warning method provided by the present invention; Figure 3 This is a schematic diagram illustrating the workflow of each model on multiple target nodes provided by the present invention. Figure 4 A schematic diagram of an embodiment of the multimodal stroke intelligent early warning system provided by the present invention; Figure 5 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0023] In this embodiment of the invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0024] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] This invention provides a multimodal intelligent early warning method and system for stroke, which will be described below.

[0027] like Figure 1 As shown, this invention provides a multimodal intelligent early warning method for stroke, the method being executed by a server-side application, the method comprising: S101. Distribute the first global multimodal fusion model based on the cross-modal attention mechanism to multiple target nodes.

[0028] Understandably, a global multimodal fusion model is initialized on the server side, namely the first global multimodal fusion model in this embodiment. This model is compatible with multimodal data types (such as medical images, electronic medical records, genetic data, etc.) from various hospitals (i.e., various target nodes). The model architecture typically adopts a cross-modal attention mechanism to ensure effective alignment and interaction of features from different modalities.

[0029] The server distributes the global multimodal fusion model parameters to each hospital node participating in federated learning via a secure communication protocol.

[0030] S102. Obtain the local model parameters returned by multiple target nodes, calculate the weighted average of the local model parameters, and update the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain the second global multimodal fusion model; the local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data.

[0031] It is understandable that the target node can be a local computer node in the hospital. The local medical data corresponding to different target nodes can be different, and the local medical data of different target nodes have different modalities of data characteristics.

[0032] This step involves federated learning between the server and each target node. After the model is trained on each target node, the local model parameters are sent back to the server, not the local medical data. This avoids the leakage of local medical data during network transmission or at the server end, thus better protecting the privacy of local medical data.

[0033] S103. Distribute the second global multimodal fusion model to multiple target nodes, adjust the parameters of the second global multimodal fusion model based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model, and perform stroke intelligent early warning based on the multimodal stroke intelligent early warning model and the local medical data to be tested of the target nodes.

[0034] It is understandable that S101-S103 is a model update iteration process in the federated learning process, and the above steps can be repeated in the corresponding order.

[0035] The purpose of this invention is to address two core issues in existing stroke prediction technologies: first, how to maintain model performance while protecting patient privacy; and second, how to efficiently integrate heterogeneous multimodal medical data for joint modeling and intelligent early warning, thereby improving the accuracy of intelligent stroke early warning. To this end, this invention proposes a multimodal intelligent stroke early warning method based on federated learning. This method achieves efficient integration of multimodal information, including MRI images (brain magnetic resonance imaging), proteogenomic data, physiological time-series signals (EEG signals), and clinical records (electronic medical records), across distributed medical institutions while protecting data privacy. This enhances the accuracy and generalization ability of the stroke early warning system.

[0036] In some embodiments, the first global multimodal fusion model is trained locally based on local medical data, including: Local medical data is cleaned, aligned, and standardized to obtain preprocessed data. The preprocessed data is input into a multimodal feature extractor for feature extraction to obtain local medical data features; The first global multimodal fusion model is trained locally based on the local medical data features. The local medical data includes any one of the following: brain imaging data, proteomics data, electroencephalogram (EEG) data, and electronic medical record data.

[0037] Understandably, standardization, that is, normalizing data, can reduce the impact of numerical range.

[0038] The types of medical data for different target nodes can be different, and therefore the multimodal feature extractors corresponding to different local medical data can also be different. For example, there are four target nodes: the local medical data of the first target node is brain imaging data, i.e., 3D brain MRI images; the local medical data of the second target node is proteomics data; the local medical data of the third target node is electroencephalogram (EEG) data; and the local medical data of the fourth target node is electronic medical record data.

[0039] In some embodiments, the multimodal feature extractor includes any one of the following: MA-3Dnet model, NOTEARS model, temporal feature extraction model, and structured table multidimensional encoder (STME); the temporal feature extraction model is composed of GAF (Gramian Angular Field) model and CNN (Convolutional Neural Network) model; The MA-3Dnet model is used for feature extraction from brain imaging data. The NOTEARS model is used for feature extraction from proteomics data; The time-series feature extraction model is used to extract features from electroencephalogram (EEG) data. The structured table multidimensional encoder is used to extract features from electronic medical record data.

[0040] It is understandable that the MA-3Dnet model includes: Multi-scale convolution path: Parallel 3D convolution kernels capture spatial features of different granularities; Spatial-channel attention (SCAM): Highlights key lesion areas through channel attention and spatial masking; Residual connections: prevent gradient vanishing in deep networks and enhance feature reuse capabilities.

[0041] NOTEARS models include: Structural equation modeling (SEM): learning the potential causal dependencies between proteins (adjacency matrix W); Causal path feature aggregation: Generate regulatory path features based on the causal graph.

[0042] GAF models include: Time-to-image conversion: Converting a one-dimensional signal into a two-dimensional GASF image while preserving its time dependence; CNN Feature Extraction: Utilizing convolutional layers to capture local temporal patterns and global trends.

[0043] Structured table multidimensional encoder, including: Column-aware attention: dynamically modeling the interaction relationships between feature columns (such as the correlation between blood pressure and blood sugar); Missing value learning and imputation: Adaptive generation of missing value representations (Learnable Tokens); Linear Transformer: Captures temporal or distributional dependencies across samples (such as vital signs records over consecutive days).

[0044] In some embodiments, the first global multimodal fusion model is trained locally based on the local medical data features, including: Before training the first global multimodal fusion model locally based on the local medical data features, noise is injected into the local first global multimodal fusion model for training. In the later stages of local training of the first global multimodal fusion model based on the local medical data features, the noise injected during the earlier training process is gradually increased. The enhanced noise satisfies the privacy budget; and during the local training of the first global multimodal fusion model, gradient pruning is used to limit the gradient magnitude of the local first global multimodal fusion model.

[0045] It is understandable that this embodiment employs gradient pruning and differential privacy noise injection: During local model parameter updates, gradient pruning is used to limit gradient magnitude and prevent excessive consumption of the privacy budget by gradient explosion. The variance of the differential privacy noise is dynamically adjusted based on the model convergence phase. Early training phase (low noise): Inject low-intensity noise to accelerate model convergence and prioritize performance improvement.

[0046] Later in the training phase (high noise): gradually increase the noise level, but strictly adhere to the privacy budget (ε-δ constraint) to ensure privacy and security.

[0047] By using staged noise control, the overall privacy loss can be controlled while reducing the negative impact on model accuracy.

[0048] In some embodiments, calculating a weighted average of the local model parameters includes: Based on the quality of local medical data at each target node and the model performance obtained after local training of the first global multimodal fusion model, the weights of the local model parameters returned by each target node are dynamically adjusted. Based on the weights of the local model parameters returned by each target node, and the local model parameters returned by each target node, a weighted average of the local model parameters returned by multiple target nodes is determined.

[0049] It is understandable that the quality of local medical data can be determined based on the consistency of local medical data labeling, and model performance can be determined by the AUC (Area Under the Curve) value obtained by inputting the local validation set into the trained model. In this embodiment, the robustness of the global model can be improved by dynamically adjusting the aggregation weights.

[0050] In some embodiments, before distributing the second global multimodal fusion model to multiple target nodes, the method includes: The obtained second global multimodal fusion model is subjected to dual verification of privacy budget audit and performance validation; If the results of the double verification determine that the second global multimodal fusion model has exceeded the privacy budget or degraded model performance, the noise intensity injected during the local training of the first global multimodal fusion model is readjusted, or the weights of the local model parameters returned by each target node are readjusted, in order to obtain the second global multimodal fusion model again.

[0051] Understandably, before the global model is deployed, a dual verification process involving privacy budget auditing and performance validation is conducted to ensure an optimal balance between privacy protection and model utility. If the privacy budget is exceeded or performance significantly degrades, an adaptive rollback mechanism is triggered to readjust the noise strategy or aggregate weights.

[0052] In some embodiments, the multimodal stroke intelligent early warning model is used for: The input local medical data to be tested is standardized and dimensionality reduced to obtain an intermediate representation; Construct the QKV mechanism matrix corresponding to the intermediate representation; Based on the QKV mechanism matrix, calculate the cross-modal attention score; Based on the cross-modal attention score and the values ​​in the QKV mechanism matrix, multiple aggregated and fused features are calculated. The multiple aggregated and fused features are then combined to obtain the combined feature; Intelligent early warning of stroke based on the splicing features.

[0053] Understandably, the QKV mechanism, or query-key-value mechanism, constructs a query (Q), key (K), and value (V) matrix: , ,

[0054] Calculate cross-modal attention score:

[0055] Aggregation and fusion characteristics:

[0056] All modal fusion outputs are concatenated to form the final representation:

[0057] In some embodiments, intelligent stroke early warning based on the splicing features includes: The probability of stroke risk is determined based on the splicing features; If the probability of stroke risk is greater than a preset probability threshold, an early warning signal is sent to the HIS system of the target node to provide intelligent early warning of stroke.

[0058] Understandably, the deployed model receives multimodal data input from patients and outputs the probability of stroke risk. The system can be linked to the Hospital Information System (HIS) to trigger early warning prompts and intervention suggestions.

[0059] In some embodiments, the multimodal stroke intelligent early warning model is also used for: Determine the contribution of each modal feature of the local medical data to the prediction results; Based on the contribution, heatmap positioning is performed in the local medical data to be tested to identify lesion areas, and causal path analysis is performed based on the local medical data to be tested to obtain key regulatory paths.

[0060] Understandably, the local medical data to be tested for different target nodes correspond to different modal features. Based on the attention weights corresponding to each modal feature, the contribution of each modal feature to the prediction results is shown. An attention heatmap is superimposed on the original MRI image (brain image data obtained from MRI) to mark high-risk lesion areas.

[0061] This embodiment also performs causal pathway analysis on proteomics data and outputs key regulatory pathways.

[0062] This embodiment integrates a relevant interpreter at the model output end to provide doctors and patients with a visual interpretation of stroke risk assessment, thereby improving the clinical usability of the system.

[0063] In some embodiments, the method provided by the present invention includes: 1. Acquire a variety of medical data, including MRI images, genomic information, physiological time-series signals, and clinical texts.

[0064] 2. Pretreatment: (i) Data cleaning and noise reduction.

[0065] (ii) Data normalization reduces the impact of numerical range.

[0066] 3. Feature extraction: The main focus is on feature extraction from four modalities: brain imaging, proteomics, time series, and tabular data. (1) Feature extraction from brain imaging data Feature extraction employs a multi-scale attention-guided 3D convolutional feature extraction network (MA-3DNet); the input data consists of uniformly sized 3D brain images; after the data is input into the MA-3DNet network, the images are first standardized (Z-score normalization) and then data augmentation is performed (such as random rotation, flipping, and affine transformation).

[0067] The MA-3Dnet network includes: a multi-scale 3D convolution module, a spatial-channel attention module (SCAM), and a residual connection and fusion module.

[0068] Multi-scale 3D convolution module: Three parallel convolution paths are set up, each using a 3D convolution kernel with a different receptive field. Each path is followed by BatchNorm and ReLU activation, and finally a 1×1×1 convolution is used for dimension fusion.

[0069] In the spatial-channel attention module, Channel Attention extracts channel importance through global average pooling and max pooling operations, and uses MLP to enhance the response of important channels; Spatial Attention performs spatial dimension pooling and convolution on the fused feature map to generate a spatial attention map, guiding the model to focus on key locations.

[0070] Residual connection and fusion module: To avoid the gradient vanishing problem, residual connections (ResNet-style) are introduced. The output of each feature extraction module is added to the input for fusion.

[0071] (2) Feature extraction from proteomics data Notears is a proteomics data feature extraction method based on Non-combinatorial Optimization via Trace Exponential and Augmented lagRangian for Structure learning. This feature extraction method, which integrates causal inference and structural path information, is particularly suitable for data modeling scenarios such as proteomics where potential regulatory relationships exist, improving interpretability while maintaining modeling efficiency and flexibility.

[0072] NOTEARS is a causal structure learning method based on continuous optimization. By introducing acyclic constraints and smoothing functions (such as the matrix exponential trace), it transforms the discrete graph structure learning problem into a differentiable optimization problem, and then uses the augmented Lagrangian method to solve it efficiently. This method overcomes the computational bottleneck of traditional combinatorial optimization methods, significantly improving the efficiency and scalability of causal graph learning.

[0073] When extracting features from proteomics data, the input data is a matrix of proteomics expression data:

[0074] in, n For the sample size, d This represents the number of proteins, with each column showing the expression level of a single protein. X For input data, It is a data matrix.

[0075] Causal Structure Learning (NOTEARS): Constructing a linear structural equation model (SEM): Assuming that the expression level of each protein satisfies the following linear model

[0076] in, Other proteins affect the first j The causal influence coefficient of a protein It is a noise term. T This indicates transpose.

[0077] Define the objective function: using the... The least squares loss of the regularization term is used as the scoring function:

[0078] in, It is the weighted adjacency matrix to be learned. This is the scoring function.

[0079] Introduce an acyclic constraint function, namely a smooth acyclic constraint function:

[0080] The solution is obtained using the augmented Lagrange method or the L-BFGS + PQN (approximate Newton) optimization algorithm.

[0081] The causal graph structure is obtained: ultimately, a sparse weighted adjacency matrix is ​​obtained. The non-zero terms indicate a potential causal relationship between proteins.

[0082] Based on the learned causal graph Feature extraction is performed. Cause-effect graph construction (based on the previous NOTEARS) ; Each element This represents the causal edge weights from protein j to i; Define the causal path characteristic function: for each protein The upstream and downstream causal feature value is defined as the weighted aggregate value of its adjacent causal path nodes:

[0083] Where ω is the edge weight threshold. For the first i The upstream regulatory polymerization value of a protein. This represents the downstream regulatory polymerization value of the i-th protein; Constructing a new feature matrix (Causal Feature Matrix): For each sample in the original data Construct new causal path feature vectors:

[0084] The output is a structure-guided feature matrix:

[0085] in These are the expression characteristics of key proteins after causal graph pruning or aggregation.

[0086] (3) Feature extraction of time series data Transforming time-series data into images for feature extraction is an effective method, among which Gramian Angular Field (GAF) is a widely used technique. GAF can preserve the temporal dependencies of time series data and utilize the image structure to allow image models such as Convolutional Neural Networks (CNNs) to model them.

[0087] Normalized time series to interval : Given the original time series: Normalize the time series to an interval This is to ensure that its value can be converted into the cosine of the angle:

[0088] in, For the normalized data, min( X ) and max( X ) are respectively X The lowest and highest values ​​in the range.

[0089] Each normalized value Mapped to polar angle : ,

[0090] This polar angle indicates that amplitude information is embedded in the angle dimension so that an angle relationship matrix can be constructed subsequently.

[0091] Constructing the Gramian Angular Summation Field (GASF): Define each pixel of GASF ( i, j )for:

[0092] Expand using trigonometric identities:

[0093] because:

[0094] Therefore, GASF can be represented as:

[0095] A symmetric matrix is ​​obtained It can be directly regarded as an image and input into a convolutional neural network (CNN).

[0096] (4) Feature extraction of tabular data An innovative feature extraction method for structured tabular data is designed, combining column-aware attention and row-wise transformation. This method maintains feature interpretability while efficiently learning heterogeneous column information, missing values, and complex cross-features, making it suitable for high-quality modeling of medical tabular data (such as stroke medical records and vital sign records). The method also includes a Structured Table Multi-View Encoder (STME).

[0097] Column Embedding: Given the original table data: ,in, Indicates the first i A row sample contains d feature columns.

[0098] To preserve the semantics of each column, define a column embedding vector:

[0099] This is concatenated with the original numerical features to form the enhanced feature input:

[0100] in, It is the value in the i-th row and j-th column. This indicates a splicing operation.

[0101] Missing value-aware imputation mechanism: For missing values Use learnable tokens As its agent:

[0102] This mechanism allows the model to automatically learn how to represent missing data during training, rather than relying on fixed imputation strategies such as mean / median.

[0103] Column-aware attention mechanism: Traditional feature extraction methods ignore the interaction relationships between different feature columns in context. This invention introduces a column-aware attention mechanism to focus on the interaction relationships between feature vectors within the same row. Perform global interaction modeling. Define attention weights:

[0104] in, , , These are the corresponding query, key, and value vectors, respectively.

[0105] The final output is:

[0106] This structure enables the model to learn the importance relationships between feature columns, making it particularly suitable for medical feature interaction modeling.

[0107] Row-wise Transformer: In batch samples (multiple rows), the Transformer is used to model distributional or temporal dependencies across samples (such as multi-day hospital vital signs records), in the form of:

[0108] Each This is the global representation of the i-th sample.

[0109] 4. Feature Fusion We propose a Federated Aware-aligned Multi-head Attention Fusion (FA-MAF) framework, which consists of three steps: Intra-modal Feature Enhancement, Cross-modal Attention Fusion, and Fusion Representation & Prediction.

[0110] (1) Intramodal feature enhancement: Data for each modality (such as brain images, time series, tables, and protein causal features) is first represented and learned using a specific deep model; Brain imaging data Extracting high-order visual features using 3D CNN ; Time series data Features are obtained by converting images using Gram Angular Field (GAF) and then applying a CNN. ; tabular data Extraction through attention mechanisms ; Protein causal characteristics Directly using causal path aggregation features ; All obtained features are normalized to the same dimension. .

[0111] (2) Cross-modal attention fusion: Define the features of each modality as a modality token and input them into the modality attention fusion block: Construct the query (Q), key (K), and value (V) matrix: , ,

[0112] Calculate cross-modal attention score:

[0113] Aggregation and fusion characteristics:

[0114] All modal fusion outputs are concatenated to form the final representation:

[0115] (3) Output module:

[0116] In other embodiments, the method provided by the present invention is as follows: Figure 2 As shown, it includes: (1) System initialization Step 1: Construction and distribution of the global model (i.e., the global multimodal fusion model) Global Model Construction: A global multimodal fusion model is initialized on the server side. This model is compatible with multimodal data types from various hospitals (such as medical images, electronic medical records, and genetic data). The model architecture typically employs a cross-modal attention mechanism to ensure effective alignment and interaction of features from different modalities.

[0117] Model broadcasting: The server distributes the initial model (global multimodal fusion model) parameters to each hospital node participating in federated learning via a secure communication protocol.

[0118] (2) Local model training phase for each target node (hospital) Step 2: Multimodal data preprocessing Data alignment and standardization: Each hospital cleans and aligns its local multimodal data and then standardizes it.

[0119] Feature extraction and fusion: Input data is processed by a multimodal feature extractor to extract features and generate a joint representation.

[0120] Step 3: Local Model Update Federated learning with local training: Each hospital updates its model parameters using stochastic gradient descent based on local data.

[0121] Key formula:

[0122] in, Here are the local model parameters for the i-th hospital, and η is the learning rate. This is the loss function.

[0123] Step 4: Upload Model Parameters Each hospital will encrypt the local model parameters ( Uploaded to the server, only parameters are transmitted, not the raw data, to ensure data privacy; Gradient clipping and differential privacy noise injection: During local model parameter updates, gradient pruning is used to limit gradient magnitude and prevent excessive consumption of the privacy budget by gradient explosion. The variance of the differential privacy noise is dynamically adjusted based on the model convergence phase. Early training phase (low noise): Inject low-intensity noise to accelerate model convergence and prioritize performance improvement.

[0124] Later in the training phase (high noise): gradually increase the noise level, but strictly adhere to the privacy budget (ε-δ constraint) to ensure privacy and security.

[0125] By using staged noise control, the overall privacy loss can be controlled while reducing the negative impact on model accuracy.

[0126] (3) Global model aggregation and update Step 5: Server Aggregation Federal Average (FedAvg): The server performs a weighted average of the N received local model parameters:

[0127] The problem of data distribution not being independent and identically distributed can be solved by introducing regularization terms or correction terms.

[0128] Dynamic weight adjustment: The aggregate weights are dynamically adjusted based on the data quality of each hospital (such as label consistency) or model performance (such as local validation set AUC) to improve the robustness of the global model.

[0129] Step 6: Global model distribution The server distributes the updated global model parameters to each hospital, completing one round of federated learning iteration. If the model size is large, parameter compression (quantization) can be used to reduce communication overhead. During the aggregation phase, client weights are dynamically adjusted based on local model performance and data quality, prioritizing the aggregation of parameters from high-quality nodes to balance privacy protection with global model performance.

[0130] (4) Model deployment and inference Step 7: Stroke Risk Assessment Local model fine-tuning: Each hospital can further fine-tune the latest global model using local data to generate personalized early warning models; Privacy-Performance Validation Module: Before global model deployment, a dual verification process of privacy budget audit and performance validation ensures an optimal balance between privacy protection and model utility. If the privacy budget is exceeded or performance significantly degrades, an adaptive rollback mechanism is triggered to readjust the noise strategy or aggregate weights. Real-time risk assessment: The deployed model receives multimodal data input from patients and outputs the probability of stroke risk. The system can be linked to the Hospital Information System (HIS) to trigger early warning prompts and intervention suggestions.

[0131] In some embodiments, the workflow of each model on multiple target nodes is as follows: Figure 3 As shown, it includes: (1) Input layer: multimodal data source Hospital N's local model receives various heterogeneous medical data inputs, including but not limited to: Modal 1: 3D brain MRI images (spatial data); Modality 2: Proteomics expression matrix (causal relationship data); Modal 3: Physiological temporal signals (EEG); Modal M: Structured clinical tabular data (electronic medical records); The number of modalities (M) can be dynamically expanded according to the actual scenario (such as adding new genome data).

[0132] (2) Feature extraction layer: Modality-specific modeling Each modal data is modeled using a dedicated network to ensure modal heterogeneity compatibility: Modus 1 (MRI images), using a multi-scale attention-guided 3D convolutional network (MA-3DNet): Multi-scale convolution path: Parallel 3D convolution kernels capture spatial features of different granularities; Spatial-channel attention (SCAM): Highlights key lesion areas through channel attention and spatial masking; Residual connections: prevent gradient vanishing in deep networks and enhance feature reuse capabilities.

[0133] Modality 2 (Proteomics), based on the NOTEARS causal inference model: Structural equation modeling (SEM): learning the potential causal dependencies between proteins (adjacency matrix W); Causal path feature aggregation: generating regulatory path features based on the causal graph; Modality 3 (time-series signal), Gramian Angular Field (GAF) + CNN: Time-to-image conversion: Converting a one-dimensional signal into a two-dimensional GASF image while preserving its time dependence; CNN feature extraction: Utilizing convolutional layers to capture local temporal patterns and global trends; Modal 4 (tabular data), Structured Tabular Multidimensional Encoder (STME): Column-aware attention: dynamically modeling the interaction relationships between feature columns (such as the correlation between blood pressure and blood sugar); Missing value learning and imputation: Adaptive generation of missing value representations (Learnable Tokens); Linear Transformer: Captures temporal or distributional dependencies across samples (such as vital signs records over consecutive days).

[0134] (3) Feature fusion layer: cross-modal attention collaboration Intramodal feature enhancement: Features from each modality are standardized and reduced in dimensionality (e.g., using fully connected layers) to be unified to the same dimension. d Generate intermediate representations: ; Cross-modal attention fusion: Query-Key-Value (QKV) mechanism: Each modality feature As a query, it interacts with other modal features:

[0135] Multi-head attention extension: Parallel multiple attention heads capture different interaction patterns, enhancing fusion robustness; Modal alignment module: Introduces a learnable alignment matrix to alleviate differences in feature space distribution (such as scale inconsistency between image features and text features). Fusion Output: Concatenate multi-head attention outputs and generate a global joint representation through a fully connected layer. As input for risk assessment.

[0136] (4) Output layer: Stroke risk assessment and interpretability Risk assessment module: Classification Header: Based on The sigmoid activation function is used to output the stroke risk probability Prisk∈[0,1]. Threshold trigger: Set a clinical threshold to trigger an early warning signal to the hospital's HIS system; Interpretability module: Attention weight visualization: Shows the contribution of each modality to the prediction results; Thermal localization: Attention thermal maps are overlaid on the original MRI images to identify high-risk lesion areas; Causal pathway analysis: Identifying key regulatory pathways from proteomics data.

[0137] (5) Federated learning adaptation mechanism Local training process: Loss function: Multi-task loss (cross-entropy loss + modality reconstruction loss) Optimize model parameters:

[0138] in, It is classification loss ( The weighting coefficients are used to adjust the model's emphasis on the stroke risk classification task, increasing... This will make the model focus more on improving prediction accuracy, but may weaken the feature reconstruction ability. It is the modal reconstruction loss ( The weighting coefficients are used to control the optimization intensity of the model's ability to reconstruct features from multimodal data. Increasing... This can enhance the model's learning of representations of input data and improve robustness, but may reduce classification performance. By adjusting... and This achieves a dynamic balance between classification accuracy and feature representation capability, ensuring that the model, within a privacy-preserving federated learning framework, can both meet the requirements for high-precision early warning and effectively capture the essential features of multimodal data.

[0139] Privacy protection: Gradient pruning + differential privacy noise injection to meet the differential privacy budget; Model upload: Quantization is used to reduce communication overhead.

[0140] This invention includes the following key technical solutions: (1) Multimodal data standardization and local feature modeling mechanism This invention employs a modular network structure, training convolutional neural networks, gene sequence analysis models, and time-series data processing networks locally at each medical node, and independently modeling each modality to effectively address the problem of modal heterogeneity.

[0141] (2) Cross-modal model aggregation mechanism under federated learning An improved federated averaging algorithm is introduced to aggregate parameters across modal sub-models, avoiding the exchange of raw data and achieving privacy protection. To address the issues of modal heterogeneity and device heterogeneity, this invention designs a modality-aware adaptive aggregation strategy.

[0142] (3) Feature cascade fusion network that integrates attention mechanisms In the global model, a multi-head attention mechanism is used to perform weighted fusion of multimodal features, and a modality alignment module is designed to alleviate the inconsistency problem of feature space, thereby achieving cross-modal collaborative perception.

[0143] (4) Explainable stroke risk prediction and early warning module This invention integrates a relevant interpreter at the model output end to provide doctors and patients with a visual interpretation of stroke risk assessment, thereby improving the clinical usability of the system.

[0144] (5) Optimization and incremental learning capabilities of heterogeneous edge devices To address the issue of limited resources at edge nodes, a model pruning and knowledge distillation mechanism is proposed, and incremental federated training is supported, which can meet the need for rapid adaptation to new modalities and new diseases.

[0145] The purpose of this invention is to provide a multimodal intelligent early warning method for stroke based on federated learning, aiming to solve the following key technical problems: Addressing the conflict between data privacy protection and model performance: This invention introduces a federated learning framework, enabling collaborative modeling of multi-institutional data through local training at each target node and sharing of model parameters. This avoids centralized storage and transmission of raw data, effectively improving model security. Simultaneously, by designing a modality-aware adaptive aggregation mechanism, it enhances modeling performance in situations with incomplete data or heterogeneous devices, overcoming the bottleneck of declining model performance caused by traditional privacy protection techniques.

[0146] The complexity of heterogeneous multimodal data fusion modeling: Stroke risk prediction involves multiple heterogeneous modalities, including MRI images, EEG and other time-series data, genetic information, and clinical text records. Existing systems struggle to achieve efficient fusion and collaborative modeling. This invention achieves effective integration and cross-modal perception of different data types through modular local feature modeling, fusion attention mechanisms, and modality alignment strategies, significantly improving prediction accuracy and system robustness.

[0147] Model deployment and adaptability issues in resource-constrained environments: Primary healthcare institutions generally face problems such as insufficient computing power and high network latency, which limit the practical application of highly complex models. This invention employs model compression, knowledge distillation, and incremental training strategies at the edge device level to effectively reduce the model's computational resource requirements while enhancing the system's ability to quickly adapt to new diseases and new modalities.

[0148] The lack of interpretable models leads to clinical trust issues: Traditional deep learning models exhibit significant black-box characteristics in medical early warning systems, lacking interpretability and limiting doctors' trust and acceptance of the model's predictions. This invention integrates an interpretability module into the model output, providing visual explanations and key feature annotations for risk assessments, assisting doctors in understanding the basis of predictions and enhancing clinical decision support capabilities.

[0149] Features of this invention: (1) Collaborative modeling and robust fusion mechanism for multimodal heterogeneous data By independently processing heterogeneous modalities such as MRI images, genetic data, and time-series signals through a modular network structure, and combining a multi-head cross-modal attention mechanism and a modality alignment module, efficient feature space fusion is achieved. Simultaneously, causal path feature extraction (NOTEARS) and elastic aggregation strategies are designed to significantly improve adaptability to scenarios with missing modalities and heterogeneous devices, solving the performance degradation problem caused by incomplete data or modality mismatch in traditional methods.

[0150] (2) Privacy-Performance Balanced Federated Learning Architecture An improved federated averaging algorithm is proposed, which introduces dynamic weight adjustment (based on data quality and local validation performance) and modality-aware aggregation strategy. Combined with gradient pruning and local differential privacy technology, it effectively balances model convergence speed and global performance while ensuring patient data privacy, thus breaking through the bottleneck of declining model utility caused by traditional privacy protection technologies.

[0151] (3) Lightweight deployment and clinical fit design with enhanced interpretability The system employs model pruning and knowledge distillation techniques to compress model size, supporting millisecond-level real-time alerts for edge devices. It integrates interpretability modules (attention heatmaps, causal path analysis) to provide visual evidence for risk prediction. Simultaneously, it supports incremental federated training, enabling rapid integration of new modalities or disease data, significantly enhancing the system's practicality and doctor trust in primary healthcare scenarios.

[0152] This invention overcomes several bottlenecks in existing stroke risk prediction technologies regarding data privacy protection, multimodal data modeling, edge device deployment, and clinical interpretability, demonstrating the following core features and technological advantages: (1) Coordinated optimization of efficient multimodal fusion and privacy protection By using modular local feature modeling and multi-head cross-modal attention mechanisms, the fusion challenge of heterogeneous data (such as MRI images, time-series signals, and genetic data) is solved. Combined with an improved federated averaging algorithm and a multimodal privacy strategy (encryption of highly sensitive data and differential privacy of low-sensitivity data), the model significantly improves performance and cross-institutional collaboration efficiency while protecting patient privacy.

[0153] (2) Lightweight edge deployment and dynamic adaptability Model pruning and knowledge distillation techniques are used to compress model size, enabling millisecond-level real-time early warning for low-computing-power devices in primary hospitals; incremental federated training is supported, which can quickly adapt to new modal data (such as newly added genomics) or new disease information, enhancing the system's flexibility and lifecycle management capabilities in complex medical scenarios.

[0154] (3) Enhanced interpretability of clinical decision support It integrates attention weight heatmap, causal path analysis and risk feature annotation modules to intuitively display the basis of model decision-making (such as MRI lesion area, key protein regulatory pathway), improve doctors' trust in the prediction results, and promote the practical implementation and application of intelligent early warning system in clinical scenarios.

[0155] like Figure 4 As shown, the present invention also provides a multimodal intelligent early warning system for stroke 400, comprising: The first distribution module 401 is used to distribute the first global multimodal fusion model based on the cross-modal attention mechanism to multiple target nodes; The model update module 402 is used to obtain local model parameters returned by multiple target nodes, calculate a weighted average of the local model parameters, and update the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain a second global multimodal fusion model; the local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data; The second distribution module 403 is used to distribute the second global multimodal fusion model to multiple target nodes, adjust the parameters of the second global multimodal fusion model based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model, and perform stroke intelligent early warning based on the multimodal stroke intelligent early warning model and the local medical data to be tested of the target nodes.

[0156] The multimodal stroke intelligent early warning system provided in the above embodiments can realize the technical solutions described in the above multimodal stroke intelligent early warning method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above multimodal stroke intelligent early warning method embodiments, which will not be repeated here.

[0157] like Figure 5 As shown, the present invention also provides an electronic device 500. The electronic device 500 includes a processor 501, a memory 502, and a display 503. Figure 5 Only some components of the electronic device 500 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0158] In some embodiments, memory 502 may be an internal storage unit of electronic device 500, such as a hard disk or memory of electronic device 500. In other embodiments, memory 502 may also be an external storage device of electronic device 500, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 500.

[0159] Furthermore, the memory 502 may include both internal storage units of the electronic device 500 and external storage devices. The memory 502 is used to store application software and various types of data installed on the electronic device 500.

[0160] In some embodiments, processor 501 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 502 or process data, such as the multimodal intelligent stroke early warning method of the present invention.

[0161] In some embodiments, display 503 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 503 is used to display information from electronic device 500 and to display a visual user interface. Components 501-503 of electronic device 500 communicate with each other via a system bus.

[0162] In some embodiments of the present invention, when the processor 501 executes the multimodal stroke intelligent early warning program in the memory 502, the following steps can be implemented: Distribute the first global multimodal fusion model, constructed based on a cross-modal attention mechanism, to multiple target nodes; The system obtains local model parameters returned by multiple target nodes, calculates a weighted average of the local model parameters, and updates the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain a second global multimodal fusion model. The local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data. A second global multimodal fusion model is distributed to multiple target nodes. The parameters of the second global multimodal fusion model are adjusted based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model. Stroke intelligent early warning is then performed based on the multimodal stroke intelligent early warning model and the local medical data to be tested from the target nodes.

[0163] It should be understood that when the processor 501 executes the multimodal stroke intelligent early warning program in the memory 502, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0164] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 500 mentioned. Electronic device 500 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, electronic device 500 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the multimodal intelligent stroke early warning method provided by the methods described above, the method comprising: Distribute the first global multimodal fusion model, constructed based on a cross-modal attention mechanism, to multiple target nodes; The system obtains local model parameters returned by multiple target nodes, calculates a weighted average of the local model parameters, and updates the parameters of the first global multimodal fusion model on the server based on the weighted average to obtain a second global multimodal fusion model. The local model parameters are obtained by each target node through local training of the first global multimodal fusion model based on local medical data. A second global multimodal fusion model is distributed to multiple target nodes. The parameters of the second global multimodal fusion model are adjusted based on the local medical data of each target node to obtain a multimodal stroke intelligent early warning model. Stroke intelligent early warning is then performed based on the multimodal stroke intelligent early warning model and the local medical data to be tested from the target nodes.

[0166] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0167] The multimodal intelligent early warning method and system for stroke provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-modal intelligent stroke early warning method, characterized in that, The method comprises the following steps: distributing a first global multi-modal fusion model constructed based on a cross-modal attention mechanism to a plurality of target nodes; obtaining local model parameters returned by the plurality of target nodes, and calculating a weighted average value of the local model parameters, and updating parameters of the first global multi-modal fusion model on the server based on the weighted average value to obtain a second global multi-modal fusion model; the local model parameters are obtained by locally training the first global multi-modal fusion model based on local medical data of each target node; distributing the second global multi-modal fusion model to the plurality of target nodes to adjust parameters of the second global multi-modal fusion model based on local medical data of each target node to obtain a multi-modal intelligent stroke early warning model, and performing intelligent stroke early warning based on the multi-modal intelligent stroke early warning model and in combination with to-be-tested local medical data of the target nodes.

2. The multi-modal intelligent stroke early warning method according to claim 1, characterized in that, The local training of the first global multi-modal fusion model based on the local medical data comprises the following steps: cleaning, aligning and standardizing the local medical data to obtain preprocessed data; inputting the preprocessed data into a multi-modal feature extractor to extract features to obtain local medical data features; locally training the first global multi-modal fusion model based on the local medical data features; wherein the local medical data comprises any one of brain image data, proteomics data, electroencephalogram data and electronic medical record data.

3. The multi-modal stroke intelligent early warning method of claim 2, wherein, The multi-modal feature extractor comprises any one of a MA-3Dnet model, a NOTEARS model, a time series feature extraction model and a structured table multi-dimensional encoder; the time series feature extraction model is composed of a GAF model and a CNN model; wherein the MA-3Dnet model is used for feature extraction of brain image data; the NOTEARS model is used for feature extraction of proteomics data; the time series feature extraction model is used for feature extraction of electroencephalogram data; the structured table multi-dimensional encoder is used for feature extraction of electronic medical record data.

4. The multi-modal stroke intelligent early warning method of claim 2, wherein, The local training of the first global multi-modal fusion model based on the local medical data features comprises the following steps: in the early stage of the local training of the first global multi-modal fusion model based on the local medical data features, injecting noise into the local first global multi-modal fusion model for training; in the later stage of the local training of the first global multi-modal fusion model based on the local medical data features, gradually enhancing the noise injected in the early training process; wherein the enhanced noise satisfies a privacy budget; and in the process of locally training the first global multi-modal fusion model, a gradient clipping technique is used to limit the gradient amplitude of the local first global multi-modal fusion model.

5. The multi-modal stroke intelligent warning method of claim 1, wherein, The calculation of the weighted average value of the local model parameters comprises the following steps: dynamically adjusting the weights of the local model parameters returned by each target node according to the quality of the local medical data of each target node and the model performance obtained after locally training the first global multi-modal fusion model. Determine a weighted average of the local model parameters returned by the plurality of target nodes based on the weights of the local model parameters returned by the plurality of target nodes and the local model parameters returned by the plurality of target nodes.

6. The multi-modal stroke intelligent early warning method of claim 5, wherein, Before distributing the second global multi-modal fusion model to the plurality of target nodes, the method comprises: Conducting double-checking of privacy budget auditing and performance verification on the second global multi-modal fusion model obtained at present; In a case where it is determined based on the results of the double-checking that the second global multi-modal fusion model has privacy budget overrun or model performance degradation, re-adjusting the noise intensity injected during the local training of the first global multi-modal fusion model or re-adjusting the weights of the local model parameters returned by the plurality of target nodes to obtain the second global multi-modal fusion model again.

7. The multi-modal intelligent pre-warning method of stroke according to any one of claims 1-6, characterized in that, The multi-modal stroke intelligent early warning model is used for: standardizing and dimensionally reducing the input local medical data to be tested to obtain intermediate representations; constructing a QKV mechanism matrix corresponding to the intermediate representations; calculating cross-modal attention scores based on the QKV mechanism matrix; calculating a plurality of aggregated fusion features based on the cross-modal attention scores and values in the QKV mechanism matrix; concatenating the plurality of aggregated fusion features to obtain concatenated features; performing intelligent early warning of stroke based on the concatenated features.

8. The multi-modal stroke intelligent warning method of claim 7, wherein, Performing intelligent early warning of stroke based on the concatenated features comprises: determining a stroke risk probability based on the concatenated features; in a case where the stroke risk probability is greater than a preset probability threshold, sending an early warning signal to a HIS system of a target node to perform intelligent early warning of stroke.

9. The multi-modal stroke intelligent warning method of claim 7, wherein, The multi-modal stroke intelligent early warning model is also used for: determining the contribution of each modality feature of the local medical data to be tested to the prediction result; performing heat map positioning in the local medical data to be tested based on the contribution to demarcate a lesion area, and performing causal path analysis based on the local medical data to be tested to obtain a key regulation path.

10. A multi-modal stroke intelligent early warning system, characterized in that, The method comprises: a first distribution module configured to distribute a first global multi-modal fusion model constructed based on a cross-modal attention mechanism to a plurality of target nodes; a model updating module configured to obtain local model parameters returned by the plurality of target nodes, calculate a weighted average of the local model parameters, and update parameters of the first global multi-modal fusion model on the server based on the weighted average to obtain a second global multi-modal fusion model; the local model parameters are obtained by local training of the first global multi-modal fusion model based on local medical data of each target node; a second distribution module configured to distribute the second global multi-modal fusion model to the plurality of target nodes, adjust parameters of the second global multi-modal fusion model based on local medical data of each target node to obtain a multi-modal stroke intelligent early warning model, and perform intelligent early warning of stroke based on the multi-modal stroke intelligent early warning model and in combination with local medical data to be tested of each target node.

Images