A prediction method for post-stroke cognitive impairment based on multimodal feature fusion

Abstract

This invention discloses a method for predicting post-stroke cognitive impairment based on multimodal feature fusion. The method first collects multimodal information from stroke patients: three-dimensional brain MRI images, EEG signals, and clinical medical records. Next, the MRI images are converted into tensors and input to a multi-scale spatiotemporal feature extraction backbone network to obtain MRI modal features; the EEG signals are processed using neural electrical signals combined with a Transformer to obtain EEG modal features; and the stroke patient's clinical medical record information is converted into semantic sentences and input into a dual-channel semantic encoder for encoding and extraction to obtain clinical medical record information features. Finally, the MRI modal features, EEG modal features, and clinical medical record information features are input into a three-modal fusion processor to obtain fused features, which are then processed by a classifier to output probability predictions. This invention achieves accurate prediction of post-stroke cognitive impairment, significantly improving robustness and medical interpretability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image analysis, and in particular to a method for predicting post-stroke cognitive impairment (PSCI) based on multimodal feature fusion. Background Technology

[0002] With the widespread application of deep learning in medical image processing, classification models based on deep neural networks have demonstrated outstanding performance in a variety of tasks.

[0003] Traditional medical image processing methods often rely on single-modal data input, such as magnetic resonance imaging (MRI) or rating scales, and extract visual features using structures like convolutional neural networks. However, this approach often struggles to comprehensively capture disease manifestations in real-world environments with incomplete information and severe noise interference, limiting the model's robustness and generalization ability. In recent years, multimodal fusion methods have been proposed to integrate complementary information from different modalities, but mainstream methods still suffer from the following shortcomings:

[0004] First, existing multimodal methods often lack effective intermodal semantic interaction mechanisms, failing to fully model cross-modal associations between MRI images, EEG signals, genomics, and clinical medical records, resulting in limited fusion feature representation capabilities. Second, most models employ static embedding or tabular modeling methods for text processing, making it difficult to leverage natural language semantic enhancement models for understanding. Third, while some works have attempted to introduce cue learning mechanisms to optimize language modality processing, most are limited to general tasks such as image-text matching, lacking customized improvements for medical scenarios.

[0005] Furthermore, existing methods lack robust intramodal modeling capabilities and feature alignment mechanisms to address challenges such as blurred boundaries, low contrast, and unstructured noise frequently encountered in medical images and neural electrical signals, making them difficult to adapt to complex and multi-source clinical data. At the modal fusion level, traditional feature concatenation or simple attention mechanisms struggle to capture deep cross-modal interaction semantics, especially in real-world scenarios with limited sample sizes, where models are prone to overfitting or information redundancy.

[0006] Therefore, how to construct a multimodal neural network model that simultaneously possesses semantic understanding capabilities, cross-modal alignment capabilities, and an efficient fusion mechanism to address the complex data distribution and practical needs in the classification of post-stroke cognitive impairment has become an urgent problem to be solved in the field of medical artificial intelligence. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a method for predicting post-stroke cognitive impairment based on multimodal feature fusion. This method, building upon a lightweight deep neural network, introduces medical language reconstruction, semantic-driven context optimization, and a dual-channel semantic encoder, enabling deep alignment and interactive fusion of clinical medical record information with modalities such as MRI images and EEG signals. Simultaneously, through a guided cross-modal semantic interaction structure, the feature distribution of MRI images and EEG signals is dynamically adjusted using text semantics as anchors, and a unified semantic representation is generated through a semantic fusion network, effectively improving information consistency and collaborative perception capabilities between modalities. Finally, by constructing a unified representation space, multimodal features are accurately mapped to cognitive state categories, achieving accurate prediction of post-stroke cognitive impairment and significantly improving the model's robustness, generalization ability, and medical interpretability.

[0008] The technical problem solved by this invention includes the following steps:

[0009] S1. Multimodal Data Acquisition: This invention first collects multimodal information from stroke patients, including three-dimensional brain MRI images, EEG signals, and clinical medical records, and constructs a dataset in a unified format.

[0010] S2. MRI Image Data Preprocessing: The raw MRI image samples acquired in step S1 are converted into single-channel data with dimensions of [dimension not specified]. , in These represent the depth, height, and width of the image, respectively. To achieve uniformity in feature extraction and adaptability to model input, this invention performs a standardized preprocessing procedure on MRI images, including image size normalization, intensity normalization, and resampling. All MRI image samples are then registered to the MNI (Montreal Neurological Institute) standard template and finally converted into tensor form.

[0011] S3. MRI Image Feature Extraction: The MRI images preprocessed in step S2 are input into a multi-scale spatiotemporal feature extraction backbone network (such as 3D ResNet18, 3D ResNet50) to extract spatial structural features and high-order semantic information layer by layer, thus obtaining high-dimensional features. This refers to MRI modal characteristics. Specific steps include:

[0012] S31. The preprocessed MRI images are input into a multi-scale spatiotemporal feature extraction backbone network. First, feature extraction is performed through an initial 3D convolutional layer. Then, the extracted features are input into a batch normalization layer, a ReLU activation function, and a max pooling layer to obtain spatial features. .

[0013] S32. Then spatial features Enter to A series of feature extraction modules - Extracting high-order semantic features .

[0014] S33. Finally, the higher-order semantic features The input is fed into a global average pooling layer, by adjusting the spatiotemporal dimension. The average of all values ​​is used to extract global semantic feature representation. Then, after flattening, it is converted into a one-dimensional feature vector. .

[0015] S4. EEG Signal Feature Extraction: The EEG signals acquired in step S1 are processed using standard neural electrical signal preprocessing procedures, and multi-channel temporal semantic features are extracted using a Transformer network structure to obtain the final features of the EEG modality. The processing procedure includes the following steps:

[0016] S41. The acquired EEG signals are preprocessed, and the acquired multi-channel EEG time-series signals are denoted as: ,in, Indicates the number of brainwave channels. This indicates the number of time steps corresponding to the sampling duration. (For EEG signals) The following preprocessing steps were performed: bandpass filtering, artifact removal, rereference, data truncation, and normalization, to obtain the preprocessed EEG signal sample tensor. ,in Indicates the number of segments. For the number of channels, The length of each time segment.

[0017] S42. For the EEG signal sample tensor obtained in step S41 Each EEG signal segment Dimension transpose Then, the EEG signal sample tensor at each time point is regarded as a The sequence, firstly, is processed through a linear projection layer to transform each... The dimension from Upgraded to Then, by adding the corresponding positional encoding, the final embedded representation of the EEG signal is obtained. .

[0018] S43. Embed the EEG signals obtained in step S42 into a representation. The input is fed into a multi-layer Transformer encoder module to model global dependencies and cross-time-slice contextual information in EEG signal sequences. This encoder consists of... The Transformer layer consists of blocks, each layer comprising three sub-modules: a multi-head self-attention mechanism, a feedforward neural network, and residual connections and layer normalization. Specifically, it includes the following steps:

[0019] S431. In the In the Transformer block, the input is the previous layer (i.e., the first layer). (layer) output Embedding EEG signals to represent The input is fed into a multi-head self-attention mechanism module to model the correlation between time slices of EEG signals over time. The multi-head self-attention mechanism maps the input embeddings to query vectors (Q), key vectors (K), and value vectors (V), respectively. Then, it calculates the scaling dot product attention for each step in the multi-head self-attention mechanism, and calculates the attention weights based on the similarity between the query vector and all key vectors. After normalization and weighted summation of all attention weights, the attention representation at each time point is obtained, with the following shape: , as follows:

[0020] To enhance the model's ability to capture diverse semantic features from different subspaces, the following is introduced: Attention calculations are performed in parallel by multiple attention heads. .

[0021] All weighted attention weights are concatenated and then unified to the original dimension through a linear transformation. This forms the output of the self-attention module in this layer. It is consistent with the input dimension, which facilitates residual connection.

[0022] S432. The output of the multi-head self-attention mechanism module obtained in step S431. After residual connections and layer normalization, the data is fed into a feedforward neural network module to enhance nonlinear expressive power and independently model the representation vector at each time step. This module contains two convolutional layers and one... Regularization operation. The specific process is as follows:

[0023] The input normalized vector is:

[0024]

[0025] in, Indicates the first The output of the layer Transformer block, This indicates a multi-head self-attention mechanism. The representation layer normalization involves performing a residual concatenation between its output and the original input, followed by applying layer normalization again to obtain the normalized vector. .

[0026] Normalized vector First, go through the first layer. Convolution expands the channel dimension, then uses... Regularization operation, then through the second layer Convolution restores the dimensionality, and finally residual connections and layer normalization are performed. The specific formula is as follows:

[0027]

[0028] in, Indicates the kernel size as One-dimensional convolution operation, It is a regularization operation. This indicates the probability of discarding (e.g., 0.1 or 0.2). This indicates a normalization operation. Indicates the first Output of the layer Transformer block.

[0029] S433. The above process is in This process is repeated within the layer, ultimately outputting a set of semantic vectors.

[0030]

[0031] in, This indicates the number of time steps in a segment of EEG brainwave signal. This represents the embedding dimension corresponding to each time point. This represents the output of each Transformer block.

[0032] S44. Set the semantic vectors output in step S433. Global average pooling is used to reduce the dimensionality of the time dimension, preserving key time-series information while reducing the number of parameters. This step reduces the dimensionality from... Reduced to The output is a semantic representation of each EEG signal segment. This is the final feature representation of the EEG mode.

[0033] S5. Clinical Medical Record Information Feature Extraction: The clinical medical record information of stroke patients collected in step S1 is transformed into unified semantic sentences through a cue learning mechanism, and then input into a dual-channel semantic encoder for encoding and extraction to obtain clinical medical record information features. Clinical medical record information includes the patient's structured scores (such as cognitive assessment scores), basic demographic information (such as gender, age, and education level), and genomic information. The specific steps are as follows:

[0034] S51. Represent the collected clinical medical record information of stroke patients as follows: The information was then mapped to natural language fragments and assembled into complete medical statements. For example: "The patient has an MMSE score of 26, is 67 years old, has 12 years of education, and exhibits mild cognitive decline." Each participant constructed a personalized clinical description statement. It is used to guide the model to obtain contextual semantic information.

[0035] S52. To enhance the clinical description statement obtained in step S51 The semantic guidance capability in text models is achieved by introducing learnable contextual cue vectors, represented as follows: :

[0036] The final input sentence is composed of a prompt vector and a clinical description sentence. .

[0037] S53. Enhance the text sequence of prompts obtained in step S52. Input to pre-training Text encoder and The encoder obtains two high-dimensional semantic representations; then, the difference between the two high-dimensional semantic representations is used to construct a semantic difference vector. Finally, semantic correction is integrated using learnable weights. Control introduction The degree of weighted fusion Semantic representation and semantic difference vector output by the encoder To achieve semantic alignment and obtain clinical medical record information features Characteristics of the obtained clinical medical record information Will be compared with MRI modal characteristics EEG modal characteristics They are input together into the three-modal fusion processor for joint modeling.

[0038] S6. Cross-modal semantic interaction fusion: Combine the MRI modal features obtained in steps S3, S4, and S5. EEG modal characteristics and clinical medical record information characteristics The input is fed into a three-modal fusion processor, where it undergoes feature extraction via a semantically guided perception module and a semantically guided interaction module to obtain the final fused features. This is used for downstream PSCI classification tasks. The semantic guidance perception module and semantic interaction guidance module construct a bidirectional attention mechanism between modalities, introduce semantic anchors to guide alignment, and use learnable weights for channel unification. The specific design of the three-modal fusion machine is as follows:

[0039] S61. Regarding MRI modal characteristics EEG modal characteristics and clinical medical record information characteristics First, use clinical medical record information features A bidirectional attention structure is established for any two modalities (e.g., MRI images and clinical medical record information) to capture the implicit correlation and complementary semantics between them. Taking the MRI modality guided by clinical medical record information as an example, the attention calculation process is as follows:

[0040] S611. Constructing a query vector for clinical medical record information MRI image key vectors Value vectors of MRI images :

[0041]

[0042]

[0043]

[0044] in, , , It is a linear transformation matrix used to map modal embeddings to the attention space.

[0045] Computational MRI image attention guided by clinical medical record information The specific formula is as follows:

[0046]

[0047] in, MRI modal characteristics Dimensions It is an activation function. Key vectors representing MRI images The transpose of .

[0048] Similarly, calculating EEG brainwave signals guided by clinical medical record information for attention... .

[0049] S62. Because clinical medical record information typically contains explicit descriptions of the patient's condition and professional terminology, it has stronger semantic clarity. Therefore, the characteristics of clinical medical record information are... As a semantic anchor, the semantic distribution of MRI images and EEG signals is guided to converge towards it. The attention guided by clinical medical record information is first differentially analyzed with the corresponding modal features to obtain an alignment residual. This residual is then scaled using a learnable coefficient γ and added to the original modal features to obtain the enhanced MRI features guided by clinical medical record information. and EEG signal characteristics .

[0050] S63. Guide the enhanced MRI features with the clinical medical record information obtained in step S62. and EEG signal characteristics After concatenation along the channel dimension, the fused features are obtained through a linear fusion module. .

[0051] S7. Fusion Feature Classification: Classify the fusion features obtained in step S6. The input is fed into a classifier, which uses a lightweight multilayer perceptron classifier to model the fused features and output the classification prediction results of PSCI. This classifier consists of two fully connected network layers with a non-linear activation function and a Dropout regularization layer inserted in between to enhance non-linear expressiveness and suppress overfitting, ultimately outputting probability predictions.

[0052] The beneficial effects of this invention are as follows: The model proposed in this invention adopts a multimodal branching structure and a deep neural network fusion strategy to extract and model features from 3D MRI images, EEG signals, and clinical medical records, respectively. This fully leverages the complementary information between different modalities, improving the model's expressive ability and discriminative performance in cognitive impairment classification tasks. The MRI branch uses a hierarchical residual feature extraction architecture built on 3D ResNet18 to extract local and global information of brain structures step by step. Combined with global average pooling, it achieves a progressive abstraction from low-level spatial features to high-order semantic representations, enhancing the model's ability to represent changes in brain tissue. The EEG branch introduces a time-channel attention mechanism, integrating a Transformer structure to model the temporal dependence and channel dimension importance of EEG signals, guiding the model to focus on key neural activity areas, suppressing redundant signal interference, and improving the quality of time series modeling. The text branch innovatively transforms medical record information into natural language guidance sentences and combines them with learnable contextual cue vectors. This paper proposes a medical language model to enhance text semantics, achieving spatial alignment of text and image semantics and improving the accuracy of medical semantics. A guided cross-modal interaction mechanism based on semantic anchors is proposed. This mechanism constructs a clinical medical record information-guided MRI image modality and a clinical medical record information-guided EEG signal modality, achieving semantic guidance and deep fusion of three modal features. This results in a unified multimodal semantic representation, improving feature consistency and modal synergy. During the inference phase, the overall model exhibits good lightweight and generalization performance. The text semantic guidance mechanism significantly improves the model's adaptability to multi-source heterogeneous medical data, and the fusion module supports flexible deployment in resource-constrained environments, demonstrating edge computing friendliness. The model output uses a multilayer perceptron classifier to accurately predict cognitive impairment states and can be extended to tasks related to other neurological diseases, demonstrating good versatility, interpretability, and application prospects. Attached Figure Description

[0053] Figure 1 This is an architecture diagram of a method for predicting post-stroke cognitive impairment based on multimodal feature fusion.

[0054] Figure 2 Structural diagram for feature extraction from MRI images;

[0055] Figure 3 This is a diagram of the residual block structure.

[0056] Figure 4 Structure diagram for feature extraction of EEG signals;

[0057] Figure 5 A structural diagram for feature extraction of clinical medical record information;

[0058] Figure 6 This is a structural diagram of a three-modal fusion converter. Detailed Implementation

[0059] The following diagram further illustrates the method for predicting post-stroke cognitive impairment based on multimodal feature fusion, and its architecture is as follows: Figure 1 As shown, it includes the following steps:

[0060] S1. Multimodal data acquisition: Collect MRI images, EEG signals and clinical medical records of the subjects, and then construct a dataset.

[0061] S2. Save the three-dimensional brain MRI scan images acquired in step S1 as... Format. Let the original MRI image sample be:

[0062]

[0063] in These represent the depth, height, and width of the image, respectively. This represents a single-channel medical image. The following preprocessing steps are performed on the raw image:

[0064] First, the MRI image dimensions are converted... The converted MRI images are then normalized, and the resolution is adjusted to 1 mm³ by resampling. All MRI images are then registered to the standard MNI space and finally converted into the tensor format required by the model.

[0065] S3. MRI Image Feature Extraction: The MRI images preprocessed in step S2 are input into a 3D ResNet18 network to extract their spatial structural features and high-order semantic information layer by layer, outputting high-dimensional features. This module, as the visual modality processing branch of this invention, possesses strong spatial modeling capabilities and can effectively characterize structural changes in brain tissue. The module's structure is as follows: Figure 2 As shown, the specific design and construction are as follows:

[0066] S31. Input the preprocessed MRI images into a multi-scale spatiotemporal feature extraction backbone network, in the following form:

[0067]

[0068] The 3D image volume, used as network input, is first processed through a network containing... 3D convolution (stride is) Fill with The initial convolutional layer is used for feature extraction, followed by a batch normalization layer. Activation function and Max pooling operation (step size is) Fill with ), to obtain preliminary spatial features The specific formula is as follows:

[0069]

[0070] in, This indicates that a step size of 1 is used. ,padding is of 3D convolution, It is a batch normalization operation. express Activation function This indicates the use of a step size of 2 and padding of 1. 3D max pooling operation, The input is three-dimensional MRI image data. These are the preliminary spatial features extracted.

[0071] S32. The preliminary spatial features obtained in step S31 are... Enter to The feature extraction module extracts high-order semantic features. This module contains two concatenated residual blocks 1, with the specific structure as follows: Figure 3 As shown, the residual block contains 3D convolution operations, each convolutional layer is equipped with batch normalization and... Activation function. This module does not perform downsampling, preserving spatial resolution and initial spatial features. After processing by two cascaded residual blocks 1, spatial features are obtained. The specific formula is shown below:

[0072]

[0073] in, This is a feature extraction method. The feature formula for residual block 1 is shown below:

[0074]

[0075] in, This indicates the output of the previous layer. It is a batch normalization operation. Represents the ReLU activation function. Indicates adoption 3D convolution.

[0076] spatial features Enter to The feature extraction module contains a residual block 1 and a residual block 2. The specific structure of residual block 2 is as follows: Figure 3 As shown, this module contains two layers in the main branch. The 3D convolution, with each convolution operation combined with batch normalization and Activation function. Used in residual branches. 3D convolution (stride is The input is downsampled and the number of channels is adjusted to ensure that the dimensions of the main branch and the residual branch are consistent before being summed. Spatial characteristics After feature extraction module After processing, the output features obtained are The specific formula is shown below:

[0077]

[0078]

[0079]

[0080] in, This indicates that a step size of 1 is used. , 3D convolution, This indicates the output of the previous layer. It is a batch normalization operation. express Activation function Indicates adoption 3D convolution.

[0081] because , , The three layers have the same structure, so they are processed together. , Output features respectively , The corresponding characteristic formula is shown below:

[0082]

[0083]

[0084] in, , This indicates that the data has been processed by residual block 1 and residual block 2. The specific processing operations are shown above. , and They are , , Output characteristics of the layer.

[0085] go through A series of feature extraction modules Then, high-order semantic features were extracted. .

[0086] S33. Apply the high-order semantic features obtained in step S32. The input is fed into the global average pooling layer, through the spatiotemporal dimension ( The average of all values ​​on the ) is used to extract the global semantic feature representation. Then, after flattening, it is converted into a one-dimensional feature vector. ,in( )for( The specific calculation formula is as follows:

[0087]

[0088]

[0089] in, This indicates a global average pooling operation. This means flattening the output into a one-dimensional vector. This represents the final image features of the MRI modality, which are used for subsequent multimodal feature fusion.

[0090] S4. EEG Signal Feature Processing: The EEG signals acquired in step S1 are processed using standard neuroelectric signal preprocessing procedures, and multi-channel temporal semantic features are extracted using a Transformer network structure to obtain the final features of the EEG modality. The specific architecture is as follows: Figure 4 As shown, the processing flow includes the following steps:

[0091] S41. Preprocess the acquired EEG signals. First, record the acquired multi-channel EEG timing signals as follows: ,in Indicates the number of brainwave channels. This indicates the number of time steps corresponding to the sampling duration.

[0092] The following preprocessing was performed on the EEG signals:

[0093] S411. Bandpass Filter: A bandpass filter is used to retain frequency components between 0.5 and 45 Hz, effectively filtering out power frequency noise and high-frequency interference.

[0094] S412. Artifact Removal: Using independent component analysis, non-EEG EEG signal interference components such as eye movement, blinking, and electromyography are separated, and then deleted or reconstructed.

[0095] S413. Rereference: Use an average reference or binaural reference strategy to unify the baseline of the entire channel electrodes and improve spatial consistency.

[0096] S414. Data Segmentation: Long-segment EEG signals are sliced ​​using a sliding window method according to fixed time windows (e.g., 2 seconds per segment, 50% overlap) to form time-series samples.

[0097] S415. Standard Deviation Standardization: Standard deviation standardization is performed on each EEG signal according to the channel dimension to normalize the signal intensity between different patients and ensure the stability of model training.

[0098] Finally, after the above steps, the standardized EEG signal sample tensor is obtained: ,in Indicates the number of segments. For the number of channels, The length of each time segment.

[0099] S42. For the EEG signal sample tensor obtained in step S41 Each EEG signal segment Dimension transpose Then treat each time point as a The sequence input is first processed through a linear projection layer to transform each... The dimension from Upgraded to Then, by adding the corresponding location encoding, we obtain the embedded representation of the EEG signal. The specific formula is as follows:

[0100]

[0101] in, For position encoding, This represents a linear transformation operation. This is the transposed input.

[0102] S43. Embed the EEG signals obtained in step S42 into a representation. The input is fed into a multi-layer Transformer encoder module to model global dependencies and cross-time-slice contextual information in EEG signal sequences. This encoder consists of... The Transformer module consists of layers, each layer comprising three sub-modules: a multi-head self-attention mechanism, a feedforward neural network, and residual connections and layer normalization. This patent uses a 4-layer Transformer module, specifically including the following steps:

[0103] S431. In the In the Transformer block, the input is the previous layer (i.e., the first layer). (layer) output EEG signal embedding representation The input is fed into a multi-head self-attention mechanism module to model the correlation between different time slices of EEG signals over time. The multi-head self-attention mechanism maps the input embeddings into query vectors (Q), key vectors (K), and value vectors (V), respectively.

[0104]

[0105]

[0106]

[0107] in, Indicates the first The output of the layer Transformer block, , , This is a learnable weight matrix. Then, scaled dot product attention is calculated, with attention weights derived from the similarity between the query vector and all key vectors. After normalization, all value vectors are weighted and summed to obtain the attention representation at each time point, with the shape as follows: .

[0108] To enhance the model's ability to capture diverse semantic features from different subspaces, the following is introduced: Attention calculations are performed in parallel by multiple attention heads. This patent uses... The specific formula for each attention point is as follows:

[0109]

[0110]

[0111] in, They represent the first The query vector, key vector, and value vector of the Transformer block. This indicates that attention calculation is being performed. Equals 8, Indicates the first The first in the layer The output of each attention head, This represents the similarity matrix between all time points. It is a scaling factor to prevent gradient explosion. This ensures that the attention weights are uniform at each time point. The core of this mechanism is that at each time point, the similarity to the entire sequence is used to determine which historical neural patterns to focus on.

[0112] The features learned from each subspace are concatenated and unified into the original dimension through a linear transformation. This forms the output of the self-attention module in this layer. It is consistent with the input dimension, which facilitates residual connection.

[0113] S432. The output of the self-attention mechanism module obtained in step S431. After residual connections and layer normalization, the data is fed into a feedforward neural network module to enhance nonlinear expressive power and independently model the representation vector at each time step. This module contains two convolutional layers and one... Regularization operation. The specific process is as follows:

[0114] The input normalized vector is:

[0115]

[0116] in, Indicates the first The output of the layer Transformer block, This indicates a multi-head self-attention mechanism. The representation layer normalization involves performing a residual concatenation between its output and the original input, followed by applying layer normalization again to obtain the normalized vector. .

[0117] Normalized vector First, go through the first layer. Convolution expands the channel dimension, then uses... Regularization operation, then through the second layer Convolution restores the dimensionality, and finally residual connections and layer normalization are performed. The specific formula is as follows:

[0118]

[0119] in, This represents a one-dimensional convolution operation with a kernel size of 1. It is a regularization operation. This indicates the probability of discarding (e.g., 0.1 or 0.2). This indicates a normalization operation. Indicates the first Output of the layer Transformer block.

[0120] S433. The above process is in This process is repeated within each layer, and the output of each layer is defined as follows:

[0121]

[0122] in, Indicates the first Layer Transformer block, It equals 4. Indicates the first The output of the layer Transformer block, Indicates the first Output of the layer Transformer block.

[0123] The final output is a set of semantic vectors:

[0124]

[0125] in, This represents the output of each Transformer block.

[0126] S44. Set the semantic vectors output in step S433. Global average pooling is used to reduce the time dimension, preserving key time-series information while reducing the number of parameters. The formula for average pooling is as follows:

[0127]

[0128] in, For the The output vector at each time step This is the final global semantic feature vector of the EEG signal. Indicates along the time dimension Global average pooling operation.

[0129] This step will change the dimension from Reduced to The output is a single semantic representation of each EEG signal segment. This is the final feature representation of the EEG mode.

[0130] S5. Clinical Medical Record Information Feature Extraction: The clinical medical record information of stroke patients collected in step S1 is transformed into unified semantic sentences through a cue learning mechanism, and then input into a dual-channel semantic encoder for encoding and extraction to obtain clinical medical record information features. Clinical medical record information includes the patient's structured scores (such as cognitive assessment scores), basic demographic information (such as gender, age, and education level), and genomic information. The specific structure is as follows: Figure 5 As shown, the steps are as follows:

[0131] S51. Represent the collected clinical medical record information of stroke patients as follows: The information was then mapped to natural language fragments and assembled into complete medical statements. For example: "The patient has an MMSE score of 26, is 67 years old, has 12 years of education, and exhibits mild cognitive decline." Each participant constructed a personalized clinical description statement. It is used to guide the model to obtain contextual semantic information.

[0132] S52. To enhance the clinical description statement obtained in step S51 The semantic guidance capability in text models is achieved by introducing learnable contextual cue vectors, represented as follows:

[0133]

[0134] in, The number of prompt words, For the embedding dimension. The final input sentence is composed of a cue vector and a clinical description statement:

[0135]

[0136] in, It concatenates two variables. It is a learnable cue vector matrix. It is a word segmenter. These are clinical descriptive statements.

[0137] S53. Enhance the text sequence of prompts obtained in step S52. Input to pre-training Text encoder and Encoder, obtain feature and feature The specific formula is as follows:

[0138]

[0139]

[0140] in Indicates use Text encoder encoding, Indicates use Encoder encoding, It is the feature dimension.

[0141] Then construct the semantic difference vector. This vector reflects the "semantic bias" between the two models in understanding the semantics of the same text. The specific formula is:

[0142]

[0143] in, yes feature, yes feature.

[0144] Finally, semantic correction is integrated through learnable weights. Control introduction To achieve semantic alignment, we obtain clinical medical record information features. The specific formula is as follows:

[0145]

[0146] in, yes feature, It is a semantic difference vector. Used to control the introduction The degree of.

[0147] Characteristics of obtained clinical medical record information Will be compared with MRI modal characteristics EEG modal characteristics They are input together into the three-modal fusion processor for joint modeling.

[0148] S6. Cross-modal semantic interaction fusion: Combine the MRI modal features obtained in steps S3, S4, and S5. EEG modal characteristics and clinical medical record information characteristics The input is fed into a three-modal fusion processor, where it undergoes feature extraction via a semantically guided perception module and a semantically guided interaction module to obtain the final fused features. This is used for downstream PSCI classification tasks. The semantic guidance perception module and semantic interaction guidance module construct a bidirectional attention mechanism between modalities, introduce semantic anchors to guide alignment, and use learnable weights for channel unification. The specific structure is as follows: Figure 6 As shown, the specific design of the three-modal fusion device is as follows:

[0149] S61. Regarding MRI modal characteristics EEG modal characteristics and clinical medical record information characteristics First, a bidirectional attention structure is established for any two modalities (e.g., MRI images and clinical medical record information) to capture the implicit correlation and complementary semantics between them. Taking the MRI modality guided by clinical medical record information as an example, the attention calculation process is as follows:

[0150] S611. Constructing a query vector for clinical medical record information MRI image key vectors Value vectors of MRI images :

[0151]

[0152]

[0153]

[0154] in, , , It is a linear transformation matrix used to map modal embeddings to the attention space.

[0155] Computational MRI image attention guided by clinical medical record information The specific formula is as follows:

[0156]

[0157] in, MRI modal characteristics Dimensions It is an activation function. Key vectors representing MRI images The transpose of .

[0158] Similarly, calculating EEG brainwave signals guided by clinical medical record information for attention... , the calculation formula is:

[0159]

[0160] in, EEG modal characteristics Dimensions It is an activation function. Key vectors representing EEG brain signals transpose, This represents a query vector for clinical medical record information. This represents the value vector of the EEG brainwave signal.

[0161] S62. Because clinical medical record information typically contains explicit descriptions of the patient's condition and professional terminology, it has stronger semantic clarity. Therefore, the characteristics of clinical medical record information are... As a semantic anchor, the semantic distribution of MRI images and EEG signals is guided to converge towards it. The attention guided by clinical medical record information is first differentially analyzed with the corresponding modal features to obtain an alignment residual. This residual is then scaled using a learnable coefficient γ and added to the original modal features to obtain the enhanced MRI features guided by clinical medical record information. and EEG signal characteristics The specific guiding formula is as follows:

[0162]

[0163]

[0164] in, These are learnable guiding coefficients used to balance the proportion of alignment information between raw modal information and clinical medical record information. This refers to attention to MRI images guided by clinical medical record information. This indicates attentional EEG signals guided by clinical medical record information.

[0165] S63. Guide the enhanced MRI features with the clinical medical record information obtained in step S62. and EEG signal characteristics After concatenation along the channel dimension, the fused features are obtained through a linear fusion module. .

[0166] S7. Fusion Feature Classification: Classify the fusion features obtained in step S6. The input is fed into a classifier, which uses a lightweight multilayer perceptron classifier to model the fused features and output the PSCI classification prediction. This classifier consists of two fully connected layers with a non-linear activation function and a Dropout regularization layer inserted in between to enhance non-linear expressiveness and suppress overfitting, ultimately outputting a probability prediction. The specific steps are as follows:

[0167] Input fusion features First, it goes through the first layer of a fully connected network, then through a... A regularization layer is used to improve generalization ability, then it passes through a second fully connected network, and finally through... The layer outputs the final probability prediction vector, as shown in the following formula:

[0168]

[0169] in, It is a linear mapping matrix. It is a bias term. It is a non-linear activation function. For the discard rate, It is a classification mapping matrix. It is the output bias. This represents the predicted probability of a sample belonging to each category, with the maximum value corresponding to the predicted label.

[0170] Example:

[0171] The steps in this embodiment are the same as those in the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0172] To verify the effectiveness of the method of this invention, this paper selected real stroke patients from a tertiary hospital as the research subjects. The experimental data included multimodal information such as brain MRI images, EEG signals, and clinical medical records, totaling 286 patients. The imaging data were raw NIfTI format structural MRI images, the EEG data sampling frequency was 256Hz, and the recording time for each patient was no less than 3 minutes. The medical record information included MMSE score, age, gender, years of education, etc. According to follow-up and neuropsychological assessment criteria, patients were categorized into post-stroke cognitive impairment (PSCI) and non-PSCI, with PSCI patients accounting for approximately 54% of the total.

[0173] The experimental results are shown in Table 1. For model training, the multimodal feature fusion network proposed in this invention was used for modeling, and various comparative models were set up for performance verification, including a 3D ResNet-based MRI monomodal network (3DResNet-MRI), a Transformer-based EEG monomodal classifier (Transformer-EEG), a simple feature concatenation model of MRI and EEG (MRI-EEG feature-level concatenation), and a multimodal attention network that integrates MRI, EEG, and clinical medical record information without introducing semantic anchoring mechanisms (anchor-free multimodal attention fusion network). All models were evaluated using a five-fold cross-validation strategy, with cross-entropy as the loss function, Adam as the optimizer, and an initial learning rate of 0.001. The experiments were run on an NVIDIA RTX 3090 GPU, with PyTorch 1.10 and CUDA version 11.8 as the development environment.

[0174] Model evaluation employed five metrics: accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). Experimental results showed that the method of this invention outperformed the comparative model across multiple evaluation metrics, achieving an accuracy of 94.6%, an F1 score of 94.1%, and an AUC of 0.947, significantly higher than the accuracy of 85.1% and AUC of 0.881 of the traditional MRI monomodal model. These results fully demonstrate the advantages of this invention in fusing multimodal features, capturing cross-modal semantic associations, and improving the predictive performance of post-stroke cognitive impairment, demonstrating promising clinical application prospects and widespread value.

[0175] Table 1. Comparison of classification performance of different models in the PSCI prediction task

[0176]

Claims

1. A method for predicting post-stroke cognitive impairment based on multimodal feature fusion, characterized in that, Includes the following steps: S1. Collect multimodal information of stroke patients, including three-dimensional brain MRI images, EEG signals and clinical medical records, and construct a dataset in a unified format; S2. Preprocess the MRI image data, convert it into tensor form and input it into the multi-scale spatiotemporal feature extraction backbone network to obtain MRI modal features; S3. The collected EEG signals are processed using a neural electrical signal preprocessing workflow, and multi-channel temporal semantic features are extracted using a Transformer network structure to obtain EEG modal features. S4. The collected clinical medical record information of stroke patients is transformed into a unified semantic sentence through a prompting learning mechanism, and then input into a dual-channel semantic encoder for encoding and extraction to obtain the clinical medical record information features; S5. Input the MRI modal features, EEG modal features, and clinical medical record information features into the three-modal fusion unit to obtain fused features. The fused features are then passed through a classifier to output probability predictions. The trimodal fusion processor is specifically implemented as follows: S51. Regarding MRI modal characteristics EEG modal characteristics and clinical medical record information characteristics First, use clinical medical record information features For any two modalities, a bidirectional attention structure is established to capture the implicit correlation and complementary semantics between them, and to calculate MRI image attention guided by clinical medical record information. And clinical medical record information-guided EEG brainwave attention The implementation is as follows: Clinical medical record information-guided MRI modality, clinical medical record information features and linear transformation matrix Multiplying them together yields the query vector for clinical medical record information. MRI modal characteristics and linear transformation matrix Multiplying them yields the key vector of the MRI image. MRI modal characteristics and linear transformation matrix Multiply to obtain the value vector of the MRI image. Based on query vector MRI image key vectors and the value vector of MRI images Calculate MRI image attention guided by clinical medical record information ; Similarly, calculating EEG brainwave signals guided by clinical medical record information for attention... ; S52. Characteristics of clinical medical record information As semantic anchors, the attention guided by clinical medical record information is first differentially analyzed with the corresponding modal features to obtain an aligned residual. This residual is then scaled using a learnable coefficient γ and added to the original modal features to obtain the enhanced MRI features guided by clinical medical record information. and EEG signal characteristics ; S53. Guide the enhanced MRI features with the clinical medical record information obtained in step S52. and EEG signal characteristics After concatenation along the channel dimension, the fused features are obtained through a linear fusion module. .

2. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 1, characterized in that, The preprocessing in step S2 includes: converting the original MRI image samples into single-channel data with dimensions of [missing information]. ,in The depth, height, and width of the image are represented respectively. A standardized preprocessing procedure is performed on the MRI images, including image size normalization, intensity normalization, and resampling. All MRI image samples are registered to the MNI space template and finally converted into tensor form.

3. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 2, characterized in that, The specific implementation process of the multi-scale spatiotemporal feature extraction backbone network is as follows: The preprocessed MRI images are input into a multi-scale spatiotemporal feature extraction backbone network. First, feature extraction is performed through an initial 3D convolutional layer. Then, the extracted features are input into a batch normalization layer, a ReLU activation function, and a max pooling layer to obtain spatial features. Then input the spatial features to A series of cascaded feature extraction modules extract high-order semantic features; Finally, the high-order semantic features are input into a global average pooling layer, and the spatiotemporal dimensions are optimized. The average of all values ​​is used to extract the global semantic feature representation, which is then flattened into a one-dimensional feature vector. This refers to MRI modal characteristics.

4. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 3, characterized in that, The specific implementation process of step S3 is as follows: S31. The acquired multi-channel electroencephalogram (EEG) time-series signals are denoted as... ,in, Indicates the number of brainwave channels. This indicates the number of time steps corresponding to the sampling duration, for EEG brain signals. Bandpass filtering, artifact removal, rereference, data truncation, and standardization are performed sequentially to obtain the preprocessed EEG signal sample tensor. ,in Indicates the number of segments. The length of each time segment; S32. For the EEG signal sample tensor, extract each EEG signal segment... Dimension transpose Then, the EEG signal sample tensor at each time point is used as a... The sequence, through a linear projection layer, transforms each The dimension from Upgraded to Then, the corresponding position code is added to obtain the embedded representation of the EEG brainwave signal; S33. The embedded representation of EEG signals is input into a multi-layer Transformer encoder module to model the global dependencies and cross-time-slice contextual information in the EEG signal sequence; this encoder consists of... The Transformer layer consists of three sub-modules: multi-head self-attention mechanism, feedforward neural network, and residual connection and layer normalization. S34. Set semantic vectors By using global average pooling, the dimension is reduced from downgraded to The output is a semantic representation of each EEG signal segment. This is the characteristic representation of the EEG mode.

5. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 4, characterized in that, The Transformer encoder module is implemented as follows: In the In a Transformer block, the input is the output of the previous layer. The EEG signals are embedded and represented as input to the multi-head self-attention mechanism module to model the correlation between different time slices of the EEG signals in the time dimension. The multi-head self-attention mechanism maps the input embeddings to query vector Q, key vector K, and value vector V, respectively. It then computes the scaled dot product attention for each step, calculating attention weights based on the similarity between the query vector and all key vectors. After normalization, all attention weights are weighted and summed to obtain the attention representation at each time point, with the shape shown. ; All weighted attention weights are concatenated and then unified to the original dimension through a linear transformation. This forms the output of the self-attention module of this layer, which is consistent with the input dimension; The output of the multi-head self-attention mechanism module is fed into the feedforward neural network module after being normalized by residual connections and layers, and the representation vector at each time step is modeled independently; this module contains two... Convolutional layer and one Regularization operation yields output features. ; The output of each Transformer block yields a set of semantic vectors: .

6. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 5, characterized in that, The specific implementation process of step S4 is as follows: The clinical medical record information of stroke patients was mapped into natural language fragments and then assembled into complete medical sentences. Each subject constructed a personalized clinical description sentence. ; Introduce learnable contextual cue vectors The number of prompt words is Embedding dimension is It consists of a concatenation of a cue vector and clinical descriptive statements. ; The enhanced text sequence will be prompted. Input to pre-training Text encoder and The encoder obtains two high-dimensional semantic representations; then, the difference between the two high-dimensional semantic representations is used to construct a semantic difference vector. Finally, semantic correction is integrated using learnable weights. Control introduction The degree of weighted fusion Semantic representation and semantic difference vector output by the encoder To achieve semantic alignment and obtain clinical medical record information features .

Images