Post-stroke cognitive impairment prediction method based on multi-modal feature fusion

Abstract

The invention discloses a post-stroke cognitive impairment prediction method based on multi-modal feature fusion. The method comprises the steps that firstly, multi-modal information of a stroke patient is collected, wherein the multi-modal information comprises a three-dimensional brain MRI image, an EEG electroencephalogram signal and clinical medical record information; secondly, converting MRI into a tensor, and inputting the tensor into a multi-scale spatial-temporal feature extraction backbone network to obtain MRI modal features; the EEG electroencephalogram signals are subjected to electroneurographic signals and are combined with Transform, and EEG modal features are obtained; the clinical medical record information of the stroke patient is converted into semantic sentences, the semantic sentences are input into a two-channel semantic encoder for encoding extraction, and clinical medical record information features are obtained. And finally, inputting the MRI modal features, the EEG modal features and the clinical medical record information features into a three-modal fusion device to obtain fusion features, and outputting probability prediction through a classifier. The post-stroke cognitive impairment prediction method achieves accurate prediction of post-stroke cognitive impairment, and significantly improves robustness and medical interpretation.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and medical image analysis, and in particular to a method for predicting post-stroke cognitive impairment (PSCI) based on multimodal feature fusion. Background Art

[0002] With the widespread application of deep learning in medical image processing, classification models based on deep neural networks have shown excellent performance in various tasks.

[0003] Traditional medical image processing methods rely on single-modality data input, such as magnetic resonance imaging (MRI) or scale scores, and extract visual features through structures such as convolutional neural networks. However, in real-world environments with incomplete information and severe noise interference, this approach often struggles to fully capture disease manifestations, limiting the model's robustness and generalization capabilities. In recent years, multimodal fusion methods have been proposed to integrate complementary information from different modalities, but mainstream approaches still suffer from the following shortcomings:

[0004] First, existing multimodal methods often lack effective semantic interaction mechanisms between modalities and cannot fully model the cross-modal associations between MRI images, EEG signals, genomics, and clinical medical records, resulting in limited fusion feature expression capabilities. Second, most models use static embedding or tabular modeling for text processing, making it difficult to leverage natural language semantics to enhance model understanding. Third, although some work has attempted to introduce prompt learning mechanisms to optimize language modality processing, most of these efforts are limited to general tasks such as image-text matching and lack customized improvements for medical scenarios.

[0005] Furthermore, existing methods lack robust intra-modal modeling and feature alignment mechanisms, facing challenges such as blurred boundaries, low contrast, and unstructured noise that frequently occur in medical images and neural electrical signals. This makes them difficult to adapt to complex and multi-source clinical data. At the modality fusion level, traditional feature concatenation or simple attention mechanisms struggle to capture the deep semantics of cross-modal interactions. This is especially true in real-world scenarios with limited sample sizes, where models are prone to overfitting and information redundancy.

[0006] Therefore, how to construct a multimodal neural network model that has semantic understanding capabilities, cross-modal alignment capabilities and efficient fusion mechanisms to cope with the complex data distribution and actual needs in the classification of post-stroke cognitive impairment has become an urgent problem that needs to be solved in the current field of medical artificial intelligence. Summary of the Invention

[0007] In response to the above problems, the present invention proposes a method for predicting post-stroke cognitive impairment based on multimodal feature fusion. Based on a lightweight deep neural network, this method introduces medical language reconstruction, semantic-driven context optimization, and a dual-channel semantic encoder, so that clinical medical record information can be deeply aligned and interactively fused with modalities such as MRI images and EEG brain signals. At the same time, through a guided cross-modal semantic interaction structure, the feature distribution of MRI images and EEG brain signals is dynamically adjusted with text semantics as anchors, and a unified semantic representation is generated through a semantic fusion network, which effectively improves the information consistency and collaborative perception capabilities between modalities. Finally, by constructing a unified representation space, multimodal features are accurately mapped to cognitive state categories, and accurate prediction of post-stroke cognitive impairment is achieved, which significantly improves the robustness, generalization ability and medical interpretability of the model.

[0008] The present invention solves the technical problem by comprising the following steps:

[0009] S1. Multimodal data acquisition: The present invention first collects multimodal information of stroke patients, including three-dimensional brain MRI images, EEG signals and clinical medical records, and constructs a data set in a unified format.

[0010] S2. MRI image data preprocessing: The original MRI image samples collected in step S1 are converted into single-channel data with a dimension of , in Representing the depth, height, and width of the image, respectively. To achieve uniform feature extraction and model input compatibility, the present invention performs a standardized preprocessing process on MRI images, including image size normalization, intensity normalization, and resampling. All MRI image samples are then registered to the standard template in the MNI space (Montreal Neurological Institute) and finally converted into tensor form.

[0011] S3. MRI image feature extraction: The MRI image preprocessed in step S2 is input into a multi-scale spatiotemporal feature extraction backbone network (such as 3D Resnet18, 3D Resnet50), and its spatial structure features and high-order semantic information are extracted layer by layer to obtain high-dimensional features. , that is, MRI modality characteristics. The specific steps include:

[0012] S31. The pre-processed MRI images are input into the multi-scale spatiotemporal feature extraction backbone network. First, the features are extracted through an initial 3D convolution layer. Then, the extracted features are input into the batch normalization layer, the ReLU activation function and the maximum pooling layer to obtain the spatial features. .

[0013] S32. Then the spatial features Input to A series of feature extraction modules - , extracting high-level semantic features .

[0014] S33. Finally, the high-level semantic features Input to the global average pooling layer, through the spatiotemporal dimension Take the average operation of all values ​​on the graph to extract the global semantic feature representation , and then converted into a one-dimensional feature vector through flattening operation .

[0015] S4.EEG signal feature extraction: The EEG signal collected in step S1 is processed using a standard neural signal preprocessing process, and combined with the Transformer network structure to extract its multi-channel temporal semantic features to obtain the final features of the EEG modality. The processing flow includes the following steps:

[0016] S41. Preprocess the collected EEG signals and record the collected multi-channel EEG time series signals as: ,in, represents the number of EEG channels, Indicates the number of time steps corresponding to the sampling duration. Perform the following preprocessing: bandpass filtering, artifact removal, re-reference, data cutting and normalization to obtain the preprocessed EEG signal sample tensor ,in Indicates the number of fragments, is the number of channels, The length of each period.

[0017] S42. For the EEG signal sample tensor obtained in step S41 , each of its EEG signal segments The dimension of , and then regard the EEG signal sample tensor at each time point as a Sequence, first through the linear projection layer to transform each The dimension from Promote to , then add the corresponding position code, and finally get the EEG signal embedding representation .

[0018] S43. Embed the EEG signal obtained in step S42 into the representation Input to the multi-layer Transformer encoder module, which is used to model the global dependencies and contextual information across time slices in the EEG signal sequence. Each layer consists of three submodules: multi-head self-attention mechanism, feedforward neural network, residual connection and layer normalization. Specifically, it includes the following steps:

[0019] S431. In the layer Transformer block, the input is the previous layer (i.e. layer) output . EEG signals are embedded into the representation The input is sent to the multi-head self-attention mechanism module to model the correlation between each time slice of the EEG signal in the time dimension. The multi-head self-attention mechanism maps the input embedding into a query vector (Q), a key vector (K), and a value vector (V). Then, each scaled dot product attention in the multi-head self-attention mechanism is calculated, and the attention weight is calculated by the similarity between the query vector and all key vectors. After normalization, all attention weights are weighted and summarized to obtain the attention representation at each time point, which is in the shape of , as follows:

[0020] In order to enhance the model's ability to capture diverse semantic features from different subspaces, we introduce Attention heads perform attention calculations in parallel .

[0021] Concatenate all weighted attention weights and unify them into the original dimension through linear transformation , forming the output of the self-attention module of this layer , which is consistent with the input dimension and facilitates residual connection.

[0022] S432. The output of the multi-head self-attention mechanism module obtained in step S431 After residual connection and layer normalization, it is sent to the feedforward neural network module to enhance the nonlinear expression ability and independently model the representation vector of each time step. This module contains two convolutional layers and a Regularization operation. The specific process is as follows:

[0023] The normalized input vector is:

[0024]

[0025] in, Indicates the The output of the layer Transformer block, represents the multi-head self-attention mechanism, Represents layer normalization, its output is residually connected with the original input, and then layer normalization is applied to obtain the normalized vector .

[0026] Normalized vector First go through the first floor Convolution expands the channel dimension, and then uses Regularization operation, then pass through the second layer Convolution restores the dimension, and finally performs residual connection and layer normalization. The specific formula is as follows:

[0027]

[0028] in, Indicates that the convolution kernel size is One-dimensional convolution operation, is a regularization operation. represents the drop probability (such as 0.1 or 0.2), represents the normalization operation, Indicates the The output of the layer Transformer block.

[0029] S433. The above process Repeat in the layer, and finally output the semantic vector set:

[0030]

[0031] in, Indicates the number of time steps in the EEG signal segment, represents the embedding dimension corresponding to each time point, Represents the output of each layer of Transformer block.

[0032] S44. The semantic vector set outputted in step S433 The time dimension is reduced by global average pooling operation, retaining key timing information while reducing the number of parameters. downgraded to , the output is the semantic representation of each EEG signal segment , which is the final feature representation of the EEG modality.

[0033] S5. Clinical medical record information feature extraction: The clinical medical record information of stroke patients collected in step S1 is converted into a unified semantic sentence through the prompt learning mechanism, and then input into the dual-channel semantic encoder for encoding extraction to obtain the clinical medical record information feature. Clinical medical record information includes the patient's structured scores (such as cognitive assessment scores), basic demographic information (such as gender, age, education level), genomic information, etc. The specific steps are as follows:

[0034] S51. The collected clinical medical records of stroke patients are expressed as The above information is then mapped into natural language fragments and spliced ​​into complete medical sentences. For example: "The patient's MMSE score is 26, age is 67, education years are 12 years, and there are mild cognitive decline symptoms." Each subject constructs a personalized clinical description sentence , used to guide the model to obtain contextual semantic information.

[0035] S52. To enhance the clinical description statement obtained in step S51 The semantic guidance capability in the text model introduces a learnable contextual hint vector, which is expressed as :

[0036] The final input sentence is composed of the prompt vector and the clinical description sentence. .

[0037] S53. The text sequence after prompt enhancement obtained in step S52 Input into the pre-trained Text encoder and Encoder, obtain two high-dimensional semantic representations; then the two high-dimensional semantic representations are subtracted to construct a semantic difference vector , and finally integrate semantic correction, through learnable weights Control introduction The degree of weighted fusion Semantic representation and semantic difference vector output by the encoder , achieve semantic alignment and obtain clinical medical record information features Characteristics of clinical medical record information obtained MRI modality characteristics , EEG modality characteristics They are input together into the tri-modal fusion device for joint modeling.

[0038] S6. Cross-modal semantic interaction fusion: MRI modality features obtained in steps S3, S4, and S5 , EEG modality characteristics and clinical medical record information characteristics Input to the trimodal fusion module, and extract features through the semantic guidance perception module and the semantic interaction guidance module to obtain the final fusion feature , used for downstream PSCI classification tasks. The semantically guided perception module and the semantic interaction guidance module construct a bidirectional attention mechanism between modalities, introduce semantic anchors to guide alignment, and use learnable weights for channel unification. The specific design of the trimodal fusion is as follows:

[0039] S61. Regarding MRI modality characteristics , EEG modality characteristics and clinical medical record information characteristics , first use the clinical medical record information features A bidirectional attention structure is established for any two modalities (e.g., MRI images and clinical medical records) to capture the implicit correlation and complementary semantics between the two. Taking the MRI modality guided by clinical medical records as an example, the attention calculation process is as follows:

[0040] S611. Constructing query vectors for clinical medical records , the key vector of the MRI image , the value vector of MRI image :

[0041]

[0042]

[0043]

[0044] in, , , is a linear transformation matrix used to map the modality embedding to the attention space.

[0045] Computational MRI image attention guided by clinical medical record information , the specific formula is:

[0046]

[0047] in, is the MRI modality characteristic Dimensions, is the activation function, Key vector representing MRI image The transpose of .

[0048] Similarly, calculating the EEG signal attention guided by clinical medical record information .

[0049] S62. Since clinical medical records usually contain clear descriptions of the patient's condition and professional terms, and have stronger semantic clarity, the clinical medical record information features As a semantic anchor, the semantic distribution of MRI images and EEG signals is guided to move closer to it. The attention guided by clinical medical record information is first differentiated from the corresponding modal features to obtain the alignment residual, which is then scaled by the learnable coefficient γ and added to the original modal features to obtain the MRI features enhanced by clinical medical record information. and EEG signal characteristics .

[0050] S63. The clinical medical record information obtained in step S62 is used to guide the enhanced MRI features and EEG signal characteristics After channel dimension splicing, the fusion feature is obtained through the linear fusion module .

[0051] S7. Fusion feature classification: The fusion feature obtained in step S6 is classified The fused features are then fed into a classifier using a lightweight multi-layer perceptron classifier to model the resulting PSCI classification predictions. This classifier consists of two fully connected layers, with a nonlinear activation function and a dropout regularization layer inserted between them to enhance nonlinear expression and suppress overfitting. The final output is a probability prediction.

[0052] Beneficial effects of the present invention: The model proposed in the present invention adopts a multimodal branch structure and a deep neural network fusion strategy to perform feature extraction and modeling on three-dimensional MRI images, EEG brain wave signals and clinical medical record information respectively, which can fully tap the complementary information between different modalities and improve the expression ability and discrimination performance of the model in cognitive impairment classification tasks. The MRI branch is based on a hierarchical residual feature extraction architecture constructed based on 3D ResNet18, which extracts local and global information of brain structure step by step, and combines the global average pooling operation to achieve progressive abstraction from underlying spatial features to high-order semantic representations, thereby enhancing the model's ability to characterize changes in brain tissue. The EEG branch introduces a time-channel attention mechanism, and integrates the Transformer structure to model the time dependence of EEG brain wave signals and the importance of channel dimensions, guiding the model to focus on key neural activity areas, suppressing redundant signal interference, and improving the quality of time series modeling. The text branch innovatively converts medical record information into natural language guide sentences, and combines it with a learnable context prompt vector. and medical language models, to enhance text semantics modeling, achieve spatial alignment of text and image semantics and improve medical semantic accuracy. A guided cross-modal interaction mechanism based on semantic anchors is proposed. By constructing clinical medical record information to guide MRI imaging modality and clinical medical record information to guide EEG brain wave signal modality, semantic guidance and deep fusion of trimodal features are achieved, thereby obtaining a unified multimodal semantic representation and improving feature consistency and modality collaboration capabilities. In the reasoning stage, the overall model has good lightweight and generalization performance. The text semantic guidance mechanism significantly improves the model's adaptability to multi-source heterogeneous medical data, and the fusion module supports flexible deployment in resource-constrained environments and is edge computing friendly. The output part of the model uses a multi-layer perceptron classifier to accurately predict the state of cognitive impairment, and can be extended to tasks of other neurological diseases. It has good versatility, interpretability and application promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an architecture diagram of a method for predicting post-stroke cognitive impairment based on multimodal feature fusion;

[0054] Figure 2 Extract structural graphs for MRI image features;

[0055] Figure 3 is the residual block structure diagram;

[0056] Figure 4 Extraction structure diagram for EEG signal features;

[0057] Figure 5 Extract structural diagrams for clinical medical record information features;

[0058] Figure 6 This is the structural diagram of the three-modal fusion device. DETAILED DESCRIPTION

[0059] The following is a further explanation of the post-stroke cognitive impairment prediction method based on multimodal feature fusion with reference to the accompanying drawings. Figure 1 As shown, the following steps are included:

[0060] S1. Multimodal data acquisition: Collect the subjects' MRI images, EEG signals, and clinical medical records, and then construct a dataset.

[0061] S2. Save the three-dimensional brain MRI scan image acquired in step S1 as Format. Assume the original MRI image sample is:

[0062]

[0063] in Represents the depth, height and width of the image respectively; Represents a single-channel medical image. The following preprocessing steps are performed on the original image:

[0064] First, the MRI image dimensions are converted into , then normalize the converted MRI images, adjust the resolution to 1mm³ through resampling, and then align all MRI images to the standard MNI space and finally convert them into the tensor format required by the model.

[0065] S3. MRI image feature extraction: The MRI image preprocessed in step S2 is input into the 3D ResNet18 network, and its spatial structure features and high-order semantic information are extracted layer by layer to output high-dimensional features. As the visual modality processing branch of the present invention, this module has strong spatial modeling capabilities and can effectively characterize the structural changes of brain tissue. The structure of the module is as follows Figure 2 The specific design and construction are as follows:

[0066] S31. Input the preprocessed MRI image into the multi-scale spatiotemporal feature extraction backbone network in the form of:

[0067]

[0068] The 3D image volume is used as the network input, first through a 3D convolution (with stride of , filled with ) for feature extraction, followed by a batch normalization layer, Activation function and Max pooling operation (step size is , filled with ), to obtain preliminary spatial features The specific formula is as follows:

[0069]

[0070] in, Indicates that the step size is 、padding is of 3D convolution, is the batch normalization operation, express activation function, Indicates that the step size is 2 and the padding is 1 Three-dimensional maximum pooling operation, is the input three-dimensional MRI image data, is the extracted preliminary spatial features.

[0071] S32. The preliminary spatial features obtained in step S31 Input to Feature extraction module, extracts high-level semantic features. This module consists of two residual blocks 1 connected in series. The specific structure is as follows Figure 3 As shown, the residual block contains Three-dimensional convolution operation, each layer of convolution is equipped with batch normalization and Activation function. This module does not perform downsampling, maintains spatial resolution, and preliminary spatial features After processing by two series-connected residual blocks 1, the spatial features are obtained , the specific formula is as follows:

[0072]

[0073] in, It is a feature extraction method. The feature formula of residual block 1 is as follows:

[0074]

[0075] in, represents the output of the previous layer, is the batch normalization operation, represents the ReLU activation function, Indicates the use of 3D convolution.

[0076] The spatial features Input to Feature extraction module, which includes a residual block 1 and a residual block 2. The specific structure of residual block 2 is as follows Figure 3 As shown, the module contains two layers in the main branch Each convolution operation is combined with batch normalization and Activation function. The residual branch uses 3D convolution (with a stride of ) Downsample the input and adjust the number of channels to keep the size of the main branch and the residual branch consistent before adding them together. Spatial features After feature extraction module After processing, the output features are , the specific formula is as follows:

[0077]

[0078]

[0079]

[0080] in, Indicates that the step size is 、 3D convolution, represents the output of the previous layer, is the batch normalization operation, express activation function, Indicates the use of 3D convolution.

[0081] because , , The three layers have the same structure, so they are processed together. , Output features separately , , the corresponding characteristic formula is as follows:

[0082]

[0083]

[0084] in, , It means that it has been processed by residual block 1 and residual block 2. The specific processing operations are shown above. 、 and They are , , The output features of the layer.

[0085] go through A series of feature extraction modules After that, high-level semantic features are extracted .

[0086] S33. The high-level semantic features obtained in step S32 , input to the global average pooling layer, through the spatiotemporal dimension ( ) and take the average operation on all values ​​to extract the global semantic feature representation , and then converted into a one-dimensional feature vector through flattening operation ,in( )for( ), the specific calculation formula is as follows:

[0087]

[0088]

[0089] in, represents the global average pooling operation, means flattening the output into a one-dimensional vector, That is the final image feature of the MRI modality, which is used for subsequent multimodal feature fusion.

[0090] S4. EEG signal feature processing: The EEG signal collected in step S1 is processed using a standard neural signal preprocessing process, and combined with the Transformer network structure to extract its multi-channel temporal semantic features to obtain the final features of the EEG modality. The specific architecture is as follows: Figure 4 As shown, the processing flow includes the following steps:

[0091] S41. Preprocess the collected EEG signals. First, record the collected multi-channel EEG time series signals as: ,in represents the number of EEG channels, Indicates the number of time steps corresponding to the sampling duration.

[0092] The following preprocessing is performed on the EEG signal:

[0093] S411. Bandpass filtering: A bandpass filter is used to retain frequency components between 0.5–45 Hz, effectively filtering out power frequency noise and high-frequency interference.

[0094] S412. Artifact removal: Use independent component analysis to separate non-EEG signal interference components such as eye movements, blinks, and electromyography, and delete or reconstruct them.

[0095] S413. Re-referencing: Use average or binaural referencing strategies to unify all-channel electrode baselines and improve spatial consistency.

[0096] S414. Data cutting: Slice the long EEG signal into fixed time windows (e.g. 2 seconds per segment, 50% overlap) to form time series samples.

[0097] S415. Standard deviation normalization: Perform standard deviation normalization on each EEG signal according to the channel dimension to normalize the signal strength between different patients and ensure the stability of model training.

[0098] Finally, after the above steps, the standardized EEG signal sample tensor is obtained: ,in Indicates the number of fragments, is the number of channels, The length of each period.

[0099] S42. For the EEG signal sample tensor obtained in step S41 , each EEG signal segment The dimension of , and then regard each time point as a Sequence input, first through the linear projection layer to transform each The dimension from Promote to , then add the corresponding position code to get the EEG signal embedding representation The specific formula is as follows:

[0100]

[0101] in, is the position code, represents a linear transformation operation, is the transposed input.

[0102] S43. Embed the EEG signal obtained in step S42 into the representation Input to the multi-layer Transformer encoder module, which is used to model the global dependencies and contextual information across time slices in the EEG signal sequence. The patent uses a 4-layer Transformer module, which specifically includes the following steps:

[0103] S431. In the layer Transformer block, the input is the previous layer (i.e. layer) output EEG signal embedding representation The input is sent to the multi-head self-attention mechanism module to model the correlation between each time slice of the EEG signal in the time dimension. The multi-head self-attention mechanism maps the input embedding into a query vector (Q), a key vector (K), and a value vector (V):

[0104]

[0105]

[0106]

[0107] in, Indicates the The output of the layer Transformer block, 、 、 is a learnable weight matrix. Then the scaled dot product attention is calculated, and the attention weight is calculated by the similarity between the query vector and all key vectors. After normalization, all value vectors are weighted and summarized to obtain the attention representation of each time point, which has the shape of .

[0108] In order to enhance the model's ability to capture diverse semantic features from different subspaces, we introduce Attention heads perform attention calculations in parallel This patent uses The specific formula is as follows:

[0109]

[0110]

[0111] in, Respectively represent The query vector, key vector, and value vector of the layer Transformer block, Indicates the execution of attention calculation, is equal to 8, Indicates the Layer The output of an attention head, represents the similarity matrix between all time points, is a scaling factor to avoid gradient explosion, Ensure that the attention weights at each time point are normalized. The core of this mechanism is to determine which historical neural patterns to focus on at each time point by judging their similarity with the entire sequence.

[0112] Concatenate the features learned in each subspace and unify them into the original dimension through linear transformation , forming the output of the self-attention module of this layer , which is consistent with the input dimension and facilitates residual connection.

[0113] S432. The output of the self-attention mechanism module obtained in step S431 After residual connection and layer normalization, it is sent to the feedforward neural network module to enhance the nonlinear expression ability and independently model the representation vector of each time step. This module contains two convolutional layers and a Regularization operation. The specific process is as follows:

[0114] The normalized input vector is:

[0115]

[0116] in, Indicates the The output of the layer Transformer block, represents the multi-head self-attention mechanism, Represents layer normalization, its output is residually connected with the original input, and then layer normalization is applied to obtain the normalized vector .

[0117] Normalized vector First go through the first floor Convolution expands the channel dimension, and then uses Regularization operation, then pass through the second layer Convolution restores the dimension, and finally performs residual connection and layer normalization. The specific formula is as follows:

[0118]

[0119] in, Represents a one-dimensional convolution operation with a convolution kernel size of 1, is a regularization operation. represents the drop probability (such as 0.1 or 0.2), represents the normalization operation, Indicates the The output of the layer Transformer block.

[0120] S433. The above process This is repeated in layers, and the output of each layer is defined as:

[0121]

[0122] in, Indicates the layer Transformer blocks, is equal to 4, Indicates the The output of the layer Transformer block, Indicates the The output of the layer Transformer block.

[0123] The final output semantic vector set is:

[0124]

[0125] in, Represents the output of each layer of Transformer block.

[0126] S44. The semantic vector set outputted in step S433 The time dimension is reduced by global average pooling, retaining key timing information while reducing the number of parameters. The formula for average pooling is as follows:

[0127]

[0128] in, For the The output vector of time steps, is the final EEG signal global semantic feature vector, Represents along the time dimension The global average pooling operation.

[0129] This step changes the dimension from downgraded to , the output is a single semantic representation of each EEG signal segment. This is the final feature representation of the EEG modality.

[0130] S5. Clinical medical record information feature extraction: The clinical medical record information of stroke patients collected in step S1 is converted into a unified semantic sentence through the prompt learning mechanism, and then input into the dual-channel semantic encoder for encoding extraction to obtain the clinical medical record information feature. Clinical medical record information includes the patient's structured scores (such as cognitive assessment scores, etc.), basic demographic information (such as gender, age, education level), genomic information, etc. The specific structure is as follows Figure 5 The steps are as follows:

[0131] S51. The collected clinical medical records of stroke patients are expressed as The above information is then mapped into natural language fragments and spliced ​​into complete medical sentences. For example: "The patient's MMSE score is 26, age is 67, education years are 12 years, and there are mild cognitive decline symptoms." Each subject constructs a personalized clinical description sentence , used to guide the model to obtain contextual semantic information.

[0132] S52. To enhance the clinical description statement obtained in step S51 The semantic guidance capability in the text model introduces a learnable contextual hint vector, which is expressed as:

[0133]

[0134] in, is the number of prompt words, is the embedding dimension. The final input sentence is composed of the prompt vector and the clinical description sentence:

[0135]

[0136] in, It is to concatenate two variables. is the learnable cue vector matrix, It is a word segmenter. It is a clinical description statement.

[0137] S53. The text sequence after prompt enhancement obtained in step S52 Input to pre-trained Text encoder and Encoder, get feature and feature , the specific formula is as follows:

[0138]

[0139]

[0140] in Indicates use Text Encoder Encoding, Indicates use Encoder encoding, is the characteristic dimension.

[0141] Then construct the semantic difference vector , this vector reflects the "semantic deviation" of the two models in their understanding of the same text semantics. The specific formula is:

[0142]

[0143] in, yes feature, yes feature.

[0144] Finally, semantic correction is integrated through learnable weights Control introduction To achieve semantic alignment and obtain clinical medical record information features , the specific formula is as follows:

[0145]

[0146] in, yes feature, is the semantic difference vector, Used to control the introduction degree.

[0147] Characteristics of clinical medical record information obtained MRI modality characteristics , EEG modality characteristics They are input together into the tri-modal fusion device for joint modeling.

[0148] S6. Cross-modal semantic interaction fusion: MRI modality features obtained in steps S3, S4, and S5 , EEG modality characteristics and clinical medical record information characteristics Input to the trimodal fusion module, and extract features through the semantic guidance perception module and the semantic interaction guidance module to obtain the final fusion feature , used for downstream PSCI classification tasks. The semantic-guided perception module and the semantic-interaction-guided module construct a bidirectional attention mechanism between modalities, introduce semantic anchors to guide alignment, and use learnable weights for channel unification. The specific structure is as follows Figure 6 As shown in Figure 2, the specific design of the three-modal fusion device is as follows:

[0149] S61. Regarding MRI modality characteristics , EEG modality characteristics and clinical medical record information characteristics First, a bidirectional attention structure is established for any two modalities (such as MRI images and clinical medical records) to capture the implicit correlation and complementary semantics between the two. Taking the MRI modality guided by clinical medical records as an example, the attention calculation process is as follows:

[0150] S611. Constructing query vectors for clinical medical records , the key vector of the MRI image , the value vector of MRI image :

[0151]

[0152]

[0153]

[0154] in, , , is a linear transformation matrix used to map the modality embedding to the attention space.

[0155] Computational MRI image attention guided by clinical medical record information , the specific formula is:

[0156]

[0157] in, is the MRI modality characteristic Dimensions, is the activation function, Key vector representing MRI image The transpose of .

[0158] Similarly, calculating the EEG signal attention guided by clinical medical record information , the calculation formula is:

[0159]

[0160] in, is the EEG modality feature Dimensions, is the activation function, Key vector representing EEG signal The transpose of A query vector representing clinical medical record information, A vector of values ​​representing an EEG signal.

[0161] S62. Since clinical medical records usually contain clear descriptions of the patient's condition and professional terms, and have stronger semantic clarity, the clinical medical record information features As a semantic anchor, the semantic distribution of MRI images and EEG signals is guided to move closer to it. The attention guided by clinical medical record information is first differentiated from the corresponding modal features to obtain the alignment residual, which is then scaled by the learnable coefficient γ and added to the original modal features to obtain the MRI features enhanced by clinical medical record information. and EEG signal characteristics The specific bootstrap formula is as follows:

[0162]

[0163]

[0164] in, is a learnable guidance coefficient used to balance the proportion of original modality information and clinical medical record information alignment information. represents MRI image attention guided by clinical medical history information, Represents EEG signal attention guided by clinical medical history information.

[0165] S63. The clinical medical record information obtained in step S62 is used to guide the enhanced MRI features and EEG signal characteristics After channel dimension splicing, the fusion feature is obtained through the linear fusion module .

[0166] S7. Fusion feature classification: The fusion feature obtained in step S6 is classified The fusion features are input to the classifier, which uses a lightweight multi-layer perceptron classifier to model the fusion features and output the PSCI classification prediction results. The classifier consists of two fully connected layers, with a nonlinear activation function and a dropout regularization layer inserted in between to enhance nonlinear expression capabilities and suppress overfitting, ultimately outputting a probability prediction. The specific steps are as follows:

[0167] Input fusion features , first through the first layer of fully connected network, and then through a Regularization layer, used to improve generalization ability, then pass through the second layer of fully connected network, and finally pass The layer outputs the final probability prediction vector, the specific formula is as follows:

[0168]

[0169] in, is the linear mapping matrix, is the bias term, is a nonlinear activation function, is the discard rate, is the classification mapping matrix, is the output bias, Indicates the predicted probability that the sample belongs to each category, and the maximum value corresponds to the predicted label.

[0170] Example:

[0171] The steps of this embodiment are the same as those of the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0172] To validate the effectiveness of the proposed method, this study used real-world stroke patient data from a tertiary hospital. The experimental data included multimodal information, including brain MRI images, EEG signals, and clinical records, covering a total of 286 patients. The imaging data consisted of structural MRI images in the original NIfTI format, with EEG data sampling at 256 Hz. Each patient was recorded for at least three minutes. Medical records included information such as MMSE scores, age, gender, and years of education. Based on follow-up and neuropsychological assessment criteria, patients were categorized as either post-stroke cognitive impairment (PSCI) or non-PSCI. PSCI patients accounted for approximately 54% of the total.

[0173] The experimental results are shown in Table 1. For model training, the proposed multimodal feature fusion network was used. Several comparison models were used for performance verification, including a 3D ResNet-based MRI unimodal network (3DResNet-MRI), a Transformer-based EEG unimodal classifier (Transformer-EEG), a simple MRI and EEG feature concatenation model (MRI-EEG feature-level concatenation), and a multimodal attention network that integrates MRI, EEG, and clinical records without semantic anchoring (anchorless multimodal attention fusion network). All models were evaluated using a 5-fold cross-validation strategy, with the cross-entropy loss function, the Adam optimizer, and an initial learning rate of 0.001. Experiments were run on an NVIDIA RTX 3090 GPU using PyTorch 1.10 and CUDA 11.8.

[0174] Model evaluation used five metrics: accuracy, precision, recall, F1 score, and area under the receiver operating characteristic (ROC) curve (AUC). Experimental results showed that the proposed method outperformed the comparison model across multiple evaluation metrics, achieving an accuracy of 94.6%, an F1 score of 94.1%, and an AUC of 0.947. These performance indicators were significantly higher than the 85.1% accuracy and 0.881 AUC of the traditional MRI single-modality model. These results fully demonstrate the advantages of the proposed method in integrating multimodal features, capturing cross-modal semantic associations, and improving the prediction performance of post-stroke cognitive impairment, demonstrating its promising clinical application prospects and potential for widespread adoption.

[0175] Table 1 Comparison of classification performance of different models in PSCI prediction task

[0176]

Claims

1. A method for predicting post-stroke cognitive impairment based on multimodal feature fusion, characterized in that: The following steps are involved: S1. Collect multimodal information from stroke patients, including 3D brain MRI images, EEG signals, and clinical records, and construct a unified dataset. S2. Preprocess the MRI image data, convert it into tensor form, and input it into the multi-scale spatiotemporal feature extraction backbone network to obtain MRI modal features; S3. The collected EEG signals are processed using a neural signal preprocessing process and combined with a Transformer network structure to extract their multi-channel temporal semantic features to obtain EEG modal features. S4. The collected clinical medical records of stroke patients are converted into unified semantic sentences through a prompt learning mechanism, and then input into a dual-channel semantic encoder for encoding and extraction to obtain clinical medical record information features; S5. Input the MRI modality features, EEG modality features and clinical medical record information features into the trimodal fusion device to obtain fusion features. The fusion features are passed through the classifier to output probability predictions.

2. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 1, characterized in that: The pre-processing in step S2 includes: converting the original MRI image sample into single-channel data with a dimension of ,in Represent the depth, height and width of the image respectively, perform standardized preprocessing on the MRI images, including image size normalization, intensity normalization and resampling, and align all MRI image samples to the MNI space template and finally convert them into tensor form.

3. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 2, characterized in that: The specific implementation process of the multi-scale spatiotemporal feature extraction backbone network is as follows: The preprocessed MRI images are input into the multi-scale spatiotemporal feature extraction backbone network. First, an initial 3D convolutional layer is used to extract features. The extracted features are then input into a batch normalization layer, a ReLU activation function, and a maximum pooling layer to obtain spatial features. Then input the spatial features into A series of feature extraction modules to extract high-level semantic features; Finally, the high-level semantic features are input into the global average pooling layer. All the values ​​on the vector are averaged to extract the global semantic feature representation, and then converted into a one-dimensional feature vector by flattening. , that is, MRI modality characteristics.

4. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 3, characterized in that: The specific implementation process of step S3 is as follows: S31. The collected multi-channel EEG time series signal is recorded as ,in, represents the number of EEG channels, Indicates the number of time steps corresponding to the sampling duration, for EEG signals Perform bandpass filtering, artifact removal, re-reference, data cutting and normalization in sequence to obtain the preprocessed EEG signal sample tensor ,in Indicates the number of fragments, is the length of each period; S32. For the EEG signal sample tensor, each EEG signal segment The dimension of , and then use the EEG signal sample tensor at each time point as a Sequence, each is transformed into The dimension from Promote to , then add the corresponding position code to get the EEG signal embedding representation; S33. The EEG signal is embedded into the multi-layer Transformer encoder module to model the global dependency and contextual information across time slices in the EEG signal sequence. Each layer consists of three submodules: multi-head self-attention mechanism, feedforward neural network, residual connection and layer normalization; S34. Set the semantic vectors By global average pooling, the dimension is changed from downgraded to , the output is the semantic representation of each EEG signal segment , which is the feature representation of the EEG modality.

5. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 4, characterized in that: The Transformer encoder module is implemented as follows: In the In the layer Transformer block, the input is the output of the previous layer ; The EEG signal is embedded into the representation input to the multi-head self-attention mechanism module to model the correlation between each time slice of the EEG signal in the time dimension; The multi-head self-attention mechanism maps the input embedding into query vector Q, key vector K and value vector V respectively. Then, each scaled dot product attention in the multi-head self-attention mechanism is calculated, and the attention weight is calculated by the similarity between the query vector and all key vectors. After normalization, all attention weights are weighted and summarized to obtain the attention representation at each time point, which is in the shape of ; Concatenate all weighted attention weights and unify them into the original dimension through linear transformation , forming the output of the self-attention module of this layer, which is consistent with the input dimension; The output of the multi-head self-attention mechanism module is sent to the feedforward neural network module after residual connection and layer normalization, and the representation vector of each time step is independently modeled; this module contains two Convolutional layer and a Regularization operation to obtain output features ; The output of each Transformer block is a set of semantic vectors: .

6. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 5, characterized in that: The specific implementation process of step S4 is as follows: The clinical medical records of stroke patients were mapped into natural language fragments and spliced ​​into complete medical sentences. Each subject constructed a personalized clinical description sentence. ; Introducing learnable contextual hint vectors , the number of prompt words is , the embedding dimension is ; Consists of the prompt vector and clinical description sentence splicing ; The enhanced text sequence Input into the pre-trained Text Encoder and Encoder, obtain two high-dimensional semantic representations; then the two high-dimensional semantic representations are subtracted to construct a semantic difference vector , and finally integrate semantic correction, through learnable weights Control introduction The degree of weighted fusion Semantic representation and semantic difference vector output by the encoder , achieve semantic alignment and obtain clinical medical record information features .

7. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 6, characterized in that: The three-modal fusion device is specifically implemented as follows: S51. Regarding MRI modality characteristics , EEG modality characteristics and clinical medical record information characteristics , first use the clinical medical record information features A bidirectional attention structure is established for any two modalities to capture the implicit correlation and complementary semantics between the two, and to calculate the MRI image attention guided by clinical medical record information. , and EEG signal attention guided by clinical medical record information ; S52. Characterize clinical medical records As a semantic anchor, the attention guided by clinical medical record information is first differentiated from the corresponding modal features to obtain the alignment residual, which is then scaled by the learnable coefficient γ and added to the original modal features to obtain the MRI features enhanced by clinical medical record information. and EEG signal characteristics ; S53. The clinical medical record information obtained in step S52 is used to guide the enhanced MRI features and EEG signal characteristics After channel dimension splicing, the fusion feature is obtained through the linear fusion module .

8. The method for predicting post-stroke cognitive impairment based on multimodal feature fusion according to claim 7, characterized in that: The step S51 is implemented as follows: Clinical medical record information guides MRI modality, clinical medical record information features and linear transformation matrix Multiply them together to get the query vector of clinical medical record information ;MRI modality characteristics and linear transformation matrix Multiply to get the key vector of the MRI image , MRI modality characteristics and linear transformation matrix Multiply to get the value vector of MRI image : Based on query vector , the key vector of the MRI image and the value vector of the MRI image , Computing MRI image attention guided by clinical medical record information ; Similarly, calculating the EEG signal attention guided by clinical medical record information .

Images