An auxiliary assessment method for brain dysfunction based on multimodal data fusion

Abstract

The present invention provides a method for assisting the assessment of brain dysfunction based on multimodal data fusion. The method comprises the following steps: acquiring functional magnetic resonance imaging, electroencephalogram, and functional near-infrared spectroscopy data; obtaining corresponding feature information based on the acquired data, respectively, based on a discrete spatiotemporal data fusion model, a continuous spatiotemporal data fusion model, and a heterogeneous spatiotemporal data fusion model; analyzing the corresponding feature information generated by the discrete spatiotemporal data fusion model, the continuous spatiotemporal data fusion model, and the heterogeneous spatiotemporal data fusion model through an assisting assessment and diagnosis method for brain dysfunction based on mechanism analysis and multimodal feature fusion, and generating the patient's brain activity status to complete the assisting assessment of brain dysfunction. The present invention facilitates assisting medical staff in diagnosing the patient's brain function.

Description

Technical Field

[0001] The present invention relates to the field of bioinformatics, and in particular to a brain dysfunction auxiliary assessment method based on multimodal data fusion. Background Art

[0002] Brain dysfunction is a disorder of movement, cognition, speech, swallowing, mental and emotional functions caused by various types of diseases, including neurological, psychiatric, and children's behavioral development. Since the brain is the center for regulating human movement, cognition, behavior, sensation, language and emotions, brain function will seriously affect the patient's life, and even lead to death in severe cases.

[0003] In the process of rehabilitation treatment for brain dysfunction, characteristic data that can characterize the activity of brain areas (such as fMRI, EEG, fNIR, etc.) are usually used to analyze the activity of brain functional areas and guide physical therapy equipment to achieve precise treatment of areas with brain dysfunction. Among them, functional magnetic resonance imaging (fMRI) usually refers to magnetic resonance imaging (BOLD-fMRI) that uses the blood oxygenation-level-dependent (BOLD) phenomenon to analyze brain activity. The blood oxygen level dependence phenomenon refers to the different magnetic properties of different binding states of hemoglobin and oxygen in the blood. When hemoglobin is combined with oxygen (oxyhemoglobin), it exhibits antimagnetism, while when hemoglobin is separated from oxygen (deoxyhemoglobin), it exhibits paramagnetism. Electroencephalography (EEG) is an electrophysiological detection method for recording electroencephalograms, which can reflect the sum of changes in postsynaptic potentials of the apical dendrites of pyramidal cells in the cerebral cortex. The stronger the detected electrical signal, the stronger the activity of the brain. Functional near-infrared spectroscopy (fNIRs) refers to the use of near-infrared spectroscopy (NIRS) for the purpose of functional neuroimaging. Its working principle is mainly to measure brain activity through hemodynamic responses related to neuronal behavior. The hemodynamic activity of the cerebral cortex is usually detected by the difference in the absorption rate of oxygenated hemoglobin and deoxygenated hemoglobin in brain tissue to near-infrared light of different wavelengths in the range of 600-900nm. By observing this hemodynamic change, the neural activity of the brain can be inferred.

[0004] In recent years, multimodal data fusion technology based on deep learning has made rapid progress. It fuses data of different modalities, uses the fitting ability of deep learning to achieve the advantages of complementarity between data, and gets rid of the limitations of time or space in single data, so as to understand the comprehensive information of the disease and ensure the reliability of doctors' decision-making. However, the existing multimodal fusion technology (Hao Aimin. Multimodal medical image fusion method based on multi-scale anisotropic decomposition and low-rank analysis: China, CN104299216A.2014.10.22.) only realizes the fusion of multimodal medical images, fails to clarify how to give individualized precise rehabilitation treatment to patients with brain dysfunction, and fails to realize comfortable and flexible individual posture treatment methods. Therefore, it is particularly important to fuse multimodal data with representation information and use these generated multimodal data to achieve precise rehabilitation treatment for individuals. Summary of the invention

[0005] The main purpose of the present invention is to provide a brain dysfunction auxiliary assessment method based on multimodal data fusion, which can achieve precise treatment of brain areas with dysfunction based on multimodal feature information.

[0006] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0007] A method for assisting assessment of brain dysfunction based on multimodal data fusion, comprising the following steps:

[0008] S1, obtain functional magnetic resonance imaging (fMRI), electroencephalogram (EEG), and functional near-infrared spectroscopy (fNIRs) data;

[0009] S2. According to the acquired data, corresponding feature information is obtained based on a discrete spatiotemporal data fusion model, a continuous spatiotemporal data fusion model, and a heterogeneous spatiotemporal data fusion model;

[0010] S3. Through the auxiliary assessment and diagnosis method of brain dysfunction based on mechanism analysis and multimodal feature fusion, the corresponding feature information generated by the discrete space-time data fusion model, the continuous space-time data fusion model and the heterogeneous space-time data fusion model is analyzed, and the patient's brain activity status is generated to complete the auxiliary assessment of brain dysfunction.

[0011] Furthermore, in step S2, a discrete spatiotemporal data fusion model based on functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG) is used to extract feature information of functional magnetic resonance imaging (fMRI) data in discrete time and feature information of electroencephalogram (EEG) data in discrete space, and generate fused feature information; the fused feature information is used to solve the shortcomings of traditional methods of using time graphs or space graphs to represent correlations, and can simultaneously represent the patient's brain activity in continuous time and continuous space;

[0012] The discrete spatiotemporal data fusion model considers that both functional magnetic resonance imaging (fMRI) data and electroencephalogram (EEG) data have the characteristics of discrete spatiotemporal data. A dynamic Transformer network model guided by a spatial graph is introduced, and a collaborative cross-temporal Encoder-Decoder model is established.

[0013] Furthermore, the spatial graph-guided dynamic Transformer network model utilizes the structure of the spatial graph to guide the Transformer network model to perform dynamic feature aggregation, as follows:

[0014] Assuming there are s discrete spatial observation points, we first construct a spatial graph adjacency matrix A∈R according to the spatial terrain and distance between different observation points. s×s , where R s×s The matrix is ​​an s-order real square matrix, and the spatial graph adjacency matrix is ​​symmetrically normalized to obtain the normalized spatial graph adjacency matrix Finally, the normalized spatial graph adjacency matrix is ​​used Guide the generation of attention scores for the Transformer network model;

[0015] Let Q∈R s×s The attention scores between the s spatial measurement points generated by the Transformer network model are added to the loss function as follows:

[0016]

[0017] The Laplace regularization term will make the generated attention score Q as smooth as possible on the graph corresponding to the spatial graph adjacency matrix A; the time series features generated at the previous moment are combined with the original features at the current moment and sent to the Transformer network model to generate the attention score matrix R, so that the Transformer network model can model the dynamic connection between different observation points.

[0018] Furthermore, the collaborative cross-series Encoder-Decoder model includes a long short-term memory neural network (LSTM) and a graph-guided Transformer network model;

[0019] The calculation process of the collaborative cross-series Encoder-Decoder model is as follows:

[0020] Firstly, the functional magnetic resonance imaging (fMRI) data and electroencephalogram (EEG) data are respectively input into the graph-guided Transformer network model. Then, the output of the Transformer network model is used as the input of the long short-term memory neural network. The fusion feature information of functional magnetic resonance imaging and electroencephalogram (EEG) is generated through the feature association of the long short-term memory neural network.

[0021] Further, in step S2, a continuous spatiotemporal data fusion model based on functional near infrared spectroscopy (fNIRs) data and electroencephalogram (EEG) data is used to generate fusion feature information of functional near infrared data and EEG data in continuous time, and the feature information in continuous time and space is generated in combination with the spatial resolution of the near infrared data;

[0022] The continuous spatiotemporal data fusion model introduces a multi-scale graph convolutional neural network model based on the attention mechanism, which is designed to perform feature extraction and feature fusion. The attention mechanism is used to reveal the correlation between brain connections. The steps of attention calculation are as follows:

[0023] First, from the input matrix X at the i-th moment i Create a query matrix H in i And the key value matrix K i , query matrix H i And the key value matrix K i It is obtained by the following formula:

[0024] H i =W H X i

[0025] K i =W K X i

[0026] Among them, W H , W K Respectively represent the generated query matrix H i The weight coefficient and key value matrix K i The weight coefficient, W H , W K Obtained by model learning; value matrix V i By weight coefficient W vFor the input matrix X at the i-th moment i The weighted summation is obtained, and the process is expressed as follows:

[0027] V i =W v X i

[0028] Among them, W v Learned by the network model; the final attention size T i It is obtained by the following formula:

[0029]

[0030] Where k is H i The output of the self-attention module in the multi-scale graph convolutional neural network model is expressed as: G i =T i F i , T i is the attention size generated by the functional near infrared spectroscopy data at the i-th moment, F i is the characteristic graph of the functional near-infrared spectroscopy (fNIRs) data at the i-th moment; considering the temporal continuity of the data, the output It is expressed as:

[0031]

[0032] Among them, δ i G i As input and learned through a single-layer MLP, w j , G j They represent the weights of the respective attention modules at the jth moment and the output of the self-attention module at the jth moment.

[0033] Furthermore, the multi-scale graph convolutional neural network model includes a graph convolutional neural network and a global average pooling layer. The calculation process of the multi-scale graph convolutional neural network model is as follows:

[0034] Firstly, the electroencephalogram (EEG) time series signal is reconstructed and the corresponding waveform profile is generated; the functional near infrared (fNIRS) data is standardized and the correlation analysis of the functional near infrared (fNIRS) data is generated, and then the feature map is generated;

[0035] Subsequently, the generated waveform profile and feature map are input into the graph convolutional neural network to generate fused feature information, where the calculation process of the graph convolutional neural network is as follows:

[0036] P l+1 =σ(LP l W l )

[0037] Among them, P l , W l , L and σ are the output, weight coefficient, adjacency matrix and nonlinear activation function of the lth network layer in the graph convolutional neural network respectively;

[0038] Finally, a global average pooling operation is performed on the fused feature information generated by the graph convolutional neural network to generate the patient's saliency map.

[0039] Furthermore, in step S2, based on the heterogeneity of functional magnetic resonance imaging (fMRI) data and functional near infrared spectroscopy (fNIRs) data in time and space information, a heterogeneous spatiotemporal data fusion model is proposed; the heterogeneous spatiotemporal data fusion model combines the characteristics of functional magnetic resonance data having a higher resolution in space and functional near infrared spectroscopy having a higher resolution in time, thereby realizing dual-modal data fusion of functional magnetic resonance imaging data and functional near infrared spectroscopy data;

[0040] The goal of the heterogeneous spatiotemporal data fusion model is to find a spatial transformation or spatial mapping between heterogeneous images, transform the image to be registered into the image space of the target image, and thus establish a corresponding relationship between the spatial positions of the images; using a feature-based method, features are first extracted from the image, and then feature matching and image registration are performed based on feature definition metrics.

[0041] Furthermore, using a feature-based approach, features are first extracted from the image, and then feature matching and image registration are performed based on feature-defined metrics, as follows:

[0042] First, the image acquired by functional magnetic resonance imaging (fMRI) is segmented, and then the MRI reference points are selected to automatically generate multiple position points, which are converted into corresponding functional near-infrared (fNIRS) digitized reference points as the positions of the optodes through rigid transformation.

[0043] The position of the optode is corrected by calculating the nearest head surface point on the straight line between the center of the head surface and the optode position, and then the midpoint of the channel source and detector is calculated as the position of the channel and projected onto the cortical grid. The correction method of the channel position is similar to that of the optode;

[0044] Subsequently, the voxel space coordinates corresponding to each channel were extracted and converted into coordinates in the standardized MNI space. The β value and BOLD signal of the functional magnetic resonance imaging (fMRI) at the coordinate point in the standardized MNI space were calculated and compared with the functional near-infrared (fNIRS) data to complete the spatial registration process.

[0045] Furthermore, in step S3, the original output modal data is processed by ConvLSTM to make the original output modal data smoother in time series, and then the connection between different feature graphs is extracted by the Transformer network model, and then the feature information is stacked in the time dimension and used together with the error pattern ε as the input of the hybrid expert model;

[0046] Among them, the correction process of the error model ε for different types of errors is as follows:

[0047]

[0048] in, represents the feature information output by ConvLSTM at time t, n represents the length of the time slice, Represents the feature information output by ConvLSTM at time 0, Expert k is the kth mixed expert model.

[0049] Furthermore, in step S3, the expert system includes k hybrid expert models Expert k , Hybrid Expert Model Expert k Based on the Transformer network model, each hybrid expert model Expert k It includes an input layer, a transformer layer, and an output layer. The input of the input layer is the feature information and error pattern ε calculated by the transformer network model and stacked in the time dimension. It is followed by a transformer layer with a dual-channel structure. Each channel includes a transformer structure. Finally, the output of the dual-channel transformer layer is used as the input of the output layer. The output layer is a single-layer fully connected neural network. Finally, a fully connected hybrid module is used to obtain the final decision result:

[0050]

[0051] in, represents the output of the expert system at time t, p k Represents the kth hybrid expert model Expert k The output of , K is the number of hybrid expert models in the expert system.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] First, the present invention considers the characteristics of functional magnetic resonance imaging data and electroencephalogram data in spatial and temporal dimensions, and introduces a discrete spatiotemporal data fusion model to extract the data characteristics of these two modalities. Subsequently, in order to further improve the temporal resolution and spatial resolution of the fusion features and based on the spatiotemporal characteristics of functional near-infrared, a heterogeneous spatiotemporal fusion model and a continuous spatiotemporal model are proposed. Finally, an expert system is introduced for accurate evaluation. Through the evaluation of the expert system, medical staff can give precise treatment plans and improve treatment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of the steps of a method for assisting in assessing brain dysfunction based on multimodal data fusion in an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the structure of a collaborative cross-sequence Encoder-Decoder model in an embodiment of the present invention.

[0056] Figure 3 This is a flowchart of the steps of the multi-scale graph convolutional neural network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0058] Embodiment 1:

[0059] A method for assisting the assessment of brain dysfunction based on multimodal data fusion, such as Figure 1 As shown, the following steps are included:

[0060] S1, obtain functional magnetic resonance imaging (fMRI), electroencephalogram (EEG), and functional near-infrared spectroscopy (fNIRs) data;

[0061] S2. According to the acquired data, corresponding feature information is obtained based on a discrete spatiotemporal data fusion model, a continuous spatiotemporal data fusion model, and a heterogeneous spatiotemporal data fusion model;

[0062] A discrete spatiotemporal data fusion model based on functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG) is used to extract the feature information of functional magnetic resonance imaging (fMRI) data in discrete time and the feature information of electroencephalogram (EEG) data in discrete space, and generate fused feature information; this fused feature information is used to solve the shortcomings of traditional methods that use time graphs or space graphs to represent correlations, and can simultaneously represent the patient's brain activity in continuous time and continuous space;

[0063] like Figure 2 As shown in the figure, the discrete spatiotemporal data fusion model considers that both functional magnetic resonance imaging (fMRI) data and electroencephalogram (EEG) data have the characteristics of discrete spatiotemporal data. A dynamic Transformer network model guided by a spatial graph is introduced, and a collaborative cross-temporal Encoder-Decoder model is established.

[0064] The spatial graph-guided dynamic Transformer network model uses the structure of the spatial graph to guide the Transformer network model to perform dynamic feature aggregation, as follows:

[0065] Assuming there are s discrete spatial observation points, we first construct a spatial graph adjacency matrix A∈R according to the spatial terrain and distance between different observation points. s×s , where R s×s Indicates that the matrix belongs to an s-order real square matrix. Symmetrically normalize the spatial graph adjacency matrix to obtain the normalized spatial graph adjacency matrix Finally, the normalized spatial graph adjacency matrix is ​​used Guide the generation of attention scores for the Transformer network model;

[0066] Specifically, let Q∈R s×s The attention scores between the s spatial measurement points generated by the Transformer network model are added to the loss function as follows:

[0067]

[0068] The Laplace regularization term will make the generated attention score Q as smooth as possible on the graph corresponding to the spatial graph adjacency matrix A; in addition, the time series features generated at the previous moment are combined with the original features at the current moment and sent to the Transformer network model to generate the attention score matrix R, so that the Transformer network model can model the dynamic connection between different observation points.

[0069] The collaborative cross-series Encoder-Decoder model includes a long short-term memory neural network (LSTM) and a graph-guided Transformer network model;

[0070] The calculation process of the collaborative cross-series Encoder-Decoder model is as follows:

[0071] Firstly, the functional magnetic resonance imaging (fMRI) data and electroencephalogram (EEG) data are respectively input into the graph-guided Transformer network model. Then, the output of the Transformer network model is used as the input of the long short-term memory neural network. The fusion feature information of functional magnetic resonance imaging and electroencephalogram (EEG) is generated through the feature association of the long short-term memory neural network.

[0072] A continuous spatiotemporal data fusion model based on functional near-infrared spectroscopy (fNIRs) data and electroencephalogram (EEG) data is used to generate fusion feature information of functional near-infrared data and EEG data in continuous time, and combine the spatial resolution of near-infrared data to generate feature information in continuous time and space;

[0073] The continuous spatiotemporal data fusion model introduces a multi-scale graph convolutional neural network model based on the attention mechanism, which is designed to perform feature extraction and feature fusion. The attention mechanism is used to reveal the correlation between brain connections. The steps of attention calculation are as follows:

[0074] First, from the input matrix X at the i-th moment i Create a query matrix H in i And the key value matrix K i , query matrix H i And the key value matrix K i It can be obtained by the following formula:

[0075] H i =W H X i

[0076] K i =W K X i

[0077] Among them, W H , W K Respectively represent the generated query matrix H i The weight coefficient and key value matrix K i The weight coefficient W H , W K It can be learned by the model; similarly, the value matrix V i The weight coefficient W v For the input matrix X at the i-th moment i The weighted summation is obtained, and the process is expressed as follows:

[0078] V i =W v X i

[0079] Where W vLearned by the network model. The final attention size T i It can be obtained by the following formula:

[0080]

[0081] Where k is H i The output of the self-attention module in the multi-scale graph convolutional neural network model is expressed as: G i =T i F i , T i is the attention size generated by the functional near infrared spectroscopy data at the i-th moment, F i is the characteristic graph of the functional near-infrared spectroscopy (fNIRs) data at the i-th moment; considering the temporal continuity of the data, the output It can be expressed as:

[0082]

[0083] Among them, δ i G i As input and learned through a single-layer MLP, w j , G j They represent the weights of the respective attention modules at time j and the output of the self-attention module at time j respectively.

[0084] The multi-scale graph convolutional neural network model includes a graph convolutional neural network and a global average pooling layer, such as Figure 3 As shown in Figure 2, the calculation process of the multi-scale graph convolutional neural network model is as follows:

[0085] Firstly, the electroencephalogram (EEG) time series signal is reconstructed and the corresponding waveform profile is generated; the functional near infrared (fNIRS) data is standardized and the correlation analysis of the functional near infrared (fNIRS) data is generated, and then the feature map is generated;

[0086] Subsequently, the generated waveform profile and feature map are input into the graph convolutional neural network to generate fused feature information, where the calculation process of the graph convolutional neural network is as follows:

[0087] P l+1 =σ(LP l W l )

[0088] Among them, P l , W l , L and σ are the output, weight coefficient, adjacency matrix and nonlinear activation function of the lth network layer in the graph convolutional neural network respectively;

[0089] Finally, a global average pooling operation is performed on the fused feature information generated by the graph convolutional neural network to generate the patient's saliency map.

[0090] Based on the heterogeneity of functional magnetic resonance imaging (fMRI) data and functional near-infrared spectroscopy (fNIRs) data in terms of temporal and spatial information, a heterogeneous spatiotemporal data fusion model is proposed; the heterogeneous spatiotemporal data fusion model combines the characteristics of functional magnetic resonance data having a higher spatial resolution and functional near-infrared spectroscopy having a higher temporal resolution, and realizes the dual-modal data fusion of functional magnetic resonance imaging data and functional near-infrared spectroscopy data;

[0091] The goal of the heterogeneous spatiotemporal data fusion model is to find a spatial transformation or spatial mapping between heterogeneous images, transform the image to be registered into the image space of the target image, and thus establish a corresponding relationship between the spatial positions of the images; using a feature-based method, first extract features from the image, and then perform feature matching and image registration based on feature definition metrics, as follows:

[0092] First, the image acquired by functional magnetic resonance imaging (fMRI) is segmented, and then the MRI reference points are selected to automatically generate multiple position points, which are converted into corresponding functional near-infrared (fNIRS) digitized reference points as the positions of the optodes through rigid transformation.

[0093] The position of the optode is corrected by calculating the nearest head surface point on the straight line between the center of the head surface and the optode position, and then the midpoint of the channel source and detector is calculated as the position of the channel and projected onto the cortical grid. The correction method of the channel position is similar to that of the optode;

[0094] Subsequently, the voxel space coordinates corresponding to each channel were extracted and converted into coordinates in the standardized MNI space. The β value and BOLD signal of the functional magnetic resonance imaging (fMRI) at the coordinate point in the standardized MNI space were calculated and compared with the functional near-infrared (fNIRS) data to complete the spatial registration process.

[0095] S3. Through the auxiliary assessment and diagnosis method of brain dysfunction based on mechanism analysis and multimodal feature fusion, the corresponding feature information generated by the discrete space-time data fusion model, the continuous space-time data fusion model and the heterogeneous space-time data fusion model is analyzed, and the patient's brain activity status is generated to complete the auxiliary assessment of brain dysfunction.

[0096] The brain dysfunction assessment and diagnosis method of the mechanism analysis and multimodal feature fusion data uses ConvLSTM to process the original output modal data so that the original output modal data is smoother in time series, and then extracts the connection between different feature maps through the Transformer network model, and then stacks the feature information in the time dimension and uses it together with the error pattern ε as the input of the hybrid expert model;

[0097] Among them, the correction process of the error model ε for different types of errors is as follows:

[0098]

[0099] in, represents the feature information output by ConvLSTM at time t, n represents the length of the time slice, Represents the feature information output by ConvLSTM at time 0, Expert k is the kth mixed expert model.

[0100] The expert system consists of k hybrid expert models Expert k , Hybrid Expert Model Expert k Based on the Transformer network model, each hybrid expert model Expert k It includes an input layer, a transformer layer, and an output layer. The input of the input layer is the feature information and error pattern ε calculated by the transformer network model and stacked in the time dimension. It is followed by a transformer layer with a dual-channel structure. Each channel includes a transformer structure. Finally, the output of the dual-channel transformer layer is used as the input of the output layer. The output layer is a single-layer fully connected neural network. Finally, a fully connected hybrid module is used to obtain the final decision result:

[0101]

[0102] in, represents the output of the expert system at time t, p k Represents the kth hybrid expert model Expert k The output of , K is the number of hybrid expert models in the expert system.

[0103] Embodiment 2:

[0104] The difference from Example 1 is that the calculation process of the multi-scale graph convolutional neural network in the continuous spatiotemporal data fusion model of step S2 is:

[0105] First, the EEG time series signal is reconstructed and the corresponding waveform profile is generated. Considering the sequential nature of the EEG in the time dimension, it is input into the recurrent neural network (RNN) to calculate and generate the feature matrix. The calculation process of the recurrent neural network is as follows:

[0106] O t =g(w o ·R t )

[0107] R t =f(w i X t +w l R t-1 )

[0108] Where g and f are nonlinear activation functions; w o 、w i 、w l are the output layer weight coefficient, input layer weight coefficient and recurrent unit weight coefficient respectively. The coefficients are obtained by model learning. t Represents the output of the recurrent neural network, that is, the feature matrix mentioned above; R t , R t-1 Represent the output of the recurrent unit at time t and time t-1 respectively.

[0109] Normalize the fNIRS data and generate correlation analysis of the fNIRS data to generate feature maps;

[0110] Subsequently, the generated feature matrix and feature map are input into the graph convolutional neural network to generate fused feature information. The calculation process of the graph convolutional neural network is as follows:

[0111] P l+1 =σ(LP l W l )

[0112] Where P l , W l , L and σ are the output of the lth network layer in the graph convolutional neural network as well as the weight coefficient, Laplacian matrix and nonlinear activation function respectively.

[0113] Finally, a global average pooling operation is performed on the fused feature information generated by the graph convolutional neural network to generate the patient's saliency map.

[0114] Embodiment 3:

[0115] The difference from Example 1 is that the brain dysfunction auxiliary assessment and diagnosis method in step S3 uses LSTM to process the original output modal data and extract the time series signal; then the connection between different feature graphs is extracted through the Transformer network model, and then the feature information is stacked in the time dimension and used together with the error pattern ε as the input of the hybrid expert model;

[0116] Among them, the correction process of the error model ε for different types of errors is as follows:

[0117]

[0118] in, represents the feature information of the output of ConvLSTM at time t, n represents the length of the time slice, Represents the output of ConvLSTM at time 0, Expert k is the kth hybrid expert model;

[0119] The expert system consists of k hybrid expert models Expert k , Hybrid Expert Model Expert k Based on the Transformer network model. Each hybrid expert model Expert k It includes input layer, transformer layer and output layer. The input of the input layer is the stacked feature information and error mode ε, followed by a transformer layer with a dual-channel structure. Each channel contains a transformer structure. Finally, the output of the dual-channel transformer layer is used as the input of the output layer. The output layer is a single-layer fully connected neural network; finally, a fully connected hybrid module is used to obtain the final decision result:

[0120]

[0121] in represents the output of the expert system at time t, p k Represents the kth hybrid expert model Expert k The output of , K is the number of hybrid expert models in the expert system.

[0122] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A brain dysfunction auxiliary assessment method based on multimodal data fusion, characterized in that: The following steps are involved: S1, obtain functional magnetic resonance imaging, electroencephalography, and functional near-infrared spectroscopy data; S2. According to the acquired data, corresponding feature information is obtained based on a discrete spatiotemporal data fusion model, a continuous spatiotemporal data fusion model and a heterogeneous spatiotemporal data fusion model respectively; a discrete spatiotemporal data fusion model based on functional magnetic resonance imaging and electroencephalogram is used to extract feature information of functional magnetic resonance imaging data in discrete time and feature information of electroencephalogram data in discrete space, and generate fused feature information; A continuous spatiotemporal data fusion model based on functional near-infrared spectroscopy data and electroencephalogram data is used to generate fusion feature information of functional near-infrared spectroscopy data and electroencephalogram data in continuous time, and to generate feature information in continuous spatiotemporal time in combination with the spatial resolution of near-infrared data; Based on the heterogeneity of functional magnetic resonance imaging data and functional near-infrared spectroscopy data in time and space information, a heterogeneous spatiotemporal data fusion model is proposed; the heterogeneous spatiotemporal data fusion model combines the characteristics of functional magnetic resonance data having a higher spatial resolution and functional near-infrared spectroscopy having a higher temporal resolution, and realizes dual-modal data fusion of functional magnetic resonance imaging data and functional near-infrared spectroscopy data; S3. Through the brain dysfunction auxiliary assessment and diagnosis method based on mechanism analysis and multimodal feature fusion, the corresponding feature information generated by the discrete spatiotemporal data fusion model, the continuous spatiotemporal data fusion model and the heterogeneous spatiotemporal data fusion model is analyzed, and the patient's brain activity status is generated to complete the brain dysfunction auxiliary assessment; the original output modal data is processed by ConvLSTM, and then the connection between different feature maps is extracted through the Transformer network model, and then the feature information is stacked in the time dimension and used together with the error pattern ε as the input of the hybrid expert model; Among them, the correction process of error mode ε for different types of errors is as follows: in, represents the feature information output by ConvLSTM at time t, n represents the length of the time slice, Represents the feature information output by ConvLSTM at time 0, Expert k is the kth hybrid expert model; p k Represents the kth hybrid expert model Expert k Output.

2. The method for assisting in assessing brain dysfunction based on multimodal data fusion according to claim 1, characterized in that: In step S2, The discrete spatiotemporal data fusion model considers that both functional MRI data and EEG data have discrete spatiotemporal characteristics. A dynamic Transformer network model guided by a spatial graph is introduced, and a collaborative cross-temporal Encoder-Decoder model is established.

3. The method for assisting in assessing brain dysfunction based on multimodal data fusion according to claim 2, characterized in that: The spatial graph-guided dynamic Transformer network model uses the structure of the spatial graph to guide the Transformer network model to perform dynamic feature aggregation, as follows: Assuming there are s discrete spatial observation points, we first construct a spatial graph adjacency matrix A∈R according to the spatial terrain and distance between different observation points. s×s , where R s×s The matrix is ​​an s-order real square matrix, and the spatial graph adjacency matrix is ​​symmetrically normalized to obtain the normalized spatial graph adjacency matrix Finally, the normalized spatial graph adjacency matrix is ​​used Guide the generation of attention scores for the Transformer network model; Let Q∈R s×s The attention scores between the s spatial measurement points generated by the Transformer network model are added to the loss function as follows: The time series features generated at the previous moment are combined with the original features at the current moment and sent to the Transformer network model to generate the attention score matrix R.

4. The method for assisting in assessing brain dysfunction based on multimodal data fusion according to claim 3, characterized in that: The collaborative cross-series Encoder-Decoder model includes a long short-term memory neural network and a graph-guided Transformer network model; The calculation process of the collaborative cross-series Encoder-Decoder model is as follows: Firstly, the functional MRI data and EEG data are respectively input into the graph-guided Transformer network model. Then, the output of the Transformer network model is used as the input of the long short-term memory neural network. The fusion feature information of functional MRI and EEG is generated through the feature association of the long short-term memory neural network.

5. The method for assisting in assessing brain dysfunction based on multimodal data fusion according to claim 1, characterized in that: In step S2, the continuous spatiotemporal data fusion model introduces a multi-scale graph convolutional neural network model based on an attention mechanism, aiming to perform feature extraction and feature fusion; wherein the attention mechanism is used to reveal the correlation between brain connections; the steps of attention calculation are as follows: First, from the input matrix X at the i-th moment i Create a query matrix H in i And the key value matrix K i , query matrix H i And the key value matrix K i It is obtained by the following formula: H i =W H X i K i =W K X i Among them, W H , W K Respectively represent the generated query matrix H i The weight coefficient and key value matrix K i The weight coefficient, W H , W K Obtained by model learning; value matrix V i By weight coefficient W v For the input matrix X at the i-th moment i The weighted summation is obtained, and the process is expressed as follows: V i =W v X i Among them, W v Learned by the network model; the final attention size T i It is obtained from the following formula: Where k is H i The output of the self-attention module in the multi-scale graph convolutional neural network model is expressed as: G i =T i F i , T i is the attention size generated by the functional near infrared spectroscopy data at the i-th moment, F i is the characteristic graph of the functional near-infrared spectral data at the i-th moment; considering the temporal continuity of the data, the output It is expressed as: Among them, δ i G i As input and learned through a single-layer MLP, w j , G j They represent the weights of the respective attention modules at the jth moment and the output of the self-attention module at the jth moment.

6. The method for assisting in assessing brain dysfunction based on multimodal data fusion according to claim 5, characterized in that: The multi-scale graph convolutional neural network model includes a graph convolutional neural network and a global average pooling layer. The calculation process of the multi-scale graph convolutional neural network model is as follows: Firstly, the EEG time series signal is reconstructed and the corresponding waveform profile is generated; the functional near infrared data is standardized and the correlation analysis of the functional near infrared data is generated, and then the feature map is generated; Subsequently, the generated waveform profile and feature map are input into the graph convolutional neural network to generate fused feature information, where the calculation process of the graph convolutional neural network is as follows: P l+1 =σ(LP l W l ) Among them, P l , W l , L and σ are the output, weight coefficient, adjacency matrix and nonlinear activation function of the lth network layer in the graph convolutional neural network respectively; Finally, a global average pooling operation is performed on the fused feature information generated by the graph convolutional neural network to generate the patient's saliency map.

7. The method for assisting in assessing brain dysfunction based on multimodal data fusion according to claim 1, characterized in that: In step S2, The goal of the heterogeneous spatiotemporal data fusion model is to find a spatial transformation or spatial mapping between heterogeneous images, transform the image to be registered into the image space of the target image, and thus establish a corresponding relationship between the spatial positions of the images; using a feature-based method, features are first extracted from the image, and then feature matching and image registration are performed based on feature definition metrics.

8. The method for assisting brain dysfunction assessment based on multimodal data fusion according to claim 7, characterized in that: Using the feature-based approach, features are first extracted from the image, and then feature matching and image registration are performed based on feature-defined metrics, as follows: First, the image acquired by functional magnetic resonance imaging is segmented, and then a plurality of position points are automatically generated by selecting magnetic resonance reference points, and then they are converted into corresponding functional near-infrared digital reference points as the positions of the optodes through rigid transformation; The position of the optode is corrected by calculating the nearest head surface point on the straight line between the center of the head surface and the optode position, and then the midpoint of the channel source and detector is calculated as the position of the channel and projected onto the cortical grid. The correction method of the channel position is similar to that of the optode; Subsequently, the voxel space coordinates corresponding to each channel were extracted and converted into coordinates in the standardized MNI space. The β value and BOLD signal of the functional magnetic resonance imaging at the coordinate point in the standardized MNI space were calculated and compared with the functional near-infrared data to complete the spatial registration process.

9. A method for assisting assessment of brain dysfunction based on multimodal data fusion according to any one of claims 1 to 8, characterized in that: In step S3, the expert system includes k hybrid expert models Expert k , Hybrid Expert Model Expert k Based on the Transformer network model, each hybrid expert model Expert k It includes an input layer, a transformer layer, and an output layer. The input of the input layer is the feature information and error pattern ε calculated by the transformer network model and stacked in the time dimension. It is followed by a transformer layer with a dual-channel structure. Each channel includes a transformer structure. Finally, the output of the dual-channel transformer layer is used as the input of the output layer. The output layer is a single-layer fully connected neural network. Finally, a fully connected hybrid module is used to obtain the final decision result: in, represents the output of the expert system at time t, and L is the number of hybrid expert models in the expert system.

Images