Multimodal deep learning source localization method and system fusing magnetoencephalography and electroencephalography

Abstract

The application discloses a multi-modal deep learning source tracing method and system combining magnetoencephalogram and electroencephalogram, relates to the field of artificial intelligence and neuroimaging technology, and inputs real magnetoencephalogram signals and electroencephalogram signals into a source tracing model after preprocessing, predicts the probability of the occurrence of corresponding regional sources, combines an imported source partition distance matrix, and obtains a final source tracing result; the training process of the source tracing model is as follows: a generative adversarial network is constructed to generate a multi-modal neuroelectrophysiological data set; the multi-modal neuroelectrophysiological data set is input into a double-branch structure residual network to perform stage hierarchical extraction and decoupling on the magnetoencephalogram signals and the electroencephalogram signals respectively; a multi-scale convolution module is used to extract features at different stages, the extracted features are input into a classifier after being fused, and a loss function is defined to update trainable parameters of the source tracing model; and the source tracing method improves the accuracy and generalization ability of source positioning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and neuroimaging technology, and in particular to a multimodal deep learning source tracing method and system that integrates magnetoencephalography (MEG) and electroencephalography (EEG). Background Technology

[0002] Magnetoencephalography (MEG) and electroencephalography (EEG), as non-invasive neuroimaging techniques, are core tools for brain source imaging. These techniques locate neural activity sources by acquiring electrical / magnetic signals from the brain and are widely used in neuroscience, clinical medicine, and cognitive research, playing a crucial role, especially in the localization of epileptic seizures and the diagnosis of neurological diseases. Compared to other non-invasive techniques such as functional near-infrared spectroscopy (fNIRS) and functional magnetic resonance imaging (fMRI), MEG / EEG possesses millisecond-level high temporal resolution, enabling precise capture of dynamic changes in neuronal electrical / magnetic signals. This gives it irreplaceable advantages in scenarios such as epilepsy monitoring and cognitive response analysis, making it an important tool for studying neurological diseases.

[0003] The problem of tracing the source of information based on magnetoencephalography (MEG) or electroencephalography (EEG) can generally be defined as:

[0004] ;

[0005] in, For the first Sensor signals from magnetoencephalography (MEG) / electroencephalography (EEG) at any given time. This represents the number of channels (i.e., the number of sensors) in a magnetoencephalogram (MEG) or electroencephalogram (EEG). Represents the real number vector space; For the guiding field matrix, The number of source space partitions is represented by 3, and 3 represents the three directions of the equivalent source dipole. The guiding field matrix reflects the mapping relationship between the magnetoencephalogram (MEG) or electroencephalogram (EEG) sensor and the electric dipole moment of the source at any location. Represents a dipole in the source space; Representing the Noise at any given moment. Considering the number of sources in the above problem ( The number of sensors is usually much greater than the number of other sensors. Therefore, solving for a source usually involves solving an ill-posed problem, making it difficult to find a unique source.

[0006] The development of traditional source tracing techniques exhibits a clear generational evolution. In the early research stage (1980s-1990s), weighted least squares was widely used as a fundamental technique for localizing neural electrical activity. In 1990, Phillips et al. achieved a technological breakthrough by introducing beamforming theory from signal processing into this field, innovatively applying the CAPON algorithm for multi-source signal classification, significantly improving the analytical capabilities for complex neural activity. With the innovation of neuroimaging technology, especially the breakthrough progress of magnetoencephalography (MEG) in monitoring brain functional activity (early 21st century), standardized low-resolution electromagnetic computed tomography (sLORETA) has gained widespread application as a method for solving classical linear inverse problems. Traditional algorithms typically use manually added constraints to obtain a unique source location, which limits the applicability and robustness of traditional algorithms. Furthermore, traditional algorithms are mainly designed for single-source tracing and perform poorly with multi-source tracing and low signal-to-noise ratio conditions.

[0007] In recent years, with the development of artificial intelligence, deep learning technology has also been proposed for use in the field of brain source localization imaging. Cui et al. proposed a method using Long Short-Term Memory (LSTM) networks to simultaneously extract spatial and temporal features from magnetoencephalography (MEG) signals acquired from the scalp, thereby achieving more accurate source localization. Sun et al. proposed a framework combining feedforward neural networks and recurrent neural networks to extract spatiotemporal features from EEG and MEG signals, respectively. This method focuses on single-modality modeling and effectively improves the model's localization ability in handling spatiotemporal dynamic changes by guiding network training under the constraints of a neural mass model.

[0008] Compared to traditional algorithms, deep learning algorithms improve performance in multi-source scenarios and low signal-to-noise ratio scenarios. However, most current deep learning-based methods only use a single modality (MEG or EEG), resulting in limited performance in low signal-to-noise ratio or multi-source scenarios. Furthermore, there are distributional differences between traditional physical simulation data and real data, which affects the model's generalization ability. Summary of the Invention

[0009] Based on the technical problems existing in the background technology, this invention proposes a multimodal deep learning source tracing method and system that integrates magnetoencephalography (MEG) and electroencephalography (EEG). It combines multi-scale convolution and channel-space attention mechanisms and uses generative adversarial networks (GANs) to generate high-fidelity simulated data, aiming to jointly improve the accuracy and generalization ability of source tracing and localization.

[0010] The multimodal deep learning source tracing method proposed in this invention integrates magnetoencephalography (MEG) and electroencephalography (EEG). After preprocessing, the real MEG and EEG signals are input into the source tracing model to predict the probability of the source appearing in the corresponding region. Combined with the imported source partition distance matrix, the final source tracing result is obtained.

[0011] The training process of the source tracing model is as follows:

[0012] A generative adversarial network was constructed, and the generated synthetic observation dataset was preprocessed to obtain a multimodal neurophysiological dataset including magnetoencephalogram (MEG) signals and electroencephalogram (EEG) signals.

[0013] The multimodal neuroelectrophysiological dataset was input into a dual-branch residual network to perform stage-by-stage extraction and decoupling of magnetoencephalography (MEG) signals and electroencephalography (EEG) signals, respectively.

[0014] Multi-scale convolution modules are used to extract features at different stages. The extracted features are then fused and input into a classifier, and a loss function is defined to update the trainable parameters of the source tracing model.

[0015] Furthermore, the construction process of the generative adversarial network is as follows:

[0016] The generator takes a random seed vector as input. and source region label vector The generator output is the synthesized source space current density timing signal;

[0017] The source space current density time-series signal is multiplied by the pre-calculated magnetoencephalogram (MEG) guidance field matrix and electroencephalogram (EEG) guidance field matrix, and then Gaussian white noise is superimposed to synthesize noisy simulated data. The simulated data includes simulated data of magnetoencephalogram (MEG) and simulated data of electroencephalogram (EEG).

[0018] The simulated data and real data are input into the discriminator to determine whether the input data comes from a real distribution or a generated distribution;

[0019] By alternately optimizing the generator and discriminator, the generator produces a label vector that matches the source region. The source activity signal is consistent with and has realistic noise characteristics, from which the magnetoencephalogram (MEG) signal, electroencephalogram (EEG) signal, and label vector of the source region are obtained. The synthetic observation dataset.

[0020] Furthermore, the step of using a multi-scale convolution module to extract features at different stages, fusing the extracted features, and then inputting them into the classifier is as follows:

[0021] The two branches are respectively The features output from each stage are input in parallel into a multi-scale convolutional module to generate the first stage. The fusion features corresponding to each stage are aligned and then summed to obtain the global context fusion feature.

[0022] The features output from the last stage are aggregated with the global context fusion features, and the resulting cross-modal weighted fusion features are concatenated with the global context fusion features to obtain cross-modal deep fusion features;

[0023] Cross-modal deep fusion features are fed into the classifier.

[0024] Furthermore, the step of inputting the multimodal neurophysiological dataset into a dual-branch residual network to perform stage-by-stage extraction and decoupling of the magnetoencephalogram (MEG) signal and electroencephalogram (EEG) signal is specifically as follows:

[0025] The residual network has two identical branch structures, each branch includes M stages, and each stage consists of W residual blocks connected in series.

[0026] One branch of the residual network is used for stage-by-stage extraction and decoupling of the magnetoencephalogram (MEG) signal to obtain the first... Electroencephalogram (MEG) characteristics output at each stage Another branch is used for stage-level extraction and decoupling of EEG signals to obtain the first... EEG characteristics of each stage , , This represents the total number of stages in one of the branches;

[0027] The residual network of the first Output characteristics of each stage .

[0028] Furthermore, the two branches respectively The features output from each stage are input in parallel into a multi-scale convolutional module to generate the first stage. The fusion features corresponding to each stage are as follows:

[0029] Will The feature map matrix is ​​obtained by fusing along the sensor channel dimension. , feature map matrix The input is fed into a multi-scale convolution module, which includes M convolution branches. Each convolution branch is sequentially connected to a batch normalization layer and an activation layer. Different parameters are set based on different convolution kernels to keep the feature shape and size of each convolution branch consistent. Each convolution branch outputs local branch features.

[0030] After concatenating the local branch features output from all convolutional branches along the channel dimension, the channel dimension is changed through the last convolutional branch to obtain the first... Fusion characteristics corresponding to each stage .

[0031] Furthermore, the step of aligning and summing the fusion features generated in all stages to obtain the global context fusion feature specifically involves:

[0032] Output features of the last stage of the residual network The feature map matrix is ​​obtained by fusing along the sensor channel dimension. ;

[0033] feature map matrix The global context weights are obtained by accumulating the pixel values ​​and then applying them to an activation function. ;

[0034] Before The fusion features corresponding to each stage are compressed using convolution operations to reduce the channel dimension, and then output after pooling layers. Each stage corresponds to a feature alignment matrix, and the dimension of each feature alignment matrix is ​​consistent with the fusion feature corresponding to the last stage.

[0035] Before The feature alignment matrix corresponding to each stage, the feature map matrix By adding each element sequentially, the global context fusion feature is obtained. .

[0036] Furthermore, the global context weights The calculation process is as follows:

[0037] ;

[0038] in, Represents the feature map matrix In the sensor dimension Location and time dimensions The value at the location, Feature map matrix The corresponding row number and column number, This is the activation function.

[0039] Furthermore, the features output from the final stage are aggregated with the global context fusion features. The resulting cross-modal weighted fusion features are then concatenated with the global context fusion features to obtain cross-modal deep fusion features, specifically:

[0040] Features Features fused with global context The features are fed into the feature aggregation module for dynamic weighting to obtain cross-modal weighted fusion features. ;

[0041] The cross-modal weighted fusion features are fed into the channel weight attention module and the spatial weight attention module, utilizing the channel attention weights. Spatial attention weights Separately with cross-modal fusion features Perform the Kronecker product to obtain cross-modal deep fusion features. .

[0042] Furthermore, the loss function Specifically:

[0043] ;

[0044] in, The number of source regions corresponds to the number of classification categories. For source region index, Describes the first in a sample The probability value of the existence of each source region is the predicted value output by the source tracing model. For a sample, the first The label vector of each source region is the real label.

[0045] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0046] The advantages of the multimodal deep learning-based source tracing method and system integrating magnetoencephalography (MEG) and electroencephalography (EEG) provided by this invention are as follows: It employs generative adversarial networks to construct the training dataset, replacing traditional physical simulation methods, to generate multimodal EEG signals that more closely approximate the statistical distribution of real data, fundamentally improving the generalization ability of the downstream source tracing model. Furthermore, this method utilizes the source tracing model to integrate features from different channels of MEG and EEG, combining stage features and multi-scale convolutional modules to fully extract the data's performance across different dimensions, significantly improving the accuracy and precision of brain source tracing and localization. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of the method of the present invention;

[0048] Figure 2 A flowchart illustrating the workflow of generative adversarial networks;

[0049] Figure 3 Examples of magnetoencephalography (MEG) and electroencephalography (EEG) data are shown below, where (a) is an example of MEG data and (b) is an example of EEG data.

[0050] Figure 4 This is a structural diagram of the source tracing network;

[0051] Figure 5 Flowchart for verifying real data. Detailed Implementation

[0052] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] like Figures 1 to 5 As shown, the multimodal deep learning source tracing method that integrates magnetoencephalography (MEG) and electroencephalography (EEG) proposed in this invention inputs real MEG and EEG signals into the source tracing model after preprocessing, predicts the probability of the source appearing in the corresponding region, and obtains the final source tracing result by combining the imported source partition distance matrix.

[0054] The training process of the source tracing model is as follows:

[0055] Step 1: Construct a generative adversarial network and preprocess the generated synthetic observation dataset to obtain a multimodal neurophysiological dataset including magnetoencephalogram (MEG) signals and electroencephalogram (EEG) signals.

[0056] Step 2: Input the multimodal neuroelectrophysiological dataset into a dual-branch residual network to perform stage-by-stage extraction and decoupling of magnetoencephalography (MEG) signals and electroencephalography (EEG) signals, respectively.

[0057] Step 3: Use a multi-scale convolution module to extract features at different stages, fuse the extracted features and input them into the classifier, define the loss function, and update the trainable parameters of the source tracing model.

[0058] This embodiment employs generative adversarial networks to construct the training dataset, replacing traditional physical simulation methods. This generates multimodal EEG signals that more closely approximate the statistical distribution of real data, fundamentally improving the generalization ability of the downstream source tracing model. Furthermore, this method utilizes the source tracing model to integrate features from different channels of magnetoencephalography (MEG) and electroencephalography (EEG), combining stage features and multi-scale convolutional modules to fully extract the data's performance across different dimensions, significantly improving the accuracy and precision of brain source tracing and localization.

[0059] In one embodiment, step one involves constructing a generative adversarial network and preprocessing the generated synthetic observation dataset to obtain a multimodal neurophysiological dataset including magnetoencephalography (MEG) signals and electroencephalography (EEG) signals. Specifically:

[0060] (a1) Construct a generative adversarial network, where the input to the generator G is a random seed vector. (Used to simulate the randomness of neural activity) and label vectors of source regions (One-hot encoded vector, representing the target source location), the output of generator G is the synthesized source spatial current density time-series signal. ,in The number of source space partitions, This represents the number of sampling points.

[0061] (a2) will generate (a1) With the pre-calculated magnetoencephalogram (MEG) guidance field matrix EEG-guided field matrix Perform matrix multiplication operations separately, and then superimpose Gaussian white noise with noise characteristics similar to those of the real data. Ultimately, synthesized, noisy simulation data is obtained.

[0062] It should be noted that, With magnetoencephalography (MEG) guidance field matrix Perform matrix multiplication, then add Gaussian white noise. Simulated data of magnetoencephalography (MEG) was obtained. With EEG guidance field matrix Perform matrix multiplication, then add Gaussian white noise. This yielded simulated electroencephalogram (EEG) data.

[0063] (a3) Construct a discriminator D. The inputs to discriminator D are the two types of simulated data obtained in (a2) and the two types of real data introduced. Its main task is to determine whether the data is simulated.

[0064] It is understandable that the two types of simulated data specifically refer to simulated magnetoencephalography (MEG) data and simulated electroencephalography (EEG) data; the two types of real data refer to real MEG data. and real data from electroencephalograms .

[0065] That is, introducing real dataset pairs into the discriminator. , Corresponding to the simulated data from magnetoencephalography (MEG), Corresponding to simulated EEG data.

[0066] (a4) Generative Adversarial Network Training and Dataset Generation. By alternately optimizing the generator G and the discriminator D, the generator G generates label vectors corresponding to the source regions. The source activity signal is consistent with and has realistic noise characteristics.

[0067] The trained generator G is used to generate a large-scale source space current density time series signal. (i.e., the source signal), after being processed by (a2) and (a3), is transformed into the final synthetic observation dataset { ,in This is a magnetoencephalogram (MEG) signal. This is an electroencephalogram (EEG) signal.

[0068] (a5) The obtained synthetic observation dataset is preprocessed sequentially through bandpass filtering for denoising, baseline correction, and time alignment to obtain a multimodal neurophysiological dataset, specifically:

[0069] Magnetoencephalograms and electroencephalograms in the synthetic observation dataset were analyzed using... Bandpass filtering parameters are used for bandpass filtering and Notch filtering.

[0070] The baseline mean was subtracted from each channel of the bandpass filtered data, transforming the data into a distribution with a mean of 0 and a standard deviation of 1. The EEG was linearly interpolated using the time synchronization marker of the magnetoencephalogram (MEG) sensor as a reference to ensure that the timestamps of the MEG and EEG were perfectly aligned, thus obtaining a multimodal neurophysiological dataset that includes both MEG and EEG signals.

[0071] This embodiment introduces generative adversarial networks (GANs) into the field of EEG / MEG data generation. It can automatically learn the complex distribution of real data and generate massive, high-quality MEG / EEG (MEG / EEG) paired data with accurate source labels. This effectively alleviates the data scarcity problem faced by deep learning models in this field and significantly improves the generalization performance and robustness of the source tracing model in real-world scenarios.

[0072] In one embodiment, step two involves inputting the multimodal neurophysiological dataset into a dual-branch residual network to perform stage-by-stage extraction and decoupling of the magnetoencephalogram (MEG) signal and electroencephalogram (EEG) signal, specifically:

[0073] The multimodal neuroelectrophysiological dataset obtained in step one is divided into segments. Magnetoencephalogram (MEG) and electroencephalogram (EEG) data of fixed-cycle sizes are input into two dual-branch structures of the residual network FAME-Net. Output features for different stages are obtained based on these dual-branch structures. The number of convolutional layers and other parameter settings for each branch structure can be referenced from ResNet18, a common implementation of ResNet. Specific network hyperparameter settings are not detailed here. The above operations specifically include (b1) to (b2):

[0074] (b1) Divide the multimodal neurophysiological dataset into a training set, a test set, and a validation set, with the following division ratio: The typical ratio is 0.7:0.2:0.1; a fixed number of rounds of magnetoencephalography (MEG) is used. With EEG The training set is input into the dual-branch structure. The batch size set for deep learning training (i.e., the number of samples per training session). These represent the number of sensor dimensions for magnetoencephalography (MEG) and electroencephalography (EEG), respectively. This represents the number of time dimensions (i.e., the number of sampling points).

[0075] (b2) The two branches of the residual network have the same structure. Each branch includes M stages, and each stage consists of W residual blocks connected in series. One branch of the residual network is used for stage-level extraction and decoupling of the magnetoencephalogram (MEG) signal to obtain the first stage. Electroencephalogram (MEG) characteristics output at each stage Another branch is used for stage-level extraction and decoupling of EEG signals to obtain the first... EEG characteristics of each stage , , The total number of stages in one of the branches; the number of stages in the residual network. Output characteristics of each stage .

[0076] This embodiment uses M = 4 and W = 2 for illustration:

[0077] The formula for the residual block is:

[0078] ;

[0079] in, For the first One residual block, For activation functions (i.e., activation layers). The kernel size is Convolutional layers, For identity mapping, For batch normalization.

[0080] The outputs of magnetoencephalography (MEG) and electroencephalography (EEG) at each stage of their respective branches are as follows:

[0081] ;

[0082] , This represents the total number of stages in one of the branches.

[0083] In one embodiment, step three involves using a multi-scale convolution module to extract features from different stages, fusing the extracted features, inputting them into a classifier, defining a loss function, and updating the trainable parameters of the source tracing model. Specifically:

[0084] (c1) respectively divide the two branches into two branches. The features output from each stage are input in parallel into a multi-scale convolutional module to generate the first stage. The fusion features corresponding to each stage are aligned and then summed to obtain the global context fusion feature.

[0085] For each ResNet stage of one of the branches ,Will The feature map matrix is ​​obtained by fusing along the sensor channel dimension. , feature map matrix The input is fed into a multi-scale convolution module, which includes M convolution branches. Each convolution branch is sequentially connected to a batch normalization layer and an activation layer. Different parameters are set based on different convolution kernels to keep the feature shape and size of each convolution branch consistent. Each convolution branch outputs local branch features.

[0086] After concatenating the local branch features output from all convolutional branches along the channel dimension, the channel dimension is changed through the last convolutional branch to obtain the first... Fusion characteristics corresponding to each stage .

[0087] Output features of the last stage of the residual network The feature map matrix is ​​obtained by fusing along the sensor channel dimension. ; the feature map matrix The global context weights are obtained by accumulating the pixel values ​​and then applying them to an activation function. .

[0088] Before Fusion characteristics corresponding to each stage Convolution operations are performed to compress channel dimensions, and the output is then processed through a pooling layer. Feature alignment matrices corresponding to each stage Each feature alignment matrix corresponds to the fused features of the last stage. Maintain consistency.

[0089] Before Feature alignment matrix corresponding to each stage The feature map matrix By adding each element sequentially, the global context fusion feature is obtained. .

[0090] Taking M=4 as an example, the feature map matrix is... Input to In the multi-scale convolution module, This refers to convolutional layers corresponding to different convolutional kernels. Each convolutional layer is sequentially connected to a batch normalization layer and an activation layer, as shown in the formula:

[0091] ;

[0092] in, for After the first Local branch features corresponding to each convolutional layer For activation function, For batch normalization layer, For the first One convolutional layer. Here The value of 4 corresponds to M being 4 above. In practical applications, M and The value of depends on the specific circumstances.

[0093] Branch features output by the four convolutional layers Perform a stitching operation along the channel dimension, then pass through Changing the channel dimension yields multimodal stage fusion features To ensure that the input and output sizes of the entire module remain constant, They are respectively go through The corresponding local branch features.

[0094] Features output from the fourth stage of the magnetoencephalography (MEG) and electroencephalography (EEG) branches The feature map matrix is ​​obtained by concatenating the features. ;

[0095] feature map matrix The pixel values ​​are accumulated and then passed through the activation layer to obtain the global context weights. The calculation is as follows:

[0096] ;

[0097] in, Represents the feature map matrix In the sensor dimension Location and time dimensions The value at the location, Feature map matrix The corresponding row number and column number, This is the activation function.

[0098] Features of the first stage of integration Second-stage integration characteristics The third stage of integration characteristics Convolutional operations are performed to compress the channel dimensions. This is followed by pooling layers, outputting the first-stage feature alignment matrix. Second-stage feature alignment matrix The feature alignment matrix of the third stage ,in , , The dimensional integration features of the fourth stage Maintain consistency; finally , , , Element-wise accumulation yields global context features. .

[0099] (c2) Aggregate the features output from the last stage with the global context fusion features, and concatenate the resulting cross-modal weighted fusion features with the global context fusion features to obtain cross-modal deep fusion features;

[0100] Specifically: features Features fused with global context The features are fed into the feature aggregation module for dynamic weighting to obtain cross-modal weighted fusion features. The cross-modal weighted fusion features are fed into the channel weight attention module and the spatial weight attention module, utilizing the channel attention weights. Spatial attention weights Separately with cross-modal fusion features Perform the Kronecker product to obtain cross-modal deep fusion features. .

[0101] Taking M as 4 as an example, the global context weights are used for explanation. Feature map matrix With global context features Adaptive weighted fusion is performed to obtain cross-modal weighted fusion features. The specific calculation formula is as follows:

[0102] ;

[0103] Cross-modal weighted fusion features The data are sequentially fed into the channel weight attention module and the spatial weight attention module to obtain the channel attention weights. Spatial attention weights The two weights are respectively associated with the cross-modal fusion features. Perform the Kronecker product. ), to obtain cross-modal deep fusion features The specific formula is:

[0104] ;

[0105] Among them, channel attention weight The calculation process is as follows: Keeping the channel dimension unchanged, the spatial dimension is compressed. Specifically, cross-modal feature fusion... The data are fed into a max pooling layer and an average pooling layer, respectively, to obtain two feature maps with a spatial dimension of 1. Then, the two feature maps (the one obtained after max pooling and the one obtained after average pooling) are summed element-wise, and the Sigmoid function is used to activate them, mapping the values ​​to between 0 and 1, thus obtaining the channel attention weights. .

[0106] Among them, spatial attention weights The calculation process is as follows: Keeping the spatial dimension unchanged, compress the channel dimension. The output of the channel attention module is then processed. The algorithm obtains two channel-compressed feature maps through max pooling and average pooling layers. Then, it performs a concatenation operation to merge the two feature maps (the one obtained through max pooling and the one obtained through average pooling). The feature map is converted to a feature map with 1 channel, and then passed through a Sigmoid activation layer to obtain spatial weights. .

[0107] It should be noted that channel attention weights The calculation process and spatial attention weights Both calculations use max pooling and average pooling layers, but the two calculation processes correspond to independent max pooling and average pooling layers, respectively.

[0108] (c3) Feed the cross-modal deep fusion features into the classifier.

[0109] A fully connected layer is used as the classifier, and the input is cross-modal deep fusion features. The output is the probability of the predicted source appearing in the corresponding region. A Sigmoid cross-entropy loss function is constructed based on the probability, and gradient backpropagation is performed based on this loss function to update the model parameters. Preserve the best model parameters based on the test set. The formula for the loss function is:

[0110] ;

[0111] in, The number of source regions corresponds to the number of classification categories. For source region index, Let be the label vector of the source region and the predicted vector of the source coordinates, respectively, representing the first _____. The true labels and model predictions for each source region.

[0112] ;

[0113] in: For learning rate, They are the first Step, First Model parameters for each iteration Indicates the first Each iteration corresponds to the model's predicted values ​​in the model parameter update process. Indicates the first Each iteration corresponds to the actual label value.

[0114] Finally, the converged source tracing model was trained using the steps, evaluation metrics were calculated on a real dataset, and a brain source tracing distribution map was drawn based on the results.

[0115] As an example:

[0116] This embodiment proposes a multimodal deep learning-based source tracing method that integrates magnetoencephalography (MEG) and electroencephalography (EEG). First, a generative adversarial network is trained to generate a multimodal neurophysiological dataset (i.e., simulated data). Then, the source tracing model is trained using the multimodal neurophysiological dataset, and the best model is retained. Finally, the trained source tracing model is used to output the corresponding prediction results and calculate the corresponding indicators to evaluate the model performance. Specifically, this can be divided into steps S1 to S7.

[0117] S1: This step creates a source-labeled simulation dataset using a generative adversarial network. The flowchart for this step can be found here. Figure 2 The specific implementation is as follows:

[0118] S11: Construct a generative adversarial network based on the StyleGAN2 architecture. The generator G uses a temporal convolutional network (TCN) as the backbone network, and the input is a random seed vector. and the label vector of the 10242-dimensional source region (That is, a one-hot encoded label vector) is used to specify the precise brain region location (corresponding to a vertex on the scalp grid) where epileptic spikes occur. The generator G calculates and outputs a synthesized source spatial current density time-series signal. , which represents the intensity of neural electrical activity at 10242 vertices of the cerebral cortex within a 2-second time window (sampling frequency 1000Hz, 2000 time points).

[0119] S12: Before training the source tracing model, use the boundary element method and import the average head model. The guiding field matrix of the magnetoencephalogram (MEG) is calculated using the `mne.make_forward_solution` function in the MNE-Python toolkit. With the guidance field matrix of EEG Compare it with the brain-derived signals obtained in S11. Matrix multiplication is performed and random Gaussian white noise is added to synthesize noisy simulated data. MNE-Python is an open-source Python toolkit mainly used for processing, analyzing and visualizing neurophysiological data such as electroencephalograms (EEG), magnetoencephalograms (MEG), and sEEG.

[0120] S13: Construct the discriminator D; use a 2D convolutional network as the backbone network. The input to the discriminator is the simulated data obtained in S12 and the introduced real data. Its main task is to distinguish whether the input data comes from the real distribution or the generated distribution.

[0121] S14: Use Adam optimization to train the network generator G and discriminator D, minimizing the following loss function:

[0122] ;

[0123] in, These are real data. It is a random seed vector. It is the label vector of the source region. These are the linear interpolation points between real and simulated data. and These are weighting coefficients (set to 10 and 0.1 respectively in this embodiment). This is a domain adaptation loss term that ensures consistency between magnetoencephalography (MEG) and electroencephalography (EEG) modalities. In this embodiment... The mean squared error (MSE) is used for calculation. The parameters of the discriminator D and the generator G are updated alternately using the Adam optimizer until the generative adversarial network converges.

[0124] The learning rate for both the generator and the discriminator was set to 0.0001. The batch size was fixed at 32. The training iterations were performed for 100,000 times until the discriminator could no longer distinguish between real and simulated data with an accuracy significantly higher than 50%, at which point the training was considered converged.

[0125] After training, a synthetic observation dataset containing 1,000,000 samples was generated using generator G, with each sample containing... , And its precise source region label vector (i.e., source label). This step provides sufficient and high-quality training samples for downstream tracing models, such as Figure 3 As shown.

[0126] S2: Preprocessing the data generated in S1. In this embodiment, since the data has already been generated by the generator, its signal-to-noise ratio and baseline are relatively stable. Therefore, the main operation is data alignment: linear interpolation is performed on the EEG data based on the timestamp of the magnetoencephalogram (MEG) sensor to ensure strict synchronization of the two modalities in the time dimension. Then, the data is divided into segments according to 2-second time windows. Specifically, this includes the following:

[0127] Using the time synchronization marker of the magnetoencephalogram (MEG) sensor as a reference, the electroencephalogram (EEG) is linearly interpolated (interpolation step size 1ms) to ensure that the timestamps of the MEG and the EEG are perfectly aligned.

[0128] The aligned continuous data was divided into segments according to a 2-second time window (2000 sampling points, 1000Hz sampling rate), with each segment having a dimension of magnetoencephalography (MEG). Electroencephalogram (EEG) .

[0129] S3: The FAMENet source tracing model is a two-branch, multi-stage hierarchical structure based on ResNet (residual neural network). The specific model architecture is as follows: Figure 4 As shown, layered feature decoupling is performed on magnetoencephalography (MEG) and electroencephalography (EEG) signals to capture neurophysiological information from basic temporal patterns to high-dimensional abstract patterns layer by layer. Steps S31 and S32 are as follows:

[0130] S31: The multimodal neuroelectrophysiological dataset is divided into fixed-round sizes (batchsize=32) and simultaneously input into the magnetoencephalography (MEG) branch (ResNet) and electroencephalography (EEG) branch (ResNet) of the residual network. (MEG input) EEG input (batch size × number of channels × number of sensors × time sampling points).

[0131] S32: Each ResNet branch consists of 4 stages (stage 1-stage 4). The number of ResNet convolutional layers and the kernel size settings can be found in ResNet18. Each stage consists of two residual blocks.

[0132] The ResNet branches of magnetoencephalography (MEG) and electroencephalography (EEG) output four phase features, denoted as... ,in For the first The brain magnetic resonance characteristics output at each stage For the first The EEG characteristics output at each stage are called stage characteristics. The changes in the size of the feature shape are shown in Table 1 below:

[0133] Table 1. Stage characteristics of magnetoencephalography (MEG) and electroencephalography (EEG)

[0134]

[0135] S4: Multi-scale convolutional modules are used to extract spatial frequency features under different receptive fields, and channel dimension compression is used to improve compactness, providing high-resolution feature support for cross-modal fusion and obtaining staged fusion features. Specific implementations include S41 and S42.

[0136] S41: Output features of each stage of the ResNet branch for magnetoencephalography / electroencephalography obtained by stitching together the sensor dimensions Then, the parallel input is fed into the system by... The multi-scale convolutional module consists of inputting the stage features in parallel into four branches with kernel sizes {1, 3, 5, 7} to extract features, followed by batch normalization (BN) based on ReLU activation, which can be represented as follows: ;

[0137] S42: Branch features output from the four convolutional branches Perform a stitching operation along the channel dimension, then pass through Changing the channel dimension yields multimodal stage fusion features This ensures that the input and output sizes of the entire module remain constant. The specific shape and size are obtained by stitching along the sensor dimension ， , , .

[0138] S5: Process the features from ResNet stage 4 of magnetoencephalography (MEG) and electroencephalography (EEG) separately. And integrate the stage fusion features obtained in S42. Obtain global context fusion features . and The cross-modal weighted fusion features are obtained by dynamically weighting the features into the feature aggregation module. Based on the channel-space attention module, global collaboration and noise suppression of multimodal features are achieved to obtain the final cross-modal deep fusion features. Specifically, it includes steps S51, S52, S53, S54, and S55.

[0139] S51: Integrating the characteristics of the fourth branch stage of magnetoencephalography (MEG) with electroencephalography (EEG). The feature map is obtained by splicing. .

[0140] S52: Calculate global context weights The specific calculation method is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] The global context weights are obtained by summing the pixel values ​​of the third and fourth dimensions and then applying softmax. The calculation is as follows:

[0141] ;

[0142] in, represent The pixel values ​​at positions a and b in the third and fourth dimensions, respectively.

[0143] S53: Yes ， , Convolution is used to change the number of channels, and pooling operations are used to change the size. To maintain consistency, the four features are then summed to obtain the global context fusion feature. .

[0144] S54: By global context weights and and Adaptive weighted fusion is performed to obtain cross-modal weighted fusion features. .

[0145] ;

[0146] S55: Weighted fusion features across modalities This is achieved through channel attention and spatial attention modules, respectively. Specifically, the channel attention module keeps the channel dimension unchanged while compressing the spatial dimension, thus reducing the input... Parallel processing through max pooling and average pooling layers yields a value of [value]. The weights are calculated by adding these two weights element-wise, and then activating the channel attention weights using a sigmoid function (activation function). .

[0147] Spatial Attention Module: Keeps the spatial dimension unchanged, compresses the channel dimension. The output of the channel attention module ( Two results are obtained through max pooling and average pooling. The feature maps are then concatenated through a concatenation operation. After being converted into a 1-channel feature map, it is activated by Sigmoid to spatial weights. Then multiply to obtain the cross-modal deep fusion features. .

[0148] ;

[0149] S6: This step simultaneously optimizes the dual-branch structure of the residual network (MEG / EEG ResNet) and the trainable parameters of the source model through classifier output, cross-entropy loss, and gradient backpropagation, achieving end-to-end training of multimodal features. The specific implementation is as follows:

[0150] Using a fully connected layer as a classifier, The data is expanded and then mapped to 10242 dimensions for classification prediction. The Sigmoid function (activation function) is used to calculate the prediction probability, and the final output is the prediction vector of the source coordinates. Then the predicted vector of the source coordinates label vector of the source region Calculate loss Here, we choose the cross-entropy loss function. Each value in the table is either 0 or 1, and each value represents whether the corresponding source region exists according to its index. This is a multi-label classification problem. The loss function is calculated as follows:

[0151] ;

[0152] Where 10242 represents the size of the source region. This indicates that the source region index in a sample is The probability value of the corresponding source coordinates existing. The source region index in a sample is The label vector of the corresponding source region.

[0153] Using the Adam optimizer (learning rate) Weight decay The parameter update formula is:

[0154] ;

[0155] The model parameters are updated in training epochs, which can be adjusted according to specific circumstances. At the end of each training epoch, a test set is used to test the model and verify its effectiveness. If the effectiveness does not meet the preset requirements, training will continue on the training set until the preset requirements are met. That is, when the performance metrics on the validation set (such as localization error) no longer improve, or the loss function decreases negligibly over multiple consecutive training epochs, the model can be considered to have converged. Specifically, when the model converges:

[0156] when hour, ;

[0157] in, The parameters saved for the final model, i.e., the optimal model. It is a fixed threshold constant, and its value is generally set to 0.0001.

[0158] S7: This step evaluates the source tracing performance of the best model using real-world datasets, such as... Figure 5 By combining data preprocessing techniques, the data is loaded into the optimal model, and evaluation indicators such as Mean Location Error (MLE), Precision, and Recall are calculated. The source tracing results are then mapped to the cerebral cortex to generate a visual source tracing map. The specific implementation is as follows:

[0159] The data preprocessing techniques have been described in step (a5) of step one, and will not be repeated here.

[0160] The data is input into the source tracing model to obtain a probability vector. This vector is then imported into the source partition distance matrix, which records the distances between the center points of each region in the brain partition. A threshold (typically set to 0.5) can be set based on the probability vector to obtain the final source result. Then, evaluation metrics such as average localization error, precision, and recall are calculated. The source tracing results are then imported into the head model. Call the mne.viz.plot_source_estimates function to generate a 3D brain source map (marking brain regions with active spike waves, such as the temporal lobe and frontal lobe).

[0161] Through steps S1 to S7, a novel multimodal source tracing network is proposed, and more importantly, an innovative paradigm for training data generation is introduced. By using generative adversarial networks, highly realistic and accurately labeled multimodal EEG data that approximates infinity can be created, significantly solving the bottleneck problems of data scarcity and inaccurate labeling faced by deep learning models in the field of brain source tracing. Experiments show that the source tracing model trained using the generated multimodal neurophysiological dataset (i.e., simulated data) significantly outperforms models trained on traditional simulated data when generalizing to real unknown data. Compared to models focusing on a single modality, improvements are seen under single-source and multi-source conditions, and at different signal-to-noise ratios. For example, at a single-source 10dB level, the precision increased from 90.90% to 92.92%, the recall increased from 91.30% to 92.60%, and the average localization error decreased from 0.983mm to 0.726mm.

[0162] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal deep learning-based source tracing method integrating magnetoencephalography (MEG) and electroencephalography (EEG), characterized in that, The real magnetoencephalogram (MEG) and electroencephalogram (EEG) signals are preprocessed and then input into the source tracing model to predict the probability of the source appearing in the corresponding region. Combined with the imported source partition distance matrix, the final source tracing result is obtained. The training process of the source tracing model is as follows: A generative adversarial network was constructed, and the generated synthetic observation dataset was preprocessed to obtain a multimodal neurophysiological dataset including magnetoencephalogram (MEG) signals and electroencephalogram (EEG) signals. The multimodal neuroelectrophysiological dataset was input into a dual-branch residual network to perform stage-by-stage extraction and decoupling of magnetoencephalography (MEG) signals and electroencephalography (EEG) signals, respectively. Multi-scale convolution modules are used to extract features at different stages. The extracted features are then fused and input into the classifier. Specifically, the features from the two branches are respectively... The features output from each stage are input in parallel into a multi-scale convolutional module to generate the first stage. The fusion features corresponding to each stage are aligned and then summed to obtain the global context fusion feature. The features output from the last stage are aggregated with the global context fusion features, and the resulting cross-modal weighted fusion features are concatenated with the global context fusion features to obtain cross-modal deep fusion features; Feed cross-modal deep fusion features into the classifier; Define the loss function and update the trainable parameters of the source tracing model.

2. The tracing method according to claim 1, characterized in that, The construction process of the generative adversarial network is as follows: The generator takes a random seed vector as input. and source region label vector The generator output is the synthesized source space current density timing signal; The source space current density time-series signal is multiplied by the pre-calculated magnetoencephalogram (MEG) guidance field matrix and electroencephalogram (EEG) guidance field matrix, and then Gaussian white noise is superimposed to synthesize noisy simulated data. The simulated data includes simulated data of magnetoencephalogram (MEG) and simulated data of electroencephalogram (EEG). The simulated data and real data are input into the discriminator to determine whether the input data comes from a real distribution or a generated distribution; By alternately optimizing the generator and discriminator, the generator produces a label vector that matches the source region. The source activity signal is consistent with and has realistic noise characteristics, from which the magnetoencephalogram (MEG) signal, electroencephalogram (EEG) signal, and label vector of the source region are obtained. The synthetic observation dataset.

3. The tracing method according to claim 1, characterized in that, The process involves inputting the multimodal neuroelectrophysiological dataset into a dual-branch residual network to perform stage-by-stage extraction and decoupling of magnetoencephalography (MEG) and electroencephalography (EEG) signals, specifically: The residual network has two identical branch structures, each branch includes M stages, and each stage consists of W residual blocks connected in series. One branch of the residual network is used for stage-by-stage extraction and decoupling of the magnetoencephalogram (MEG) signal to obtain the first... Electroencephalogram (MEG) characteristics output at each stage Another branch is used for stage-level extraction and decoupling of EEG signals to obtain the first... EEG characteristics of each stage , , This represents the total number of stages in one of the branches; The residual network of the first Output characteristics of each stage .

4. The tracing method according to claim 3, characterized in that, The two branches are respectively The features output from each stage are input in parallel into a multi-scale convolutional module to generate the first stage. The fusion features corresponding to each stage are as follows: Will The feature map matrix is ​​obtained by fusing along the sensor channel dimension. , feature map matrix The input is fed into a multi-scale convolution module, which includes M convolution branches. Each convolution branch is sequentially connected to a batch normalization layer and an activation layer. Different parameters are set based on different convolution kernels to keep the feature shape and size of each convolution branch consistent. Each convolution branch outputs local branch features. After concatenating the local branch features output from all convolutional branches along the channel dimension, the channel dimension is changed through the last convolutional branch to obtain the first... Fusion characteristics corresponding to each stage .

5. The tracing method according to claim 4, characterized in that, The process of aligning and summing the fusion features generated in all stages to obtain the global context fusion feature is as follows: Output features of the last stage of the residual network The feature map matrix is ​​obtained by fusing along the sensor channel dimension. ; feature map matrix The global context weights are obtained by accumulating the pixel values ​​and then applying them to an activation function. ; Before The fusion features corresponding to each stage are compressed using convolution operations to reduce the channel dimension, and then output after pooling layers. Each stage corresponds to a feature alignment matrix, and the dimension of each feature alignment matrix is ​​consistent with the fusion feature corresponding to the last stage. Before The feature alignment matrix corresponding to each stage, the feature map matrix By adding each element sequentially, the global context fusion feature is obtained. .

6. The tracing method according to claim 5, characterized in that, global context weight The calculation process is as follows: ； in, Represents the feature map matrix In the sensor dimension Location and time dimensions The value at the location, Feature map matrix The corresponding row number and column number, This is the activation function.

7. The tracing method according to claim 5, characterized in that, The features output from the final stage are aggregated with the global context fusion features. The resulting cross-modal weighted fusion features are then concatenated with the global context fusion features to obtain cross-modal deep fusion features, specifically: feature map matrix Features fused with global context The features are fed into the feature aggregation module for dynamic weighting to obtain cross-modal weighted fusion features. ; The cross-modal weighted fusion features are fed into the channel weight attention module and the spatial weight attention module, utilizing the channel attention weights. Spatial attention weights Separately with cross-modal fusion features Perform the Kronecker product to obtain cross-modal deep fusion features. .

8. The tracing method according to claim 1, characterized in that, The loss function Specifically: ； in, The number of source regions corresponds to the number of classification categories. For source region index, Describes the first in a sample The probability value of the existence of each source region is the predicted value output by the source tracing model. For a sample, the first The label vector of each source region is the real label.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-8.

Images