A multi-source brain inter-synchronization information fusion method and system considering space and spectrum

Abstract

The application provides a multi-source inter-brain synchronization information fusion method and system considering space and spectrum, comprising: constructing an inter-brain synchronization information fusion model for realizing multi-person collaborative behavior classification, in the aspect of electroencephalogram feature representation, considering information in different frequency domains and correlation in space; in the aspect of data fusion, considering features among various channels and topological structures through graph convolution and graph pooling, grasping spatial dependence in electroencephalogram data, and enhancing the ability of the model in frequency domain transformation in extracting features through BiGRU, and further improving the expression ability and distinguishability of overall electroencephalogram features under small group cooperation through a TFN module. The application can extract electroencephalogram features with higher expression ability and robustness by stage-by-stage feature fusion, paying attention to electroencephalogram information at different levels and scales.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to a method and system for multi-source brain synchronous information fusion that takes into account spatial and spectral factors. Background Technology

[0002] With the emergence and continuous development of new technologies, education is increasingly intersecting and merging with fields such as artificial intelligence and neuroscience. This multidisciplinary integration is becoming a source of knowledge innovation for the modernization of education. Computer-Supported Collaborative Learning (CSCL) is a learning method that organizes learners into groups or teams. Compared to individual learning, collaborative learning is more effective in promoting learners' higher-order thinking skills, knowledge acquisition, and the formation of positive learning attitudes. In recent years, the field of CSCL has focused on understanding the complexity of collaborative learning from a fine-grained and multi-dimensional perspective. Its main analytical dimensions include social, cognitive, metacognitive, behavioral, and temporal aspects. Furthermore, the development of brain imaging technology has provided a foundation for further exploring the neurophysiological mechanisms behind collaborative learning. Traditional collaborative learning analysis relies primarily on manual observation, which suffers from high subjectivity and low efficiency. Therefore, intelligent methods are needed to improve the performance of cognitive analysis in multi-person collaboration. By integrating EEG data collected in collaborative contexts, researchers can more objectively interpret the implicit cognitive processing of learners in collaboration, achieving more accurate learning analysis and promoting personalized and precise adaptive learning and instructional feedback during the learning process.

[0003] Understandably, EEG signals exhibit spatial, temporal, and spectral patterns associated with specific cognitive processes. For example, EEG signals are typically divided into different frequency bands, including δ (0.5-4Hz), θ (4-8Hz), α (8-13Hz), β (13-30Hz), and γ (30Hz and above), each associated with different cognitive processes. Phase Locking Value (PLV) represents the absolute value of the average phase difference between any two signals, reflecting the interaction between different brain regions. In cognitive processes, phase synchronization between different brain regions plays a crucial role in information transmission and collaborative work. PLV can provide information about the degree of interaction between different brain regions, thus revealing brain network activity during cognitive processes. In recent studies, to obtain better representations of EEG features, researchers have begun to fuse temporal, spatial, and spectral features of multi-domain EEGs to obtain more complementary and discriminative emotional features. However, EEG is a complex nonlinear signal, and the non-stationary nature of EEG signals can easily lead to distributional biases among different subjects. Furthermore, EEG signals typically contain more noise than images.

[0004] Therefore, extracting more distinctive and reliable features from EEG signals and building a robust and universal deep learning brain cognition recognition model remains a challenge. Summary of the Invention

[0005] This invention provides a method and system for multi-source brain synchronous information fusion that takes into account spatial and spectral factors, in order to overcome the deficiencies in the prior art.

[0006] In a first aspect, the present invention provides a method for fusing multi-source interbrain synchronous information that takes into account both spatial and spectral factors, comprising:

[0007] The subject's conversational text data and EEG signal data were collected, and the conversational text data and EEG signal data were preprocessed to obtain preprocessed conversational text data and preprocessed EEG signal data.

[0008] Based on the preprocessed conversation text data, the preprocessed EEG signal data is segmented to obtain multiple EEG signal data segments;

[0009] Based on the segmentation of the multiple EEG signal data, EEG feature data is calculated, and EEG data acquisition node information is obtained when multiple subjects collaborate.

[0010] The EEG feature data and the EEG data acquisition node information are input into the multi-source brain synchronization information fusion initial model that takes into account space and spectrum for training. A loss function is constructed to optimize the model and obtain the multi-source brain synchronization information fusion model.

[0011] The EEG feature data of the test subject and the corresponding EEG data acquisition node information are input into the multi-source brain synchronization information fusion model, and the brain synchronization information fusion result of the test subject is output.

[0012] According to the present invention, a multi-source interbrain synchronization information fusion method considering spatial and spectral aspects is provided, which involves collecting conversational text data and electroencephalogram (EEG) signal data from subjects, preprocessing the conversational text data and the EEG signal data to obtain preprocessed conversational text data and preprocessed EEG signal data, including:

[0013] The conversation text data is recorded according to a preset recording format. The recorded conversation text data is divided into conversation text time periods, and the divided conversation text data is encoded to obtain the preprocessed conversation text data.

[0014] The EEG signal data is sequentially processed using the EEGLAB toolbox, which performs channel localization, filtering, downsampling, segmentation, and baseline correction. Bad segments and artifact time periods are recorded. The EEGLAB toolbox is then used to remove the bad segments and artifact time periods to obtain the preprocessed EEG signal data.

[0015] According to the present invention, a multi-source brain synchronization information fusion method considering spatial and spectral aspects is provided, which segments the preprocessed EEG signal data based on the preprocessed conversational text data to obtain multiple EEG signal data segments, including:

[0016] Using the conversation text time period, the bad segment, and the artifact time period, a new conversation text time period is recalculated;

[0017] The preprocessed EEG signal data was reconnected using the EEGLAB toolbox.

[0018] According to the new conversation text time period, the preprocessed EEG signal data after connection is segmented to obtain the multiple EEG signal data segments.

[0019] According to the present invention, a multi-source brain synchronization information fusion method considering spatial and spectral aspects is provided, which calculates brainwave feature data based on the segmentation of multiple brainwave signal data, and obtains brainwave data acquisition node information when multiple subjects collaborate, including:

[0020] Determine the EEG frequency band parameters and calculate the phase lock value (PLV) between every two EEG signals in the multiple EEG signal data segments;

[0021] Based on the EEG frequency band parameters, multiple PLV values ​​are organized and classified into multiple PLV value classification sets;

[0022] Use the EEGLAB toolbox to obtain the EEG electrode names and electrode layout files from the EEG dataset;

[0023] The electrode spatial arrangement matrix is ​​determined from the electrode arrangement file, and the coordinates of the EEG electrodes in the EEG dataset corresponding to the electrode spatial arrangement matrix are obtained to construct the EEG electrode node coordinates.

[0024] According to the present invention, a multi-source inter-brain synchronization information fusion method considering spatial and spectral aspects is provided. The method involves inputting the EEG feature data and the EEG data acquisition node information into an initial multi-source inter-brain synchronization information fusion model considering spatial and spectral aspects for training, constructing a loss function to optimize the model, and obtaining a multi-source inter-brain synchronization information fusion model, including:

[0025] An adjacency matrix is ​​obtained based on the coordinates of the EEG electrode nodes, and a node feature matrix is ​​obtained based on the EEG feature data.

[0026] The adjacency matrix and the node feature matrix are input into the channel fusion network to obtain the channel-fused features;

[0027] The channel-fused features are input into the frequency band fusion network to obtain the frequency band fused features.

[0028] The fused frequency band features are input into a group fusion network to obtain comprehensive EEG fusion features;

[0029] The integrated EEG fusion features are input into the fully connected layer and optimized using the cross-entropy loss function to obtain the multi-source brain synchronous information fusion model.

[0030] According to the present invention, a multi-source interbrain synchronization information fusion method considering spatial and spectral aspects is provided, which obtains an adjacency matrix based on the coordinates of EEG electrode nodes and a node feature matrix based on the EEG feature data, comprising:

[0031] The node feature matrix is ​​constructed using the EEG feature data of any two paired subjects as matrix elements;

[0032] Obtain the distance between any two EEG electrode nodes, and construct an undirected graph structure matrix based on the distance between the two EEG electrode nodes;

[0033] Determine the pruning distance hyperparameter, and use the pruning distance hyperparameter to filter adjacent electrodes in the undirected graph structure matrix to obtain the adjacency matrix.

[0034] According to the present invention, a multi-source interbrain synchronization information fusion method considering spatial and spectral aspects is provided, wherein the adjacency matrix and the node feature matrix are input into a channel fusion network to obtain channel-fused features, including:

[0035] The channel fusion network is defined to include graph convolutional blocks and graph pooling blocks, wherein each graph convolutional block consists of 5 parallel graph convolutional layers (GCN) and a linear rectified function (ReLU), and each graph pooling block consists of 5 parallel self-attention pooling (SAGPool) layers and an output readout layer.

[0036] The adjacency matrix and the node feature matrix are input into the graph convolution block to obtain intermediate output features;

[0037] The intermediate output features are input into the graph pooling block, and the channel-fused features are output.

[0038] According to the present invention, a multi-source interbrain synchronization information fusion method considering both spatial and spectral aspects is provided, wherein the channel-fused features are input into a frequency band fusion network to obtain frequency band fused features, including:

[0039] The frequency band fusion network is determined to include a bidirectional gated cyclic unit (BiGRU).

[0040] The features of the channels corresponding to multiple EEG frequency bands are spliced ​​together to form an input feature sequence, and the input feature sequence is input into the frequency band fusion network to obtain the frequency band fused features.

[0041] According to the present invention, a multi-source inter-brain synchronization information fusion method considering both spatial and spectral aspects is provided, wherein the fused frequency band features are input into a group fusion network to obtain comprehensive EEG fusion features, including:

[0042] The group fusion network is determined to include a Tensor Fusion Network (TFN) module;

[0043] The fused frequency band features are input into the TFN to obtain the integrated EEG fusion features.

[0044] According to the present invention, a multi-source inter-brain synchronous information fusion method considering spatial and spectral aspects is provided. The integrated EEG fusion features are input into a fully connected layer and optimized using a cross-entropy loss function to obtain the multi-source inter-brain synchronous information fusion model, comprising:

[0045] The feature dimensions of the integrated EEG fusion features are converted into classification category dimensions to obtain multidimensional input data, which is then input into the fully connected layer. The multidimensional input data is then converted into multidimensional output data using the connecting neurons of the fully connected layer.

[0046] The error between the multidimensional output data and the label value is calculated using a multivariate cross-entropy function, and the multi-source brain synchronization information fusion model is output.

[0047] Secondly, the present invention also provides a multi-source brain-to-brain synchronous information fusion system that takes into account both spatial and spectral aspects, comprising:

[0048] The acquisition and preprocessing module is used to acquire conversational text data and EEG signal data from the subjects, and to preprocess the conversational text data and the EEG signal data to obtain preprocessed conversational text data and preprocessed EEG signal data.

[0049] The segmentation module is used to segment the preprocessed EEG signal data based on the preprocessed conversation text data to obtain multiple EEG signal data segments;

[0050] The calculation module is used to calculate the EEG feature data based on the segmentation of the multiple EEG signal data, and to obtain the EEG data acquisition node information when multiple subjects collaborate.

[0051] The training module is used to input the EEG feature data and the EEG data acquisition node information into the multi-source brain synchronization information fusion initial model that takes into account space and spectrum for training, construct a loss function to optimize the model, and obtain the multi-source brain synchronization information fusion model.

[0052] The fusion module is used to input the EEG feature data of the test subject and the corresponding EEG data acquisition node information into the multi-source brain synchronization information fusion model, and output the subject's brain synchronization information fusion result.

[0053] This invention provides a multi-source inter-brain synchronous information fusion method and system that considers both spatial and spectral dimensions. By constructing a multi-source inter-brain synchronous information fusion model that considers both spatial and spectral dimensions, this model, through staged feature fusion, focuses on EEG information at different levels and scales, enabling the extraction of more expressive and robust EEG features. In terms of EEG representation, it considers both information from different frequency domains and spatial correlations. Regarding data fusion, graph convolution and graph pooling are used to consider the features and topological structure between each node (i.e., each channel), capturing the spatial dependencies in the EEG data. Furthermore, BiGRU enhances the model's ability to extract features in frequency domain transformations, and TFN further improves the expressive power and discriminative power of overall EEG features in collaborative group settings. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is one of the flowcharts of the multi-source brain synchronous information fusion method that takes into account space and spectrum provided by the present invention;

[0056] Figure 2 This is the second flowchart of the multi-source brain synchronous information fusion method that takes into account space and spectrum provided by the present invention;

[0057] Figure 3 This is a structural diagram of the multi-source brain synchronous information fusion model provided by the present invention;

[0058] Figure 4 This is an example diagram of the channel fusion network provided by the present invention;

[0059] Figure 5 This is an example diagram of the frequency band fusion network provided by the present invention;

[0060] Figure 6 This is an example diagram of the group fusion network provided by the present invention;

[0061] Figure 7 This is a schematic diagram of the structure of the multi-source brain-to-brain synchronous information fusion system that takes into account both spatial and spectral aspects, provided by the present invention.

[0062] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0064] To address the shortcomings of existing technologies in EEG information fusion processing, this invention utilizes the spatial and spectral features of EEG to construct a multi-source inter-brain synchronous information fusion model.

[0065] Figure 1 This is one of the flowcharts illustrating the multi-source interbrain synchronous information fusion method considering spatial and spectral aspects provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0066] Step 100: Collect the subject's conversational text data and EEG signal data, and preprocess the conversational text data and EEG signal data to obtain preprocessed conversational text data and preprocessed EEG signal data;

[0067] Step 200: Segment the preprocessed EEG signal data based on the preprocessed conversation text data to obtain multiple EEG signal data segments;

[0068] Step 300: Based on the segmentation of the multiple EEG signal data, calculate the EEG feature data and obtain the EEG data acquisition node information when multiple subjects collaborate;

[0069] Step 400: Input the EEG feature data and the EEG data acquisition node information into the multi-source brain synchronization information fusion initial model that takes into account space and spectrum for training, construct a loss function to optimize the model, and obtain the multi-source brain synchronization information fusion model;

[0070] Step 500: Input the EEG feature data of the subject to be tested and the corresponding EEG data acquisition node information into the multi-source brain synchronization information fusion model, and output the subject's brain synchronization information fusion result.

[0071] Specifically, such as Figure 2As shown, this embodiment of the invention first records the time period of the conversation text and the corresponding tags, acquires and preprocesses the EEG signals of each subject, and then segments the preprocessed EEG signals of each subject according to the time period of the conversation text. Based on the obtained segmented data, the PLV of each frequency band paired up for each subject is calculated as EEG feature data. The EEG electrodes used in the EEG data acquisition during multi-person collaboration are recorded to construct node coordinates. The aforementioned node coordinates and EEG features are input into the constructed initial model of multi-source inter-brain synchronization information fusion that takes into account space and spectrum, and a loss function is constructed to optimize the classification results, resulting in a trained multi-source inter-brain synchronization information fusion model. Finally, the EEG feature data of the test subjects and the corresponding EEG data acquisition node information are input into the multi-source inter-brain synchronization information fusion model to output the final inter-brain synchronization information fusion result of the subjects.

[0072] This invention constructs a multi-source brain synchronous information fusion model that considers both spatial and spectral aspects. This model, through staged feature fusion, focuses on EEG information at different levels and scales, enabling the extraction of more expressive and robust EEG features. In terms of EEG representation, it considers both information from different frequency domains and spatial correlations. Regarding data fusion, graph convolution and graph pooling are used to consider the features and topological structure between nodes (channels), capturing the spatial dependencies in EEG data. Furthermore, BiGRU enhances the model's ability to extract features in frequency domain transformations, and TFN further improves the expressive power and discriminative power of overall EEG features in collaborative group settings.

[0073] Based on the above embodiments, conversational text data and electroencephalogram (EEG) signal data of subjects are collected, and the conversational text data and EEG signal data are preprocessed to obtain preprocessed conversational text data and preprocessed EEG signal data, including:

[0074] The conversation text data is recorded according to a preset recording format. The recorded conversation text data is divided into conversation text time periods, and the divided conversation text data is encoded to obtain the preprocessed conversation text data.

[0075] The EEG signal data is sequentially processed using the EEGLAB toolbox, which performs channel localization, filtering, downsampling, segmentation, and baseline correction. Bad segments and artifact time periods are recorded. The EEGLAB toolbox is then used to remove the bad segments and artifact time periods to obtain the preprocessed EEG signal data.

[0076] Specifically, the preprocessing method for the conversational text data and EEG signal data output by the subjects in this embodiment of the invention is as follows:

[0077] Acquire conversation text data, recording formats including date, time, name, text content, and seconds, initially dividing the conversation text into time periods and performing encoding operations.

[0078] For EEG signal data, the MATLAB EEGLAB toolbox was used to perform operations such as channel localization, filtering, downsampling, segmentation, and baseline correction, recording the bad segments and artifact time periods of the EEG signals. For example, in a three-person collaboration scenario, EEG signals were acquired from subjects A, B, and C. Bad segments and artifact time periods were statistically analyzed, and the EEGLAB toolbox was used to remove bad segments and artifacts from the subjects' EEG signals.

[0079] Based on the above embodiments, the preprocessed EEG signal data is segmented based on the preprocessed conversation text data to obtain multiple EEG signal data segments, including:

[0080] Using the conversation text time period, the bad segment, and the artifact time period, a new conversation text time period is recalculated;

[0081] The preprocessed EEG signal data was reconnected using the EEGLAB toolbox.

[0082] According to the new conversation text time period, the preprocessed EEG signal data after connection is segmented to obtain the multiple EEG signal data segments.

[0083] Specifically, in this embodiment of the invention, the conversation text time periods are recalculated according to the initially divided conversation text time periods and the bad segments and artifact time periods of the recorded EEG signals; the segmented EEG signals are reconnected using the EEGLAB toolbox; and the connected EEG signals are segmented according to the conversation text time periods.

[0084] Based on the above embodiments, according to the segmentation of the multiple EEG signal data, EEG feature data is calculated, and EEG data acquisition node information is obtained when multiple subjects collaborate, including:

[0085] Determine the EEG frequency band parameters and calculate the phase lock value (PLV) between every two EEG signals in the multiple EEG signal data segments;

[0086] Based on the EEG frequency band parameters, multiple PLV values ​​are organized and classified into multiple PLV value classification sets;

[0087] Use the EEGLAB toolbox to obtain the EEG electrode names and electrode layout files from the EEG dataset;

[0088] The electrode spatial arrangement matrix is ​​determined from the electrode arrangement file, and the coordinates of the EEG electrodes in the EEG dataset corresponding to the electrode spatial arrangement matrix are obtained to construct the EEG electrode node coordinates.

[0089] Specifically, this embodiment of the invention still uses MATLAB to calculate the PLV between each pair of subjects in the test group based on the obtained segmented EEG signals, setting frequency band parameters. The obtained PLV data is then sorted according to frequency bands and organized into corresponding folders, forming multiple PLV value classification sets.

[0090] Similarly, using the EEGLAB toolbox, the names of the EEG electrodes used in the EEG dataset and the electrode layout files used are recorded, such as the 10-20 standard electrode layout system. A spatial layout matrix K for the electrodes is defined based on the electrode layout file, and the coordinates of the EEG electrodes used in the EEG dataset within the spatial layout matrix K are recorded as the node coordinates of the graph.

[0091] Based on the above embodiments, the EEG feature data and the EEG data acquisition node information are input into a multi-source inter-brain synchronization information fusion initial model that takes into account both spatial and spectral aspects for training. A loss function is constructed to optimize the model, resulting in a multi-source inter-brain synchronization information fusion model, including:

[0092] An adjacency matrix is ​​obtained based on the coordinates of the EEG electrode nodes, and a node feature matrix is ​​obtained based on the EEG feature data.

[0093] The adjacency matrix and the node feature matrix are input into the channel fusion network to obtain the channel-fused features;

[0094] The channel-fused features are input into the frequency band fusion network to obtain the frequency band fused features.

[0095] The fused frequency band features are input into a group fusion network to obtain comprehensive EEG fusion features;

[0096] The integrated EEG fusion features are input into the fully connected layer and optimized using the cross-entropy loss function to obtain the multi-source brain synchronous information fusion model.

[0097] Specifically, in this embodiment of the invention, the node coordinates and EEG features obtained in the foregoing embodiments are input into a multi-source brain synchronization information fusion model that takes into account both spatial and spectral aspects. A loss function is then constructed to optimize the classification results. The model structure is as follows: Figure 3 As shown, three subjects, A, B, and C, are used as an example.

[0098] First, a graph signal is constructed, including the construction of the graph structure and the extraction of the graph signal. The graph structure uses the spatial arrangement of electrodes to construct a brain network, that is, the extracted node coordinates are used as input and processed to obtain the adjacency matrix A; the graph signal is obtained by using the input EEG features as the node feature matrix.

[0099] Then, the EEG feature dataset D brainIn the input model, the combined features corresponding to each pair of pairs are X. u (u∈U), where U represents the pairing of subjects, each input to the corresponding channel fusion module to obtain the channel-fused features. Next, the EEG features from the five channel-fused frequency bands are concatenated and used as the input sequence into the frequency band fusion network. Then, the features output from the frequency band fusion of different pairing combinations are input into the group fusion network to obtain the comprehensive EEG fusion features.

[0100] Finally, the EEG fusion features H after group fusion were analyzed. brain Input the fully connected layer to predict the classification result. The loss function used is cross-entropy.

[0101] This invention extracts features and selects appropriate representation methods based on the data characteristics of different frequency bands and different channels, taking into account both information from different frequency domains and spatial correlation.

[0102] Based on the above embodiments, an adjacency matrix is ​​obtained based on the coordinates of the EEG electrode nodes, and a node feature matrix is ​​obtained based on the EEG feature data, including:

[0103] The node feature matrix is ​​constructed using the EEG feature data of any two paired subjects as matrix elements;

[0104] Obtain the distance between any two EEG electrode nodes, and construct an undirected graph structure matrix based on the distance between the two EEG electrode nodes;

[0105] Determine the pruning distance hyperparameter, and use the pruning distance hyperparameter to filter adjacent electrodes in the undirected graph structure matrix to obtain the adjacency matrix.

[0106] Specifically, the graph signal construction includes graph structure construction and graph signal extraction, with each channel acting as a node. The graph structure uses the spatial arrangement of electrodes to construct a brain network, that is, the extracted node coordinates are used as input and processed to obtain the adjacency matrix A; the graph signal uses the input EEG features as the node feature matrix.

[0107] The obtained PLV matrix of phase-locked values ​​between each pair of subjects is used as the EEG feature dataset D. brain ={X u,b |X u,b ∈R N×N In the input model, {u∈U, b∈B}, where U={AB,AC,BC} represents the combination of subjects paired up when taking three people as an example, B={δ,θ,α,β,γ} represents the frequency band, and N represents the number of channels. The EEG characteristics of the paired subjects A and B are X. AB ={X AB,α ,X AB,β ,XAB,δ ,X AB,θ ,X AB,γ}, X AB,b ∈R N×N (b∈B), while X AC and X BC Similarly.

[0108] First, the distance between each electrode is calculated according to formula (1) and the input node coordinates, resulting in an undirected graph structure with added self-connections—matrix M∈R. N×N (N represents the number of nodes, i.e., the number of channels). Secondly, given that the influence of brain activity decreases with the increase of the relative distance between electrodes, a hyperparameter τ is introduced as the pruning distance to control the distance of signal propagation between electrodes. The matrix M is filtered according to formula (2) to obtain the adjacency matrix A∈R. N×N And convert the adjacency matrix A into a sparse matrix format.

[0109]

[0110] Where d(j,k) represents the shortest distance between electrodes j and k.

[0111] The adjacency matrix A is obtained by filtering the adjacent electrodes in matrix M using τ, as shown in formula (2):

[0112]

[0113] Among them, A (j,k) τ represents the influence of the j-th channel on the k-th channel, and τ represents the pruning distance.

[0114] Based on the above embodiments, the adjacency matrix and the node feature matrix are input into the channel fusion network to obtain the channel-fused features, including:

[0115] The channel fusion network is defined to include graph convolutional blocks and graph pooling blocks, wherein each graph convolutional block consists of 5 parallel graph convolutional layers (GCN) and a linear rectified function (ReLU), and each graph pooling block consists of 5 parallel self-attention pooling (SAGPool) layers and an output readout layer.

[0116] The adjacency matrix and the node feature matrix are input into the graph convolution block to obtain intermediate output features;

[0117] The intermediate output features are input into the graph pooling block, and the channel-fused features are output.

[0118] Specifically, the EEG feature dataset D brain In the input model, X AB X AC X BCEach input Figure 3 The channel fusion network shown yields the fused channel features. Taking the combination of subject A and subject B as an example, the EEG features X of subject A and subject B in 5 frequency bands are obtained. AB And the adjacency matrix A is input to Figure 3 In the channel fusion network shown, H is obtained. AB,b ∈R F (b∈B), where F represents the dimension of the output node features. Similarly, we obtain H. AC,b ∈R F (b∈B) and H BC,b ∈R F (b∈B).

[0119] Taking a three-person collaboration as an example, the channel fusion module contains three channel fusion networks. Each channel fusion network consists of graph convolutional blocks and graph pooling blocks. Each graph convolutional block comprises five parallel graph convolutional network (GCN) layers and a linear rectification function (ReLU). Each graph pooling block comprises five parallel self-attention pooling (SAGPool) layers and a readout layer. These modules process the input, fusing features between channels; the five channels correspond to five EEG signal frequency bands.

[0120] The first step is to X-ray the brainwave characteristics. AB The convolutional block of the input graph with the adjacency matrix A yields the intermediate output O. AB X AB EEG characteristics of 5 frequency bands AB,α X AB,β X AB,δ X AB,θ X AB,γ The node features of the five graphs are input into the corresponding graph convolutional layer (GCN), and then passed through the ReLU activation function to obtain the output O. AB,α O AB,β O AB,δ O AB,θ O AB,γ The GCN layer receives the adjacency matrix A and node feature matrix X from the EEG and performs graph convolution to extract spatial features. Each node aggregates the features of its neighboring nodes and updates its own node features. Then, the ReLU activation function is used for non-linear transformation to alleviate overfitting. The specific calculation of the graph convolution layer using ReLU activation is as follows:

[0121]

[0122] Where Z∈RN×C Represents the output feature matrix, X∈R N×F This represents the input feature matrix. C represents the input feature dimension, F represents the output feature dimension, and N represents the number of nodes. It is the adjacency matrix of an undirected graph with added self-connections, I N It is the identity matrix, and A represents the input adjacency matrix. yes The degree matrix. Θ∈R C×F Let be the learnable parameter matrix. σ(·) represents the activation function, i.e., ReLU(·) = max(0,·).

[0123] The second step is to output the intermediate O. AB Input graph pooling block, obtain output H AB,b ∈R F (b∈B), O AB,b (b∈B) Input Figure 4 H is obtained in the pooling layer shown. AB,b (b∈B). The pooling layer consists of five parallel SAGPool layers and a readout layer. The SAGPool layer (Self-Attention Pooling) is an implementation of hierarchical pooling. Then, the readout layer performs a one-time aggregation operation on the clustered nodes to output the representation of the entire graph.

[0124] The SAGPool layer adaptively learns the importance of nodes from the graph through graph convolution, and then uses the TopK mechanism to discard nodes. The SAGPool layer calculation formula is as follows:

[0125]

[0126] idx = top-rank(Z, [kN]) mask =Z idx

[0127] X′=X idx,: X out =X′⊙Z mask A out =A idx.idx

[0128] First, graph convolution is used to obtain the self-attention score Z. Where Θ att ∈R N×1 , representing the weight parameter, is also the only parameter in the SAGPool layer. The rest are similar to the GCN layer, where X∈R N×F The input features are those of a graph with N nodes and F-dimensional features, where σ(·) represents an activation function. It is an adjacency matrix with self-connection. yes The degree matrix.

[0129] Secondly, based on the node importance score Z, the top [kN] nodes are retained. Here, `top-rank` is a function that returns the indices of the top [kN] nodes, which are selected based on the value of Z. idx It is an index operation, Z mask This is a feature attention mask. The pooling rate k∈(0,1] is a hyperparameter that determines the number of nodes to retain.

[0130] Finally, the pooled node feature matrix and adjacency matrix are obtained based on idx. Where X... idx,: It is a feature matrix with indexes arranged row by row (each row represents the feature vector of a node), A idx.idx It is an adjacency matrix indexed by row and column. X out and A out These are the updated feature matrix and the corresponding adjacency matrix, respectively. ⊙ represents the Hadamard product.

[0131] The readout layer is primarily used to aggregate node features for a fixed-size representation. The readout mechanism is similar to the global pooling operation commonly used in CNN models. The pooled node feature matrices are then aggregated into a vector. The specific formula is as follows:

[0132]

[0133] Where s represents the output. i Let N represent the feature of the i-th node. N represents the number of nodes. || represents the feature concatenation operation.

[0134] This invention employs self-attention pooling, which adaptively learns the importance weights between different channels, thereby better capturing important spatial features in EEG signals. The self-attention mechanism also reduces information redundancy and noise, improving the anti-interference and robustness of EEG features.

[0135] Based on the above embodiments, the channel-fused features are input into the frequency band fusion network to obtain the frequency band fused features, including:

[0136] The frequency band fusion network is determined to include a bidirectional gated cyclic unit (BiGRU).

[0137] The features of the channels corresponding to multiple EEG frequency bands are spliced ​​together to form an input feature sequence, and the input feature sequence is input into the frequency band fusion network to obtain the frequency band fused features.

[0138] Specifically, the EEG features of the five frequency bands after channel fusion are spliced ​​together and used as the input sequence into the frequency band fusion network. Taking subject A and subject B as a pair, the EEG features of subject A and subject B in the five frequency bands are spliced ​​together as H'. AB =(H AB,α H AB,β H AB,δ H AB,θ H AB,γ ), input into the frequency band fusion network, to obtain H″ AB Similarly, we obtain H″ AC and H″ BC .

[0139] The frequency band fusion network consists of bidirectional BiGRUs, a type of GRU-based network that can simultaneously process the input sequence both forward and backward. For example... Figure 5 As shown, assuming the input of BiGRU is a sequence X of length n and an initial hidden state h0, the output of BiGRU is a sequence H of length n and h0. n h n The output of the entire frequency band fusion network is the concatenation of the forward and reverse hidden layer outputs at the last moment. The specific calculation is as follows:

[0140]

[0141]

[0142]

[0143] First, calculate the forward and reverse outputs at time t. and These represent the hidden states from left to right and from right to left, respectively. GRU represents the GRU cell, and x... t This represents the t-th element of the input sequence.

[0144] Then, the hidden layer outputs from both directions are concatenated to obtain the final output h. t , where [·; ·] denotes the vector concatenation operation.

[0145] Based on the above embodiments, the frequency band fusion features are input into the group fusion network to obtain comprehensive EEG fusion features, including:

[0146] The group fusion network is determined to include a Tensor Fusion Network (TFN) module;

[0147] The fused frequency band features are input into the TFN to obtain the integrated EEG fusion features.

[0148] Specifically, in this embodiment of the invention, taking a three-person collaboration as an example, the characteristic H″ output from the three paired groups after frequency band fusion is... AB H″ AC H″ BC The input group fusion network yields the final EEG fusion feature H. brain .

[0149] The group fusion network employs a Tensor Fusion Network (TFN) module. TFN represents an early stage of fusion and is a typical multimodal network that fuses features through matrix operations, such as... Figure 6 As shown. The specific formula is as follows:

[0150]

[0151] The TFN module first expands the dimension of each mode by 1, and then calculates the Cartesian product of the different modes. Here, z... l z v z a F represents three eigenvectors. l F v F a This represents the dimension of the three feature vectors. Representing vector and The outer product between them represents a 3D cube with 7 semantically distinct subregions. There are three single-modal interactions, three bimodal interactions, and one trimodal interaction in tensor fusion. H″ is used in the computation. AB H″ AC H″ BC Considered as z l z v z a .

[0152] This invention employs a TFN module to fuse multi-source features, jointly modeling and fusing features from different groups. Through the TFN module, features from different groups can influence and interact with each other, thereby improving the overall feature expressiveness and discriminative power.

[0153] Based on the above embodiments, the integrated EEG fusion features are input into the fully connected layer and optimized using the cross-entropy loss function to obtain the multi-source inter-brain synchronous information fusion model, including:

[0154] The feature dimensions of the integrated EEG fusion features are converted into classification category dimensions to obtain multidimensional input data, which is then input into the fully connected layer. The multidimensional input data is then converted into multidimensional output data using the connecting neurons of the fully connected layer.

[0155] The error between the multidimensional output data and the label value is calculated using a multivariate cross-entropy function, and the multi-source brain synchronization information fusion model is output.

[0156] Specifically, the EEG fusion features H after group fusion brain Input a fully connected layer to predict the classification result.

[0157] EEG fusion features H brain ∈R K The feature dimension K is converted into the classification category dimension M to facilitate the calculation of cross-entropy. The fully connected layer, by connecting neurons, uses weight parameters and bias parameters to transform multidimensional input data into multidimensional output data. The specific calculation is as follows:

[0158] y = Wx + b

[0159] Where y is the result of the fully connected layer, W is the weight matrix, and b is the bias term.

[0160] The model loss function used here is cross-entropy, one of the most commonly used loss functions in machine learning. It measures the error between the model's predicted value and the label value using a multivariate cross-entropy function. The specific formula is as follows:

[0161]

[0162] in, Let y be the model's prediction for the i-th sample in the dataset. i Let be the label value of the i-th sample after one-hot encoding, and C be the number of categories of the sample label. During calculation, the EEG fusion feature H after group fusion is used. brain The calculation results after the fully connected layer are considered as The real label is y i .

[0163] This invention employs a three-stage feature fusion approach: first, node-level fusion; then, graph-level fusion; and finally, group-level fusion. This multi-layered fusion enables a more comprehensive capture of key information in EEG signals, thereby better reflecting the inherent patterns and structures of EEG signals and improving feature representation and discriminative power.

[0164] The following describes the multi-source inter-brain synchronous information fusion system that takes into account space and spectrum provided by the present invention. The multi-source inter-brain synchronous information fusion system that takes into account space and spectrum described below can be referred to in correspondence with the multi-source inter-brain synchronous information fusion method that takes into account space and spectrum described above.

[0165] Figure 7 This is a schematic diagram of the structure of a multi-source interbrain synchronous information fusion system that takes into account both spatial and spectral aspects, as provided in an embodiment of the present invention. Figure 7As shown, it includes: a data acquisition and preprocessing module 71, a segmentation module 72, a calculation module 73, a training module 74, and a fusion module 75, wherein:

[0166] The acquisition and preprocessing module 71 is used to acquire the subject's conversational text data and EEG signal data, and preprocess the conversational text data and the EEG signal data to obtain preprocessed conversational text data and preprocessed EEG signal data; the segmentation module 72 is used to segment the preprocessed EEG signal data based on the preprocessed conversational text data to obtain multiple EEG signal data segments; the calculation module 73 is used to calculate EEG feature data based on the multiple EEG signal data segments and obtain EEG data acquisition node information when multiple subjects collaborate; the training module 74 is used to input the EEG feature data and the EEG data acquisition node information into the multi-source inter-brain synchronization information fusion initial model that takes into account space and spectrum for training, construct a loss function to optimize the model, and obtain the multi-source inter-brain synchronization information fusion model; the fusion module 75 is used to input the EEG feature data and corresponding EEG data acquisition node information of the subject to be tested into the multi-source inter-brain synchronization information fusion model and output the subject's inter-brain synchronization information fusion result.

[0167] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a multi-source inter-brain synchronization information fusion method that considers space and spectrum. This method includes: collecting conversational text data and EEG signal data from subjects; preprocessing the conversational text data and the EEG signal data to obtain preprocessed conversational text data and preprocessed EEG signal data; segmenting the preprocessed EEG signal data based on the preprocessed conversational text data to obtain multiple EEG signal data segments; calculating EEG feature data based on the multiple EEG signal data segments to obtain EEG data acquisition node information when multiple subjects collaborate; inputting the EEG feature data and the EEG data acquisition node information into an initial multi-source inter-brain synchronization information fusion model that considers space and spectrum for training; constructing a loss function to optimize the model to obtain a multi-source inter-brain synchronization information fusion model; inputting the EEG feature data and corresponding EEG data acquisition node information of the subjects to be tested into the multi-source inter-brain synchronization information fusion model, and outputting the subject's inter-brain synchronization information fusion result.

[0168] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusing multi-source interbrain synchronous information considering spatial and spectral aspects, characterized in that, include: The subject's conversational text data and EEG signal data were collected, and the conversational text data and EEG signal data were preprocessed to obtain preprocessed conversational text data and preprocessed EEG signal data. Based on the preprocessed conversation text data, the preprocessed EEG signal data is segmented to obtain multiple EEG signal data segments; Based on the segmentation of the multiple EEG signal data, EEG feature data is calculated, and EEG data acquisition node information is obtained when multiple subjects collaborate. The EEG feature data and the EEG data acquisition node information are input into an initial model for multi-source inter-brain synchronization information fusion that takes into account both spatial and spectral aspects for training. A loss function is constructed to optimize the model, resulting in a multi-source inter-brain synchronization information fusion model, including: An adjacency matrix is ​​obtained based on the coordinates of the EEG electrode nodes, and a node feature matrix is ​​obtained based on the EEG feature data. The adjacency matrix and the node feature matrix are input into the channel fusion network to obtain the channel-fused features; The channel-fused features are input into the frequency band fusion network to obtain the frequency band fused features. The fused frequency band features are input into a group fusion network to obtain comprehensive EEG fusion features; The integrated EEG fusion features are input into the fully connected layer and optimized using the cross-entropy loss function to obtain the multi-source brain synchronous information fusion model. The EEG feature data of the test subject and the corresponding EEG data acquisition node information are input into the multi-source brain synchronization information fusion model, and the brain synchronization information fusion result of the test subject is output.

2. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, Conversational text data and electroencephalogram (EEG) signal data of subjects were collected. The conversational text data and EEG signal data were preprocessed to obtain preprocessed conversational text data and preprocessed EEG signal data, including: The conversation text data is recorded according to a preset recording format. The recorded conversation text data is divided into conversation text time periods, and the divided conversation text data is encoded to obtain the preprocessed conversation text data. The EEG signal data is sequentially processed using the EEGLAB toolbox, which performs channel localization, filtering, downsampling, segmentation, and baseline correction. Bad segments and artifact time periods are recorded. The EEGLAB toolbox is then used to remove the bad segments and artifact time periods to obtain the preprocessed EEG signal data.

3. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 2, characterized in that, Based on the preprocessed conversation text data, the preprocessed EEG signal data is segmented to obtain multiple EEG signal data segments, including: Using the conversation text time period, the bad segment, and the artifact time period, a new conversation text time period is recalculated; The preprocessed EEG signal data was reconnected using the EEGLAB toolbox. According to the new conversation text time period, the preprocessed EEG signal data after connection is segmented to obtain the multiple EEG signal data segments.

4. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, Based on the segmentation of the multiple EEG signal data, EEG feature data is calculated, and EEG data acquisition node information is obtained when multiple subjects collaborate, including: Determine the EEG frequency band parameters and calculate the phase lock value (PLV) between every two EEG signals in the multiple EEG signal data segments; Based on the EEG frequency band parameters, multiple PLV values ​​are organized and classified into multiple PLV value classification sets; Use the EEGLAB toolbox to obtain the EEG electrode names and electrode layout files from the EEG dataset; The electrode spatial arrangement matrix is ​​determined from the electrode arrangement file, and the coordinates of the EEG electrodes in the EEG dataset corresponding to the electrode spatial arrangement matrix are obtained to construct the EEG electrode node coordinates.

5. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, An adjacency matrix is ​​obtained based on the coordinates of the EEG electrode nodes, and a node feature matrix is ​​obtained based on the EEG feature data, including: The node feature matrix is ​​constructed using the EEG feature data of any two paired subjects as matrix elements; Obtain the distance between any two EEG electrode nodes, and construct an undirected graph structure matrix based on the distance between the two EEG electrode nodes; Determine the pruning distance hyperparameter, and use the pruning distance hyperparameter to filter adjacent electrodes in the undirected graph structure matrix to obtain the adjacency matrix.

6. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, The adjacency matrix and the node feature matrix are input into the channel fusion network to obtain the channel-fused features, including: The channel fusion network is defined to include graph convolutional blocks and graph pooling blocks, wherein each graph convolutional block consists of 5 parallel graph convolutional layers (GCN) and a linear rectified function (ReLU), and each graph pooling block consists of 5 parallel self-attention pooling (SAGPool) layers and an output readout layer. The adjacency matrix and the node feature matrix are input into the graph convolution block to obtain intermediate output features; The intermediate output features are input into the graph pooling block, and the channel-fused features are output.

7. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, The channel-fused features are input into a frequency band fusion network to obtain frequency band fused features, including: The frequency band fusion network is determined to include a bidirectional gated cyclic unit (BiGRU). The features of the channels corresponding to multiple EEG frequency bands are spliced ​​together to form an input feature sequence, and the input feature sequence is input into the frequency band fusion network to obtain the frequency band fused features.

8. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, The fused frequency band features are input into a group fusion network to obtain comprehensive EEG fusion features, including: The group fusion network is determined to include a Tensor Fusion Network (TFN) module; The fused frequency band features are input into the TFN to obtain the integrated EEG fusion features.

9. The multi-source interbrain synchronous information fusion method considering spatial and spectral aspects according to claim 1, characterized in that, The integrated EEG fusion features are input into a fully connected layer and optimized using a cross-entropy loss function to obtain the multi-source inter-brain synchronous information fusion model, including: The feature dimensions of the integrated EEG fusion features are converted into classification category dimensions to obtain multidimensional input data, which is then input into the fully connected layer. The multidimensional input data is then converted into multidimensional output data using the connecting neurons of the fully connected layer. The error between the multidimensional output data and the label value is calculated using a multivariate cross-entropy function, and the multi-source brain synchronization information fusion model is output.

Images