A multi-user, multi-device brain-computer interface and semantic communication system

Abstract

This invention discloses a multi-user, multi-device brain-computer interface and semantic communication system, comprising an EEG signal acquisition module, an EEG feature extraction and encoding module, and a fringe agent decoding and task execution module. The EEG signal acquisition module acquires brain signal data. The EEG feature extraction and encoding module performs feature learning and encoding of the brain signal data into EEG commands. The EEG feature extraction and encoding module includes a transmitter module and a wireless channel module. The transmitter module includes an EEG data feature extraction unit and a channel encoding unit. The wireless channel module includes a channel transmission unit and a neural network cross-entropy model. The fringe agent decoding and task execution module reconstructs and recognizes the EEG commands and generates different execution actions based on different user commands. This enables personalized EEG learning and improves the accuracy of personalized EEG cognition and classification for users.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) data technology, and in particular to a multi-user, multi-device brain-computer interface and semantic communication system. Background Technology

[0002] The Metaverse is emerging as one of the most promising platforms for the future internet, with BCI (Brain-Computer Interface) being a key technology. Compared to invasive BCIs that are implanted directly into the brain during neurosurgery, non-invasive BCIs, primarily based on EEG (Electroencephalography) principles, are relatively inexpensive and have greater potential for market penetration. Currently, our approach focuses on non-invasive BCIs and multi-brain / multi-robot interactions, which have been largely unresolved in previous work. However, there is an increasing demand for intelligent agents and human-object interactions in metaverse scenarios such as co-creation, autonomous manufacturing, unmanned factories, immersive technologies, or extended reality (XR). Therefore, the scenarios defined in our framework are meaningful for both research and practice. Furthermore, we propose a general framework for EEG-adaptive neural networks and semantic communication for brain signal analysis and feedback on heterogeneous or multimodal data generated by various sensing devices. Its design and optimization scheme provide a reference case integrating sensing, computation, communication, and control, which can benefit industry and academia working on 6G and beyond. However, most existing technologies only consider one-to-one brain-computer interface control, which is insufficient for future multi-user-agent interaction scenarios; at the same time, existing technologies do not have a good way to overcome scenarios involving collaboration of various types of brain-computer devices. Summary of the Invention

[0003] To address the above technical problems, this invention discloses a multi-user, multi-device brain-computer interface and semantic communication system. It achieves personalized EEG learning through meta-learning, improving the accuracy of personalized EEG cognition and classification for users. The dynamic encoding and decoding process solves the problem of real-time access to EEG signals from multiple users and devices. Semantic communication solves the problem of transmitting EEG commands to multiple users in wireless signal-noise environments. Code mapping solves the problem of recognizing commands from multiple users by multiple robots.

[0004] A multi-user, multi-device brain-computer interface and semantic communication system includes an EEG signal acquisition module, an EEG feature extraction and encoding module, and an edge agent decoding and task execution module.

[0005] The EEG signal acquisition module is used to acquire brain signal data;

[0006] The EEG feature extraction and encoding module is used to learn features from the brain signal data and encode them into EEG commands. The EEG feature extraction and encoding module includes a transmitter module and a wireless channel module. The transmitter module includes an EEG data feature extraction unit and a channel encoding unit. The wireless channel module includes a channel transmission unit and a neural network cross-entropy model.

[0007] The edge intelligence decoding and task execution module is used to reconstruct and recognize the EEG commands and generate different execution actions according to different user commands.

[0008] Furthermore, the EEG feature extraction and encoding module also includes a lightweight transmitter module, which includes an EEG data feature extraction unit and a channel encoding unit after model compression processing.

[0009] Furthermore, the edge agent decoding and task execution module includes a receiver module, which includes an edge decoding unit and an execution unit.

[0010] Furthermore, the EEG data feature extraction unit includes a common feature extraction module, a low-dimensional feature extraction model, a high-dimensional feature extraction model, and a dynamic decoding unit; the low-dimensional feature extraction model is used to improve the recognition accuracy of low-dimensional EEG data and extract the features of low-dimensional EEG data, the high-dimensional feature extraction model is used to extract the features of high-feature EEG data, and the dynamic decoding unit is used to adjust the selection of the low-dimensional feature extraction model and the high-dimensional feature extraction model.

[0011] Furthermore, the common feature extraction module includes a convolutional layer, an embedding layer, a spacing mask layer, and a common feature extraction layer.

[0012] Furthermore, the wireless channel module is used to simulate interference by superimposing noise onto the incoming channel coding features. The wireless channel module uses noise simulation to simulate transmission interference in the wireless environment.

[0013] The beneficial effects of this invention are:

[0014] 1. Compared with existing technologies, this invention proposes a method to extract joint features from heterogeneous EEG data based on a discrete attention mechanism, and then uses dynamic feature integration to improve classification accuracy, thereby achieving accurate extraction of homogeneous information from mixed-type EEG data.

[0015] 2. This invention designs a dynamic semantic autoencoder, using a brain converter and a corresponding semantic autodecoder to inherit the semantic information of the transmitter; simultaneously, a semantic performance index is established through mutual information function and cross-entropy function to measure the performance of the channel encoder and decoder, and to measure the performance of the semantic encoder and decoder, accurately recognizing instructions from multi-dimensional EEG data.

[0016] 3. Regarding deployment and agent recognition instructions, this invention deploys a model compression scheme with pruning, weight sharing, and quantization to support the transmitter running the semantic encoder model on resource-constrained edge devices; a lightweight channel decoder and a semantic decoder with residual technology and a fully connected layer are also deployed at the receiver; in addition, code graphs representing various commands are provided for multiple users to control multiple intelligent agents. Attached Figure Description

[0017] Appendix Figure 1 This is a schematic diagram of a multi-user, multi-device brain-computer interface and semantic communication system according to an embodiment of the present invention;

[0018] Appendix Figure 2 This is a schematic diagram of the dynamic EEG feature extractor in an embodiment of the present invention;

[0019] Appendix Figure 3 This is a schematic diagram of the common feature extraction mechanism in an embodiment of the present invention;

[0020] Appendix Figure 4 This is a schematic diagram of the Brain Transformer module of the low-dimensional feature extraction model in an embodiment of the present invention.

[0021] Appendix Figure 5 This is a schematic diagram of the semantic communication process in an embodiment of the present invention.

[0022] Appendix Figure 6 The attached diagram illustrates a real-time brain-computer interface involving multiple users and multiple agents, as described in this invention. Figure 7 This is a schematic diagram illustrating the accuracy of various models in the brainwave command classification experiment under mixed data input from multiple types of brain-computer interfaces in this embodiment of the invention.

[0023] Appendix Figure 8 This is a schematic diagram illustrating the comparative experimental results of mixed data classification accuracy under various types of wireless channel noise interference in an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the embodiments do not constitute a limitation on the scope of protection of the present invention.

[0025] Example 1

[0026] like Figure 1 As shown, this invention discloses a multi-user, multi-device brain-computer interface and semantic communication system, including an EEG signal acquisition module, an EEG feature extraction and encoding module, and an edge agent decoding and task execution module.

[0027] The EEG signal acquisition module is used to acquire brain signal data. Multiple users can wear various EEG devices, such as the Brainlink series, Emotiv series, and 64-channel electrode caps. These EEG devices will acquire EEG signals from different brain regions and with different numbers of channels, forming raw EEG data of different dimensions, which will serve as input to the EEG feature extraction and encoding module.

[0028] The EEG feature extraction and encoding module is used for feature learning and encoding of brain signal data. This module includes a transmitter module and a wireless channel module. The transmitter module includes an EEG data feature extraction unit and a channel encoding unit; the wireless channel module includes a channel transmission unit and a neural network cross-entropy model.

[0029] The edge agent decoding and task execution module is used to decode EEG commands transmitted through a wireless channel. The edge agent decoding and task execution module includes a receiver module, which includes an edge decoding unit and an execution unit.

[0030] To achieve accurate classification of multi-type and multi-dimensional EEG signals and generate EEG commands, such as... Figure 2 As shown, the present invention provides a dynamic EEG feature extraction unit, which mainly includes a common feature extraction module, a low-dimensional feature extraction model, a high-dimensional feature extraction model, and a dynamic decoding unit.

[0031] In the common feature extraction module, EEG data of different dimensions are mapped into the same latent space through a convolutional layer. The mapped data is then input into the embedding layer for initial data learning. The embedded features output by the embedding layer are subjected to interval masking to cause destructive interference to the EEG data. For example, three EEG data points x1[3,150], x2[4,260], and x3[64,640] are mapped together into the latent space of [64,256]. That is, the data dimensions x1', x2', and x3' after convolution are all [64,256]. x1', x2', and x3' are then placed into the common feature extraction module for similarity feature learning and output as influencing factors in the low / high-dimensional feature extraction model for utilization. The embedding layer is a convolutional layer with a kernel of 1, and the input and output dimensions are the same. The reason for performing interval masking is that the inventors discovered through experiments that EEG data not only has time and space dependence, but also a trend in the damaged data. This trend may improve the model's cognitive ability to EEG data, which we call trend dependence. Based on this, the data is processed by interval masking, that is, non-zero values ​​are set to 0 at intervals to highlight the trend dependence that they all have.

[0032] Subsequently, a multi-head attention mechanism was used to extract common features from the interval-masked data. The multi-head attention features were used to find pairwise correspondences between EEG values ​​at different sampling time points to obtain their temporal correlation. Recent research indicates that traditional self-attention requires higher memory complexity and secondary dot product calculations, which is its main drawback in predictive ability. This study also proposes that the importance scores of the calculated values ​​in sparse self-attention form a long-tailed distribution, meaning that only a few dot products contribute to the main attention calculations, while other dot product pairs are negligible. It is assumed that important dot product pairs are uniformly distributed. The probability distribution formula and uniform distribution of the traditional attention mechanism are derived using the KL divergence formula:

[0033]

[0034]

[0035] in, It is the probability distribution of the attention mechanism. It is a uniform distribution. It is an asymmetric exponential kernel function. Substituting it into the divergence formula to calculate KL(q||p) and discarding the constant term, we obtain the dispersion evaluation of the i-th query value in the attention mechanism, as follows: If we determine a value of U, and let U = L Q lnL K This significantly reduces the computational complexity of the attention mechanism while ensuring that important dot product pairs are retained in subsequent calculations without affecting computational performance. Here, U is... Figure 3 The system selects K=30 to retain valid dot product pairs to form a dot product pair matrix. To ensure matrix dimension adaptation during the calculation process, the other dimensions of the matrix are filled with the corresponding query average to form an intermediate common feature. The intermediate common feature is then activated by a convolutional layer as the final extracted common feature of each signal, and then sent to the next module for learning and utilization.

[0036] To address the challenges of low information content and classification difficulty in low-dimensional EEG data (corresponding to low-cost, high-noise EEG devices), a low-dimensional feature extraction model is used to improve recognition accuracy. This model consists of a Deepconvnet and a BrainTransformer. Figure 2 It consists of the upper right region. By utilizing a parallel learning structure and applying a residual mechanism, the classification accuracy of low-dimensional EEG signals can be improved. For example... Figure 4As shown, the Brain Transformer embeds a temporal feature transformer, whose main mechanism is a multi-head attention mechanism to efficiently extract the temporal characteristics of the data. After multiple layers of normalization and multiple residual fusions, the data flows to an integration layer composed of two convolutional layers and one fully connected layer. After integration, a classification prediction feature is output. Before the output of the low-dimensional feature extraction model, two fully connected layers are introduced to learn the weights of the sum of two parameters. The extracted low-dimensional features and the common features output by the common feature extraction module are used as the two summing parameters input to these two fully connected layers. This allows the low-dimensional features and common features to influence each other in a self-learning optimal way to obtain the optimal features before fusion, finally outputting a fused classification result.

[0037] In the high-dimensional feature extraction model, this model ( Figure 2 The lower right half of the EEG data is mainly improved based on EEGNet (a neural network structure). The pooling layer built into EEGNet is replaced with an average pooling layer. During pooling, the data is averaged, which helps to retain more background information, that is, to retain the temporal mutual influence of EEG data. Two fully connected layers are introduced before EEGNet to learn the weights of the sum of two parameters. That is, the input raw high-dimensional data and common features are self-learned and weighted to obtain the optimal feature pre-fusion. The fused features are then learned.

[0038] The dynamic decoding unit is used to adjust the selection of low-dimensional and high-dimensional feature extraction models, ensuring good learning and classification capabilities for different EEG data and avoiding accuracy degradation caused by information conflicts between data. The low-dimensional and high-dimensional feature extraction models are selectively updated and learned. The dynamic decoding unit automatically freezes parameters based on the dimensionality of the input data. For example, if low-dimensional data is input into the EEG feature extraction unit, the parameters of the high-dimensional feature extraction model will be frozen and will not participate in the learning and prediction tasks, and vice versa.

[0039] The channel coding unit consists of two convolutional layers. It also reorganizes dimensions to integrate the classification results of instructions output by multiple EEG devices into an ordered one-dimensional matrix, and then outputs the channel coding unit.

[0040] To facilitate deployment, the EEG feature extractor unit and channel coding unit are lightweighted through model compression to overcome the limitations of small computational load in embedded intelligent units. From the feature extraction model to any resource-constrained server or intelligent unit, the model is compressed using model pruning (pruning less important model branches), weight sharing, and quantization, resulting in a 60% reduction in memory usage and a 45% reduction in running speed, while maintaining the same accuracy.

[0041] The wireless channel module is used to simulate interference by adding noise to the incoming channel-coded features. During the transmission of processed EEG signal features from the transmitter module to the receiver module via broadband wireless communication, the instructions are generally significantly affected by noise interference during wireless transmission, leading to distortion. To address this issue, a noise simulation method is used to simulate three wireless environments (additive white Gaussian noise, Ricean channel, and Rayleigh fading channel) to train our framework and demonstrate its robustness under different wireless noise conditions, simulating noise addition interference to the incoming channel-coded features.

[0042] In this process, the present invention provides a multi-information loss entropy calculation model to learn the loss function to better obtain good communication reconstruction performance and achieve a more accurate semantic communication process. The calculated loss value is used for backpropagation calculation of the overall model loss. Specifically, a small neural network cross-mutual information entropy model is established to calculate the signal data z emitted from the transmitter module that is not affected by noise and the signal data z that has been affected by noise but has not yet entered the receiver module. The neural network cross-entropy model first calculates a cross-entropy between the two data points, then swaps half of each data point to form two new data points and calculates another cross-entropy. The two cross-entropies are added together to form the loss value, which is used for backpropagation calculation of the overall deep learning model.

[0043] The receiver module includes an edge decoding unit and an execution unit.

[0044] Interference features processed by the wireless channel module are transmitted to the edge decoding unit in the receiver module for decoding. The edge decoding unit is configured on each agent and mainly includes a semantic decoder and a channel decoder to reconstruct and recognize the EEG commands issued by the user. The receiver module has limited computing resources but requires fast transmission speeds; therefore, a lightweight module is needed to implement the command decoding process. Specifically, a semantic decoder consisting of a dimension adjustment layer and two convolutional layers is used to reshape the one-dimensional data from the wireless channel and decode its original semantic commands. Following this, the channel decoder, consisting of a fully connected layer and a sigmoid activation layer, activates and outputs each semantic command. Thus, multiple commands from multiple users are sorted according to input time and transmitted to all edge agents in a two-dimensional table format, meaning the edge agents can receive all user commands.

[0045] Finally, the user instructions are arranged according to input time and user. Each edge agent searches for its own task list in the input instructions according to the preset instruction list shown in Table 1. The execution units on each agent execute the tasks of each robot one by one according to the time, thereby achieving the brain-computer interaction process between multiple users and multiple agents. Each agent has its own execution unit, which produces different execution actions in response to different user instructions. After receiving all the user instruction information, the agent searches for its own task according to the instruction list and implements the preset behavioral tasks according to the instruction classification, ultimately achieving multi-user collaborative control of multi-agent interaction.

[0046] Table 1 Instruction Table

[0047]

[0048] An experiment was conducted using four datasets to test the accuracy of instruction classification. The four datasets are: 1. Brainlink dataset: This dataset was collected using Brainlink Lite for continuous EEG measurements. 2. Eyestate dataset: This dataset was obtained using the Emotiv EEG Neuroheadset for continuous EEG measurements. 3. BCI-2000 competition dataset: This dataset collected EEG signals from 64 channels from volunteers. 4. Mixed dataset: This dataset is a combination of the previous three datasets and contains ten categories. The mixed training data consists of all training sets from the three datasets, including 400-800 samples randomly selected from each dataset.

[0049] The following seven models are used for data comparison:

[0050] EEGNet is a compact fully convolutional network for EEG-based BCI.

[0051] Compact-CNN, a compact ConvNet, is used to decode signals from the 12-class SSVEP dataset without requiring user-specific calibration.

[0052] DeepConvNet and ShallowConvNet are not only novel and promising tools in the EEG decoding toolkit, but also combine with innovative visualization techniques.

[0053] EEG-TCNet, a novel temporal ConvNet, achieves excellent accuracy with only a few trainable parameters.

[0054] ResNet, a residual learning scheme, simplifies the training of deep neural networks.

[0055] DeepBrain, a transformer-based variant, excels in low-dimensional EEG feature extraction.

[0056] Table 2

[0057]

[0058] As shown in Table 2, the performance of all models was tested on a single dataset (corresponding to a scenario with a single EEG device). Our model outperformed the other models in classification accuracy.

[0059] The three datasets mentioned above are mixed; different types of data can interfere with the model's learning, thus significantly affecting the model's classification accuracy. The multi-user, multi-device brain-computer interface and semantic communication system (Our) of this invention employs a dynamic encoding and decoding process. This process effectively avoids interference between data from different devices, resulting in experimental accuracy far exceeding that of other methods. The results are as follows... Figure 7 As shown, the results demonstrate that the multi-user, multi-device brain-computer interface and semantic communication system of the present invention has the best command recognition capability in scenarios where multiple brain-computer devices simultaneously input and collaboratively control the intelligent agent.

[0060] Furthermore, to demonstrate the good performance of our multi-user, multi-device brain-computer interface and semantic communication system in various wireless noise environments, we tested our framework using a mixed dataset in three different wireless channel environments and compared it with an improved semantic communication framework, such as... Figure 8 As shown, the results demonstrate that the multi-user, multi-device brain-computer interface and semantic communication system (Our) of the present invention achieves superior instruction accuracy and robustness in three different noise environments.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-user, multi-device brain-computer interface and semantic communication system, characterized in that, It includes an EEG signal acquisition module, an EEG feature extraction and encoding module, and a limbic agent decoding and task execution module; The EEG signal acquisition module is used to acquire brain signal data; The EEG feature extraction and encoding module is used to learn features from the brain signal data and encode them into EEG commands. The EEG feature extraction and encoding module includes a transmitter module and a wireless channel module. The transmitter module includes an EEG data feature extraction unit and a channel encoding unit. The wireless channel module includes a channel transmission unit and a neural network cross-entropy model. The edge intelligence decoding and task execution module is used to reconstruct and recognize the EEG commands, and generate different execution actions according to different user commands; The EEG data feature extraction unit includes a common feature extraction module, a low-dimensional feature extraction model, a high-dimensional feature extraction model, and a dynamic decoding unit. The low-dimensional feature extraction model is used to improve the recognition accuracy of low-dimensional EEG data and extract the features of low-dimensional EEG data. The high-dimensional feature extraction model is used to extract the features of high-dimensional EEG data. The dynamic decoding unit is used to adjust the selection of the low-dimensional feature extraction model and the high-dimensional feature extraction model. The common feature extraction module includes a convolutional layer, an embedding layer, a spacing mask layer, and a common feature extraction layer; The neural network cross-information entropy model will retrieve the signal data emitted by the transmitter module that is free from noise interference. and signal data that has not yet entered the receiver module after being interfered with by noise The first cross-entropy is calculated, and then half of the two data points are swapped and the second cross-entropy is calculated again. The first cross-entropy and the second cross-entropy are added together to form the loss value, which is used to perform backpropagation calculation on the overall deep learning model.

2. The multi-user, multi-device brain-computer interface and semantic communication system as described in claim 1, characterized in that, The EEG feature extraction and encoding module also includes a lightweight transmitter module, which comprises an EEG data feature extraction unit and a channel encoding unit after model compression.

3. The multi-user, multi-device brain-computer interface and semantic communication system as described in claim 1, characterized in that, The wireless channel module is used to simulate interference by superimposing noise onto the incoming channel coding features. The wireless channel module uses noise simulation to simulate transmission interference in the wireless environment.

4. The multi-user, multi-device brain-computer interface and semantic communication system as described in claim 1, characterized in that, The edge intelligent agent decoding and task execution module includes a receiver module, which includes an edge decoding unit and an execution unit; the edge decoder unit is used to restore and recognize the EEG commands issued by the user, and the edge decoder unit includes a semantic decoding unit and a channel decoding unit; the execution unit arranges the user commands and generates execution actions according to the command list.

Images