A brain signal basic processing system and method for multi-modal neuroelectric signals

Abstract

The application relates to a brain signal basic processing system and method for multi-modal neuroelectric signals, which comprises: a neuroelectric signal anomaly detection module configured to output qualified multi-modal neuroelectric signals; an intracerebral and extracerebral unified spatial position coding module configured to output multi-modal neuroelectric signals with position coding; a dynamic brain region spatial convolution module configured to adaptively weight the multi-modal neuroelectric signals and convert heterogeneous multi-channel neuroelectric signals into homogeneous single-channel feature sequences; and a composite self-supervised learning module configured to perform self-supervised training of mask prediction combined with autoregressive prediction on the homogeneous single-channel feature sequences, output a pre-training completed brain signal basic model, and complete downstream task evaluation based on the pre-training completed brain signal basic model. The application improves the performance of multi-dimensional downstream tasks in the field of brain-computer interfaces and reduces the demand of downstream tasks for data scale.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of brain science and artificial intelligence, and to a brain signal basic processing system and method for multimodal neural electrical signals. Background Technology

[0002] With the development of artificial intelligence, deep learning technology has been widely applied in the field of neuroscience, demonstrating superior performance compared to previous feature extraction methods that relied heavily on prior human design. As a core area for exploring human cognition and neural activity, neuroscience research has significant application value in areas such as brain-computer interfaces, neurological disease diagnosis, and human-computer intelligent interaction. Neural electrical signals are the core carriers reflecting brain neural activity, mainly including various modalities such as electroencephalography (EEG), magnetoencephalography (MEG), and invasive electroencephalography (iEEG). Different modalities of signals characterize the state of neural activity in the brain from macroscopic and microscopic dimensions, each with its own technological advantages and application scenarios. However, the development and application of basic brain signal models still face many technical bottlenecks. Labeled datasets in the field of neuroscience are small in scale and costly to acquire. Furthermore, there are significant differences in neural responses between individuals, and the data heterogeneity of signals from different devices and modalities is obvious, making it difficult to train basic brain signal models with sufficient capacity and strong generalization ability.

[0003] Existing technologies have the following drawbacks: 1) Existing technologies mainly focus on single EEG modalities; 2) Lack of exploration of pre-training paradigms: Existing methods mainly use mask reconstruction to learn the intrinsic knowledge of data in an unsupervised manner. For brain signals with low signal-to-noise ratios, using the original signal with high noise levels as the final learning target may constrain the upper limit of model performance; 3) Weak model generalization ability: Existing basic brain signal models still rely on full fine-tuning in downstream applications, which may lead to problems such as catastrophic forgetting of knowledge, and indicates that existing models are still insufficient to generate features with strong expressive power; 4) Low model temporal precision: The model windowing time is too long, making it impossible to capture rapidly changing neural activity at the tens of millisecond level; 5) Narrow coverage of downstream task applications: Downstream applications mainly focus on sequence classification tasks such as disease classification, emotion recognition, and imagined action classification, with relatively coarse granularity, and insufficient attention is paid to the performance of the basic model in fine-grained tasks such as language decoding and visual reconstruction. How to expand the data scale and model size, and use self-supervised methods to efficiently learn the intrinsic knowledge structure of multiple brain signals, thereby empowering the development of downstream tasks of brain-computer interfaces, is an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a brain signal basic processing system and method for multimodal neural electrical signals. It aims to construct a neural representation system with strong generalization ability through multimodal integration and dual-objective learning, so as to establish a basic brain model covering multiple neural electrical signals such as EEG, MRI, and invasive EEG.

[0005] This invention primarily addresses the following core challenges of current basic brain signal models in processing multimodal neural electrical signals:

[0006] 1) Bottleneck in modal integration and heterogeneous device adaptation: Although neural electrical signals are excited by common neural activity, existing models are mostly limited to a single modality or specific device, failing to effectively integrate multiple neural electrical signals including EEG, MEG and iEEG, and cannot adapt to heterogeneous acquisition scenarios where the number, type and location of sensors change frequently.

[0007] 2) Data quality and artifact interference: Open source data contains a large number of unlabeled bad sectors and artifacts, and lacks high-quality automated cleaning algorithms, which causes pre-trained models to fit false features;

[0008] 3) Limitations of the pre-training paradigm: Existing mask reconstruction paradigms focus on fine-grained completion of the original signal. Due to the low signal-to-noise ratio of brain signals, it is difficult to capture deep cognitive semantics by relying solely on physical reconstruction, and it is also prone to fitting noise.

[0009] 4) Weak generalization ability of the model: When existing models are transferred to downstream tasks, they often need to be fine-tuned in full. This not only leads to "catastrophic forgetting" of the model, but also indicates that its representation ability is insufficient and its generalization ability is limited.

[0010] 5) Low model temporal accuracy: The window length of the existing model is about one second, which makes it difficult to capture rapidly changing fine neural activities and does not reflect the improvement of important brain-computer interface applications with fine granularity such as language decoding and visual reconstruction.

[0011] This invention achieves unified modeling across modalities and devices through the encoding of intraocular and extraocular sensor locations. After learning the discretized representation of brain signals using a self-reconstruction task, it combines mask prediction and autoregressive prediction as a self-supervised learning paradigm to autonomously learn efficient and precise representations of brain signals in large-scale multimodal neural electrical signal data. This improves the performance of downstream tasks in the field of brain-computer interfaces, such as neurological disease diagnosis, emotion recognition, motor imagery, language decoding, and visual reconstruction, while reducing the data scale requirements of downstream tasks.

[0012] To address the aforementioned technical problems, this invention provides a brain signal basic processing system for multimodal neural electrical signals, comprising:

[0013] The neural electrical signal abnormality detection module is configured to label bad channels, clean and separate artifacts from the original multimodal neural electrical signals, remove unqualified signal segments, and output qualified multimodal neural electrical signals.

[0014] The intrabrain and extrabrain unified spatial location coding module is configured to map the multimodal sensors corresponding to the qualified multimodal neural electrical signals to a unified three-dimensional coordinate system, fuse the sensor spatial coordinates and physical properties to generate learnable location codes, and output multimodal neural electrical signals with location codes.

[0015] A dynamic brain region spatial convolution module is configured to generate a dynamic spatial convolution kernel based on the position-encoded multimodal neural electrical signals, adaptively spatially weight the multimodal neural electrical signals, and convert heterogeneous multichannel neural electrical signals into homogeneous single-channel feature sequences; and

[0016] The composite self-supervised learning module is configured to perform self-supervised training combining mask prediction and autoregressive prediction on the isomorphic single-channel feature sequence, learn the contextual structured information and future state prediction ability of brain signals, and output a pre-trained basic model of brain signals. Based on the pre-trained basic model of brain signals, it processes multimodal neural electrical signals related to downstream tasks, and completes brain signal analysis, decoding and recognition for downstream tasks.

[0017] Furthermore, the unqualified signal segments include segments with extremely low signal-to-noise ratios or severely distorted waveforms. Extremely low signal-to-noise ratios are caused by eye movement, electromyography, external electromagnetic disturbances, or sensor malfunctions or poor contact, resulting in very little effective information in the signal.

[0018] Furthermore, the physical properties include the spatial location and orientation of the sensor, and the type of sensor (scalp potential sensor, magnetic field gradient sensor, magnetic field amplitude sensor, ECoG sensor, and SEEG sensor).

[0019] Furthermore, the neural electrical signal anomaly detection module is based on a lightweight residual one-dimensional convolutional network. It performs sliding window detection on the original multimodal neural electrical signals in both the time and frequency domains, automatically identifying bad channels and severe artifact interference. The module also scores signal quality to remove substandard signal segments. A lightweight time-frequency detection module is constructed within this module to automatically label bad channels and remove artifacts from large-scale training data. In the signal quality scoring process, low-quality signal segments are first manually labeled. The training objective of the anomaly detection module is to output the probability of low signal quality, with a set confidence threshold; for example, a probability greater than 0.8 indicates that the segment needs to be removed.

[0020] Furthermore, the multimodal neural electrical signals include electroencephalography (EEG), magnetoencephalography (MEG), and invasive EEG.

[0021] Furthermore, in the unified intrabrain and extrabrain spatial location coding module, the multimodal sensors include scalp EEG electrodes, magnetoencephalography (MEG) sensors, and invasive EEG contacts. Specifically, they include scalp potential sensors, magnetic field gradient sensors, magnetic field amplitude sensors, ECoG sensors, and SEEG sensors.

[0022] Furthermore, in the unified intrabrain and extrabrain spatial location coding module, generating the location code includes the following steps:

[0023] 3D coordinate alignment: The multimodal sensors are mapped to a unified MNI standard brain 3D coordinate system, enabling the model to perceive the relative positions of different sensors in anatomical space;

[0024] Electrode coding for physical sensing: A linear layer is used to encode the spatial coordinates of the sensors, and the sensor type (scalp location, magnetic field amplitude, magnetic field gradient, brain potential, etc.) is converted into learnable codes; and

[0025] After fusing the position and physical property features of the electrodes, a multilayer perceptron is used for feature fusion, and the final position code is obtained after normalization. A neural network is then used to transform the electrode's three-dimensional coordinates and physical properties into a learnable position code, enabling unified modeling of signals across devices and modalities. This code can be used to obtain device and modal information for each input signal, thus achieving unified modeling across devices and modalities.

[0026] Furthermore, the dynamic brain region spatial convolution module includes:

[0027] Electrode position encoding, its shape being [number of electrode channels, feature dimension]; and

[0028] The learnable inversion matrix, whose shape is [number of spatial convolution kernels, feature dimension], is used in conjunction with the electrode position encoding to obtain the dynamic spatial convolution kernel.

[0029] Furthermore, the dynamic brain region spatial convolution module performs matrix multiplication between the inversion matrix and the transpose of the electrode position encoding, and uses exponential normalization to convert the result into an attention score, obtaining a dynamic spatial convolution kernel with the shape of [number of spatial convolution kernels, number of electrode channels]. This dynamic spatial convolution kernel consists of multiple dynamic spatial filters, each learning different weighted filtering methods for sensors located in different brain regions to transform heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences. This dynamic brain region spatial convolution module constructs multiple dynamic spatial filters for input signals from different devices and modalities, with each filter learning specific spatial weighting methods for channels located in different brain regions. For neural electrical signals where the number of channels and the physical meaning of the channels vary, this method can construct adaptive channel combinations to transform heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences.

[0030] Furthermore, in the composite self-supervised learning module, mask prediction involves splitting the brain signal sequence into front, middle, and back segments and rearranging them, then predicting discrete labels for the masked parts based on context using causal attention. Autoregressive prediction involves calculating the probability distribution of the discrete label corresponding to the next position based on the feature sequence after causal attention. Combining mask prediction and autoregressive prediction tasks for self-supervised learning allows the model to simultaneously learn the structured information of the context and the ability to predict future states, improving the model's understanding and generation capabilities of neural electrical signals. The discrete labels originate from a discrete autoencoder network pre-trained using a self-reconstruction task: first, the discrete autoencoder of neural electrical signals is trained separately using a self-reconstruction task; then, the discrete labels provided by this discrete autoencoder are used as learning targets, enabling the model to predict units with stronger semantics and reducing noise interference with the learning targets.

[0031] Furthermore, the downstream tasks include coarse-grained discrimination tasks and fine-grained generation tasks; the coarse-grained discrimination tasks include disease-assisted diagnosis, emotional state recognition, sleep staging, and motor imagery classification, and the fine-grained generation tasks include language decoding, visual reconstruction, and auditory decoding.

[0032] Furthermore, it also includes:

[0033] The downstream task evaluation module is configured to perform performance verification and representation ability assessment on the pre-trained brain signal baseline model. It covers a task matrix ranging from coarse-grained discrimination (disease / emotion / action) to fine-grained generation (language decoding / image reconstruction) to comprehensively evaluate the model's capabilities.

[0034] This invention also provides a basic brain signal processing method for multimodal neural electrical signals, comprising the following steps:

[0035] S1. Input the raw data of multimodal neural electrical signals into the neural electrical signal abnormality detection module to complete the automatic identification and cleaning of bad channels and artifacts, remove unqualified signal segments, and output qualified multimodal neural electrical signals.

[0036] S2. Input the qualified multimodal neural electrical signal into the unified spatial coding module inside and outside the brain to complete the three-dimensional coordinate alignment of the multimodal sensor, fuse the sensor spatial coordinates and physical properties to generate a learnable position code, and output the multimodal neural electrical signal with position code.

[0037] S3. Input the multimodal neural electrical signals with position encoding into the dynamic brain region spatial convolution module to generate a dynamic spatial convolution kernel, and perform adaptive spatial weighting on the heterogeneous multi-channel neural electrical signals to convert them into homogeneous single-channel feature sequences.

[0038] S4. Input the isomorphic single-channel feature sequence into the composite self-supervised learning module to perform self-supervised training combining mask prediction and autoregressive prediction. This allows the model to simultaneously learn the contextual structure understanding ability and future state prediction generation ability of brain signals, thereby completing model pre-training and outputting the pre-trained basic model of brain signals; and

[0039] S5. Use the pre-trained brain signal baseline model to process multimodal neural electrical signals related to downstream tasks.

[0040] Furthermore, in step S5, for classification and retrieval tasks, a small task head (outputting classification probabilities or vectors for retrieval) is connected to the high-quality features output by the brain signal base model. With the base model frozen or slightly fine-tuned, the downstream task network is primarily trained to achieve high performance. Since the base model itself provides good representations, there is no need for extensive feature learning from scratch, reducing the need for downstream data. For finer-grained decoding tasks (such as phoneme sequence decoding), an LSTM-based network can be connected to the high temporal resolution features output by this model for sequence decoding, improving performance. Current base models typically have a temporal resolution of 1 second, making them unsuitable for similar high-precision sequence decoding tasks.

[0041] Furthermore, it also includes:

[0042] S6. Input the pre-trained brain signal baseline model into the multi-dimensional downstream task evaluation module, and perform performance verification through a downstream task system of coarse-grained discrimination and fine-grained generation. This includes:

[0043] A multi-layered task matrix was constructed to verify the model's generality and generalization ability, and to evaluate its performance on coarse-grained and fine-grained tasks: 1) Coarse-grained cognitive and discrimination tasks: covering traditional disease-aided diagnosis (such as epilepsy and Alzheimer's disease), emotional state recognition, and motor imagery classification. In these tasks, the model mainly utilizes global semantic features for sequence-level classification. 2) Fine-grained neural decoding tasks: covering various important applications of brain-computer interfaces such as language decoding and visual reconstruction, exploring the performance gains of the basic model in streaming language decoding and visual stimulus reconstruction. In these tasks, the model requires high-resolution fine-grained features for information decoding.

[0044] This invention proposes a brain signal basic processing system for multimodal neural electrical signals, which can simultaneously process multiple neural electrical signals such as electroencephalography (EEG), magnetoencephalography (MEG), and invasive electroencephalography (iEEG). Through unified spatial location coding and a shared backbone network, it achieves universal modeling capabilities across modalities, devices, and sensor configurations.

[0045] This invention proposes a method for detecting and automatically cleaning abnormal neural electrical signals. By jointly modeling in the time and frequency domains, it can automatically detect and remove bad channels, saturation artifacts, and non-neural source interference such as electromyography and eye movement, thereby ensuring the quality of input data in the unsupervised large-scale pre-training stage.

[0046] The dynamic brain region spatial convolution module proposed in this invention constructs multiple dynamic spatial filters for input signals from different devices and modalities, learns specific spatial weighting methods for channels located in different brain regions, and adaptively transforms heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences.

[0047] This invention combines a self-supervised paradigm of mask prediction and autoregressive prediction, enabling the model to learn both mask prediction and autoregressive prediction tasks simultaneously, as well as the ability to learn context-based structured understanding and the ability to predict future states, thereby improving the model's representation quality.

[0048] The present invention provides a downstream application system for brain signals that comprehensively evaluates both coarse-grained and fine-grained tasks. During the comprehensive evaluation, a complete evaluation system is constructed that covers coarse-grained discrimination tasks such as disease diagnosis, emotion recognition, and motor imagery classification, as well as fine-grained generation tasks such as language decoding and image reconstruction. This system is used to comprehensively evaluate the basic model's representation and transfer capabilities.

[0049] This invention has at least the following beneficial effects: 1) Existing technologies mainly focus on single-modality EEG, with limited exploration of MRI, invasive EEG, and combined multimodal approaches. This invention offers more comprehensive modality coverage and significantly enhanced adaptability. It constructs a unified modeling system covering multiple neural electrical signals such as EEG, MRI, and invasive EEG, significantly improving generalization capabilities across devices and acquisition scenarios, allowing the model to simultaneously learn the precise dynamics (invasive) and macroscopic structure (non-invasive) of neural activity; 2) Compared to existing methods that directly use raw unsupervised data for pre-training, this invention's processing system introduces an automated anomaly detection and cleaning module at the output model front end, effectively filtering bad channels and strong artifact segments. This technique reduces noise pollution of the potential representation space, improves data quality from the source, reduces noise interference with pre-trained representations, and improves pre-training efficiency and the reliability of the final representation; 3) Existing mask reconstruction methods mainly use the original signal as the reconstruction or prediction target, making it difficult to learn high-level cognitive semantics under low signal-to-noise ratio conditions. This invention discretizes brain signals... Furthermore, by combining mask prediction and autoregressive prediction as a self-supervised learning paradigm, the model learns a structured understanding based on context and the ability to predict future states, thus better learning the semantics of brain signals, breaking through the limitations of reconstruction targets, and improving semantic modeling capabilities; 4) This invention is pre-trained on large-scale neural electrical signal data of more than 30,000 hours to learn more efficient and general feature representations. In downstream tasks, it can achieve good results through efficient parameter fine-tuning (such as partial layer updates or lightweight adaptation modules), reducing the dependence on full fine-tuning, reducing the risk of catastrophic forgetting, improving the practicality of the model in multi-task scenarios, significantly enhancing the model's transferability, and reducing downstream fine-tuning costs; 5) This invention improves the temporal precision of features from the current mainstream one second to about sixty milliseconds, enabling the learning of more fine-grained brain state changes, improving the performance of high-difficulty fine-grained tasks such as language decoding, image retrieval and reconstruction of brain signals, providing more promising basic model support for complex brain-computer interface applications, significantly improving feature temporal precision, and improving the performance of fine-grained decoding applications. Furthermore, this invention pertains to the specific application of innovative algorithms in a particular technological field, namely brain science and artificial intelligence. The resulting technical effect is to improve the performance of downstream tasks in the field of brain-computer interfaces and reduce the data scale requirements of downstream tasks. Attached Figure Description

[0050] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the embodiments of the invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0051] Figure 1A schematic diagram of the principle of the neural electrical signal abnormality detection module in some embodiments of the present invention is shown;

[0052] Figure 2 A schematic diagram illustrating the workflow of the unified intrabrain and extrabrain spatial location coding module in some embodiments of the present invention is shown.

[0053] Figure 3 A schematic diagram illustrating the workflow of the dynamic brain region spatial convolution module in some embodiments of the present invention is shown.

[0054] Figure 4 A schematic diagram illustrating the workflow of the composite self-supervised pre-training module in some embodiments of the present invention is shown;

[0055] Figure 5 A comprehensive brain signal downstream task evaluation system is shown in some embodiments of the present invention. Detailed Implementation

[0056] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0057] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0058] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0059] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0060] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0061] It should also be noted that in the description of this invention, the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not explicitly or implicitly suggest that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0062] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by a corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by a corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by a corresponding output device (such as a display, speaker, etc.), and so on.

[0063] Furthermore, the embodiments of the present invention describe the process steps in a specific order. However, this is only for the convenience of distinguishing each step, and is not a limitation on the order of each step. In different embodiments of the present invention, the order of each step can be adjusted according to the process.

[0064] The following embodiment provides a basic brain signal processing system for multimodal neural electrical signals, comprising:

[0065] The neural electrical signal abnormality detection module based on multi-scale features is configured to perform bad channel labeling, artifact cleaning and separation, and remove unqualified signal segments (unqualified signal segments are low-quality signal segments, including segments with extremely low signal-to-noise ratio or severely distorted waveforms) on the original multimodal neural electrical signals (EEG, MRI, and invasive EEG), and output qualified multimodal neural electrical signals. Figure 1 A schematic diagram of the principle of the neural electrical signal abnormality detection module is shown. Figure 1In the image, the four waveforms on the left correspond to different types of raw EEG signals, encompassing two typical problems. From top to bottom, these are the first, second, third, and fourth waveforms. The first two represent normal channels with artifacts / noise interference, exhibiting abnormal spikes and baseline drift. The latter two are completely abnormal / failed channels, displaying severe distortions such as large jumps, saturation, and disconnections. After the raw multimodal neural electrical signals are input into the neural signal anomaly detection module, artifact separation is performed on the effective signals containing artifacts, outputting a denoised, clean signal that preserves the characteristics of real brain activity. Abnormal channels such as electrode detachment and poor contact are automatically identified, marked, and removed. This neural electrical signal anomaly detection module can automatically discard bad channels and separate noise artifacts from the signal, thereby performing large-scale data cleaning. Based on a lightweight residual one-dimensional convolutional network, the module performs sliding window detection on the original multimodal neural electrical signals in both the time and frequency domains, automatically identifying bad channels and severe artifact interference. The module also scores the signal quality to remove unqualified signal segments. Furthermore, a lightweight time-frequency detection module is constructed within this module to automatically label bad channels and remove artifacts from large-scale training data.

[0066] The unified intrabrain and extrabrain spatial location coding module is configured to map the multimodal sensors (scalp EEG electrodes, magnetoencephalogram sensors, and invasive EEG contacts) corresponding to qualified multimodal neural electrical signals to a unified MNI standard three-dimensional brain coordinate system, enabling the model to perceive the relative positions of different sensors in anatomical space. Figure 2 This diagram illustrates the workflow of a unified intracranial and extracranial spatial location coding module. Specifically, it showcases the network structure for electrode channel coding that supports cross-EEG, MRI, and invasive EEG modeling. The module incorporates both intracranial and extracranial measurement devices into a standard space. Linear layers encode the location information of each electrode, maintaining a learnable vector for the physical quantities measured by each electrode. Ultimately, the physical information of the electrodes is transformed into learnable codes. Linear layers encode the spatial coordinates of sensors and convert the sensor type (scalp location, magnetic field amplitude, magnetic field gradient, intracranial potential, etc.) into learnable codes. After fusing the electrode location and physical attribute features, a multilayer perceptron is used for feature fusion. After normalization, the final location code is obtained. A neural network is then used to transform the electrode's three-dimensional coordinates and physical attributes into learnable location codes, achieving unified modeling across devices and modalities. This coding method can acquire device and modal information for each input signal, thus enabling unified modeling across devices and modalities.

[0067] The dynamic brain region spatial convolution module is configured to generate dynamic spatial convolution kernels based on multimodal neural electrical signals with positional encoding. It adaptively spatially weights the multimodal neural electrical signals, transforming heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences. The module includes: electrode position encoding with the shape [number of electrode channels, feature dimension]; and a learnable inversion matrix with the shape [number of spatial convolution kernels, feature dimension]. The inversion matrix is ​​then multiplied by the transpose of the electrode position encoding, and the result is converted into an attention score using exponential normalization. This yields a dynamic spatial convolution kernel with the shape [number of spatial convolution kernels, number of electrode channels]. Each dynamic spatial convolution kernel is a set of multiple dynamic spatial filters. Each kernel learns different weighted filtering methods for sensors located in different brain regions to transform heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences. This dynamic brain region spatial convolution module constructs multiple dynamic spatial filters for input signals from different devices and modalities. Each filter learns a specific spatial weighting method for channels located in different brain regions. For neural electrical signals where the number of channels and the physical meaning of the channels vary, this method can construct adaptive channel combination methods, transforming heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences. Figure 3 A schematic diagram illustrating the workflow of the dynamic brain region spatial convolution module is provided, demonstrating its structure. Through matrix multiplication operations using inversion matrices and electrode position encoding, along with exponentially normalized attention calculations, dynamic spatial convolution kernels are generated for input signals from different devices and modalities. The module learns different weighting methods for electrode channels in different brain regions. This module can transform heterogeneous multichannel brain signals into homogeneous single-channel feature sequences; and

[0068] The composite self-supervised learning module is configured to perform self-supervised training on isomorphic single-channel feature sequences, combining mask prediction and autoregressive prediction. It learns the contextual structure information and future state prediction capabilities of brain signals, outputting a pre-trained basic model of brain signals. This pre-trained model is then used to process multimodal neural electrical signals related to downstream tasks, including coarse-grained discrimination tasks and fine-grained generation tasks. Coarse-grained discrimination tasks include disease-assisted diagnosis, emotion state recognition, and motor imagery classification; fine-grained generation tasks include language decoding and visual reconstruction. In the composite self-supervised learning module, mask prediction involves splitting the brain signal sequence into pre-, mid-, and post-sequence segments and rearranging them, then predicting discrete labels for the masked portions based on context using causal attention. Autoregressive prediction involves calculating the probability distribution of the discrete label corresponding to the next position based on each position of the feature sequence after causal attention. Combining mask prediction and autoregressive prediction tasks for self-supervised learning allows the model to simultaneously learn the structured information of the context and the ability to predict future states, improving the model's understanding and generation capabilities of neural electrical signals. Figure 4 The diagram illustrates the workflow of the composite self-supervised pre-training module, showcasing a self-supervised learning paradigm that combines autoregressive prediction and mask prediction. After dynamic brain region spatial convolution processing, the device layout and input signals are pre-trained using both autoregressive and mask prediction methods. The paradigm training constructs stacked causal attention blocks for the model, ultimately outputting a pre-trained basic model of brain signals. For autoregressive prediction, after causal attention calculation, each position in the feature sequence attempts to predict the discrete label corresponding to the next position. For mask prediction, the input sequence is segmented and rearranged, allowing the model to attempt to predict the discrete label of the masked portion given the known context. This method enables the model to simultaneously learn context-based understanding and the ability to predict and generate future states, improving the quality of model representations.

[0069] Based on the pre-trained brain signal model, this embodiment also constructs a multi-level task matrix to verify the model's universality and generalization ability, and to evaluate the model's performance on coarse-grained and fine-grained tasks. Figure 5 It demonstrates a comprehensive assessment system for downstream tasks involving brain signals, including coarse-grained tasks (such as neurological disease diagnosis, emotion classification, and motor classification) and fine-grained tasks (visual decoding, phoneme decoding, and language decoding):

[0070] Coarse-grained cognitive and discriminative tasks: covering traditional disease-aided diagnosis (such as epilepsy and Alzheimer's disease), emotional state recognition, sleep staging, and motor imagery classification. In these tasks, the model primarily utilizes global semantic features for sequence-level classification.

[0071] Fine-grained neural decoding tasks: Covering various important brain-computer interface applications such as visual decoding, auditory decoding, and language decoding, this study explores the performance gains of the base model for tasks such as language decoding, phoneme decoding, auditory stimulus reconstruction, visual stimulus reconstruction, and visual retrieval. In these tasks, the model requires high-resolution, fine-grained features for information decoding.

[0072] The following embodiments also provide a basic brain signal processing method for multimodal neural electrical signals, including the following steps:

[0073] S1. Input the raw data of multimodal neural electrical signals into the neural electrical signal abnormality detection module to complete the automatic identification and cleaning of bad channels and artifacts, remove unqualified signal segments, and output qualified multimodal neural electrical signals.

[0074] S2. Input qualified multimodal neural electrical signals into the unified spatial coding module inside and outside the brain to complete the three-dimensional coordinate alignment of the multimodal sensor, fuse the sensor spatial coordinates and physical properties to generate learnable position codes, and output multimodal neural electrical signals with position codes.

[0075] S3. Input the multimodal neural electrical signals with position encoding into the dynamic brain region spatial convolution module to generate a dynamic spatial convolution kernel, and perform adaptive spatial weighting on the heterogeneous multi-channel neural electrical signals to transform them into homogeneous single-channel feature sequences.

[0076] S4. Input the isomorphic single-channel feature sequence into the composite self-supervised learning module, and perform self-supervised training combining mask prediction and autoregressive prediction. This allows the model to simultaneously learn the contextual structure understanding ability and future state prediction generation ability of brain signals, thereby completing model pre-training and outputting the pre-trained basic model of brain signals; and

[0077] S5. Use the pre-trained brain signal baseline model to process multimodal neural electrical signals related to downstream tasks.

[0078] The above embodiments address the artifact and bad sector issues present in large-scale unsupervised pre-training data by designing a pre-processed automated data cleaning module:

[0079] Automatic identification of bad sectors and artifacts: Using a lightweight residual one-dimensional convolutional network, a sliding window detection is performed on the original signal in the time and frequency domains to automatically identify bad sectors and severe artifact interference.

[0080] Large-scale data cleaning strategy: This model automatically removes segments with extremely low signal-to-noise ratios or severely distorted waveforms by scoring the signal quality. This mechanism ensures the quality of the data input to the base model, avoids the contamination of the model's latent space by noisy signals, and lays the foundation for subsequent high-quality representation learning.

[0081] To break down the physical barriers between heterogeneous acquisition devices, different modalities, and invasive and non-invasive signals, this embodiment introduces a unified coding mechanism based on physical coordinates:

[0082] 3D coordinate alignment: All sensors (including scalp EEG electrodes, MEG sensors, and invasive iEEG contacts) are mapped to a unified MNI standard brain 3D coordinate system, enabling the model to perceive the relative positions of different sensors in anatomical space.

[0083] Electrode encoding for physical sensing: A linear layer is used to encode the spatial coordinates of the sensor, and the sensor type (scalp location, magnetic field amplitude, magnetic field gradient, brain potential, etc.) is converted into a learnable code. After fusing the electrode's location and physical attribute features, a multilayer perceptron is used for feature fusion, and the final location code is obtained after normalization. The model can use this encoding to obtain device and modal information for each input signal, thereby achieving unified modeling across devices and modalities.

[0084] In this embodiment, an adaptive dynamic spatial convolution module for input signals from different devices and modalities is also proposed. This module consists of electrode location encoding and a learnable inversion matrix. The shape of the electrode location encoding is [number of electrode channels, feature dimension], and the shape of the inversion matrix is ​​[number of spatial convolution kernels, feature dimension]. Matrix multiplication is performed between the inversion matrix and the transpose of the electrode location encoding, and exponential normalization is used to convert the result into an attention score, resulting in a dynamic spatial convolution kernel with the shape [number of spatial convolution kernels, number of electrode channels]. Each dynamic spatial convolution kernel learns different weighted filtering methods for sensors located in different brain regions, thereby adaptively transforming heterogeneous multi-channel input signals into homogeneous single-channel feature sequences.

[0085] In this embodiment, a self-supervised learning paradigm combining autoregression and mask prediction is used:

[0086] Autoregressive prediction: After the feature sequence is processed by causal attention, the probability distribution of the discrete label corresponding to the next position is predicted at each position.

[0087] Masking prediction: To enable the model to learn the contextual structure of brain signals and build better understanding, a signal segment can be split into its beginning, middle, and end parts, rearranged, and then computed using causal attention. This allows the model to predict the label of the masked portion given the known context.

[0088] This invention has collected approximately 30TB of EEG, MRI, and invasive EEG data, and constructed 20 downstream tasks:

[0089] EEG: ad65 (Alzheimer's disease classification), FACED (emotion classification), SEED-V (emotion classification), SEED-VII (emotion classification), PhysioNet-MI (motor imagery), BCIC-IV-2a (motor imagery), BCIC-2020-3 (verbal imagery), SHU-MI (motor imagery), ISRUC (sleep stages), MODMA (depression diagnosis), MAT (cognitive load), THINGS-EEG (visual retrieval);

[0090] Magnetoencephalography (MEG): asd74 (autism, auditory stimulation), MEG-MMI (depression, mood induction), Geometric-MEG (geometric shape visual stimulation), Omega (Parkinson's disease, chronic pain, health classification), Somato (motor response), LibriBrain (phoneme classification, phoneme streaming decoding, word retrieval), Armeni (word retrieval), MEG-MASC (word retrieval), THINGS-MEG (visual retrieval);

[0091] Invasive EEG: Chinese tone classification based on private datasets ecog and seeg, Omni-iEEG (epilepsy region identification).

[0092] In collaboration with BrainOmni [1] With the same amount of pre-training data, the present invention has achieved performance improvements in tasks such as MAT (cognitive load), MDD (diagnosis of depression), PhysioNet-MI (motor imagery), BCICIV-2a (motor imagery), ISRUC (sleep staging), ad65 (diagnosis of Alzheimer's disease), and Somato. Among them, [1] Xiao, Q., Cui, Z., Zhang, C., et al. BrainOmni: A brain-based model for unifying EEG and MEG signals [EB / OL]. (2025-05-18) [2026-03-25]. arXiv:2505.18185v1 [eess.SP]. https: / / arxiv.org / abs / 2505.18185.

[0093] In this embodiment, the dynamic brain region spatial convolution module essentially learns the spatial dimension features of neural electrical signals, enabling the model to understand variable-length and semantically different channel signals from different devices, and transforming heterogeneous input signals into homogeneous and unified feature sequences. The dynamic brain region spatial convolution module generates dynamic spatial convolution kernels through a learnable inversion matrix and the positional encoding of each channel sensor, thereby adaptively spatially weighting the input signals and transforming variable-length and semantically different input signals into homogeneous one-dimensional feature sequences.

[0094] In some embodiments, the dynamic brain region spatial convolution module can also employ an attention scheme based on learnable query vectors to achieve the same spatial feature learning function. Specifically, for an input signal with the shape [number of electrode channels, number of sampling points], the input waveform is first windowed and linearly projected, and then summed with the positional encoding corresponding to each electrode channel. This allows the model to acquire the physical meaning of each channel, resulting in features of [number of electrode channels, number of time windows, feature dimension]. Then, a set of learnable query vectors with the shape [number of query vectors, feature dimension] are used together with the aforementioned features for self-attention calculation in the spatial dimension. This allows the query vectors to learn the spatial relationships of neural electrical signals during the attention interaction process and extract key information from each channel. Finally, the features extracted from the query vectors are taken as the calculation result, with the shape [number of query vectors, number of time windows, feature dimension], thereby converting the heterogeneous input signal into a homogeneous feature sequence output.

[0095] In this embodiment, the composite self-supervised learning module essentially completes a pre-training task of learning efficient representations from multimodal neural electrical signals without supervision by setting a pre-supervised task—performing self-supervised training that combines mask prediction and autoregressive prediction on isomorphic single-channel feature sequences, so that the model can learn efficient representations from the data without supervision through this pre-training task.

[0096] In some embodiments, the composite self-supervised learning module can also employ a composite self-supervised learning approach combining mask prediction and contrastive learning to achieve the same unsupervised, efficient representation learning function, specifically as follows:

[0097] The original causal attention structure is replaced with a bidirectional attention structure. For mask prediction tasks, there is no need for prefix, infix, or suffix mask design. The mask is directly applied to the original signal, allowing the model to learn high-quality representations using the contextual information of the neural electrical signals. At the same time, a contrastive learning method is combined: the complete signal is input into the model to obtain the teacher representation, and the signal that has been damaged (e.g., masked, truncated to a shorter time segment, or randomly discarded channels) is input into the model again to obtain the student features. Through contrastive learning, the student features are made to approximate the teacher representation, enabling the model to predict the whole from the local and improving the quality of representation learning.

[0098] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.

Claims

1. A brain signal base processing system for multi-modal neuroelectric signals, characterized by, include: The neural electrical signal abnormality detection module is configured to label bad channels, clean and separate artifacts from the original multimodal neural electrical signals, remove unqualified signal segments, and output qualified multimodal neural electrical signals. The intrabrain and extrabrain unified spatial location coding module is configured to map the multimodal sensors corresponding to the qualified multimodal neural electrical signals to a unified three-dimensional coordinate system, fuse the sensor spatial coordinates and physical properties to generate learnable location codes, and output multimodal neural electrical signals with location codes. The dynamic brain region spatial convolution module is configured to generate a dynamic spatial convolution kernel based on the position-coded multimodal neural electrical signals, adaptively spatially weight the multimodal neural electrical signals, and convert heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences. as well as The composite self-supervised learning module is configured to perform self-supervised training combining mask prediction and autoregressive prediction on the isomorphic single-channel feature sequence, and output a pre-trained brain signal basic model to process multimodal neural electrical signals related to downstream tasks based on the pre-trained brain signal basic model.

2. The brain signal processing system for multi-modal neuroelectric signals according to claim 1, wherein, The neural electrical signal anomaly detection module is based on a residual one-dimensional convolutional network. It performs sliding window detection on the original multimodal neural electrical signals in the time and frequency domains to automatically identify bad channels and severe artifact interference. The neural electrical signal anomaly detection module scores the signal quality to remove unqualified signal segments.

3. The brain signal processing system for multi-modal neuroelectric signals according to claim 1, wherein, The multimodal neural electrical signals include electroencephalography (EEG), magnetoencephalography (MEG), and invasive EEG.

4. The brain signal processing system for multimodal neural electrical signals according to claim 1, characterized in that, In the unified intrabrain and extrabrain spatial location coding module, the multimodal sensors include scalp EEG electrodes, magnetoencephalography (MEG) sensors, and invasive EEG contacts.

5. The brain signal processing system for multimodal neural electrical signals according to claim 4, characterized in that, In the unified intracranial and extracranial spatial location coding module, generating the location code includes the following steps: The multimodal sensors are mapped to a unified MNI standard brain three-dimensional coordinate system; The spatial coordinates of the sensor are encoded using a linear layer, and the sensor type is converted into a learnable code; and After fusing the position and physical property features of the electrodes, a multilayer perceptron is used for feature fusion, and the final position code is obtained after normalization.

6. The brain signal processing system for multimodal neural electrical signals according to claim 1, characterized in that, The dynamic brain region spatial convolution module includes: Electrode position coding; and The learnable inversion matrix, when combined with the electrode position encoding, yields the dynamic spatial convolution kernel.

7. The brain signal processing system for multimodal neural electrical signals according to claim 6, characterized in that, The dynamic brain region spatial convolution module performs matrix multiplication on the inversion matrix and the transpose of the electrode position code, and uses exponential normalization to convert the result into an attention score, thus obtaining a dynamic spatial convolution kernel. The dynamic spatial convolution kernel consists of multiple dynamic spatial filters. Each dynamic spatial convolution kernel learns different weighted filtering methods for sensors located in different brain regions to convert heterogeneous multi-channel neural electrical signals into homogeneous single-channel feature sequences.

8. The brain signal processing system for multimodal neural electrical signals according to claim 1, characterized in that, In the composite self-supervised learning module, mask prediction involves splitting the brain signal sequence into front, middle, and back segments and rearranging them, then predicting the discrete labels of the masked parts based on context using causal attention; autoregressive prediction involves calculating the feature sequence using causal attention and then predicting the probability distribution of the discrete label corresponding to the next position based on each position.

9. The brain signal processing system for multimodal neural electrical signals according to claim 1, characterized in that, The downstream tasks include coarse-grained discrimination tasks and fine-grained generation tasks; the coarse-grained discrimination tasks include disease-assisted diagnosis, emotion state recognition, sleep staging, and motor imagery classification; and the fine-grained generation tasks include language decoding, visual reconstruction, and auditory decoding.

10. A method for basic brain signal processing oriented towards multimodal neural electrical signals, characterized in that, Includes the following steps: S1. Input the raw data of multimodal neural electrical signals into the neural electrical signal abnormality detection module to complete the automatic identification and cleaning of bad channels and artifacts, remove unqualified signal segments, and output qualified multimodal neural electrical signals. S2. Input the qualified multimodal neural electrical signal into the unified spatial coding module inside and outside the brain to complete the three-dimensional coordinate alignment of the multimodal sensor, fuse the sensor spatial coordinates and physical properties to generate a learnable position code, and output the multimodal neural electrical signal with position code. S3. Input the multimodal neural electrical signals with position encoding into the dynamic brain region spatial convolution module to generate a dynamic spatial convolution kernel, and perform adaptive spatial weighting on the heterogeneous multi-channel neural electrical signals to convert them into homogeneous single-channel feature sequences. S4. Input the isomorphic single-channel feature sequence into the composite self-supervised learning module, perform self-supervised training combining mask prediction and autoregressive prediction to complete model pre-training, and output the pre-trained basic model of brain signals; and S5. Use the pre-trained brain signal baseline model to process multimodal neural electrical signals related to downstream tasks.

Images