Brain-computer interface signal enhancement method, device, equipment and storage medium

Abstract

The present application relates to a brain-computer interface signal enhancement method, device, equipment and storage medium. The method includes: respectively forming a multimodal brain data pair with the set conditional modal brain data and each target modal brain data, outputting the heterogeneous spatial features of the multimodal brain data pair through the brain data pre-training feature extraction module, and obtaining the unified spatial features of the aligned multimodal brain data pair using the transformation matrix through the hyper-dimensional alignment module; obtaining the unique hot encoding of the multimodal brain data pair, inputting the unique hot encoding and the aligned unified spatial features into the trained brain data unified representation module, outputting the corresponding cross-modal generation prediction representation through the brain data unified representation module, and outputting the generated target modal brain data through the functional representation decoding module. The present application realizes the effective conversion from low-cost modal data to high-cost modal data by constructing a multimodal brain data generation architecture that supports flexible conversion between arbitrary modalities, thereby improving the versatility of the technology.

Description

Technical Field

[0001] The present application belongs to the technical field of brain science and neuroengineering, and in particular relates to a method, apparatus, device and storage medium for enhancing brain-computer interface signals. Background Art

[0002] The rapid development of brain science and neuroengineering has created an urgent need for a deeper understanding of complex brain functions such as cognition, emotion, decision-making, and behavioral regulation, as well as for advancing the next generation of brain-computer interface (BCI) technology. In this context, multimodal functional neuroimaging, by integrating information from diverse imaging and recording methods, has become an indispensable core tool for revealing the spatiotemporal dynamics of the brain, accurately decoding user intent, and building high-performance BCI systems.

[0003] Currently, the neuroimaging and signal recording technologies available for BCI research are becoming increasingly diverse, such as electrophysiological / magnetophysiological signals such as electrocorticography (ECoG), magnetoencephalography (MEG), stereoelectroencephalography (SEEG), event-related optical signal (EROS), functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), functional photoacoustic imaging (fPAI), magnetic particle imaging (MPI), positron emission tomography (PET), and single photon emission computed tomography (SPECT). Each technique has its own unique advantages and inherent limitations. For example, EEG and MEG can capture rapid neural activity, making them ideal for real-time BCIs. However, their relatively low spatial resolution makes signal tracing challenging. Invasive techniques such as ECoG and SEEG utilize electrodes placed on the cortical surface or deep within the brain, providing signals with both high spatiotemporal resolution and high signal-to-noise ratio. However, their invasive nature restricts their application to specific clinical patients and prevents their widespread adoption in healthy individuals. fNIRS, as a relatively portable and low-cost optical technique, offers an alternative to blood oximetry. However, its low temporal resolution, expensive equipment, and demanding environmental requirements make it unsuitable for real-time BCI applications. Emerging techniques such as nuclear medicine and MPI, PET, and SPECT, which track radioactive tracers to measure brain metabolism and neurotransmitter activity, offer unique insights into the pathophysiology of BCI-related neurological disorders. However, their invasive nature, high cost, and extremely low temporal resolution preclude their application for dynamic BCI signal acquisition.

[0004] While technological diversity presents opportunities for comprehensive brain analysis, it also poses significant challenges to the effective integration and analysis of multimodal brain data in BCI development. Existing BCI applications suffer from high costs for collecting multimodal brain function data, insufficient representation of data from specific populations, and bottlenecks in cross-modal data fusion and generation. These limitations severely restrict the decoding accuracy, robustness, and fairness of BCI systems, primarily in the following areas:

[0005] 1. Different brain signal modalities have very different characteristics, and existing BCI systems have difficulty achieving efficient multimodal information fusion. Existing methods mostly rely on a single modality, making it difficult to effectively fuse while retaining their respective advantages, resulting in information silos. Although the concept of multimodal brain data fusion has been widely recognized, and some feature-level fusion technologies have emerged, the acquisition of original high-quality, multimodal neuroimaging data, especially for modalities such as ECoG that rely on large and expensive equipment, complex operating procedures, or are invasive, remains a fundamental bottleneck due to its high economic cost and strict implementation conditions, which restricts large-scale data collection, multi-center research collaboration, and the promotion of technology to resource-poor areas.

[0006] Second, existing models generally use an end-to-end training approach, which relies heavily on paired datasets. This requires a large amount of paired, multimodal brain data collected synchronously from the same subjects. However, such paired datasets are much rarer and more valuable than single-modal data, making them extremely difficult to obtain. This strong reliance on such data makes existing models not only difficult to train but also prone to overfitting on limited data, resulting in poor generalization. This significantly hinders the development and application of efficient and robust generative models.

[0007] 3. The physiological authenticity of generated data is insufficient. The neural data produced by existing generation methods are often difficult to fully reproduce the complex spatiotemporal dynamics and fine neurophysiological characteristics of real brain activity. For example, the generated signals may differ significantly from the real data in the accuracy of spatial activation patterns, the morphology of blood oxygen response functions, or the correlation between cross-modal neural oscillations. This is mainly due to the failure to effectively align the deep semantic features between heterogeneous modalities, and the insufficient expression ability of the generation model to capture complex data distributions, resulting in limited application value of these generated data in downstream tasks requiring high fidelity.

[0008] Fourth, most current technical solutions focus on one-way generation ("one-to-one") between two specific modalities, and lack a unified framework that is widely compatible with multiple heterogeneous brain data.

[0009] 5. Brain data of specific groups such as patients with specific neuropsychiatric diseases, people in specific age groups or ethnic minorities are often underrepresented or severely missing in existing databases. Existing data augmentation methods often simply copy or interpolate the original data distribution, failing to generate high-quality, diverse datasets that truly reflect the unique neurophysiological patterns of specific groups. BCI decoding models trained based on such unbalanced and biased datasets will have significantly reduced generalization ability and prediction accuracy on new samples or unseen groups, which will not only cause systematic performance bias and damage the fairness and universality of the technology, but is also more likely to produce misleading results in clinical applications. Summary of the Invention

[0010] The present application provides a brain-computer interface signal enhancement method, device, equipment and storage medium, which aims to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0011] In order to solve the above problems, this application provides the following technical solutions:

[0012] A brain-computer interface signal enhancement method, comprising:

[0013] Constructing a multimodal brain function data enhancement model and training the multimodal brain function data enhancement model using a multimodal brain dataset; the multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyperdimensional alignment module, a brain data unified representation module, and a function representation decoding module;

[0014] The set conditional modality brain data and each target modality brain data are respectively combined to form a multimodal brain data pair, and the multimodal brain data pair is input into a trained brain data pre-training feature extraction module, and the brain data pre-training feature extraction module outputs the heterogeneous spatial features of the multimodal brain data pair;

[0015] Inputting the heterogeneous spatial features into a trained brain data hyperdimensional alignment module, the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space, thereby obtaining a unified spatial feature after alignment of the multimodal brain data pairs;

[0016] Obtaining one-hot encoding of the multimodal brain data pairs, inputting the one-hot encoding and the aligned unified spatial features into a trained brain data unified representation module, and outputting a cross-modal generation prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module;

[0017] The cross-modality generated prediction representation is input into a trained functional representation decoding module, and the functional representation decoding module outputs the target modality brain data generated by the conditional modality brain data.

[0018] The technical solution adopted in the embodiment of the present application also includes: using the multimodal brain dataset to train the multimodal brain function data enhancement model, including:

[0019] The brain data pre-training feature extraction module and the function representation decoding module are trained. The training process of the brain data pre-training feature extraction module and the function representation decoding module includes:

[0020] Dividing the multimodal brain dataset into a first training set and a first validation set according to a first set ratio, and setting training parameters such as a first number of training times;

[0021] Preprocess the first training set to obtain preprocessed data , the data Input the brain data pre-training feature extraction module’s input layer to obtain the initial features of each modality’s brain data ;

[0022] According to the characteristics of each modality of brain data, the temporal attention main network of the brain data pre-training feature extraction module is adjusted separately Dimension , and respectively convert the initial features of each modality brain data Input the adjusted temporal attention main network for feature extraction to obtain the heterogeneous spatial features of each modality brain data ;

[0023] The heterogeneous spatial features are input into the brain data hyperdimensional alignment module to obtain the aligned unified spatial features. ;

[0024] The unified spatial features Input the brain data unified representation module, and generate the cross-modal generative prediction representation of the corresponding modality through the brain data unified representation module ;

[0025] Generate the cross-modal prediction representation Dimensionality expansion input layer of the input feature representation decoding module , obtain the high-dimensional predictive coding of the target modality brain data ;

[0026] The high-dimensional predictive coding After the semantic decoding layer, the generated target modality brain data is obtained ;

[0027] The preprocessed data For supervision, the parameters of the brain data pre-trained feature extraction module and the functional representation decoding module are updated according to the following formula:

[0028]

[0029] in, represents the target modality brain data;

[0030] The multimodal brain function data enhancement model is iteratively trained according to the first training times and verified using the first verification set. After the iteration, the optimal parameters are selected according to the verification results as the brain data pre-training feature extraction module and functional representation decoding module corresponding to each modality type.

[0031] The technical solution adopted in the embodiment of the present application further includes: using the multimodal brain dataset to train the multimodal brain function data enhancement model, and further includes:

[0032] The brain data hyperdimensional alignment module is trained; the training process of the brain data hyperdimensional alignment module includes: constructing a brain data hyperdimensional alignment module using physiological prior knowledge of all conditional modal brain data and target modal brain data, inputting the heterogeneous spatial features output by the brain data pre-training feature extraction module into the brain data hyperdimensional alignment module, and the brain data hyperdimensional alignment module designs exclusive transformation matrices for the heterogeneous spatial features of each modal brain data based on the physiological prior knowledge of detecting brain data activities of different modal types. And map the heterogeneous spatial features to the preset unified space through interpolation or average pooling method to obtain the aligned unified spatial features :

[0033]

[0034] in, is a nonlinear activation function, is the bias term.

[0035] The technical solution adopted in the embodiment of the present application further includes: using the multimodal brain dataset to train the multimodal brain function data enhancement model, and further includes:

[0036] The brain data unified representation module is trained. The training process of the brain data unified representation module includes:

[0037] Set conditional modal brain data, input the conditional modal brain data into the brain data pre-training feature extraction module of the corresponding modality type, and obtain the unique hot encoding of the conditional modal brain data ;

[0038] Input the brain data of each target modality into the brain data pre-training feature extraction module under the corresponding modality type to obtain the unique hot encoding of each target modality brain data ;

[0039] obtaining a training data pair consisting of the conditional modality brain data and each target modality brain data, preprocessing the training data pair, dividing the training data pair into a second training set and a second validation set according to a second set ratio, and setting a second number of training times;

[0040] The second training set is input into the brain data hyperdimensional alignment module to obtain the unified spatial features of each training data pair in the second training set. and ;in, is the unified spatial feature of conditional modality brain data, is the unified spatial feature of the target modal data;

[0041] The conditional modality brain data is one-hot encoded , one-hot encoding of brain data of each target modality and unified spatial features and Input the conditional input layer of the unified representation module of brain data respectively to obtain the comprehensive conditional coding ;

[0042] Randomly sampling the unified spatial features, is the noise level of random sampling, and the comprehensive condition is encoded and noise levels Input the temporal attention network of the unified representation module of brain data , get the Step prediction noise , and use a denoising algorithm to reduce the predicted noise Perform denoising to obtain the denoised prediction representation ;

[0043] The unified brain data representation module is iteratively trained according to the second training times, and is verified using the second verification set after each iterative training. After the iteration, the optimal parameters are selected according to the verification results as the final unified brain data representation module corresponding to each modality type.

[0044] The technical solution adopted in the embodiment of the present application further includes: after randomly sampling the unified spatial features, it also includes:

[0045] Update the parameters of the brain data unified representation module according to the following formula:

[0046]

[0047] in, is the noise level of random sampling, 1 to Integers between For the preset total number of diffusion steps, from the standard Gaussian distribution Randomly sampled noise vector in .

[0048] The technical solution adopted in the embodiment of the present application also includes: using a denoising algorithm to denoise the predicted noise Perform denoising, specifically:

[0049]

[0050] The technical solution adopted in the embodiment of the present application also includes: the brain data modality types in the multimodal brain data pair include EEG, ECoG, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET, and SPECT; the conditional modality brain data is EEG; the target modality brain data includes ECoG, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET, and SPECT; the target modality brain data output by the functional characterization decoding module includes ECoG brain data , MEG brain data , SEEG brain data , EROS brain data , FMRI brain data , FNIRS brain data , FPAI brain data , MPI brain data , PET brain data and SPECT brain data .

[0051] Another technical solution adopted in the embodiment of the present application is: a brain-computer interface signal enhancement device, comprising:

[0052] Model training module: used to construct a multimodal brain function data enhancement model and train the multimodal brain function data enhancement model using a multimodal brain dataset; the multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyperdimensional alignment module, a brain data unified representation module, and a function representation decoding module;

[0053] A brain data pre-training feature extraction module is used to form multimodal brain data pairs with the set conditional modality brain data and the target modality brain data, and input the multimodal brain data pairs into the trained brain data pre-training feature extraction module, and output the heterogeneous spatial features of the multimodal brain data pairs through the brain data pre-training feature extraction module;

[0054] Brain data hyperdimensional alignment module: used to input the heterogeneous spatial features into the trained brain data hyperdimensional alignment module, and the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space to obtain the unified spatial features of the aligned multimodal brain data;

[0055] A brain data unified representation module is configured to obtain one-hot encodings of the multimodal brain data pairs, input the one-hot encodings and aligned unified spatial features into the trained brain data unified representation module, and output a cross-modal generation prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module;

[0056] Functional representation decoding module: used to input the cross-modal generated prediction representation into the trained functional representation decoding module, and output the target modality brain data generated by the conditional modality brain data through the functional representation decoding module.

[0057] Another technical solution adopted by the embodiment of the present application is: a device, the device comprising a processor and a memory coupled to the processor, wherein:

[0058] The memory stores program instructions for implementing the brain-computer interface signal enhancement method;

[0059] The processor is used to execute the program instructions stored in the memory to control the brain-computer interface signal enhancement method.

[0060] Another technical solution adopted in the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the brain-computer interface signal enhancement method.

[0061] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the brain-computer interface signal enhancement method, device, equipment and storage medium of the embodiments of the present application construct a multimodal brain data generation architecture that supports flexible conversion between arbitrary modalities. The generation architecture integrates an end-to-end system of four modules: feature extraction, alignment, unified representation generation and decoding, and realizes the effective conversion from low-cost modal data to high-cost modal data. Ultimately, high-quality target modal data that can truly reflect brain activity is generated, which greatly improves the versatility of the technology. It can also synthesize missing high-quality and diverse brain data for underrepresented groups, thereby expanding and balancing the training set, effectively correcting model bias, and significantly improving the fairness and generalization ability of brain-computer interface decoding. This application not only supports flexible data enhancement and generation, but also provides a powerful platform for the development of more accurate and adaptable BCI systems and the exploration of new neural biomarkers. It aims to significantly reduce the application threshold of high-cost brain data in the BCI field and maximize its scientific research and clinical value. It provides a powerful data generation and enhancement tool for neuroscience research, clinical auxiliary diagnosis and advanced BCI development, showing a more direct and broader translation application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a method for enhancing brain-computer interface signals according to an embodiment of the present application;

[0063] Figure 2 This is a schematic diagram of the multimodal brain function data enhancement model framework of an embodiment of the present application;

[0064] Figure 3 This is a schematic diagram of the training process of the unified brain data representation module in an embodiment of the present application;

[0065] Figure 4 Radar chart for comparing performance of downstream brain decoding tasks;

[0066] Figure 5 This is a schematic structural diagram of a brain-computer interface signal enhancement device according to an embodiment of the present application;

[0067] Figure 6 This is a schematic diagram of the device structure of an embodiment of the present application;

[0068] Figure 7 A schematic diagram of the structure of the storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0070] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features identified. Therefore, features identified as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional designations in the embodiments of this application (such as up, down, left, right, front, back, etc.) are intended only to illustrate the relative positional relationships and movement of components in a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional designations will also change accordingly. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to such process, method, product, or apparatus.

[0071] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] Specifically, see Figure 1 , is a flow chart of the brain-computer interface signal enhancement method of the embodiment of the present application. The brain-computer interface signal enhancement method of the embodiment of the present application comprises the following steps:

[0073] S100: Constructing a multimodal brain function data enhancement model;

[0074] In this step, the multimodal brain function data enhancement model constructed includes a brain data pre-training feature extraction module, a brain data hyper-dimensional alignment module, a brain data unified representation module, and a function representation decoding module. Figure 2The figure shows a schematic diagram of the framework of the multimodal brain function data enhancement model of the embodiment of the present application. Among them, the brain data pre-training feature extraction module adopts a decoupled pre-training strategy, and uses its large-scale single-modal data set for training for each modality of brain data to improve the ability to extract representative neural activity features from the original, high-dimensional signals, and solve the technical problems of unstable training and insufficient feature extraction caused by the strong dependence of the existing end-to-end model on rare "paired" multimodal brain data sets. The brain data super-dimensional alignment module uses the physiological prior knowledge about the intrinsic correlation between different modal brain data to design an exclusive transformation matrix for each modal data, map the heterogeneous data to a unified representation space, generate aligned multimodal features, solve the "semantic gap" or modal heterogeneity problem caused by the different physical principles, spatiotemporal resolution and data structure of different modal brain data, improve the quality of data generation, and ensure the authenticity of the generated data. The unified representation module of brain data jointly distributes the aligned multimodal features in a unified space, and adopts a conditional diffusion model internally. During the data generation process, it fuses the complementary information of multiple source modal data as conditions to achieve accurate reconstruction of the temporal dynamics and fine texture of the target modality brain data. This solves the technical problems of conversion distortion, information loss and pattern mismatch caused by the lack of a common "language" in cross-modal generation of current technical solutions, and makes it difficult to support reliable generation between multiple heterogeneous modalities, thereby solving the limitations of the model in flexible cross-modal conversion. The functional representation decoding module uses the learned universal predictive representation to generate high-quality and diverse supplementary data for special groups that are underrepresented, solving the problem that existing data enhancement methods cannot generate brain data that reflects the unique neurophysiological patterns of the group, and realizing cross-modal data generation from low-cost data to high-cost data.

[0075] S110: Acquire a multimodal brain dataset consisting of conditional modality brain data and target modality brain data, and use the multimodal brain dataset to train a brain data pre-training feature extraction module, a brain data super-dimensional alignment module, a brain data unified representation module, and a functional representation decoding module, respectively, to obtain trained brain data pre-training feature extraction modules, brain data super-dimensional alignment modules, brain data unified representation modules, and functional representation decoding modules corresponding to each modality type;

[0076] In this step, the brain data modality types in the multimodal brain data set include but are not limited to EEG, ECoG, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET, SPECT, etc., the conditional modality brain data is any one of all modality types, and the target modality brain data is brain data of other modal types except the conditional modality brain data. For example, if the conditional modality brain data is set to EEG, then the target modality brain data is ECoG, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET and SPECT. Based on the acquired multimodal brain data set, the training processes of the brain data pre-training feature extraction module, the brain data hyperdimensional alignment module, the brain data unified representation module and the functional representation decoding module are as follows:

[0077] The brain data pre-training feature extraction module and the function characterization decoding module are trained. The training process of the brain data pre-training feature extraction module and the function characterization decoding module in the embodiment of the present application includes the following steps:

[0078] Step 1-1: Divide the multimodal brain dataset into a first training set and a first validation set according to a first set ratio, and set training parameters such as a first training number;

[0079] Step 1-2: Preprocess the first training set to obtain preprocessed data , the data Input the brain data pre-training feature extraction module’s input layer to obtain the initial features of each modality’s brain data ;

[0080] Steps 1-3: Adjust the temporal attention main network of the brain data pre-training feature extraction module according to the characteristics of each modality of brain data Dimension , and the initial features of each modality brain data are Input the adjusted temporal attention main network for feature extraction to obtain the heterogeneous spatial features of each modality brain data :

[0081] (1)

[0082] Finally, the brain data pre-training feature extraction module extracts the heterogeneous spatial feature set of all modal brain data ,in, is the number of modal types in the multimodal brain dataset.

[0083] Step 1-4: Transform heterogeneous spatial features Input brain data into the hyperdimensional alignment module to obtain the unified spatial features after alignment ;

[0084] Steps 1-5: Unify spatial features Input the brain data unified representation module, and generate the cross-modal generative prediction representation of the corresponding modality through the brain data unified representation module ;

[0085] Steps 1-6: Generate predictive representations across modalities Dimensional expansion input layer of the functional representation decoding module corresponding to the input modality , obtain the high-dimensional predictive coding of the target modality brain data ;

[0086] Step 1-7: Encode the high-dimensional prediction After the semantic decoding layer, the generated target modality brain data is obtained :

[0087] (2)

[0088] in 、 are the weight matrix and bias vector of the semantic decoding layer, is a non-linear activation function.

[0089] Step 1-8: Preprocessed data For supervision, the parameters of the brain data pre-trained feature extraction module and functional representation decoding module are updated according to the following formula:

[0090] (3)

[0091] in, represents the target modality brain data;

[0092] Step 1-9: Repeat steps 1-2 to 1-8, and iteratively train the multimodal brain function data enhancement model according to the set first training times. Use the first validation set for verification in each iteration, and select the optimal parameters as the brain data pre-training feature extraction module and functional representation decoding module corresponding to each modality type based on the verification results after the iteration.

[0093] The brain data hyperdimensional alignment module is trained; the training process of the brain data hyperdimensional alignment module includes: using the physiological prior knowledge of all conditional modal brain data and target modal brain data to construct the brain data hyperdimensional alignment module, and the heterogeneous spatial feature set output by the brain data pre-training feature extraction module Input brain data hyperdimensional alignment module, brain data hyperdimensional alignment module detects the physiological prior knowledge of brain data activity according to different modal types (such as the correspondence between the spatial coordinates of different electrodes in EEG and the spatial coordinates of standard voxels in the preset unified space) as the heterogeneous spatial features of brain data of each modality Design a dedicated transformation matrix , by interpolation or average pooling method, the heterogeneous spatial features are mapped into a preset unified space to obtain the aligned unified spatial features :

[0094] (4)

[0095] in, is a nonlinear activation function, is the bias term. The unified spatial features after alignment have the same embedding dimensions.

[0096] Train the unified representation module of brain data; e.g. Figure 3 FIG. 1 is a schematic diagram of the training process of the unified brain data representation module according to an embodiment of the present application. The specific training process includes the following steps:

[0097] Step 2-1: Set the conditional modal brain data, input the conditional modal brain data into the brain data pre-training feature extraction module under the corresponding modality type, and obtain the unique hot encoding of the conditional modal brain data ; It can be understood that the modality type of the conditional modality brain data can be set according to the actual application scenario. For example, in the embodiment of the present application, EEG is set as the conditional modality brain data;

[0098] Step 2-2: Input the brain data of each target modality into the brain data pre-training feature extraction module under the corresponding modality type to obtain the unique hot encoding of each target modality brain data , expressed as:

[0099]

[0100] It can be understood that the target modality brain data is other modality brain data except the conditional modality brain data. For example, if EEG is set as the conditional modality brain data, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET, SPECT, etc. are used as the target modality brain data to be generated;

[0101] Step 2-3: Obtain training data pairs consisting of the conditional modality brain data and each target modality brain data, preprocess the training data pairs, divide them into a second training set and a second validation set according to a second set ratio, and set training parameters such as the second number of training times;

[0102] Step 2-4: Input the second training set into the brain data hyperdimensional alignment module under the corresponding modality type to obtain the unified spatial features of each training data pair in the second training set and ;in, is the unified spatial feature of conditional modality brain data, is the unified spatial feature of the target modal data;

[0103] Step 2-5: One-hot encode the conditional modality brain data , one-hot encoding of brain data of each target modality and unified spatial features and Input the conditional input layer of the unified representation module of brain data respectively, and obtain Figure 3 The comprehensive conditional coding shown ;

[0104] Step 2-6: Randomly sample the uniform spatial features to obtain the noise vector sampled in step t and prediction noise , and update the parameters of the brain data unified representation module according to the following formula:

[0105] (5)

[0106] in, is the noise level of random sampling, 1 to Integers between For the preset total number of diffusion steps, from the standard Gaussian distribution Randomly sampled noise vector in .

[0107] Step 2-7: Encode the comprehensive conditions and noise levels Temporal Attention Network for Unified Representation of Brain Data , get the Step prediction noise , and use denoising algorithm to predict noise Perform step-by-step denoising to finally obtain the denoised prediction representation ; Among them, the denoising algorithm formula is:

[0108] (6)

[0109] Step 2-8: Iteratively train the unified brain data representation module according to the second training times, verify it with the second validation set after each iterative training, and select the optimal parameters as the final unified brain data representation module corresponding to each modality type based on the verification results after the iteration.

[0110] S120: The set conditional modality brain data and each target modality brain data are combined into multimodal brain data pairs, and the multimodal brain data pairs are respectively input into the trained brain data pre-training feature extraction module corresponding to each modality type, and the brain data pre-training feature extraction module outputs the heterogeneous spatial features of each modality type;

[0111] In this step, assuming that the conditional modal brain data is EEG, the multimodal brain data pairs composed of the conditional modal brain data and each target modal brain data include: EEG and ECoG multimodal brain data pairs, EEG and MEG multimodal brain data pairs, EEG and SEEG multimodal brain data pairs, EEG and EROS multimodal brain data pairs, EEG and fMRI multimodal brain data pairs, EEG and fNIRS multimodal brain data pairs, EEG and fPAI multimodal brain data pairs, EEG and MPI multimodal brain data pairs, EEG and PET multimodal brain data pairs, and EEG and SPECT multimodal brain data pairs. It can be understood that the extraction process of heterogeneous spatial features is the same as the training process of the brain data pre-training feature extraction module, and will not be repeated in this step.

[0112] S130: Inputting the heterogeneous spatial features into the trained brain data hyperdimensional alignment module, the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space, and obtains a unified spatial feature after alignment of each modality type;

[0113] In this step, it can be understood that the generation process of the unified spatial features is the same as the training process of the brain data hyperdimensional alignment module, and will not be repeated in this step.

[0114] S140: Inputting the one-hot encoding of the conditional modality brain data, the one-hot encoding of the target modality brain data, and the aligned unified spatial features of each multimodal brain data pair into a brain data unified representation module of the corresponding modality type after training, and outputting the cross-modal generation prediction representation corresponding to each multimodal brain data pair after denoising through the brain data unified representation module;

[0115] In this step, the cross-modal prediction representation corresponding to each multimodal brain data pair after denoising output by the brain data unified representation module is expressed as 、 、 、 、 、 、 、 、 as well as ,in, represents the cross-modal generation of predictive representations from EEG to ECoG, represents cross-modal generation of predictive representations from EEG to MEG, represents the cross-modal generation of predictive representations from EEG to SEEG, represents the cross-modal generation of predictive representations from EEG to EROS, represents cross-modal generation of predictive representations from EEG to fMRI, represents cross-modality generation of predictive representations from EEG to fNIRS, represents the cross-modal generation of predictive representations from EEG to fPAI, represents the cross-modal generation of predictive representations from EEG to MPI, represents cross-modal generation of predictive representations from EEG to PET, It represents the cross-modal generation of predictive representations from EEG to SPEC. It can be understood that the generation process of cross-modal generation of predictive representations is the same as the training process of the brain data unified representation module, and will not be repeated in this step.

[0116] S150: Inputting the denoised cross-modality prediction representations of each multimodal brain data pair into the trained functional representation decoding module corresponding to each modality type, and outputting the target modality brain data generated by the conditional modality brain data through the functional representation decoding module;

[0117] In this step, the cross-modal prediction representations generated by denoising each multimodal brain data pair are input into the functional representation decoding module under the corresponding modality type after training. The high-cost invasive target modality brain data generated by the low-cost non-invasive conditional modality brain data is output by the functional representation decoding module and represented as: ECoG brain data , MEG brain data , SEEG brain data , EROS brain data , FMRI brain data , FNIRS brain data , FPAI brain data , MPI brain data , PET brain data and SPECT brain data It can be understood that the generation process of the target modality brain data is the same as the training process of the functional representation decoding module, and will not be repeated in this step.

[0118] Furthermore, this application can be further extended to the broader field of physiological signal processing. Its basic principles are applicable to the conversion and generation of any multi-channel signals that have inherent physiological coupling but heterogeneous data characteristics (such as physical units, spatiotemporal resolution, and data dimensions). For example, this framework can be applied to functional signals closely related to central nervous system activity, such as electrocardiogram (ECG), electrooculogram (EOG), and electromyography (EMG).

[0119] In order to verify the feasibility and effectiveness of the embodiments of the present application, experiments were conducted through specific downstream applications. The experiments took EEG and fMRI as examples to conduct a typical downstream task - Brain Decoding comparison experiment. Taking low-cost electroencephalogram (EEG) data as input, the embodiment of the present application was used to generate the corresponding high-cost functional magnetic resonance imaging (fMRI). The data from different sources (original EEG, original fMRI, fMRI generated by this application, and a combination of generated fMRI and original EEG) were used to decode the brain activity state, evaluate its performance in the classification of specific cognitive tasks, and use four key indicators of accuracy (ACC), precision (PRE), sensitivity (SEN) and F1 score (F1-Score) to measure performance. The experimental results are as follows Figure 4 The following is a radar chart comparing the performance of downstream brain decoding tasks. Figure 4 It can be concluded that using only the fMRI signals generated by the generative framework of this application for decoding significantly outperforms using only the original EEG signals in all four performance metrics. This demonstrates that the generative framework of this application successfully reconstructs high-information fMRI signals from low-cost EEG signals. These signals contain richer spatial feature information than the original EEG, which is crucial for the decoding task. Crucially, when the original EEG signals are combined with the fMRI signals generated by the generative framework of this application as input, the decoding performance is further significantly improved. The results are very close to the upper bound of the performance using real fMRI data in all metrics. Experimental results demonstrate that this application can generate high-quality target modality data containing rich task-related information from low-cost modality data. The supplementary data generated by the generative framework of this application can complement the source data, significantly improving the performance of downstream decoding models. This provides a cost-effective and effective alternative to the acquisition of expensive neuroimaging data, successfully achieving the ultimate goals of this application: to solve core technical problems, lower the application threshold, and improve application performance.

[0120] Based on the above, the brain-computer interface signal enhancement method of the embodiment of the present application constructs a multimodal brain data generation architecture that supports flexible conversion between arbitrary modalities. The generation architecture integrates an end-to-end system of four major modules: feature extraction, alignment, unified representation generation and decoding. It realizes the effective conversion from low-cost modal data to high-cost modal data, and ultimately generates high-quality target modal data that can truly reflect brain activity, greatly improving the versatility of the technology. It can also synthesize missing high-quality and diverse brain data for underrepresented groups, thereby expanding and balancing the training set, effectively correcting model bias, and significantly improving the fairness and generalization ability of brain-computer interface decoding. This application not only supports flexible data enhancement and generation, but also provides a powerful platform for the development of more accurate and adaptable BCI systems and the exploration of new neural biomarkers. It aims to significantly reduce the application threshold of high-cost brain data in the BCI field and maximize its scientific research and clinical value, thereby providing powerful data generation and enhancement tools for neuroscience research, clinical auxiliary diagnosis and advanced BCI development, showing a more direct and broader transformation application prospect.

[0121] See also Figure 5 , is a schematic diagram of the structure of the brain-computer interface signal enhancement device according to an embodiment of the present application. The brain-computer interface signal enhancement method device 40 according to an embodiment of the present application includes:

[0122] Model training module 41: used to construct a multimodal brain function data enhancement model and train the multimodal brain function data enhancement model using a multimodal brain dataset; the multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyperdimensional alignment module, a brain data unified representation module, and a function representation decoding module;

[0123] The brain data pre-training feature extraction module 42 is used to form multimodal brain data pairs with the set conditional modality brain data and the target modality brain data, and input the multimodal brain data pairs into the trained brain data pre-training feature extraction module, and output the heterogeneous spatial features of the multimodal brain data pairs through the brain data pre-training feature extraction module;

[0124] Brain data hyperdimensional alignment module 43: used for inputting the heterogeneous spatial features into the trained brain data hyperdimensional alignment module, and the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space to obtain the unified spatial features of the aligned multimodal brain data pairs;

[0125] A brain data unified representation module 44 is configured to obtain one-hot encodings of the multimodal brain data pairs, input the one-hot encodings and aligned unified spatial features into a trained brain data unified representation module, and output a cross-modal prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module;

[0126] Functional representation decoding module 45: used to input the cross-modal generated prediction representation into a trained functional representation decoding module, and output the target modality brain data generated by the conditional modality brain data through the functional representation decoding module.

[0127] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0128] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0129] See also Figure 6 , is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

[0130] A memory 51 storing executable program instructions;

[0131] a processor 52 connected to the memory 51;

[0132] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: construct a multimodal brain function data enhancement model, and use the multimodal brain data set to train the multimodal brain function data enhancement model; the multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyper-dimensional alignment module, a brain data unified representation module and a function representation decoding module; the set conditional modality brain data and each target modality brain data are respectively combined into a multimodal brain data pair, and the multimodal brain data pair is input into the trained brain data pre-training feature extraction module, and the heterogeneous spatial feature of the multimodal brain data pair is output through the brain data pre-training feature extraction module. Characteristic; input the heterogeneous spatial features into the trained brain data super-dimensional alignment module, and the super-dimensional alignment module uses the transformation matrix to map the heterogeneous spatial features to a preset unified space to obtain the unified spatial features of the multimodal brain data pairs after alignment; obtain the one-hot encoding of the multimodal brain data pairs, input the one-hot encoding and the aligned unified spatial features into the trained brain data unified representation module, and output the cross-modal generation prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module; input the cross-modal generation prediction representation into the trained functional representation decoding module, and output the target modality brain data generated by the conditional modality brain data through the functional representation decoding module.

[0133] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0134] See also Figure 7, which is a structural diagram of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: construct a multimodal brain function data enhancement model, and use a multimodal brain data set to train the multimodal brain function data enhancement model; the multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyper-dimensional alignment module, a brain data unified representation module and a function representation decoding module; the set conditional modality brain data and each target modality brain data are respectively combined to form a multimodal brain data pair, and the multimodal brain data pair is input into the trained brain data pre-training feature extraction module, and the heterogeneous spatial features of the multimodal brain data pair are output through the brain data pre-training feature extraction module. ; Input the heterogeneous spatial features into a trained brain data hyperdimensional alignment module, and the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space to obtain the unified spatial features of the aligned multimodal brain data pairs; obtain the one-hot encoding of the multimodal brain data pairs, input the one-hot encoding and the aligned unified spatial features into a trained brain data unified representation module, and output the cross-modal generation prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module; input the cross-modal generation prediction representation into a trained functional representation decoding module, and output the target modality brain data generated from the conditional modality brain data through the functional representation decoding module. The program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for causing a device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage media include: USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media that can store program instructions, or terminal devices such as computers, servers, mobile phones, and tablets. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0136] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A brain-computer interface signal enhancement method, characterized in that: include: Constructing a multimodal brain function data enhancement model, and training the multimodal brain function data enhancement model using a multimodal brain dataset; The multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyper-dimensional alignment module, a brain data unified representation module, and a function representation decoding module; The set conditional modality brain data and each target modality brain data are respectively combined to form a multimodal brain data pair, and the multimodal brain data pair is input into a trained brain data pre-training feature extraction module, and the brain data pre-training feature extraction module outputs the heterogeneous spatial features of the multimodal brain data pair; Inputting the heterogeneous spatial features into a trained brain data hyperdimensional alignment module, the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space, thereby obtaining a unified spatial feature after alignment of the multimodal brain data pairs; Obtaining one-hot encoding of the multimodal brain data pairs, inputting the one-hot encoding and the aligned unified spatial features into a trained brain data unified representation module, and outputting a cross-modal generation prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module; The cross-modality generated prediction representation is input into a trained functional representation decoding module, and the functional representation decoding module outputs target modality brain data generated from the conditional modality brain data; wherein: The multimodal brain function data enhancement model is trained using the multimodal brain dataset, comprising: The brain data unified representation module is trained. The training process of the brain data unified representation module includes: Set conditional modal brain data, input the conditional modal brain data into the brain data pre-training feature extraction module of the corresponding modality type, and obtain the unique hot encoding of the conditional modal brain data ; Input the brain data of each target modality into the brain data pre-training feature extraction module under the corresponding modality type to obtain the unique hot encoding of each target modality brain data ; obtaining a training data pair consisting of the conditional modality brain data and each target modality brain data, preprocessing the training data pair, dividing the training data pair into a second training set and a second validation set according to a second set ratio, and setting a second number of training times; The second training set is input into the brain data hyperdimensional alignment module to obtain the unified spatial features of each training data pair in the second training set. and ;in, is the unified spatial feature of conditional modality brain data, is the unified spatial feature of the target modal data; The conditional modality brain data is one-hot encoded , one-hot encoding of brain data of each target modality and unified spatial features and Input the conditional input layer of the unified representation module of brain data respectively to obtain the comprehensive conditional coding ; Randomly sampling the unified spatial features, is the noise level of random sampling, and the comprehensive condition is encoded and noise levels Input the temporal attention network of the unified representation module of brain data , get the Step prediction noise , and use a denoising algorithm to reduce the predicted noise Perform denoising to obtain the denoised prediction representation ; The unified brain data representation module is iteratively trained according to the second training times, and is verified using the second verification set after each iterative training. After the iteration, the optimal parameters are selected according to the verification results as the final unified brain data representation module corresponding to each modality type.

2. The brain-computer interface signal enhancement method according to claim 1, characterized in that: The method of training the multimodal brain function data enhancement model using the multimodal brain dataset further includes: The brain data pre-training feature extraction module and the function representation decoding module are trained. The training process of the brain data pre-training feature extraction module and the function representation decoding module includes: Dividing the multimodal brain dataset into a first training set and a first validation set according to a first set ratio, and setting a first training number training parameter; Preprocess the first training set to obtain preprocessed data , the data Input the brain data pre-training feature extraction module’s input layer to obtain the initial features of each modality’s brain data ; According to the characteristics of each modality of brain data, the temporal attention main network of the brain data pre-training feature extraction module is adjusted separately Dimension , and respectively convert the initial features of each modality brain data Input the adjusted temporal attention main network for feature extraction to obtain the heterogeneous spatial features of each modality brain data ; The heterogeneous spatial features are input into the brain data hyperdimensional alignment module to obtain the aligned unified spatial features. ; The unified spatial features Input the brain data unified representation module, and generate the cross-modal generative prediction representation of the corresponding modality through the brain data unified representation module ; Generate the cross-modal prediction representation Dimensionality expansion input layer of the input feature representation decoding module , obtain the high-dimensional predictive coding of the target modality brain data ; The high-dimensional predictive coding After the semantic decoding layer, the generated target modality brain data is obtained ; The preprocessed data For supervision, the parameters of the brain data pre-trained feature extraction module and the functional representation decoding module are updated according to the following formula: in, represents the target modality brain data; The multimodal brain function data enhancement model is iteratively trained according to the first training times and verified using the first verification set. After the iteration, the optimal parameters are selected according to the verification results as the brain data pre-training feature extraction module and functional representation decoding module corresponding to each modality type.

3. The brain-computer interface signal enhancement method according to claim 2, characterized in that: The method of training the multimodal brain function data enhancement model using the multimodal brain dataset further includes: The brain data hyperdimensional alignment module is trained; the training process of the brain data hyperdimensional alignment module includes: constructing a brain data hyperdimensional alignment module using physiological prior knowledge of all conditional modal brain data and target modal brain data, inputting the heterogeneous spatial features output by the brain data pre-training feature extraction module into the brain data hyperdimensional alignment module, and the brain data hyperdimensional alignment module designs exclusive transformation matrices for the heterogeneous spatial features of each modal brain data based on the physiological prior knowledge of detecting brain data activities of different modal types. And map the heterogeneous spatial features to the preset unified space through interpolation or average pooling method to obtain the aligned unified spatial features : in, is a nonlinear activation function, is the bias term.

4. The brain-computer interface signal enhancement method according to claim 3, characterized in that: After randomly sampling the unified spatial features, the method further includes: Update the parameters of the brain data unified representation module according to the following formula: in, is the noise level of random sampling, 1 to Integers between For the preset total number of diffusion steps, from the standard Gaussian distribution Randomly sampled noise vector in .

5. The brain-computer interface signal enhancement method according to claim 4, characterized in that: The noise reduction algorithm is used to reduce the noise of the prediction Perform denoising, specifically: 。 6. The brain-computer interface signal enhancement method according to any one of claims 1 to 5, characterized in that: The brain data modality types in the multimodal brain data pair include EEG, ECoG, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET, and SPECT; the conditional modality brain data is EEG; the target modality brain data includes ECoG, MEG, SEEG, EROS, fMRI, fNIRS, fPAI, MPI, PET, and SPECT; and the target modality brain data output by the functional characterization decoding module includes ECoG brain data. , MEG brain data , SEEG brain data , EROS brain data , FMRI brain data , FNIRS brain data , FPAI brain data , MPI brain data , PET brain data and SPECT brain data .

7. A brain-computer interface signal enhancement device using the brain-computer interface signal enhancement method according to claim 1, characterized in that: include: Model training module: used to build a multimodal brain function data enhancement model and train the multimodal brain function data enhancement model using a multimodal brain dataset; The multimodal brain function data enhancement model includes a brain data pre-training feature extraction module, a brain data hyper-dimensional alignment module, a brain data unified representation module, and a function representation decoding module; A brain data pre-training feature extraction module is used to form multimodal brain data pairs with the set conditional modality brain data and the target modality brain data, and input the multimodal brain data pairs into the trained brain data pre-training feature extraction module, and output the heterogeneous spatial features of the multimodal brain data pairs through the brain data pre-training feature extraction module; Brain data hyperdimensional alignment module: used to input the heterogeneous spatial features into the trained brain data hyperdimensional alignment module, and the hyperdimensional alignment module uses a transformation matrix to map the heterogeneous spatial features to a preset unified space to obtain the unified spatial features of the aligned multimodal brain data; A brain data unified representation module is configured to obtain one-hot encodings of the multimodal brain data pairs, input the one-hot encodings and aligned unified spatial features into the trained brain data unified representation module, and output a cross-modal generation prediction representation corresponding to each multimodal brain data pair through the brain data unified representation module; Functional representation decoding module: used to input the cross-modal generated prediction representation into the trained functional representation decoding module, and output the target modality brain data generated by the conditional modality brain data through the functional representation decoding module.

8. An electronic device, characterized in that: The device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the brain-computer interface signal enhancement method according to any one of claims 1 to 6; The processor is used to execute the program instructions stored in the memory to control the brain-computer interface signal enhancement method.

9. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the brain-computer interface signal enhancement method described in any one of claims 1 to 6.

Images