Brain-computer interface instruction issuing method, device and equipment based on regulation and enhancement simulation

Abstract

The application relates to the technical field of brain-computer interfaces, and provides a brain-computer interface instruction issuing method, device and equipment based on regulation and enhancement simulation, which comprises: a brain electrical decoding model comprising an encoder, a feature enhancer and a task classifier; the encoder encodes real-time brain electrical signals to obtain compressed representations before neural regulation; the feature enhancer performs feature enhancement on the compressed representations to obtain enhanced representations; and the task classifier classifies the enhanced representations to obtain brain electrical decoding results. In the method, the feature enhancer is obtained by joint state discriminator training based on sample brain electrical signals collected before neural regulation and real state labels after neural regulation, the feature enhancer is driven to learn the feature migration relationship between the compressed features before neural regulation and the features after neural regulation, and the feature representation capability of the brain electrical decoding model for brain electrical signals is significantly improved, so that the decoding robustness for weak stimulation signals is enhanced without relying on high-intensity external stimulation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain-computer interface, and particularly relates to a brain-computer interface instruction issuing method, device and equipment based on regulation and enhancement simulation. BACKGROUND

[0002] Brain-Computer Interface (BCI) is a new discipline crossing brain science and information science, which studies how to establish a direct communication and control channel between the brain and external devices to realize information exchange between the brain and the device, so as to realize replacement, recovery, enhancement, complement or improvement of the interaction between the central nervous system and the internal and external environment, and has broad application prospects in medical rehabilitation, intelligent interaction, national defense and military fields. The working process of the BCI system mainly includes three core links: first, the neural electrophysiological signals of the user are collected through invasive or non-invasive technology; second, the user's intention encoded in the brain activity is decoded through signal processing steps such as feature extraction and conversion algorithm; finally, the decoding result is converted into specific device control instructions according to the specific application scene, to realize the direct interaction between the brain and the external environment.

[0003] Steady-State Visual Evoked Potentials (SSVEP) is an exogenous electroencephalogram response. When a person's eyes fix on a periodically flickering visual stimulus, the occipital lobe of the brain will produce neural oscillations synchronized with the stimulus frequency and its harmonic frequencies. Studies have shown that visual stimuli in the range of 1-90Hz can induce SSVEP responses, and the response amplitude induced by low-frequency (4-12Hz) and medium-frequency (12-30Hz) stimuli is significantly higher than that of high-frequency (>30Hz), so these two frequency bands are most widely used in SSVEP-BCI systems. Thanks to the higher response amplitude and Signal-to-Noise Ratio (SNR) of SSVEP, SSVEP-BCI systems have the advantages of large instruction set, high Information Transfer Rate (ITR), etc., and show the best performance in EEG (Electroencephalography)-based BCI systems, and are often applied to online typing interfaces, control function selection interfaces and other brain-computer interface forms.

[0004] Current high-speed SSVEP-BCI systems usually use 8-15.8 Hz as the main encoding frequency band, and the stimulation in this frequency band can easily induce brain responses with high signal-to-noise ratio, which is beneficial to realize high-precision decoding. However, such stimulation is often accompanied by strong flicker, which leads to poor user experience and easily causes visual fatigue, limiting the long-term stable operation of the system. In order to improve user comfort, in recent years, some studies have tried to use peripheral visual field stimulation, high-frequency stimulation and other weak stimulation methods. Although these methods can alleviate discomfort to some extent, they still face the problem of low decoding accuracy due to the low signal-to-noise ratio of the induced brain response and poor command discrimination.

[0005] In summary, although the existing SSVEP-BCI system has made significant progress in transmission rate, it still faces challenges in improving user comfort and system stability, especially in the key technical bottlenecks of insufficient feature extraction ability of weak stimulation signal by decoding model and low command recognition accuracy. SUMMARY

[0006] The application provides a brain-computer interface command issuing method, device and equipment based on regulation and enhancement simulation, to solve the defects of insufficient feature extraction ability of weak stimulation signal by decoding model and low command recognition accuracy in the prior art.

[0007] The application provides a brain-computer interface command issuing method based on regulation and enhancement simulation, comprising the following steps:

[0008] Obtaining real-time electroencephalogram signals of a user;

[0009] Inputting the real-time electroencephalogram signals into an electroencephalogram decoding model to obtain an electroencephalogram decoding result output by the electroencephalogram decoding model;

[0010] Issuing a command based on the electroencephalogram decoding result;

[0011] The electroencephalogram decoding model comprises an encoder, a feature enhancer and a task classifier; the encoder is used to encode the real-time electroencephalogram signals to obtain compressed representations before neural regulation; the feature enhancer is used to enhance the features of the compressed representations to obtain enhanced representations; and the task classifier is used to classify the enhanced representations to obtain the electroencephalogram decoding result.

[0012] The feature enhancer is trained based on sample electroencephalogram signals collected before neural regulation and real state labels after neural regulation, and a state discriminator.

[0013] According to the brain-computer interface command issuing method based on regulation and enhancement simulation provided by the application, the training step of the feature enhancer comprises:

[0014] obtaining an initial electroencephalogram decoding model comprising an initial feature enhancer, and initializing parameters of an initial encoder in the initial electroencephalogram decoding model based on parameters of a pre-training encoder in a pre-training electroencephalogram decoding model; the parameters of the state discriminator are initialized based on parameters of a pre-training state discriminator in the pre-training electroencephalogram decoding model;

[0015] inputting the sample electroencephalogram signal collected before the neural regulation into the initial electroencephalogram decoding model, encoding the sample electroencephalogram signal by the initial encoder to obtain a sample compressed representation before the neural regulation, enhancing the sample compressed representation by the initial feature enhancer to obtain a sample enhanced representation, and discriminating the state of the sample enhanced representation by the state discriminator to obtain a state prediction result;

[0016] based on the difference between the state prediction result and the true state label, determining a state discrimination loss, and training the initial feature enhancer based on the state discrimination loss to obtain the feature enhancer.

[0017] According to the brain-computer interface instruction issuing method based on the regulation enhancement simulation provided by the application, the obtaining step of the pre-training electroencephalogram decoding model comprises:

[0018] obtaining a sample electroencephalogram signal set comprising a first electroencephalogram signal set collected before the neural regulation and a second electroencephalogram signal set collected after the neural regulation, and an original electroencephalogram decoding model; the original electroencephalogram decoding model comprises an original encoder, an original decoder, an original task classifier and an original state discriminator;

[0019] inputting the sample electroencephalogram signal set into the original electroencephalogram decoding model, encoding the sample electroencephalogram signal set by the original encoder to obtain compressed features, classifying the compressed features by the original task classifier to obtain a predicted class distribution, and predicting the state of the compressed features by the original state discriminator to obtain a state prediction probability;

[0020] based on the difference between the predicted class distribution and the label category corresponding to the sample electroencephalogram signal set, determining a task classification loss;

[0021] based on the difference between the state prediction probability and the true state label corresponding to the sample electroencephalogram signal set, determining a state classification loss;

[0022] based on the task classification loss and the state classification loss, determining a target loss, and training the original electroencephalogram decoding model based on the target loss to obtain the pre-training electroencephalogram decoding model.

[0023] According to the brain-computer interface instruction issuing method based on the regulation and enhancement simulation provided by the application, the target loss is determined based on the task classification loss and the state classification loss, and the target loss comprises:

[0024] The orthogonality constraint loss is determined based on the correlation between the internal dimensions of the compressed features.

[0025] The target loss is determined based on the task classification loss, the state classification loss, and the orthogonality constraint loss.

[0026] According to the brain-computer interface instruction issuing method based on the regulation and enhancement simulation provided by the application, the method further comprises:

[0027] The sample electroencephalogram set is subjected to frequency domain conversion to obtain frequency domain input features;

[0028] The frequency domain input features are subjected to local masking to obtain frequency domain masked features;

[0029] The frequency domain masked features are encoded by the original encoder to obtain the compressed features;

[0030] The compressed features are subjected to frequency spectrum reconstruction by the original decoder to obtain reconstructed features;

[0031] The reconstruction loss is determined based on the difference between the reconstructed features and the frequency domain input features;

[0032] The target loss is determined based on the reconstruction loss, the task classification loss, the state classification loss, and the orthogonality constraint loss.

[0033] According to the brain-computer interface instruction issuing method based on the regulation and enhancement simulation provided by the application, the encoder is a harmonic alignment encoder;

[0034] The harmonic alignment encoder comprises a fundamental frequency feature extraction module, a multiple frequency feature extraction module, and a self-attention feature mining module; the fundamental frequency feature extraction module is used to extract the fundamental frequency features from the real-time electroencephalogram; the multiple frequency feature extraction module is used to extract the multiple frequency features from the real-time electroencephalogram; and the self-attention feature mining module is used to perform self-attention calculation based on the fundamental frequency features and the multiple frequency features to obtain the compressed features.

[0035] According to the brain-computer interface instruction issuing method based on the regulation and enhancement simulation provided by the application, the fundamental frequency feature extraction module performs convolution calculation on the fundamental frequency part of the real-time electroencephalogram by using a first convolution kernel with a first expansion rate to obtain the fundamental frequency features;

[0036] The frequency multiplication feature extraction module obtains the frequency multiplication feature by performing convolution calculation on the frequency multiplication part of the real-time electroencephalogram signal by using a second convolution kernel with a second expansion rate.

[0037] The size of the second convolution kernel on the frequency axis is greater than the first convolution kernel, and the second expansion rate is greater than the first expansion rate.

[0038] The application also provides a brain-computer interface instruction issuing device based on regulation enhancement simulation, comprising the following units:

[0039] An acquisition unit is configured to acquire a real-time electroencephalogram signal of a user.

[0040] An input unit is configured to input the real-time electroencephalogram signal into an electroencephalogram decoding model to obtain an electroencephalogram decoding result output by the electroencephalogram decoding model.

[0041] An instruction issuing unit is configured to issue an instruction based on the electroencephalogram decoding result.

[0042] The electroencephalogram decoding model comprises an encoder, a feature enhancer and a task classifier. The encoder is configured to encode the real-time electroencephalogram signal to obtain a compressed representation before neural regulation. The feature enhancer is configured to enhance the feature of the compressed representation to obtain an enhanced representation. The task classifier is configured to classify the enhanced representation to obtain the electroencephalogram decoding result.

[0043] The feature enhancer is trained based on a sample electroencephalogram signal collected before neural regulation and a real state label after neural regulation, and a state discriminator.

[0044] The application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor.

[0045] The application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the brain-computer interface instruction issuing method based on regulation enhancement simulation.

[0046] The application also provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the brain-computer interface instruction issuing method based on regulation enhancement simulation.

[0047] The application provides a brain-computer interface instruction issuing method based on regulation enhancement simulation, a device and equipment. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the application or prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0049] Figure 1 FIG. 1 is a flowchart of the brain-computer interface instruction issuing method based on regulation enhancement simulation provided by the application.

[0050] Figure 2 FIG. 4 is a flowchart of the acquisition of the sample electroencephalogram signal set.

[0051] Figure 3 FIG. 7 is a training schematic diagram of the pre-trained electroencephalogram decoding model.

[0052] Figure 4 FIG. 8 is a training schematic diagram of the electroencephalogram decoding model.

[0053] Figure 5 FIG. 9 is a schematic diagram of the harmonic alignment encoder.

[0054] Figure 6 FIG. 11 is a schematic diagram of the overall architecture of the electroencephalogram signal processing system.

[0055] Figure 7 FIG. 13 is a structural schematic diagram of the brain-computer interface instruction issuing device based on regulation enhancement simulation provided by the application.

[0056] Figure 8Fig. 1 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection scope of the present application.

[0058] The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" and the like are generally of a kind.

[0059] In related technologies, in recent years, it has been proved that the neuromodulation technology can effectively enhance the neural response of the visual cortex of the brain, and shows important potential in improving the performance of BCI under weak stimulation conditions. However, the neuromodulation experiment usually needs expensive equipment and strict safety standards, and the technical threshold, time and economic cost limit its popularization in daily brain-computer interface applications.

[0060] Figure 1 Fig. 1 is a structural schematic diagram of an electronic device provided by the present application. Figure 1 As shown in the figure, the method comprises steps 110, 120 and 130.

[0061] Step 110, acquiring real-time electroencephalogram of a user.

[0062] Specifically, in one specific implementation scenario, the present application can be applied to a steady-state visual evoked potential brain-computer interface system. First, the real-time electroencephalogram of the user can be acquired, wherein the real-time electroencephalogram refers to the electrical signal reflecting the neural activity of the brain of the user collected in real time from the scalp of the user by an electroencephalogram acquisition device, for example, an electroencephalogram cap with multiple electrodes. For example, when the user gazes at a visual stimulation target flickering at a specific frequency on the computer screen, the visual cortex of the brain will produce a response synchronized with the stimulation frequency and its harmonic frequency, that is, the SSVEP signal, which is a typical electroencephalogram signal.

[0063] Here, after obtaining the real-time brain electrical signals of the user, a series of preprocessing operations can be performed on the real-time brain electrical signals, such as filtering operations to remove power frequency interference and baseline drift, bad lead interpolation, downsampling, independent component analysis (ICA), and the like, which are not specifically limited by the embodiments of the present application.

[0064] The independent component analysis is used to remove artifacts such as eye movement and electromyogram.

[0065] Step 120, inputting the real-time brain electrical signals into the electroencephalogram decoding model to obtain an electroencephalogram decoding result output by the electroencephalogram decoding model;

[0066] Step 130, issuing an instruction based on the electroencephalogram decoding result.

[0067] The electroencephalogram decoding model comprises an encoder, a feature enhancer, and a task classifier; the encoder is configured to encode the real-time brain electrical signals to obtain compressed representations before neural regulation; the feature enhancer is configured to perform feature enhancement on the compressed representations to obtain enhanced representations; and the task classifier is configured to classify the enhanced representations to obtain the electroencephalogram decoding result.

[0068] The feature enhancer is trained based on sample brain electrical signals collected before neural regulation and real state labels after neural regulation, and a state discriminator.

[0069] Specifically, after obtaining the real-time brain electrical signals of the user, the real-time brain electrical signals can be input into the electroencephalogram decoding model to obtain an electroencephalogram decoding result output by the electroencephalogram decoding model.

[0070] The electroencephalogram decoding model comprises an encoder, a feature enhancer, and a task classifier; the encoder is configured to encode the real-time brain electrical signals to obtain compressed representations before neural regulation; the feature enhancer is configured to perform feature enhancement on the compressed representations to obtain enhanced representations; and the task classifier is configured to classify the enhanced representations to obtain the electroencephalogram decoding result. The electroencephalogram decoding model is used to map complex real-time brain electrical signals with low signal-to-noise ratio to clear instruction categories that can be understood by a computer. The electroencephalogram decoding result output by the electroencephalogram decoding model is the recognition result of the user's intention, for example, in 40 flashing targets, the electroencephalogram decoding model determines that the user is staring at the 5th target, and the electroencephalogram decoding result is "target 5".

[0071] Here, the role of the encoder is to extract and encode the features of the input real-time electroencephalogram signal, compress the real-time electroencephalogram signal from a high-dimensional original signal space to a low-dimensional, more information-intensive feature space, and obtain a compressed representation before neural regulation. Here, before neural regulation refers to the natural state of the user's electroencephalogram signal without physical neural regulation, such as transcranial magnetic stimulation (TMS), in the actual use stage.

[0072] Among them, the compressed representation is a feature that retains the key features in the original electroencephalogram signal that are most relevant to the task, while filtering out part of the noise and irrelevant information. Here, the key features can be frequency and phase information in the SSVEP signal, etc., and the embodiments of the present application do not make specific limitations.

[0073] In specific implementation, the encoder can adopt various neural network structures. The encoder can be a multi-layer convolutional neural network (CNN) with a cascade structure, a deep neural network (DNN), a combination structure of CNN and DNN, or a Transformer, etc., and the embodiments of the present application do not make specific limitations.

[0074] Here, the role of the feature enhancer is to receive the compressed representation before neural regulation from the encoder and perform feature enhancement on the compressed representation to simulate a brain electroencephalogram signal feature similar to that after physical neural regulation, i.e., an enhanced representation. It should be understood that physical neural regulation can enhance the activity of a specific brain area of the user, thereby improving the signal-to-noise ratio and decodability of the electroencephalogram signal. However, it is not realistic to perform physical neural regulation on the user in daily use. The feature enhancer of the present application achieves the gain effect of neural regulation in a virtual or simulated manner by transforming in the feature space.

[0075] Here, the feature enhancer can be a feedforward neural network, a residual network (RS), or other modules capable of learning nonlinear mapping, etc., and the embodiments of the present application do not make specific limitations.

[0076] The enhanced representation can be defined as a compressed representation obtained after a nonlinear transformation in the feature space, which simulates the ideal brain response state. Specifically, the enhanced representation, on the one hand, retains the core task information inherent in the compressed representation before neural regulation, which is directly related to the user's instruction intention, for example, in the SSVEP task, the core task information indicates the target frequency that the user is staring at; on the other hand, the enhanced representation enhances the saliency and distinguishability of these core task information through the mapping of the feature enhancer, while suppressing the interference of noise and irrelevant features, thereby simulating and reflecting the signal state information with high signal-to-noise ratio and high decodability similar to that after physical neural regulation at the feature level. Therefore, the enhanced representation is a high-quality feature that not only contains the user's original intention but also has the feature distribution characteristics under ideal decoding conditions, providing an easier-to-distinguish input for the subsequent task classifier.

[0077] Here, the role of the task classifier is to receive the enhanced representation from the feature enhancer and perform the final classification and discrimination on the enhanced representation, and output the electroencephalogram decoding result. Since the input enhanced representation has simulated high-quality signal features with high signal-to-noise ratio, the task classifier can more accurately and stably complete the classification task. The task classifier is usually composed of one or more fully connected layers (Fully Connected, FC) and a final classification output layer.

[0078] In order to enable the feature enhancer to learn how to simulate the effect of neural regulation, the feature enhancer is trained based on the sample electroencephalogram signals collected before neural regulation and the real state labels after neural regulation, in combination with the joint state discriminator. The joint state discriminator training is an adversarial or generative training strategy. Specifically, in the training phase, an additional auxiliary model, i.e., the state discriminator, is introduced. The state discriminator is trained to distinguish two types of electroencephalogram features: one from the signal before neural regulation (Pre state), and the other from the signal after neural regulation (Post state). The input of the feature enhancer is the feature representation obtained by processing the sample electroencephalogram signals collected before neural regulation by the encoder, and the output of the feature enhancer is the enhanced feature. In the training process, the enhanced feature is assigned a label after neural regulation (Post state) as a supervision signal. The optimization goal of the feature enhancer is to make the enhanced feature output by the feature enhancer be identified as a feature belonging to the post-neural regulation state in the judgment of the state discriminator.

[0079] Through this mechanism, the feature enhancer is guided to learn the mapping relationship from the pre-neural regulation feature to the post-neural regulation feature, thereby simulating the enhancement effect of neural regulation.

[0080] Here, the state discriminator is similar to a binary classifier, used to determine whether a given compressed representation originates from a pre-modulation (Pre) or post-modulation (Post) state. This state discriminator is a simple binary classification network, with the compressed representation output by the encoder as its input. The output is a probability value. , This represents the probability that the representation comes from the Post state. The specific structure of the state discriminator can be several fully connected layers plus a sigmoid activation function; however, this embodiment of the invention does not impose specific limitations on this.

[0081] The steps for acquiring the sample EEG signals are as follows:

[0082] First, a 40-target instruction system was designed based on mid-to-high frequency visual stimulation. Centered on a 50Hz visual stimulus, the system was expanded forward and backward in 0.2Hz increments to form the 40-target instruction system, with a phase interval of 0.5π between adjacent instructions. Users were required to fixate on the response locations according to the on-screen prompts, thereby inducing brain responses under different targets. After completing a set of SSVEP tasks, users underwent transcranial magnetic stimulation (TMS) neuromodulation to enhance their visual attention, thereby improving the signal-to-noise ratio of brain responses in subsequent SSVEP tasks and facilitating the reconstruction of high-quality brain signals in subsequent decoding and reconstruction tasks. After the modulation was completed, SSVEP data of the user under high visual attention levels was collected again.

[0083] Finally, after obtaining the EEG decoding results, commands can be issued based on these results.

[0084] Understandably, the feature enhancer is trained using sample EEG signals acquired before neural modulation and real-state labels after neural modulation, in conjunction with a state discriminator. This allows for virtual enhancement of weak, latent SSVEP signals in new subjects under zero-training conditions, thereby improving decoding accuracy and robustness without additional physical modulation. This approach effectively reduces the user burden, enhances user experience, and ensures long-term stable system operation.

[0085] The method provided by the embodiment of the application comprises an electroencephalogram decoding model, the electroencephalogram decoding model comprises an encoder, a feature enhancer and a task classifier, the encoder is configured to encode real-time electroencephalogram signals to obtain compressed representations before neuromodulation, the feature enhancer is configured to perform feature enhancement on the compressed representations to obtain enhanced representations, and the task classifier is configured to classify the enhanced representations to obtain electroencephalogram decoding results. In the method, the feature enhancer is trained based on sample electroencephalogram signals collected before neuromodulation and real state labels after neuromodulation, in combination with a state discriminator. The training manner drives the feature enhancer to learn a feature migration relationship between the compressed features before neuromodulation and the features after neuromodulation, simulates an enhancement effect of neuromodulation, and can significantly improve the feature representation capability of the electroencephalogram decoding model on the electroencephalogram signals, thereby enhancing the decoding robustness of the system to weak stimulation signals without relying on high-intensity external stimulation, and finally improving the accuracy of brain-computer interface instruction recognition and the overall stability of the system while ensuring user comfort.

[0086] Based on the above embodiment, the training step of the feature enhancer comprises:

[0087] In step 210, an initial electroencephalogram decoding model comprising an initial feature enhancer is obtained, and the parameters of an initial encoder in the initial electroencephalogram decoding model are initialized based on the parameters of a pre-trained encoder in a pre-trained electroencephalogram decoding model; the parameters of the state discriminator are initialized based on the parameters of a pre-trained state discriminator in the pre-trained electroencephalogram decoding model;

[0088] In step 220, the sample electroencephalogram signals collected before neuromodulation are input into the initial electroencephalogram decoding model, the sample electroencephalogram signals are encoded by the initial encoder to obtain sample compressed representations before neuromodulation, the sample compressed representations are subjected to feature enhancement by the initial feature enhancer to obtain sample enhanced representations, and the sample enhanced representations are subjected to state discrimination by the state discriminator to obtain state prediction results.

[0089] In step 230, a state discrimination loss is determined based on the difference between the state prediction results and the real state labels, and the initial feature enhancer is trained based on the state discrimination loss to obtain the feature enhancer.

[0090] Specifically, first, an initial electroencephalogram decoding model comprising an initial feature enhancer is obtained, and the parameters of an initial encoder in the initial electroencephalogram decoding model are initialized based on the parameters of a pre-trained encoder in a pre-trained electroencephalogram decoding model. The parameters of the state discriminator are initialized based on the parameters of a pre-trained state discriminator in the pre-trained electroencephalogram decoding model.

[0091] In addition, the parameters of the initial decoder in the initial electroencephalogram decoding model can be initialized based on the parameters of the pre-training decoder in the pre-training electroencephalogram decoding model, and the parameters of the initial task classifier in the initial electroencephalogram decoding model can be initialized based on the parameters of the pre-training task classifier in the pre-training electroencephalogram decoding model.

[0092] Here, the only difference between the model structure of the initial electroencephalogram decoding model and the pre-training electroencephalogram decoding model is that the initial electroencephalogram decoding model includes an initial feature enhancer between the initial encoder and the initial decoder, that is, the initial electroencephalogram decoding model includes the initial encoder, the initial feature enhancer, the initial task classifier, the initial state discriminator, and the initial decoder. The pre-training electroencephalogram decoding model includes the pre-training encoder, the pre-training decoder, the pre-training task classifier, and the pre-training state discriminator.

[0093] Here, the parameters of the initial feature enhancer can be pre-set or randomly generated, and the embodiments of the present application do not make specific limitations thereto.

[0094] After obtaining the initial electroencephalogram decoding model, the sample electroencephalogram signal collected before neural regulation can be input into the initial electroencephalogram decoding model, the sample electroencephalogram signal is encoded by the initial encoder to obtain a sample compressed representation before neural regulation, the sample compressed representation is enhanced by the initial feature enhancer to obtain a sample enhanced representation, and the state discriminator discriminates the sample enhanced representation to obtain a state prediction result. Here, the state prediction result is a probability value between 0 and 1, indicating the confidence that the sample enhanced representation belongs to the state after neural regulation (Post state).

[0095] Finally, after obtaining the state prediction result, the state discrimination loss can be determined based on the difference between the state prediction result and the true state label, and the initial feature enhancer can be trained based on the state discrimination loss to obtain the feature enhancer.

[0096] It can be understood that the greater the difference between the state prediction result and the true state label, the greater the state discrimination loss; the smaller the difference between the state prediction result and the true state label, the smaller the state discrimination loss.

[0097] Here, the true state label can be set to 1, the state discrimination loss can be a binary classification cross-entropy loss, and the formula of the state discrimination loss is as follows:

[0098]

[0099] wherein, represents the state discrimination loss, represents the true state label, represents the state prediction result, that is, the output probability of the state discriminator.

[0100] It should be noted that when calculating the gradient by back propagation, the parameters of the initial encoder, the initial decoder, the initial task classifier and the state discriminator are kept unchanged (frozen) in the embodiment of the application, and only the gradient is used to update the parameters of the initial feature enhancer. Through continuous iteration of this process, the initial feature enhancer will be forced to learn how to adjust its mapping function, so that the sample enhanced representation output by it is closer and closer to the real neuro-modulated features in the feature distribution, so that the state discriminator is more and more difficult to distinguish between true and false, and finally the state discriminator loss converges. After training, the initial feature enhancer with optimized parameters is the final required feature enhancer.

[0101] The method provided by the embodiment of the application acquires an initial electroencephalogram decoding model including an initial feature enhancer, and initializes the parameters of the initial encoder in the initial electroencephalogram decoding model based on the parameters of the pre-training encoder in the pre-training electroencephalogram decoding model; the parameters of the state discriminator are initialized based on the parameters of the pre-training state discriminator in the pre-training electroencephalogram decoding model, which provides a high-quality starting point for the initial electroencephalogram decoding model training and accelerates the convergence of the initial electroencephalogram decoding model; at the same time, the feature enhancer is directly guided to learn the mapping from the feature space before neuro-modulation to the feature space after neuro-modulation through the adversarial signal provided by the state discriminator. This mechanism ensures that the feature enhancer can efficiently and accurately learn the ability to simulate the effect of neuro-modulation, thereby significantly improving the feature extraction performance and decoding robustness of the final electroencephalogram decoding model in a real scenario.

[0102] Based on the above embodiment, the step of acquiring the pre-training electroencephalogram decoding model comprises:

[0103] Step 310: acquiring a sample electroencephalogram signal set including a first electroencephalogram signal set collected before neuro-modulation and a second electroencephalogram signal set collected after neuro-modulation, and an original electroencephalogram decoding model; the original electroencephalogram decoding model includes an original encoder, an original decoder, an original task classifier and an original state discriminator;

[0104] Step 320: inputting the sample electroencephalogram signal set into the original electroencephalogram decoding model, encoding the sample electroencephalogram signal set by the original encoder to obtain compressed features, classifying the compressed features by the original task classifier to obtain a predicted class distribution, and predicting the state of the compressed features by the original state discriminator to obtain a state prediction probability;

[0105] Step 330: determining a task classification loss based on the difference between the predicted class distribution and the label category corresponding to the sample electroencephalogram signal set;

[0106] Step 340, determining a state classification loss based on the difference between the state prediction probability and the true state label corresponding to the sample electroencephalogram signal set;

[0107] Step 350, determining a target loss based on the task classification loss and the state classification loss, and training the original electroencephalogram decoding model based on the target loss to obtain the pre-trained electroencephalogram decoding model.

[0108] Specifically, first, a sample electroencephalogram signal set including a first electroencephalogram signal set collected before neuromodulation and a second electroencephalogram signal set collected after neuromodulation can be obtained, and an original electroencephalogram decoding model can be obtained.

[0109] Among them, the first electroencephalogram signal set can be represented by EEG-Pre data, that is, the SSVEP electroencephalogram data collected by the user before using TMS and other neuromodulation methods. The second electroencephalogram signal set can be represented by EEG-Post data, that is, the electroencephalogram signal set collected by the same user after receiving TMS modulation and in the case of improved attention and other states. The first electroencephalogram signal set and the second electroencephalogram signal set constitute the sample electroencephalogram signal set.

[0110] In an optional embodiment, Figure 2 is the acquisition process diagram of the sample electroencephalogram signal set provided by the present application, as Figure 2 shown, a 40-target high-frequency SSVEP visual stimulation interface is designed using an SSVEP neuromodulation data set. Six groups (blocks) of SSVEP data are collected, each group collects 40 trials, of which 40 targets appear once in a random order, and each trial contains 0.5s prompt + 6s visual stimulation + 0.5s rest. The first electroencephalogram signal set is denoted as EEG-Pre. Then, the subject is subjected to rTMS neuromodulation, 5 Hz rTMS is used to stimulate the left brain V1 of the subject, a total of 1200 pulses are applied, and the signal-to-noise ratio of the subsequent SSVEP response is improved by improving the visual attention of the subject. After the rTMS stimulation is completed, the above-mentioned SSVEP task state data acquisition process is repeated again, and the process is consistent with Pre. The second electroencephalogram signal set is denoted as EEG-Post.

[0111] Among them, the original electroencephalogram decoding model is a complete model architecture for pre-training, and the parameters thereof are usually randomly initialized at the beginning of training. The original electroencephalogram decoding model includes an original encoder, an original decoder, an original task classifier, and an original state discriminator.

[0112] After obtaining the sample electroencephalogram signal set, the sample electroencephalogram signal set can be input into the original electroencephalogram decoding model, and the original encoder is used to encode the sample electroencephalogram signal set to obtain compressed features.

[0113] To guide the compressed features To more effectively capture task features related to stimulus categories in SSVEP signals, this embodiment of the invention introduces a parallel-trained original task classifier. This original task classifier uses... As input, output predicted class distribution ,in This represents the number of frequency categories for the SSVEP task. The original task classifier is jointly trained with the original encoder, and backpropagation prompts the encoder to extract latent features that are discriminative for the SSVEP task category, thereby improving the accuracy of downstream instruction recognition.

[0114] The original state discriminator performs state prediction on the compressed features to obtain the state prediction probability.

[0115] Here, the original encoder can be a cascaded multilayer convolutional neural network, a deep neural network, a combination of CNN and DNN, or a Transformer, etc. The embodiments of the present invention do not specifically limit this.

[0116] Here, the specific structure of the original state discriminator can be several fully connected layers plus a sigmoid activation function, and the embodiments of the present invention do not specifically limit this.

[0117] After obtaining the predicted category distribution and state prediction probabilities, the task classification loss can be determined based on the difference between the predicted category distribution and the label categories corresponding to the sample EEG signal set. Here, the task classification loss is the categorical cross-entropy loss, and the formula for the task classification loss is as follows:

[0118]

[0119] in, Indicates task classification loss. This indicates the label category corresponding to the sample EEG signal set, i.e., the true one-hot encoded label. This indicates the predicted class distribution. For example, if there are 40 SSVEP targets, then the actual one-hot encoded labels would be 0, 1, 2, 3, ... 39, a total of forty classes.

[0120] It is understandable that the greater the difference between the predicted category distribution and the label category corresponding to the sample EEG signal set, the greater the task classification loss; the smaller the difference between the predicted category distribution and the label category corresponding to the sample EEG signal set, the smaller the task classification loss.

[0121] Furthermore, the state classification loss can be determined based on the difference between the predicted state probability and the true state label corresponding to the sample EEG signal set. The formula for the state classification loss is as follows:

[0122]

[0123] in, Represents the state classification loss. Indicates the actual state label. This represents the probability of state prediction.

[0124] It is understandable that the greater the difference between the state prediction probability and the true state label corresponding to the sample EEG signal set, the greater the state classification loss; the smaller the difference between the state prediction probability and the true state label corresponding to the sample EEG signal set, the smaller the state classification loss.

[0125] Finally, after obtaining the task classification loss and the state classification loss, the target loss can be determined based on the task classification loss and the state classification loss. The original EEG decoding model can then be trained based on the target loss, and the trained original EEG decoding model can be used as a pre-trained EEG decoding model.

[0126] Here, the target loss can be determined based on the sum of the task classification loss and the state classification loss, or it can be determined based on the weighted sum of the task classification loss and the state classification loss. This embodiment of the invention does not specifically limit the method.

[0127] It should be noted that the methods described in the above embodiments are not limited to the SSVEP paradigm, but can also be applied to other brain-computer interface paradigms, such as motor imagery (MI) and event-related potentials (P300). As long as two different states, such as low concentration / high concentration and fatigue / awake state, can be collected, the same methods as those in the embodiments of this invention can be used for model training and command issuance.

[0128] The method provided in this invention, by combining task classification loss and state classification loss to train the original EEG decoding model for multiple tasks, effectively solves the problems of insufficient model feature learning and weak generalization ability in single-task training. Task classification loss ensures that the pre-trained EEG decoding model masters basic EEG command decoding capabilities, while state classification loss forces the pre-trained EEG decoding model to distinguish the intrinsic differences between signals before and after neural modulation at the feature level, guiding the original encoder to extract general features that are sensitive to and discriminative of changes in neural modulation state. This multi-task collaborative optimization mechanism ensures that the final pre-trained encoder and pre-trained state discriminator contain high-quality, robust feature prior knowledge, laying a solid foundation for the efficient training of subsequent feature enhancers, thereby improving the overall performance and stability of the EEG decoding system.

[0129] Based on the above embodiments, step 350, determining the target loss based on the task classification loss and the state classification loss, includes:

[0130] Step 351: Determine the orthogonality constraint loss based on the correlation between the internal dimensions of the compressed features;

[0131] Step 352: Determine the target loss based on the task classification loss, the state classification loss, and the orthogonality constraint loss.

[0132] Specifically, to improve compression features To address the structural decoupling and interpretability of neural activity, embodiments of the present invention further add orthogonality constraints to each dimension. Orthogonality constraints promote... The different dimensions are independent and non-redundant, avoiding multiple dimensions from expressing the same EEG feature components. Assume that the compressed features of NNN samples in a mini-batch are concatenated into a matrix. The goal of orthogonality constraints is to make That is, the covariance matrix is ​​close to the identity matrix. The formula for the orthogonality constraint loss is as follows:

[0133]

[0134] in, This represents the loss due to orthogonality constraints. Denotes the Frobenius norm. The identity matrix is ​​used. This orthogonality constraint loss, by penalizing non-orthogonal components, guides the encoder to learn more physically interpretable and more separable feature dimensions. This allows different dimensions to capture different neural components in the EEG signal, such as frequency bands and modulation changes, thereby improving the interpretability and stability of the pre-trained EEG decoding model.

[0135] After obtaining the orthogonality constraint loss, the target loss can be determined based on the task classification loss, state classification loss, and orthogonality constraint loss.

[0136] Here, the target loss can be determined based on the sum of task classification loss, state classification loss, and orthogonality constraint loss, or based on the weighted sum of task classification loss, state classification loss, and orthogonality constraint loss.

[0137] Based on the above embodiments, the method further includes:

[0138] Step 410: Perform frequency domain transformation on the sample EEG signal set to obtain frequency domain input features;

[0139] Step 420: Perform local masking on the frequency domain input features to obtain frequency domain masked features;

[0140] Step 430: The frequency domain mask features are encoded by the original encoder to obtain the compressed features;

[0141] Step 440: The original decoder performs spectral reconstruction on the compressed features to obtain the reconstructed features;

[0142] Step 450: Determine the reconstruction loss based on the difference between the reconstructed features and the frequency domain input features;

[0143] Step 460: Determine the target loss based on the reconstruction loss, the task classification loss, the state classification loss, and the orthogonality constraint loss.

[0144] Specifically, to enhance the neural network's attention to the frequency harmonics information in SSVEP, a frequency domain transformation is performed on the sample EEG signal set to obtain frequency domain input features. That is, Fourier transform or spectral analysis is applied to the sample EEG signals of each trial to generate frequency domain input features. .

[0145] To enhance the robustness of the pre-trained EEG decoding model to local information loss and to construct a reconstruction task, the frequency domain input features are locally masked to obtain frequency domain masked features. Specifically, a portion of the frequency domain input features is set to 0 or random noise to simulate information loss. The frequency domain masked features can be used... express.

[0146] Furthermore, the frequency domain mask features are encoded by the original encoder to obtain compressed features. The compressed features are then reconstructed using the spectrum by the original decoder to obtain reconstructed features. This process reconstructs the masked frequency domain signal. The original decoder structure is symmetrical to the original encoder and is constructed using dedilation convolution and Transformer modules.

[0147] After obtaining the reconstructed features, the reconstruction loss can be determined based on the difference between the reconstructed features and the frequency domain input features. The formula for the reconstruction loss is as follows:

[0148]

[0149] in, Indicates the losses incurred during reconstruction. Indicates reconstruction features, This represents the frequency domain input characteristics.

[0150] It is understandable that the greater the difference between the reconstructed features and the frequency domain input features, the greater the reconstruction loss; the smaller the difference between the reconstructed features and the frequency domain input features, the smaller the reconstruction loss.

[0151] Figure 3 This is a training diagram of the pre-trained EEG decoding model provided by the present invention, as shown below. Figure 3 As shown,

[0152] The target loss can be determined based on reconstruction loss, task classification loss, state classification loss, and orthogonality constraint loss. The formula for the target loss is as follows:

[0153]

[0154] in, Indicates target loss. Indicates the losses incurred during reconstruction. Indicates task classification loss. Represents the state classification loss. This represents the loss due to orthogonality constraints. , , , These are all weighting coefficients, set according to the performance of the validation set.

[0155] Here, both the encoder and decoder use a parameter-sharing mechanism, meaning that the preprocessing and postprocessing branches use the same encoder and the same decoder.

[0156] Through the above-mentioned joint training strategy, the embodiments of the present invention effectively improve the task discrimination ability and structural decoupling ability of compressed feature z, providing a stable, controllable and interpretable feature foundation for subsequent feature enhancement and instruction recognition.

[0157] Understandably, the multi-task auxiliary module consists of three modules: the task classifier, the state discriminator, and the orthogonality constraint. Since the sample EEG signal set includes a first EEG signal set collected before neural modulation and a second EEG signal set collected after neural modulation, the compressed features correspondingly exist in two forms: one is the pre-neural modulation... One characteristic is after neural modulation. Features, and Features and The features are input into three modules: task classifier, state discriminator, and orthogonality constraint, which determine the task classification loss, state classification loss, and orthogonality constraint loss, respectively.

[0158] Figure 4 This is a training diagram of the EEG decoding model provided by the present invention, as shown below. Figure 4 As shown, the EEG decoding model includes an encoder, a feature enhancer, and a task classifier. The encoder is used to encode real-time EEG signals to obtain a compressed representation before neural modulation. The feature enhancer is used to enhance the features of the compressed representation, resulting in an enhanced representation. The task classifier is used to classify the enhanced representations to obtain the EEG decoding results.

[0159] Similar to the target loss during the training process of the pre-trained EEG decoding model, the training loss of the second-stage EEG decoding model is as follows:

[0160]

[0161] in, Indicates training loss, Indicates the losses incurred during reconstruction. Indicates task classification loss. Indicates the loss in the state determination. This represents the loss due to orthogonality constraints. , , , These are all weighting coefficients, set according to the performance of the validation set.

[0162] It should be noted that the training loss of the EEG decoding model is only used for gradient backpropagation and is not used for parameter updates. That is, the parameters of the encoder, decoder, task classifier and state discriminator in the EEG decoding model are fixed.

[0163] In summary, the training of the EEG decoding model is divided into two stages: the first stage is the learning and optimization of the autoencoded compressed representation, and the second stage is the construction and training of the feature enhancer.

[0164] Phase 1 constructs a self-supervised autoencoder network with structural constraints and task orientation, using frequency domain EEG as input for compression and reconstruction. Through local masking and frequency domain modeling, the extraction capability of high-frequency and harmonic response features is effectively improved. Task classification constraints (distinguishing stimulus frequencies), state classification constraints (distinguishing signals before and after modulation), and orthogonality structural constraints are applied to the compressed features to make them discriminative, decoupled, and interpretable, providing stable features for subsequent modeling.

[0165] Phase two involves the construction and training of the feature enhancer. Building upon compressed features, a feature enhancer module is introduced to simulate the feature transfer relationship from the pre-modulation state to the post-modulation state. A nonlinear mapping is constructed using a residual network to achieve feature style transfer and task feature preservation, enabling the model to generate more discriminative virtual brain responses. This significantly improves the representation quality and decoding performance of signals under weak stimulation conditions.

[0166] Based on the above embodiments, the encoder is a harmonic alignment encoder;

[0167] The harmonic alignment encoder includes a fundamental frequency feature extraction module, a harmonic frequency feature extraction module, and a self-attention feature mining module; the fundamental frequency feature extraction module is used to extract fundamental frequency features from the real-time EEG signal; the harmonic frequency feature extraction module is used to extract harmonic frequency features from the real-time EEG signal; the self-attention feature mining module is used to perform self-attention calculation based on the fundamental frequency features and the harmonic frequency features to obtain the compressed representation.

[0168] Specifically, the encoder can be a harmonic alignment encoder, which can include a fundamental frequency feature extraction module, an overtone feature extraction module, and a self-attention feature mining module. The fundamental frequency feature extraction module is used to extract fundamental frequency features from real-time EEG signals, the overtone feature extraction module is used to extract overtone features from real-time EEG signals, and the self-attention feature mining module is used to perform self-attention calculation based on fundamental frequency features and overtone features to obtain a compressed representation.

[0169] Here, the fundamental frequency feature is used to reflect the fundamental frequency information in the real-time EEG signal, and the harmonic frequency feature is used to reflect the harmonic frequency information in the real-time EEG signal.

[0170] The method provided in this invention, a harmonic alignment encoder, effectively addresses the problems of insufficient utilization of the harmonic structure and weak feature representation ability of traditional methods by setting a fundamental frequency feature extraction module, a harmonic frequency feature extraction module, and a self-attention feature mining module. The fundamental frequency feature extraction module and the harmonic frequency feature extraction module decouple the fundamental frequency and its harmonic components directly related to the visual stimulus frequency from the real-time EEG signal, ensuring the integrity of the core frequency information. The self-attention feature mining module adaptively mines and fuses the intrinsic correlations and weights among these harmonic components, thereby constructing a more discriminative compressed representation. This collaborative mechanism significantly improves the ability to capture and represent key patterns of SSVEP signals, laying a solid foundation for high-precision EEG decoding.

[0171] Based on the above embodiments, the fundamental frequency feature extraction module performs convolution calculation on the fundamental frequency part of the real-time EEG signal using a first convolution kernel with a first dilation rate to obtain the fundamental frequency feature;

[0172] The frequency octave feature extraction module obtains the frequency octave features by performing convolution calculation on the frequency octave portion of the real-time EEG signal using a second convolution kernel with a second dilation rate.

[0173] Wherein, the second convolution kernel has a larger size on the frequency axis than the first convolution kernel, and the second dilation rate is greater than the first dilation rate.

[0174] Specifically, in the frequency domain analysis of steady-state visual evoked potential signals, although the fundamental band and the octave band have the same spectral resolution, the number of sampling points between adjacent targets increases exponentially with the frequency order because the target stimulus interval is a multiple of each other in the octave band.

[0175] Therefore, if a uniform convolution kernel structure is used directly in different frequency bands, it will lead to an imbalance in the ability to capture local spectral patterns, making it difficult to meet the characterization requirements of the fundamental frequency and harmonics.

[0176] Accordingly, in this embodiment of the invention, the input signal is first segmented in the frequency domain into a fundamental frequency band and a second harmonic frequency band to ensure that different frequency bands have independent modeling paths during feature extraction. The fundamental frequency feature extraction module performs convolution calculation on the fundamental frequency portion of the real-time EEG signal using a first convolution kernel with a first dilation rate to obtain the fundamental frequency feature; the harmonic frequency feature extraction module performs convolution calculation on the harmonic frequency portion of the real-time EEG signal using a second convolution kernel with a second dilation rate to obtain the harmonic frequency feature.

[0177] The second convolution kernel has a larger size on the frequency axis than the first convolution kernel, and the second dilation rate is greater than the first dilation rate.

[0178] In an alternative embodiment, Figure 5 This is a schematic diagram of the harmonic alignment encoder provided by the present invention, as shown below. Figure 5 As shown, for the fundamental frequency band, a size of [size missing] is used. The convolutional kernel extracts the local correlation between adjacent sampling points, thus preserving the continuity of the narrow interval features of the fundamental frequency; while for the second octave band, a kernel with a size of [missing information] is used. The convolutional kernels are adapted to accommodate wider frequency intervals and cross-target relationships. Then, multi-dilated convolutional structures are further introduced in different frequency bands to achieve scale alignment. A convolutional kernel with a dilation rate of 1 is used for the fundamental band to preserve its local continuity; while for the second harmonic band, a convolutional kernel with a dilation rate of 2 is used to align the sampling points corresponding to the fundamental and harmonic frequencies within the receptive field. The convolutional kernels establish equivariance across harmonics on the frequency axis, effectively compensating for the inability of traditional methods to guarantee scale consistency in multi-band modeling. Subsequently, learnable parameters are introduced. and A harmonic weighted aggregation mechanism is implemented to weight and fuse the features of the fundamental frequency band and the octave band, and dynamically adjust the contribution ratio of different frequency band features in the global representation.

[0179] Finally, a three-layer stacked self-attention feature mining module is introduced, which can simultaneously model global dependencies in the frequency and lead dimensions, capture long-range correlations across frequency bands and spatial synergistic effects across brain regions, thereby generating compressed features with discriminative, robust, and interpretable characteristics. This provides a high-quality feature base for subsequent brain signal reconstruction and target classification tasks.

[0180] Here, frequency domain mask features can also be used. The input is fed into the harmonic alignment encoder to obtain the compression characteristics output by the harmonic alignment encoder.

[0181] Here, the self-attention feature mining module may include a normalization layer and a self-attention module.

[0182] Based on any of the above embodiments Figure 6 This is a schematic diagram of the overall architecture of the EEG signal processing system provided by the present invention, as shown below. Figure 6 As shown, the system first begins with the signal acquisition step. Figure 6 The top left side shows the head-mounted data acquisition device worn by the subject. This module is responsible for acquiring raw electroencephalogram (EEG) signals. Figure 6 The raw data is labeled in the image. This raw data appears as a chaotic waveform, representing the initial physiological signal state.

[0183] Subsequently, the system enters the signal processing stage. The acquired raw data is sent to the signal processing module, one of the core functions of which is to extract the effective signal segments related to steady-state visual evoked potentials, namely the SSVEP segment shown in the figure. The processed and normalized waveform lays the foundation for subsequent decoding and analysis.

[0184] Then, the system performs a weak implicit decoding process. The preprocessed and feature-enhanced SSVEP signal is input to the weak implicit decoding module, whose task is to accurately extract the user intent information hidden in the weak signal with a low signal-to-noise ratio.

[0185] The decoding results are ultimately sent to the target recognition stage. The system converts the decoded intent information into specific control commands, thereby enabling the recognition and selection of specific targets on external devices or interfaces, such as... Figure 6 The grid options displayed on the central display screen.

[0186] In addition, the system includes a result feedback path. The result of target recognition can be fed back to the user or other parts of the system, thus forming a closed-loop control loop from signal acquisition, processing, decoding, recognition to feedback, optimizing the interactive performance and stability of the entire system.

[0187] In one embodiment, firstly, a high-frequency transcranial magnetic stimulation (TSVEP) experimental paradigm oriented towards weak implicit stimulation conditions is designed, constructing a dual-domain dataset covering the states before and after neural response enhancement to collect EEG signals induced by real physical modulation. Subsequently, a neural network modeling framework is proposed to simulate the brain response enhancement effect. Specifically, this invention designs a harmonic alignment feature mining module to align the fundamental and harmonic components, and introduces a task classifier, state discriminator, and orthogonality constraints into the compressed representation to guide the encoder to learn the key features of SSVEP enhancement. Innovatively, the enhanced SSVEP signal is used as the decoder reconstruction target, thereby realizing the virtual enhancement and reconstruction of weak responses in the feature space. Finally, a weak implicit SSVEP brain-computer interface command issuing device is constructed to achieve accurate decoding and command output of weak brain responses. This device can be widely applied in scenarios such as brain-controlled typing, neurorehabilitation training, and intelligent interaction under zero-training conditions. Compared with existing methods, this invention can still significantly improve the recognition accuracy, signal-to-noise ratio, and robustness of brain-computer interfaces under weak implicit stimulation conditions, providing a novel technical solution for efficient and natural human-computer interaction under low-interference conditions.

[0188] The following describes the brain-computer interface command issuing device based on modulation enhancement simulation provided by the present invention. The brain-computer interface command issuing device based on modulation enhancement simulation described below and the brain-computer interface command issuing method based on modulation enhancement simulation described above can be referred to in correspondence with each other.

[0189] Based on any of the above embodiments, the present invention provides a brain-computer interface command issuing device based on modulation enhancement simulation. Figure 7 This is a schematic diagram of the brain-computer interface command issuing device based on modulation enhancement simulation provided by the present invention, as shown below. Figure 7 As shown, the device includes:

[0190] Acquisition unit 710 is used to acquire the user's real-time EEG signals;

[0191] The input unit 720 is used to input the real-time EEG signal into the EEG decoding model to obtain the EEG decoding result output by the EEG decoding model;

[0192] The instruction issuing unit 730 is used to issue instructions based on the EEG decoding results;

[0193] The EEG decoding model includes an encoder, a feature enhancer, and a task classifier; the encoder is used to encode the real-time EEG signal to obtain a compressed representation before neural modulation; the feature enhancer is used to enhance the compressed representation to obtain an enhanced representation; the task classifier is used to classify the enhanced representation to obtain the EEG decoding result.

[0194] The feature enhancer is obtained by training a state discriminator based on sample EEG signals collected before neural modulation and real state labels after neural modulation.

[0195] The device provided in this invention includes an EEG decoding model comprising an encoder, a feature enhancer, and a task classifier. The encoder encodes real-time EEG signals to obtain a compressed representation before neural modulation. The feature enhancer enhances the compressed representation to obtain an enhanced representation. The task classifier classifies the enhanced representation to obtain the EEG decoding result. In this method, the feature enhancer is trained using sample EEG signals acquired before neural modulation and real-state labels after neural modulation, in conjunction with a state discriminator. This training method drives the feature enhancer to learn the feature transfer relationship between the compressed features before neural modulation and the features after neural modulation, simulating the enhancement effect of neural modulation. This significantly improves the EEG decoding model's ability to represent EEG signals by features, thereby enhancing the system's robustness in decoding weak stimulus signals without relying on high-intensity external stimuli. Ultimately, while ensuring user comfort, it improves the accuracy of brain-computer interface command recognition and the overall stability of the system.

[0196] Based on any of the above embodiments, a first training unit is further included, wherein the first training unit is specifically used for:

[0197] An initial EEG decoding model including an initial feature enhancer is obtained, and the parameters of the initial encoder in the initial EEG decoding model are initialized based on the parameters of the pre-trained encoder in the pre-trained EEG decoding model; the parameters of the state discriminator are initialized based on the parameters of the pre-trained state discriminator in the pre-trained EEG decoding model.

[0198] The sample EEG signal collected before the neural modulation is input into the initial EEG decoding model. The initial encoder encodes the sample EEG signal to obtain a compressed representation of the sample before neural modulation. The initial feature enhancer enhances the compressed representation of the sample to obtain an enhanced representation of the sample. The state discriminator discriminates the enhanced representation of the sample to obtain a state prediction result.

[0199] Based on the difference between the state prediction result and the true state label, a state discrimination loss is determined, and the initial feature enhancer is trained based on the state discrimination loss to obtain the feature enhancer.

[0200] Based on any of the above embodiments, a second training unit is further included, the second training unit specifically including:

[0201] The sample acquisition unit is used to acquire a sample EEG signal set including a first EEG signal set acquired before neural modulation and a second EEG signal set acquired after neural modulation, as well as an original EEG decoding model; the original EEG decoding model includes an original encoder, an original decoder, an original task classifier, and an original state discriminator;

[0202] The prediction unit is used to input the sample EEG signal set into the original EEG decoding model, where the original encoder encodes the sample EEG signal set to obtain compressed features, the original task classifier classifies the compressed features to obtain a predicted category distribution, and the original state discriminator performs state prediction on the compressed features to obtain a state prediction probability.

[0203] A task classification loss unit is determined, which is used to determine the task classification loss based on the difference between the predicted category distribution and the label category corresponding to the sample EEG signal set;

[0204] A state classification loss unit is defined to determine the state classification loss based on the difference between the state prediction probability and the true state label corresponding to the sample EEG signal set.

[0205] A target loss unit is defined to determine a target loss based on the task classification loss and the state classification loss, and to train the original EEG decoding model based on the target loss to obtain the pre-trained EEG decoding model.

[0206] Based on any of the above embodiments, the target loss determination unit is specifically used for:

[0207] Based on the correlation between the internal dimensions of the compressed features, the orthogonality constraint loss is determined;

[0208] The target loss is determined based on the task classification loss, the state classification loss, and the orthogonality constraint loss.

[0209] Based on any of the above embodiments, a joint training unit is further included, wherein the joint training unit is specifically used for:

[0210] The sample EEG signal set is frequency domain transformed to obtain frequency domain input features;

[0211] The frequency domain input features are locally masked to obtain the frequency domain masked features;

[0212] The compressed features are obtained by encoding the frequency domain mask features using the original encoder;

[0213] The original decoder performs spectral reconstruction on the compressed features to obtain the reconstructed features;

[0214] The reconstruction loss is determined based on the difference between the reconstructed features and the frequency domain input features;

[0215] The target loss is determined based on the reconstruction loss, the task classification loss, the state classification loss, and the orthogonality constraint loss.

[0216] Based on any of the above embodiments, the encoder is a harmonic alignment encoder;

[0217] The harmonic alignment encoder includes a fundamental frequency feature extraction module, a harmonic frequency feature extraction module, and a self-attention feature mining module; the fundamental frequency feature extraction module is used to extract fundamental frequency features from the real-time EEG signal; the harmonic frequency feature extraction module is used to extract harmonic frequency features from the real-time EEG signal; the self-attention feature mining module is used to perform self-attention calculation based on the fundamental frequency features and the harmonic frequency features to obtain the compressed representation.

[0218] Based on any of the above embodiments, the fundamental frequency feature extraction module performs convolution calculation on the fundamental frequency part of the real-time EEG signal using a first convolution kernel with a first dilation rate to obtain the fundamental frequency feature;

[0219] The frequency octave feature extraction module obtains the frequency octave features by performing convolution calculation on the frequency octave portion of the real-time EEG signal using a second convolution kernel with a second dilation rate.

[0220] Wherein, the second convolution kernel has a larger size on the frequency axis than the first convolution kernel, and the second dilation rate is greater than the first dilation rate.

[0221] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logical instructions in the memory 830 to execute a brain-computer interface instruction issuance method based on modulation enhancement simulation. This method includes: acquiring the user's real-time EEG signal; inputting the real-time EEG signal into an EEG decoding model to obtain the EEG decoding result output by the EEG decoding model; and issuing instructions based on the EEG decoding result. The EEG decoding model includes an encoder, a feature enhancer, and a task classifier. The encoder encodes the real-time EEG signal to obtain a compressed representation before neural modulation. The feature enhancer enhances the compressed representation to obtain an enhanced representation. The task classifier classifies the enhanced representation to obtain the EEG decoding result. The feature enhancer is trained based on sample EEG signals acquired before neural modulation and real-state labels after neural modulation, combined with a state discriminator.

[0222] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the brain-computer interface instruction issuance method based on modulation enhancement simulation provided by the above methods. The method includes: acquiring the user's real-time EEG signal; inputting the real-time EEG signal into an EEG decoding model to obtain the EEG decoding result output by the EEG decoding model; and issuing instructions based on the EEG decoding result. The EEG decoding model includes an encoder, a feature enhancer, and a task classifier. The encoder is used to encode the real-time EEG signal to obtain a compressed representation before neural modulation. The feature enhancer is used to enhance the compressed representation to obtain an enhanced representation. The task classifier is used to classify the enhanced representation to obtain the EEG decoding result. The feature enhancer is trained based on sample EEG signals acquired before neural modulation and real state labels after neural modulation, in conjunction with a state discriminator.

[0224] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a brain-computer interface instruction issuing method based on modulation enhancement simulation provided by the above methods. The method includes: acquiring a user's real-time EEG signal; inputting the real-time EEG signal into an EEG decoding model to obtain an EEG decoding result output by the EEG decoding model; and issuing an instruction based on the EEG decoding result. The EEG decoding model includes an encoder, a feature enhancer, and a task classifier. The encoder is used to encode the real-time EEG signal to obtain a compressed representation before neural modulation. The feature enhancer is used to enhance the compressed representation to obtain an enhanced representation. The task classifier is used to classify the enhanced representation to obtain the EEG decoding result. The feature enhancer is obtained by training a state discriminator based on sample EEG signals acquired before neural modulation and real state labels after neural modulation.

[0225] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A brain-computer interface command issuance method based on modulated augmentation simulation, characterized in that, include: Acquire the user's real-time brainwave signals; The real-time EEG signal is input into the EEG decoding model to obtain the EEG decoding result output by the EEG decoding model; Based on the EEG decoding results, commands are issued; The EEG decoding model includes an encoder, a feature enhancer, and a task classifier; the encoder is used to encode the real-time EEG signal to obtain a compressed representation before neural modulation. The feature enhancer is used to enhance the features of the compressed representation to obtain an enhanced representation; The task classifier is used to classify the enhanced representation to obtain the EEG decoding result; The feature enhancer is obtained by training a state discriminator based on sample EEG signals acquired before neural modulation and real state labels after neural modulation. The encoder is a harmonic alignment encoder; The harmonic alignment encoder includes a fundamental frequency feature extraction module, a harmonic frequency feature extraction module, and a self-attention feature mining module. The fundamental frequency feature extraction module is used to extract fundamental frequency features from the real-time EEG signal. The harmonic frequency feature extraction module is used to extract harmonic frequency features from the real-time EEG signal. The self-attention feature mining module is used to perform self-attention calculation based on the fundamental frequency features and the harmonic frequency features to obtain the compressed representation. The fundamental frequency feature extraction module performs convolution calculation on the fundamental frequency part of the real-time EEG signal using a first convolution kernel with a first dilation rate to obtain the fundamental frequency feature. The frequency octave feature extraction module obtains the frequency octave features by performing convolution calculation on the frequency octave portion of the real-time EEG signal using a second convolution kernel with a second dilation rate. Wherein, the second convolution kernel has a larger size on the frequency axis than the first convolution kernel, and the second dilation rate is greater than the first dilation rate.

2. The brain-computer interface command issuance method based on enhanced modulation simulation according to claim 1, characterized in that, The training steps of the feature enhancer include: An initial EEG decoding model including an initial feature enhancer is obtained, and the parameters of the initial encoder in the initial EEG decoding model are initialized based on the parameters of the pre-trained encoder in the pre-trained EEG decoding model; the parameters of the state discriminator are initialized based on the parameters of the pre-trained state discriminator in the pre-trained EEG decoding model. The sample EEG signal collected before the neural modulation is input into the initial EEG decoding model. The initial encoder encodes the sample EEG signal to obtain a compressed representation of the sample before neural modulation. The initial feature enhancer enhances the compressed representation of the sample to obtain an enhanced representation of the sample. The state discriminator discriminates the enhanced representation of the sample to obtain a state prediction result. Based on the difference between the state prediction result and the true state label, a state discrimination loss is determined, and the initial feature enhancer is trained based on the state discrimination loss to obtain the feature enhancer.

3. The brain-computer interface command issuance method based on enhanced modulation simulation according to claim 2, characterized in that, The steps for obtaining the pre-trained EEG decoding model include: The sample EEG signal set is acquired, including a first EEG signal set collected before neural modulation and a second EEG signal set collected after neural modulation, as well as an original EEG decoding model; the original EEG decoding model includes an original encoder, an original decoder, an original task classifier, and an original state discriminator; The sample EEG signal set is input into the original EEG decoding model. The original encoder encodes the sample EEG signal set to obtain compressed features. The original task classifier classifies the compressed features to obtain the predicted category distribution. The original state discriminator performs state prediction on the compressed features to obtain the state prediction probability. Based on the difference between the predicted category distribution and the label categories corresponding to the sample EEG signal set, the task classification loss is determined; The state classification loss is determined based on the difference between the state prediction probability and the true state label corresponding to the sample EEG signal set. Based on the task classification loss and the state classification loss, a target loss is determined, and the original EEG decoding model is trained based on the target loss to obtain the pre-trained EEG decoding model.

4. The brain-computer interface command issuance method based on enhanced modulation simulation according to claim 3, characterized in that, The determination of the target loss based on the task classification loss and the state classification loss includes: Based on the correlation between the internal dimensions of the compressed features, the orthogonality constraint loss is determined; The target loss is determined based on the task classification loss, the state classification loss, and the orthogonality constraint loss.

5. The brain-computer interface command issuance method based on modulated augmentation simulation according to claim 4, characterized in that, The method further includes: The sample EEG signal set is frequency domain transformed to obtain frequency domain input features; The frequency domain input features are locally masked to obtain the frequency domain masked features; The compressed features are obtained by encoding the frequency domain mask features using the original encoder; The original decoder performs spectral reconstruction on the compressed features to obtain the reconstructed features; The reconstruction loss is determined based on the difference between the reconstructed features and the frequency domain input features; The target loss is determined based on the reconstruction loss, the task classification loss, the state classification loss, and the orthogonality constraint loss.

6. A brain-computer interface command issuing device based on modulation enhancement simulation, characterized in that, include: The acquisition unit is used to acquire the user's real-time EEG signals; The input unit is used to input the real-time EEG signal into the EEG decoding model to obtain the EEG decoding result output by the EEG decoding model; The instruction issuing unit is used to issue instructions based on the EEG decoding results; The EEG decoding model includes an encoder, a feature enhancer, and a task classifier; The encoder is used to encode the real-time EEG signal to obtain a compressed representation before neural modulation. The feature enhancer is used to enhance the features of the compressed representation to obtain an enhanced representation; The task classifier is used to classify the enhanced representation to obtain the EEG decoding result; The feature enhancer is obtained by training a state discriminator based on sample EEG signals collected before neural modulation and real state labels after neural modulation. The encoder is a harmonic alignment encoder; The harmonic alignment encoder includes a fundamental frequency feature extraction module, a harmonic frequency feature extraction module, and a self-attention feature mining module; the fundamental frequency feature extraction module is used to extract fundamental frequency features from the real-time EEG signal; the harmonic frequency feature extraction module is used to extract harmonic frequency features from the real-time EEG signal; the self-attention feature mining module is used to perform self-attention calculation based on the fundamental frequency features and the harmonic frequency features to obtain the compressed representation; The fundamental frequency feature extraction module performs convolution calculation on the fundamental frequency part of the real-time EEG signal using a first convolution kernel with a first dilation rate to obtain the fundamental frequency feature. The frequency octave feature extraction module obtains the frequency octave features by performing convolution calculation on the frequency octave portion of the real-time EEG signal using a second convolution kernel with a second dilation rate. Wherein, the second convolution kernel has a larger size on the frequency axis than the first convolution kernel, and the second dilation rate is greater than the first dilation rate.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the brain-computer interface instruction issuing method based on modulation enhancement simulation as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the brain-computer interface instruction issuing method based on the modulation enhancement simulation as described in any one of claims 1 to 5.

Images