Control method, device and equipment based on electroencephalogram signals

Abstract

The invention provides a control method, device and equipment based on electroencephalogram signals. The method comprises the steps of outputting a first visual signal under the condition that a user establishes a connection with a signal acquisition device, the first visual signal being used for guiding the user to perform motor imagery about a first task; receiving an electroencephalogram signal of motor imagery of the user in the process of outputting the first visual signal; the electroencephalogram signals are analyzed through a pre-trained classification model, labels of the electroencephalogram signals and a first instruction are obtained, and the first instruction is a control instruction for the first task; sending a first instruction to equipment corresponding to the first task; the classification model is obtained by training based on a first fusion signal, and the first fusion signal is a signal obtained by fusing an electroencephalogram signal induced by a visual signal, an electroencephalogram signal induced by an auditory signal and a physical sign signal. According to the method, the intention of the user can be accurately recognized, the correct control instruction is generated, and the external equipment is accurately controlled according to the electroencephalogram signal.

Description

Technical Field

[0001] This application relates to the field of data services, and in particular to a control method, device, and equipment based on electroencephalogram (EEG) signals. Background Technology

[0002] Brainwaves possess unique temporal and frequency characteristics, reflecting the state of the brain. The more frequent the brain activity, the higher the frequency of electrical activity. Brainwaves from various senses and activities exhibit unique spatial characteristics. Different regions of the brain have different functions, and specific correlations exist between these regions. Brain-computer interfaces (BCIs) enable the control of upper and lower limb movement and communication functions.

[0003] Brain activity can be stimulated by voluntary imagination, specifically motor imagery. This means that even when a person imagines their limbs (or muscles) moving without actual physical movement, certain brain regions will still be activated. By analyzing brain signals and detecting the activation effects of different brain regions, the user's intentions can be determined, thereby enabling direct communication and control between the human brain and external devices.

[0004] Currently, most EEG testing equipment is still in the academic research and clinical diagnostic stages. One characteristic is that it has high requirements for the signal acquisition environment. The vast majority of EEG signals are spontaneous and random signals. In particular, untrained subjects have difficulty controlling their own motor imagery process and emitting clean and clear motor imagery signals. Therefore, the signal-to-noise ratio of the acquired signals is low, resulting in a lower accuracy in recognizing user intentions. Summary of the Invention

[0005] The purpose of this application is to provide a control method, device, and equipment based on electroencephalogram (EEG) signals, which solves the problem of low user recognition accuracy based on EEG signals in the prior art.

[0006] To achieve the above objectives, embodiments of this application provide a control method based on electroencephalogram (EEG) signals, comprising:

[0007] Once a connection is established between the user and the signal acquisition device, a first visual signal is output, which is used to guide the user to visualize the movement of the first task.

[0008] Receive the EEG signals of the user during the first visual signal output process to perform motor imagery;

[0009] The EEG signal is analyzed using a pre-trained classification model to obtain a label for the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task.

[0010] Send the first instruction to the device corresponding to the first task;

[0011] The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

[0012] Optionally, before analyzing the EEG signals using a pre-trained classification model, the method further includes:

[0013] Determine whether the EEG signal is in a motion state;

[0014] If the EEG signal is in motion, then the pre-trained classification model is used to analyze the EEG signal.

[0015] Optionally, the method further includes:

[0016] If the EEG signal is in an idle state, the EEG signal is analyzed according to the Steady-State Visual Evoked Potentials (SSVEP) algorithm to obtain the analysis results;

[0017] If the analysis results indicate that the SSVEP response contains a preset frequency, then the first command is output.

[0018] Optionally, the method further includes:

[0019] Acquire offline training data from multiple users, including the users' electroencephalogram (EEG) signals and vital signs;

[0020] The offline training data is classified and modeled to obtain classification models that match different users.

[0021] Optionally, the method further includes:

[0022] When the user is trained offline, a second visual signal and an auditory signal are output, which are used to prompt the user to perform motion visualization.

[0023] The system continuously outputs third visual signals during the first time period and receives EEG signals and vital signs signals of the user during the third visual signal output process. The EEG signals include EEG signals of motor imagination of N body parts, and the vital signs signals include vital signs signals of N body parts during the motor imagination process, where N is a positive integer.

[0024] The electroencephalogram (EEG) signal and the vital signs signal are fused to obtain a first fused signal;

[0025] A classification model matching the user is generated based on the first fused signal.

[0026] Optionally, the second visual signal corresponds to an image indicating multiple directions, and the image in each direction flashes at a preset frequency.

[0027] Optionally, the EEG signals obtained from offline training of the user have corresponding labels, which are used to indicate the content of the user's motor imagery.

[0028] To achieve the above objectives, embodiments of this application provide a control device based on electroencephalogram (EEG) signals, comprising:

[0029] The first output module is used to output a first visual signal when the user establishes a connection with the signal acquisition device. The first visual signal is used to guide the user to imagine the movement of the first task.

[0030] The first receiving module is used to receive the electroencephalogram (EEG) signals of the user's motor imagery during the first visual signal output process;

[0031] The first processing module is used to analyze the EEG signal using a pre-trained classification model to obtain the label of the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task.

[0032] The first sending module is used to send the first instruction to the device corresponding to the first task;

[0033] The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

[0034] To achieve the above objectives, embodiments of this application provide an electronic device, including a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the steps of the above-described control method based on electroencephalogram (EEG) signals.

[0035] To achieve the above objectives, embodiments of this application provide a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps of the control method based on electroencephalogram (EEG) signals as described above.

[0036] To achieve the above objectives, embodiments of this application provide a computer program product that stores a program or instructions, including computer instructions, which, when executed by a processor, implement the steps of the control method based on electroencephalogram (EEG) signals described above.

[0037] The beneficial effects of the above technical solution in this application are as follows:

[0038] The embodiments of this application employ a hybrid experimental paradigm, primarily based on motor imagery and secondarily on steady-state visual induction, to effectively induce EEG signals and ensure the effective operation of the brain-computer interface system. A user-matched classification model is constructed, trained on EEG and physiological signals under multimodal stimuli such as visual and auditory stimuli. This reduces the user's dependence on the screen and enhances the freedom of the user's motor imagery. Analyzing the collected EEG signals using the pre-trained classification model accurately identifies the user's intentions, thereby generating correct control commands and enabling accurate control of external devices based on EEG signals. Attached Figure Description

[0039] Figure 1 This is one of the flowcharts illustrating the control method based on electroencephalogram (EEG) signals according to an embodiment of this application.

[0040] Figure 2 This is a second schematic flowchart of the control method based on electroencephalogram (EEG) signals according to an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of a visual cue signal according to an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of another visual cue signal according to an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the electrodes of the signal acquisition device according to an embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the signal output during the testing process of an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the control device based on electroencephalogram (EEG) signals according to an embodiment of this application;

[0046] Figure 8 This is one of the structural schematic diagrams of the electronic device according to an embodiment of this application;

[0047] Figure 9 This is a second schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0048] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0049] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0050] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0051] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0052] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0053] like Figure 1 As shown in the figure, this application provides a control method based on electroencephalogram (EEG) signals, which can be executed by a cloud application server. The method includes:

[0054] Step 101: When the user establishes a connection with the signal acquisition device, output a first visual signal. The first visual signal is used to guide the user to imagine the motion of the first task.

[0055] Step 102: Receive the EEG signal of the user's motor imagery during the first visual signal output process;

[0056] Step 103: Analyze the EEG signal using a pre-trained classification model to obtain the label of the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task.

[0057] Step 104: Send the first instruction to the device corresponding to the first task;

[0058] The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

[0059] In this embodiment, when collecting the user's EEG signals, a cloud application server is connected to an external EEG signal acquisition device, enabling uninterrupted collection and recording of EEG signals. The signal acquisition device may have a body area network (BNB) data center for processing the collected data. The cloud application server has a display for showing the output visual signals. The cloud application server can connect to external devices, and can analyze the collected EEG signals to obtain the user's intentions, thereby controlling the external devices. For example, when the user views an image corresponding to a visual signal and performs a motor imagery related to a first task, the visual signal can be associated with the first task. For instance, if the first task is to turn off the lights, the image corresponding to the first visual signal moving to the left indicates turning off the lights. The user can imagine the image corresponding to the first visual signal moving during motor imagery. Based on the EEG signals collected during the user's motor imagery, if the user's intention is to turn off the lights, the light switch can be controlled to turn off.

[0060] The first visual signal can be displayed as an arrow, a square, or other graphic to guide the user to imagine movement. For example, the display interface can present a visual task that is completed through movement imagination, such as controlling the movement of a small square. When the user is trying to complete the task, they will perform a series of regular movement imaginations, and the small square is responsible for controlling the output command of "backward".

[0061] When a user is dissatisfied with the position of the small square and needs to revert, they can trigger motor imagery by gazing at the small square. During the motor imagery process, idle state detection and steady-state visual evoked signals are performed, which can collect a series of regular, categorizable unlabeled electroencephalogram (EEG) signals. In the experimental paradigm, after collecting these EEG signals, they are input into a classification model. The classification model calculates and assigns labels to the EEG signals, and generates control commands to instruct the completion of the corresponding task (reverting the position of the small square) until the end.

[0062] The classification model is obtained through offline training. Different users can be assigned different classification models. These models analyze the collected EEG signals to identify user intentions. EEG signal labels indicate the content of the user's motor imagery, such as moving left or right, or movement indicators related to identity, such as moving the left hand, right hand, or right leg. During offline training, multiple training sets can be performed on multiple users to obtain multiple training datasets. These training datasets can then be used to derive the corresponding classification models.

[0063] During offline training, visual and auditory signals induce motor imagery in users, thereby collecting the resulting electroencephalogram (EEG) and physiological signals. These physiological signals can include signals from multiple body parts, such as the left hand, right hand, feet, and tongue. Synchronous physiological signals under different evoked stimuli can be recorded using different sensors; for example, when a second visual signal is output, physiological signals from the left hand, right hand, feet, and tongue can be collected separately. By fusing and analyzing the synchronized EEG and physiological signals evoked by visual and auditory stimuli, a classification model matching the user is obtained. During offline training, the use of different types of multimodal evoked stimuli, including visual, auditory, and somatosensory stimuli, can simultaneously activate brain regions responsible for corresponding sensory functions, thereby enhancing the user's response to external stimuli. The use of multimodal stimulation enhances the brain's response to stimulus events, and the classification model obtained by training with the collected brainwave signals under these conditions can more accurately identify the user's intentions.

[0064] The embodiments of this application employ a hybrid experimental paradigm, primarily based on motor imagery and secondarily on steady-state visual induction, to effectively induce EEG signals and ensure the effective operation of the brain-computer interface system. A user-matched classification model is constructed, trained on EEG and physiological signals under multimodal stimuli such as visual and auditory stimuli. This reduces the user's dependence on the screen and enhances the freedom of the user's motor imagery. Analyzing the collected EEG signals using the pre-trained classification model accurately identifies the user's intentions, thereby generating correct control commands and enabling accurate control of external devices based on EEG signals.

[0065] Optionally, before analyzing the EEG signals using a pre-trained classification model, the method further includes:

[0066] Determine whether the EEG signal is in a motion state;

[0067] If the EEG signal is in motion, then the pre-trained classification model is used to analyze the EEG signal.

[0068] Optionally, the method further includes:

[0069] If the EEG signal is in an idle state, the EEG signal is analyzed according to the steady-state visual evoked potential (SSVEP) algorithm to obtain the analysis results;

[0070] If the analysis results indicate that the SSVEP response contains a preset frequency, then the first command is output.

[0071] In this embodiment, when a user is taking a test, the EEG signals over a period of time can be analyzed to determine whether the EEG signals during that period are in a motion state. If they are determined to be in a motion state, a classification model calculates and identifies the user's intention and outputs a corresponding command. If they are determined to be in an idle state, a Steady-State Visual Evoked Potentials (SSVEP) algorithm processes them. If the classification result is an SSVEP response with a preset frequency, a first command is output. The preset frequency is, for example, 10Hz.

[0072] As an optional embodiment, the method further includes:

[0073] Acquire offline training data from multiple users, including the users' electroencephalogram (EEG) signals and vital signs;

[0074] The offline training data is classified and modeled to obtain classification models that match different users.

[0075] In this embodiment, offline training can be performed on multiple users. Users are induced to engage in motor imagery through various means, including visual signals, auditory signals, and somatosensory stimulation, thereby collecting EEG and physiological signals during the motor imagery process. The obtained EEG and physiological signals are then classified and modeled to obtain classification models corresponding to different users. For example, during offline training, a population comparison experiment is conducted. First, multiple groups are established, with one group corresponding to each of the four categories. Each category contains 12 trials, including tests in four different directions. The categories can refer to task categories. Signals can be collected multiple times at predetermined intervals, such as twice, with an interval of 7 days, and each collection period containing 6 groups. EEG and physiological signals from multiple users with normal vision and limb function are repeatedly collected. A sufficient training set is used to train the model to obtain a classification model.

[0076] As an optional embodiment, the method further includes:

[0077] When the user is trained offline, a second visual signal and an auditory signal are output, which are used to prompt the user to perform motion visualization.

[0078] The system continuously outputs third visual signals during the first time period and receives EEG signals and vital signs signals of the user during the third visual signal output process. The EEG signals include EEG signals of motor imagination of N body parts, and the vital signs signals include vital signs signals of N body parts during the motor imagination process, where N is a positive integer.

[0079] The electroencephalogram (EEG) signal and the vital signs signal are fused to obtain a first fused signal;

[0080] A classification model matching the user is generated based on the first fused signal.

[0081] In this embodiment, during offline training, visual and auditory signals can be used to induce the user to prepare for motor imagery. At the start of the test, visual signals are output and corresponding EEG and physical signs are collected. The EEG and physical signs are fused to obtain a fused signal, which is then used to train the model and obtain a classification model.

[0082] Optionally, the second visual signal corresponds to an image indicating multiple directions, and each image flashes at a preset frequency. The second visual signal, for example, is an arrow pointing in multiple directions, which can flash at a preset frequency to visually stimulate the user when guiding them to visualize movement.

[0083] Optionally, the EEG signals obtained from offline training of the user are labeled accordingly, and the labels are used to indicate the content of the user's motor imagery. Examples of such labels include: the user imagines left and right movement, the user imagines right-hand movement, etc.

[0084] This embodiment allows for the construction of a multimodal EEG signal model (i.e., the classification model) for an individual user. During spontaneous motor imagery, different types of multimodal evoked stimuli, including visual, auditory, and somatosensory stimuli, are used to simultaneously activate the brain regions responsible for the corresponding sensory functions, thereby enhancing the individual's response to external stimuli. The use of multimodal stimulation enhances the brain's responsiveness to stimulus events, thus improving the target recognition accuracy of the spontaneous brain-computer interface (BCI) during imagery.

[0085] The following example illustrates the implementation process of the control method based on electroencephalogram (EEG) signals according to an embodiment of this application.

[0086] like Figure 2 As shown, it can include an offline training part and an online testing part, wherein:

[0087] The offline training component includes: static state cues, motor imagery cues, and cessation of motor imagery cues, and outputs a 10Hz steady-state visual evoked signal; it also collects the user's EEG signals during the above process and processes these signals to obtain the offline training model. The specific steps are as follows:

[0088] Step 21: Prepare for offline training by using the body area network data center and cloud application server.

[0089] 21-1. Preparation of steady-state visual evoked paradigm.

[0090] Eye movement elimination was performed before the experiment: Subjects sat in comfortable chairs in front of a computer screen (the visual signals output from the cloud application server were displayed on the screen). The eye movement elimination procedure was set up before each group of experiments: subjects kept their eyes open for two minutes while looking at the screen, closed their eyes for one minute, and finally performed eye movements for one minute. This was performed once before each test group. At the start of the experiment (t=0s), a fixed cross was displayed on the screen, such as... Figure 3 As shown.

[0091] During the subject's motor imagery exercise, the steady-state visual evoked interface displays small squares indicating up, down, left, and right. Figure 4 As shown, each small square on the black screen flashes at a different frequency, stimulating the subject to perform corresponding motion imagery.

[0092] Electroencephalogram (EEG) signals can be continuously recorded near the cerebral cortex of the subject using EEG acquisition equipment, and the signals are transmitted to a storage medium via wired or wireless transmission. Under single or multiple paradigms of stimulation, the activity of specific areas of the subject's brain will be recorded.

[0093] The device connected to the subject for acquiring EEG signals can be a non-invasive 8-channel EEG acquisition unit, consisting of an EEG cap, 8-channel electrodes, an amplifier, and transmission lines. It runs on a Windows platform with accompanying recording software. The 8-channel electrodes are as follows... Figure 5 As shown, the user wears an EEG cap and stands in front of a computer screen. Visual signals output from the cloud application server are displayed on the screen, stimulating the user to visualize movement. Signal acquisition devices collect the user's EEG signals. The non-invasive brain-computer interface (BCI) measures EEG through the scalp.

[0094] The cloud application server system is designed to connect to external EEG acquisition devices (i.e., signal acquisition devices) to continuously acquire and record EEG signals.

[0095] The monitor is set to a black background and uses text, arrows, and other visual aids to guide the subject in visualizing actions. This facilitates guidance by having the subject sit in a comfortable chair, wearing an EEG cap, and facing the monitor.

[0096] The experiment involves grouping multiple subjects. Because EEG experiments require a high degree of concentration from the subjects, and they expend energy quickly during the experiment, the data collection task is usually divided into multiple groups when a large amount of training data is needed.

[0097] 21-2. Preparation for auditory evoked potentials.

[0098] The cloud application server outputs a short alarm tone. For example, two seconds after the sound is output (t=2 seconds), an arrow-shaped cue is displayed on the screen, pointing left, right, down, or up, and remains on the screen for 1.25 seconds. This prompts the subject to perform the required motor imagery task until the fixed cross on the screen disappears at t=6 seconds, the screen goes black again, and a brief pause occurs. Figure 6 As shown.

[0099] During the experiment, the cloud application server outputs corresponding instructions to guide the subjects to complete the corresponding motor imagery, and the system records the steady-state somatosensory evoked signals.

[0100] Step 22: The cloud server provides experimental guidance.

[0101] 22-1. At the 0th second of the experiment, a crosshair appears in the center of the screen, indicating to the subject that the experiment has begun.

[0102] 22-2. At the 2nd second of the experiment, a buzzer sound is emitted to remind the subject to concentrate.

[0103] 22-3. At the 3-second mark of the experiment, an arrow is displayed at the location of the crosshair on the screen. The direction of the arrow indicates the type of task being performed; different directions represent different tasks. For example, an arrow pointing left indicates left-hand movement, and an arrow pointing right indicates right-hand movement. Subjects are guided to continuously imagine the movement of the corresponding body part according to this instruction until the crosshair disappears. This state of motion visualization lasts approximately 4 seconds.

[0104] After the motor imagery session ends, a short rest period is taken before the experiment can resume. Simultaneously, a small dot flashing at a frequency of 10Hz is added to the upper right corner of the screen. Users can trigger idle state detection and 10Hz steady-state visual evoked responses by looking at this dot. Optionally, this control is used relatively infrequently and is considered an assistive function, so subjects do not rely heavily on the screen during use.

[0105] Finally, the EEG signals of the subjects during motor imagery were collected, recorded, and labeled accordingly.

[0106] Step 23: Collect vital signs during the process of motor imagery.

[0107] 23-1. Setting up a single body part experiment (trail): Imagining movement in a single direction for a single body part.

[0108] 23-2. The body parts for motor imagery include: left hand, right hand, both feet, and tongue.

[0109] 23-3. Based on motor imagery, extract and fuse brain signals from the brain in four directions (up, down, left, and right) in a multimodal manner.

[0110] Even when a subject imagines moving their limbs (or muscles) but does not actually move them, certain brain regions are still activated.

[0111] Step 24: Perform data fusion, which can be performed by the body area network data center or by the cloud application server.

[0112] Multiple body parts were classified into four categories: left hand, right hand, both feet, and tongue. Different sensors recorded synchronized vital signs under different evoked stimuli.

[0113] The integrated control experiment involved multimodal active imagination and visual and auditory evoked stimuli, forming a signal under a fused EEG signal paradigm.

[0114] By detecting and identifying the activation effects of different brain regions, the user's intention can be determined, thereby enabling direct communication and control between the human brain and external devices.

[0115] For the same subject, different physical sign collection terminals perform physical sign signal alignment and fusion, feature extraction, and output the physiological location topographic heat map feature matrix of EEG and related physical sign signals at each moment according to the sampling rate.

[0116] Step 25: Data processing and offline training.

[0117] After completing a series of experiments in multiple rounds, several labeled EEG signals were obtained, and their features could be extracted for classification.

[0118] The signal is preprocessed, and the locally acquired data will undergo processes including but not limited to filtering, elimination of electrooculography interference, artifact removal, and correction.

[0119] After preprocessing, feature extraction, and classification of this data, a classification model that matches the subject can be obtained.

[0120] Step 27: Population comparison experiment.

[0121] Group: Each group contains one of four categories. Each category has 12 trials, including four different directions.

[0122] Data collection sessions: 2 times, with a 7-day interval. Each session contains 6 groups.

[0123] Nine subjects with normal corrected vision and normal limb function were repeatedly sampled.

[0124] Local information is packaged and uploaded to the cloud platform, where it is classified and modeled. Once a sufficient training set is available, newly collected EEG data from new users is uploaded to the cloud platform, which quickly matches appropriate existing twin models to analyze and process the new users' information.

[0125] II. The online testing section includes:

[0126] Step 28: The cloud application server outputs visual signals during the online testing process to guide the user to complete the test.

[0127] Step 28-1: Design an experimental paradigm interface that presents a visual task that is completed through motion and imagination, such as controlling the movement of small squares in the interface.

[0128] Step 28-2: Load the classification model of the subject, which was calculated during offline training, into the experimental paradigm.

[0129] Step 28-3: While the subject is trying to complete the task, they will engage in a series of regular motion visualizations, and the small squares that flash at a frequency of 10Hz are responsible for controlling the output command of retraction.

[0130] For example, when a subject is not satisfied with the current position of the small square and needs to back away, they can trigger motion imagery idle state detection and 10Hz steady-state visual evoked by looking at the small square in the upper right corner.

[0131] Step 28-4: A series of regular, classifiable unlabeled EEG signals were generated. The experimental paradigm collected these EEG signals and input them into the classification model.

[0132] Step 28-5: After the classification model calculates the label of this EEG signal, it generates instructions to complete the task corresponding to the label until the end.

[0133] Step 29: Analyze and judge the data.

[0134] Step 29-1: For the EEG signal within the 2-second time window, first determine whether the EEG signal within the 2-second window is in a motion state;

[0135] Step 29-2: If the state is determined to be in motion, the classification model will perform calculations and output corresponding instructions.

[0136] Step 29-3: If it is determined to be in an idle state, it is processed by the SSVEP algorithm. If the classification result is a 10Hz SSVEP response, the instruction to backtrack one unit is output.

[0137] The test set yielded preliminary performance results across four different aspects. The system achieved an average accuracy of 90.78%, with all individual parts achieving accuracy above 80%, which is 20% higher than a single-mode test using standard EEG.

[0138] In this embodiment, using motor imagery as the main experimental paradigm allows subjects to reduce their reliance on screens and perform basic operations on external devices solely through motor imagery. While motor imagery offers limited freedom, steady-state visual evoked responses can compensate for this.

[0139] In this application, multimodal stimulation can simultaneously activate brain regions responsible for corresponding sensory functions, thereby enhancing a person's response to external stimuli. This phenomenon is called multimodal sensory integration. Therefore, using multimodal stimulation enhances the brain's response to stimulus events, thereby improving the target recognition accuracy of spontaneous motor imagery (BCI). This includes assisting with different types of eliciting stimuli during spontaneous motor imagery, including visual, auditory, and somatosensory stimuli.

[0140] During spontaneous motor imagery, different types of evoked stimuli, including visual, auditory, and somatosensory stimuli, are used. Multimodal stimulation can simultaneously activate brain regions responsible for corresponding sensory functions, thereby enhancing the individual's response to external stimuli and achieving multimodal sensory integration. Since multimodal EEG data do not carry entirely identical information, multimodal fusion can achieve information complementarity, and multimodal EEG data fusion helps improve the signal quality of brain-computer interfaces.

[0141] This embodiment proposes a method for simultaneously inducing multimodal EEG signals. A hybrid experimental paradigm is employed, primarily based on motor imagery and secondarily on steady-state visual evoked signals, allowing the two to complement each other's strengths. Negative impacts are minimized by appropriately setting the degrees of freedom within the experimental paradigm. Applying steady-state visual evoked signals to infrequently used operations requiring only single-use control, such as turning machines on and off or pausing them, minimizes screen dependence. The experimental paradigm is further improved based on the correlation characteristics between motor imagery and steady-state visual evoked signals, further reducing screen dependence, increasing the paradigm's degrees of freedom, and achieving high-precision, accurate, and rapid control.

[0142] This embodiment's multimodal fusion EEG signal control method can extract stable and valuable features from EEG signals. Based on the characteristics of motor imagery EEG signals and combined with other EEG signals, a universally applicable MI-BCI hybrid experimental paradigm is proposed, which minimizes dependence on external stimuli such as screens while maximizing degrees of freedom. This achieves multimodal EEG signal enhancement. Effective induction of EEG signals is achieved by stimulating brain-computer interface signals with signals from multiple modalities. Simultaneous acquisition of signals from multiple modalities enables signal fusion in everyday environments.

[0143] The embodiments of this application employ a hybrid experimental paradigm, primarily based on motor imagery and secondarily on steady-state visual induction, to effectively induce EEG signals and ensure the effective operation of the brain-computer interface system. A user-matched classification model is constructed, trained on EEG and physiological signals under multimodal stimuli such as visual and auditory stimuli. This reduces the user's dependence on the screen and enhances the freedom of the user's motor imagery. Analyzing the collected EEG signals using the pre-trained classification model accurately identifies the user's intentions, thereby generating correct control commands and enabling accurate control of external devices based on EEG signals.

[0144] like Figure 7 As shown in the figure, this application embodiment also provides a control device 700 based on electroencephalogram (EEG) signals, including:

[0145] The first output module 710 is used to output a first visual signal when the user establishes a connection with the signal acquisition device. The first visual signal is used to guide the user to imagine the movement of the first task.

[0146] The first receiving module 720 is used to receive the electroencephalogram (EEG) signals of the user's motor imagery during the first visual signal output process;

[0147] The first processing module 730 is used to analyze the EEG signal using a pre-trained classification model to obtain the label of the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task.

[0148] The first sending module 740 is used to send the first instruction to the device corresponding to the first task;

[0149] The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

[0150] Optionally, the device further includes:

[0151] The judgment module is used to determine whether the EEG signal is in a motion state;

[0152] The second processing module is used to analyze the EEG signal using a pre-trained classification model if the EEG signal is in a motion state.

[0153] Optionally, the device further includes:

[0154] The third processing module is used to analyze the EEG signal according to the steady-state visual evoked potential (SSVEP) algorithm if the EEG signal is in an idle state, and obtain the analysis results.

[0155] If the analysis results indicate that the SSVEP response contains a preset frequency, then the first command is output.

[0156] Optionally, the device further includes:

[0157] The acquisition module is used to acquire offline training data from multiple users, including the users' electroencephalogram (EEG) signals and vital signs signals.

[0158] The fourth processing module is used to classify and model the offline training data to obtain classification models that match different users.

[0159] Optionally, the device further includes:

[0160] The second output module is used to output a second visual signal and an auditory signal when the user is being trained offline. The second visual signal and the auditory signal are used to prompt the user to perform motion visualization.

[0161] The second receiving module is used to continuously output a third visual signal during a first time period, and to receive the EEG signals and vital signs signals of the user during the output of the third visual signal; the EEG signals include EEG signals of motor imagination of N body parts, and the vital signs signals include vital signs signals of N body parts during the motor imagination process, where N is a positive integer.

[0162] The fifth processing module is used to fuse the electroencephalogram (EEG) signal and the vital signs signal to obtain a first fused signal;

[0163] The sixth processing module is used to generate a classification model that matches the user based on the first fused signal.

[0164] Optionally, the second visual signal corresponds to an image indicating multiple directions, and the image in each direction flashes at a preset frequency.

[0165] Optionally, the EEG signals obtained from offline training of the user have corresponding labels, which are used to indicate the content of the user's motor imagery.

[0166] It should be noted that the apparatus provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0167] like Figure 8 As shown, an electronic device 800 according to an embodiment of this application includes a processor 810 and a transceiver 820;

[0168] The transceiver 820 is configured to: output a first visual signal when a connection is established between the user and the signal acquisition device, the first visual signal being used to guide the user to perform motor imagery about a first task; and receive the electroencephalogram (EEG) signals of the user performing motor imagery during the output of the first visual signal.

[0169] The processor 810 is configured to: analyze the EEG signal using a pre-trained classification model to obtain a label for the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task;

[0170] The transceiver 820 is also used to: send the first instruction to the device corresponding to the first task;

[0171] The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

[0172] Optionally, the processor is further configured to:

[0173] Determine whether the EEG signal is in a motion state;

[0174] If the EEG signal is in motion, then the pre-trained classification model is used to analyze the EEG signal.

[0175] Optionally, the processor is further configured to:

[0176] If the EEG signal is in an idle state, the EEG signal is analyzed according to the steady-state visual evoked potential (SSVEP) algorithm to obtain the analysis results;

[0177] If the analysis results indicate that the SSVEP response contains a preset frequency, then the first command is output.

[0178] Optionally, the processor is further configured to:

[0179] Acquire offline training data from multiple users, including the users' electroencephalogram (EEG) signals and vital signs;

[0180] The offline training data is classified and modeled to obtain classification models that match different users.

[0181] Optionally, the transceiver is also used for:

[0182] When the user is trained offline, a second visual signal and an auditory signal are output, which are used to prompt the user to perform motion visualization.

[0183] The system continuously outputs third visual signals during the first time period and receives EEG signals and vital signs signals of the user during the third visual signal output process. The EEG signals include EEG signals of motor imagination of N body parts, and the vital signs signals include vital signs signals of N body parts during the motor imagination process, where N is a positive integer.

[0184] The processor is further configured to: fuse the electroencephalogram (EEG) signal and the vital signs signal to obtain a first fused signal; and generate a classification model matching the user based on the first fused signal.

[0185] Optionally, the second visual signal corresponds to an image indicating multiple directions, and the image in each direction flashes at a preset frequency.

[0186] Optionally, the EEG signals obtained from offline training of the user have corresponding labels, which are used to indicate the content of the user's motor imagery.

[0187] It should be noted that the device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0188] Another embodiment of the electronic device in this application, such as Figure 9 As shown, it includes a transceiver 910, a processor 900, a memory 920, and a program or instructions stored in the memory 920 and executable on the processor 900; when the processor 900 executes the program or instructions, it implements the above-mentioned control method based on EEG signals.

[0189] The transceiver 910 is used to receive and send data under the control of the processor 900.

[0190] Among them, Figure 9 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 900) and memory (memory 920). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 910 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 can store data used by the processor 900 during operation.

[0191] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the control method based on EEG signals as described above and achieve the same technical effect. To avoid repetition, this will not be repeated here.

[0192] The processor mentioned above is the processor in the network device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] This application also provides a computer program product that stores a program or instructions, including computer instructions. When these computer instructions are executed by a processor, they implement the steps in the control method based on EEG signals as described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0194] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0195] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0196] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0197] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0198] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A control method based on electroencephalogram (EEG) signals, characterized in that, include: Once a connection is established between the user and the signal acquisition device, a first visual signal is output, which is used to guide the user to visualize the movement of the first task. Receive the EEG signals of the user during the first visual signal output process to perform motor imagery; The EEG signal is analyzed using a pre-trained classification model to obtain a label for the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task. Send the first instruction to the device corresponding to the first task; The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

2. The method according to claim 1, characterized in that, Before analyzing the EEG signals using a pre-trained classification model, the method further includes: Determine whether the EEG signal is in a motion state; If the EEG signal is in motion, then the pre-trained classification model is used to analyze the EEG signal.

3. The method according to claim 2, characterized in that, The method further includes: If the EEG signal is in an idle state, the EEG signal is analyzed according to the steady-state visual evoked potential (SSVEP) algorithm to obtain the analysis results; If the analysis results indicate that the SSVEP response contains a preset frequency, then the first command is output.

4. The method according to claim 1, characterized in that, The method further includes: Acquire offline training data from multiple users, including the users' electroencephalogram (EEG) signals and vital signs; The offline training data is classified and modeled to obtain classification models that match different users.

5. The method according to claim 1 or 4, characterized in that, The method further includes: When the user is trained offline, a second visual signal and an auditory signal are output, which are used to prompt the user to perform motion visualization. The system continuously outputs third visual signals during the first time period and receives EEG signals and vital signs signals of the user during the third visual signal output process. The EEG signals include EEG signals of motor imagination of N body parts, and the vital signs signals include vital signs signals of N body parts during the motor imagination process, where N is a positive integer. The electroencephalogram (EEG) signal and the vital signs signal are fused to obtain a first fused signal; A classification model matching the user is generated based on the first fused signal.

6. The method according to claim 5, characterized in that, The second visual signal corresponds to an image indicating multiple directions, and the image in each direction flashes at a preset frequency.

7. The method according to claim 5, characterized in that, The EEG signals obtained from offline training of the user are labeled with corresponding tags, which are used to indicate the content of the user's motor imagery.

8. A control device based on electroencephalogram (EEG) signals, characterized in that, include: The first output module is used to output a first visual signal when the user establishes a connection with the signal acquisition device. The first visual signal is used to guide the user to imagine the movement of the first task. The first receiving module is used to receive the electroencephalogram (EEG) signals of the user's motor imagery during the first visual signal output process; The first processing module is used to analyze the EEG signal using a pre-trained classification model to obtain the label of the EEG signal and a first instruction, wherein the first instruction is a control instruction for the first task. The first sending module is used to send the first instruction to the device corresponding to the first task; The classification model is obtained by training based on a first fusion signal, which is a fusion of EEG signals induced by visual signals, EEG signals induced by auditory signals, and vital signs signals.

9. An electronic device, characterized in that, It includes a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the steps of the control method based on electroencephalogram (EEG) signals as described in any one of claims 1 to 7.

10. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the control method based on EEG signals as described in any one of claims 1-7.

11. A computer program product having a program or instructions stored thereon, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the control method based on electroencephalogram (EEG) signals as described in any one of claims 1-7.

Images