Multi-mode brain-computer interaction method and system

Abstract

The invention discloses a multi-mode brain-computer interaction method and system. The multi-mode brain-computer interaction system is composed of a signal acquisition module, a control module and a human-computer interaction interface module. The signal acquisition module is used for acquiring various signals, which can be used for man-machine interaction, of a user, including electroencephalogram, myoelectricity, electrooculogram, voice, posture, somatosensory human body signals, position, motion posture and temperature human body surrounding environment signals, and combining the channels to form a multi-channel parallel input information group; the information is transmitted to the control module in a wireless communication mode; and the control module performs digital filtering, signal processing, classification detection and fusion algorithm identification according to the acquired multi-mode signal, and outputs a corresponding instruction according to an identification result for corresponding interactive operation.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal brain-computer interaction, and in particular to a multimodal brain-computer interaction method and system. Background Art

[0002] In the current field of science and technology, human-computer interaction technology is developing rapidly, among which brain-computer interaction (BCI) technology has received particular attention. Traditional brain-computer interaction technology mainly relies on electroencephalogram (EEG) signals to achieve direct communication between humans and machines. However, these technologies have some limitations and challenges:

[0003] Limitations of signal acquisition: Existing technologies usually only collect a single type of biological signal, such as EEG signals, while ignoring other potentially useful biological signals, such as electromyography and electrooculography, which limits the accuracy and richness of the interaction;

[0004] Poor environmental adaptability: Many existing BCI systems are sensitive to environmental noise and easily affected by external interference, resulting in reduced accuracy of signal recognition;

[0005] Real-time and efficiency issues: In some application scenarios, such as virtual reality (VR) or emergency medical response, the real-time requirements of the system are extremely high. Existing technologies may have delays when processing and analyzing signals, affecting the smoothness of interaction;

[0006] User training and adaptability: Traditional BCI systems usually require users to undergo long-term training to adapt to the system's interaction mode, which limits the popularity and ease of use of the technology;

[0007] Challenges of multimodal fusion: Although multimodal signal fusion can theoretically improve the accuracy and robustness of interaction, existing technologies still face technical difficulties in achieving effective signal fusion. Summary of the invention

[0008] In order to solve the above-mentioned technical problems, the present invention provides a multimodal brain-computer interaction method and system.

[0009] The technical solution of the present invention is achieved in this way:

[0010] A multimodal brain-computer interaction system, comprising a signal acquisition module, a control module, and a human-computer interaction interface module;

[0011] The signal acquisition module is used to collect various signals of the user that can be used for human-computer interaction, including EEG, EMG, EOG, voice, posture, body sense human body signals, and human surrounding environment signals such as position, movement posture, and temperature, and combine these channels to form a multi-channel parallel input information group, which is transmitted to the control module by wireless communication;

[0012] The control module includes a communication unit, a digital filtering unit, a signal processing unit, a classification detection unit, a fusion algorithm recognition unit, and an API interface unit. Various input signals are first classified and recognized according to the human-computer interaction type, and then different weights are assigned or different stages of interaction work are undertaken according to the requirements of the interaction. Finally, the recognition results are combined to form the corresponding instructions for output;

[0013] The human-machine interaction interface module is composed of a human-machine interface and a communication interface (API), and is used to match control instructions of the control module under different hardware environments and run the entire operation interface.

[0014] Furthermore, the signal acquisition module is a wearable device, which is worn on the corresponding positions of the user's head and torso, and the signal acquisition module includes a front-end signal acquisition sensor group, a signal conversion and processing unit, a microprocessor unit and a wireless communication unit;

[0015] The signal acquisition module amplifies and filters the collected signals and converts them into digital signals. The resulting raw data passes through the wireless communication network; the multi-channel different modal signals are transmitted synchronously, that is, after inserting synchronization markers, they are fused to form multi-channel raw data.

[0016] Furthermore, the signal acquisition module also introduces real-time data marking during the synchronous transmission of signals of different modalities. Specifically, when a stimulus occurs, the time when the stimulus occurs is recorded and a mark is generated. The OpenDevices operation is performed through the API interface to form a stimulus marking channel MarkerChannel, and then the marking information SendMarkerChannel is sent to the lower computer, that is, the signal acquisition module. After receiving the marker channel data, the signal acquisition module inserts it into the collected data channel hBCIDateChannel to become a marked data channel hBCIMarkerDateChannel. The signal acquisition module then sends the marked data to the host and performs classification and identification operations on the data through OpenSevers. Among them, API is a standardized program interface, which connects various devices or obtains service support through list-based management.

[0017] Furthermore, after receiving the marker channel data, the signal acquisition module inserts the data into the acquired data channel hBCIDateChannel.

[0018] A multimodal brain-computer interaction method comprises the following steps:

[0019] S1: A software client is provided, and the human-computer interaction interface module of the client is connected to the signal acquisition module and the control module in a wired or wireless manner. The user wears the acquisition device and completes the acquisition preparation (such as placing several electrodes close to the user's skin to collect bioelectric signals, placing a microphone near the mouth, and aiming the camera at the human body, etc.);

[0020] S2: Setting parameters of the program in the human-computer interaction interface (GUI), including but not limited to the type of human-computer signal used for interaction, the prompt (stimulation) method of the signal, the requirements for command output (synchronous or asynchronous mode, speed, accuracy, etc.), and the human-computer interaction paradigm;

[0021] S3: Before the system starts running, a training operation is performed first. Specifically, the user looks at the training interface and uses the interaction set in S2 to perform corresponding operations. The acquisition device synchronously acquires the interactive signal, performs bandpass filtering on the acquisition, and intercepts the data into sampling points of a certain length of time;

[0022] S4: Execute the recognition operation of human-computer interaction, prompt (stimulate) each virtual object or button in the GUI in a certain order, and when the user needs to select the virtual object or button, determine the target by the operation in S3, and the acquisition device synchronously acquires various modal signals, performs bandpass filtering on the acquisition, and intercepts the data into sampling points of a certain length of time;

[0023] S5: Compare the feature vector generated by S4 with the training pattern using a certain algorithm. If the similarity exceeds a threshold, it is determined to be a target button, and the serial number of the target button is sent to the display module.

[0024] S6: The client completes the recognition and identification of the target button, transmits the result to the human-computer interaction interface module, and executes the function of the option.

[0025] Furthermore, in S2, when there are multiple user graphical interfaces GUI, the ID number of the currently displayed GUI is transmitted back to the control module to provide a reference for the client's signal recognition. Thereafter, the GUI interface generates options with different functions for the user to choose from. Each option corresponds to a virtual object or button, and the user completes the corresponding operation by selecting the required virtual object or button.

[0026] Furthermore, in S3, the sampling points are downsampled, and the downsampled data is used to form a feature vector. The computer stores the feature vector corresponding to each interaction; the result obtained after the feature vector is averaged is used as a threshold to establish a training model for the user in preparation for subsequent recognition operations.

[0027] Furthermore, in S4, the sampling points are downsampled and waveform detection is performed. If the features are detected by waveform, it is considered that there is a potential target, which is transformed by a certain algorithm to form a feature vector; the computer stores the corresponding feature vector when each virtual object or button flashes.

[0028] The multimodal brain-computer interaction method and system proposed in the present invention provide the following beneficial effects in view of the limitations of the above-mentioned prior art:

[0029] Multimodal signal acquisition: By integrating the acquisition of multiple biological signals such as EEG, EMG, EOG, voice, posture, and body sensation, the present invention can more comprehensively capture the user's intention and status, and improve the accuracy and reliability of interaction;

[0030] Enhanced environmental adaptability: Using advanced digital filtering and signal processing technology, it effectively reduces the interference of environmental noise and improves the accuracy of signal recognition;

[0031] Improve real-time performance and efficiency: By optimizing algorithms and adopting efficient data processing technology, the present invention significantly reduces the delay of signal processing and meets application scenarios with high real-time requirements;

[0032] Simplify the user training process: Through intelligent training models and adaptive learning algorithms, the present invention reduces the time and difficulty of user training and improves the ease of use and popularity of the system;

[0033] Effective multi-modal signal fusion: The present invention adopts an innovative signal fusion algorithm, which can reasonably integrate signals of different modes and improve the robustness and intelligence of the interactive system;

[0034] Wearable device design: The signal acquisition module adopts a wearable design, which allows users to use the system more conveniently and improves the comfort and naturalness of signal acquisition;

[0035] Real-time data labeling method: The developed real-time data labeling method ensures the synchronization and accuracy of data, avoids data shift, and further improves the accuracy of signal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a principle framework diagram of a multimodal brain-computer interaction method and system of this embodiment 1;

[0037] Figure 2 A flow chart of real-time data labeling introduced by the signal acquisition module in the present invention;

[0038] Figure 3 The figure is a schematic diagram of a method for forming raw data with time stamp by a signal acquisition module in the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1

[0041] This embodiment discloses a multimodal brain-computer interaction system, including a signal acquisition module, a control module, and a human-computer interaction interface module;

[0042] The signal acquisition module is used to collect various signals of the user that can be used for human-computer interaction, including EEG, EMG, EOG, voice, posture, body sense human body signals, and human surrounding environment signals such as position, movement posture, and temperature, and combine these channels to form a multi-channel parallel input information group, which is transmitted to the control module by wireless communication;

[0043] The control module includes a communication unit, a digital filtering unit, a signal processing unit, a classification detection unit, a fusion algorithm recognition unit, and an API interface unit. Various input signals are first classified and recognized according to the human-computer interaction type, and then different weights are assigned or different stages of interaction work are undertaken according to the requirements of the interaction. Finally, the recognition results are combined to form the corresponding instructions for output;

[0044] The human-machine interaction interface module is composed of a human-machine interface and a communication interface (API), and is used to match control instructions of the control module under different hardware environments and run the entire operation interface.

[0045] Furthermore, the signal acquisition module is a wearable device, which is worn on the corresponding positions of the user's head and torso, and the signal acquisition module includes a front-end signal acquisition sensor group, a signal conversion and processing unit, a microprocessor unit and a wireless communication unit;

[0046] The signal acquisition module amplifies and filters the collected signals and converts them into digital signals. The resulting raw data passes through the wireless communication network; the multi-channel different modal signals are transmitted synchronously, that is, after inserting synchronization markers, they are fused to form multi-channel raw data.

[0047] Furthermore, the process of the signal acquisition module fusing and forming multi-channel raw data is as follows:

[0048] (1) Convert the original signals of different modes into original digital signals;

[0049] (2) adding type coding information to these original digital signals to form digital signals with type coding;

[0050] (3) Various digital signals are integrated into a mixed signal and sent to the control module through the wireless module;

[0051] (4) In the control module, the transmitted mixed signal is split into a single-mode digital signal through encoding information for use and recognition by other sub-modules.

[0052] Furthermore, the signal acquisition module also introduces real-time data marking during the synchronous transmission of signals of different modalities. Specifically, when a stimulus occurs, the time when the stimulus occurs is recorded and a mark is generated. The OpenDevices operation is performed through the API interface to form a stimulus marking channel MarkerChannel, and then the marking information SendMarkerChannel is sent to the lower computer, that is, the signal acquisition module. After receiving the marker channel data, the signal acquisition module inserts it into the collected data channel hBCIDateChannel to become a marked data channel hBCIMarkerDateChannel. The signal acquisition module then sends the marked data to the host and performs classification and identification operations on the data through OpenSevers. Among them, API is a standardized program interface, which connects various devices or obtains service support through list-based management.

[0053] Furthermore, after the signal acquisition module receives the marker channel data, the specific process of inserting it into the acquired data channel hBCIDateChannel is as follows:

[0054] (1) The marker information module Send Marker Channel will provide time-stamped signal codes for different signal types;

[0055] (2) The received signal is encoded and matched to the corresponding information channel, and the corresponding encoding code is inserted into the original data with a time axis to form the original data with a time mark.

[0056] Furthermore, when the signal acquisition module sends the marked data to the host, data mark detection is also introduced, specifically,

[0057] (1) Set the interval time ti for the lower computer to send data to the upper computer;

[0058] (2) Always check whether the received data is marked;

[0059] (3) If the data is marked, determine whether the current moment is a sampling interval. The sampling interval refers to the current sampling stage when the hBCI data sampling and reading have been completed but the sampling data has not yet been transmitted to the host computer;

[0060] If the current period is in the sampling interval, the markers are inserted into the sampled data according to the time interval requirements; otherwise, the number of markers continues to increase according to the time interval, but is temporarily saved and not output. After the sampling is completed, the temporarily stored markers are inserted into the sampled data at one time, and the process returns to step 2.

[0061] (4) If the data is not marked, no processing is done and the process returns to step 2;

[0062] In step (3), Di is the delay error, ti is the sampling time point, Ni is the marking time point, and the time error has two major influencing factors: system response delay and process inherent delay. The controller starts multiple threads at the same time to complete data collection and marking tasks, reducing response delay and process inherent delay. The delay time Di is inferred, Di <T:

[0063]

[0064] The delay error Di is controlled within one sampling period, and the data mark Ni will not be mistakenly operated as the next mark Ni+1, that is, data shift will not occur.

[0065] This embodiment collects various types of biological signals and environmental signals through the signal acquisition module, such as EEG, EMG, EOG, voice, posture, body sensation, etc., as well as environmental signals such as position, motion posture and temperature. This multi-channel parallel input information group is transmitted to the control module through wireless communication, which increases the accuracy and richness of the interaction.

[0066] The collected signals are filtered by the digital filter unit to reduce the impact of environmental noise. In addition, the fusion algorithm recognition unit performs comprehensive analysis on different modal signals to improve the system's robustness to external interference.

[0067] The signal processing unit and classification detection unit in the control module can quickly process and analyze signals to achieve real-time requirements. At the same time, the matching of the API interface unit and the hardware environment ensures the rapid execution of control instructions and improves the fluency of interaction. The fusion algorithm recognition unit learns and adapts to the user's interactive signals, reducing the user's training time. In addition, the human-computer interaction interface module can be adjusted according to the different needs of users, improving the ease of use of the system.

[0068] Furthermore, the fusion algorithm recognition unit assigns weights and stages to signals of different modes, thus achieving effective signal fusion. This fusion of multimodal signals improves the accuracy and robustness of interaction. Real-time data labeling and synchronous transmission technology ensure the synchronization and time accuracy of signals of different modes, which helps to improve the accuracy of signal processing, especially in interactive scenarios that require precise time alignment.

[0069] The data marker detection mechanism is introduced to ensure the accurate insertion of data markers during the sampling interval, avoiding data shift and delay errors, and further improving the real-time performance and accuracy of signal processing;

[0070] The signal acquisition module of this embodiment is designed as a wearable device, which improves the user's comfort and the portability of the system. This design allows users to conveniently use the BCI system in their daily lives.

[0071] Example 2

[0072] This embodiment discloses a multimodal brain-computer interaction method, comprising the following steps:

[0073] S1: A software client is provided, and the human-computer interaction interface module of the client is connected to the signal acquisition module and the control module in a wired or wireless manner. The user wears the acquisition device and completes the acquisition preparation (such as placing several electrodes close to the user's skin to collect bioelectric signals, placing a microphone near the mouth, and aiming the camera at the human body, etc.);

[0074] S2: Setting parameters of the program in the human-computer interaction interface (GUI), including but not limited to the type of human-computer signal used for interaction, the prompt (stimulation) method of the signal, the requirements for command output (synchronous or asynchronous mode, speed, accuracy, etc.), and the human-computer interaction paradigm;

[0075] S3: Before the system starts running, a training operation is performed first. Specifically, the user looks at the training interface and uses the interaction set in S2 to perform corresponding operations. The acquisition device synchronously acquires the interactive signal, performs bandpass filtering on the acquisition, and intercepts the data into sampling points of a certain length of time;

[0076] S4: Execute the recognition operation of human-computer interaction, prompt (stimulate) each virtual object or button in the GUI in a certain order, and when the user needs to select the virtual object or button, determine the target by the operation in S3, and the acquisition device synchronously acquires various modal signals, performs bandpass filtering on the acquisition, and intercepts the data into sampling points of a certain length of time;

[0077] S5: Compare the feature vector generated by S4 with the training pattern using a certain algorithm. If the similarity exceeds a threshold, it is determined to be a target button, and the serial number of the target button is sent to the display module.

[0078] S6: The client completes the recognition and identification of the target button, transmits the result to the human-computer interaction interface module, and executes the function of the option.

[0079] Furthermore, in S2, when there are multiple user graphical interfaces GUI, the ID number of the currently displayed GUI is transmitted back to the control module to provide a reference for the client's signal recognition. Thereafter, the GUI interface generates options with different functions for the user to choose from. Each option corresponds to a virtual object or button, and the user completes the corresponding operation by selecting the required virtual object or button.

[0080] Furthermore, in S3, the sampling points are downsampled, and the downsampled data is used to form a feature vector. The computer stores the feature vector corresponding to each interaction; the result obtained after the feature vector is averaged is used as a threshold to establish a training model for the user in preparation for subsequent recognition operations.

[0081] Furthermore, in S4, the sampling points are downsampled and waveform detection is performed. If the features are detected by waveform, it is considered that there is a potential target, which is transformed by a certain algorithm to form a feature vector; the computer stores the corresponding feature vector when each virtual object or button flashes.

[0082] This embodiment enhances the accuracy and richness of the interaction by collecting multiple types of biological signals (such as EEG signals, EMG signals, EOG signals, etc.). By fusing multimodal signals, the user's intention and state can be understood more comprehensively, thereby improving the accuracy of the interaction.

[0083] Through bandpass filtering technology, this method can reduce the impact of environmental noise on signal acquisition, improve signal clarity and recognition accuracy. In addition, the fusion of multimodal signals also helps to improve the system's adaptability and robustness to environmental changes.

[0084] In S3 and S4, downsampling and waveform detection technology are used to quickly extract key features, reduce data processing time, and improve the real-time performance of the system. At the same time, through the establishment of training models, the user's interaction intention can be quickly identified, further improving the efficiency of interaction;

[0085] Through the training operation mentioned in S3, users can adapt to the system's interaction mode in a short time. By collecting users' interaction signals and establishing personalized training models, the system can better adapt to the operating habits of different users and reduce users' learning costs;

[0086] This embodiment realizes the effective fusion of multimodal signals through the construction of feature vectors and analogy algorithms. By storing and analyzing the features of different modal signals by computer, the system can more accurately identify the user's interaction intention and improve the accuracy and robustness of the interaction.

[0087] In S2, users can choose different human-computer interaction paradigms and signal prompt methods according to their needs, which improves the personalization and flexibility of the interaction. At the same time, by returning the GUI ID number to the control module, the system can provide more accurate signal recognition based on the current interaction interface and user operating habits.

Claims

1. A multimodal brain-computer interaction system, characterized in that: Including signal acquisition module, control module, and human-computer interaction interface module; The signal acquisition module is used to collect various signals of the user that can be used for human-computer interaction, including EEG, EMG, EOG, voice, posture, body sense human body signals, and human surrounding environment signals such as position, movement posture, and temperature, and combine these channels to form a multi-channel parallel input information group, which is transmitted to the control module by wireless communication; The control module includes a communication unit, a digital filtering unit, a signal processing unit, a classification detection unit, a fusion algorithm recognition unit, and an API interface unit. Various input signals are first classified and recognized according to the human-computer interaction type, and then different weights are assigned or different stages of interaction work are undertaken according to the requirements of the interaction. Finally, the recognition results are combined to form the corresponding instructions for output; The human-machine interaction interface module is composed of a human-machine interface and a communication interface (API), and is used to match control instructions of the control module under different hardware environments and run the entire operation interface.

2. A multimodal brain-computer interaction system according to claim 1, characterized in that: The signal acquisition module is a wearable device, which is worn on the corresponding positions of the user's head and torso. The signal acquisition module includes a front-end signal acquisition sensor group, a signal conversion and processing unit, a microprocessor unit and a wireless communication unit; The signal acquisition module amplifies and filters the collected signals and converts them into digital signals. The resulting raw data passes through the wireless communication network; the multi-channel different modal signals are transmitted synchronously, that is, after inserting synchronization markers, they are fused to form multi-channel raw data.

3. A multimodal brain-computer interaction system according to claim 2, characterized in that: The process of the signal acquisition module fusing and forming multi-channel raw data is as follows: (1) Convert the original signals of different modes into original digital signals; (2) adding type coding information to these original digital signals to form digital signals with type coding; (3) Various digital signals are integrated into a mixed signal and sent to the control module through the wireless module; (4) In the control module, the transmitted mixed signal is split into a single-mode digital signal through encoding information for use and recognition by other sub-modules.

4. A multimodal brain-computer interaction method and system according to claim 2, characterized in that: The signal acquisition module also introduces real-time data marking during the synchronous transmission of signals of different modalities. Specifically, when a stimulus occurs, the time when the stimulus occurs is recorded and a mark is generated. The OpenDevices operation is performed through the API interface to form a stimulus marking channel MarkerChannel, and then the marking information SendMarkerChannel is sent to the lower computer, that is, the signal acquisition module. After receiving the marker channel data, the signal acquisition module inserts it into the collected data channel hBCIDateChannel to become a marked data channel hBCIMarkerDateChannel. Then the signal acquisition module sends the marked data to the host and performs classification and identification operations on the data through OpenSevers. Among them, API is a standardized program interface, which connects various devices or obtains service support through list-based management.

5. A multimodal brain-computer interaction system according to claim 4, characterized in that: After the signal acquisition module receives the marker channel data, the specific process of inserting it into the acquired data channel hBCIDateChannel is as follows: (1) The marker information module Send Marker Channel will provide time-stamped signal codes for different signal types; (2) The received signal is encoded and matched to the corresponding information channel, and the corresponding encoding code is inserted into the original data with a time axis to form the original data with a time mark.

6. A multimodal brain-computer interaction system according to claim 5, characterized in that: When the signal acquisition module sends the marked data to the host, data mark detection is also introduced, specifically, (1) Set the interval time ti for the lower computer to send data to the upper computer; (2) Always check whether the received data is marked; (3) If the data is marked, determine whether the current moment is a sampling interval. The sampling interval refers to the current sampling stage when the hBCI data sampling and reading have been completed but the sampling data has not yet been transmitted to the host computer; If the current period is in the sampling interval, the markers are inserted into the sampled data according to the time interval requirements; otherwise, the number of markers continues to increase according to the time interval, but is temporarily saved and not output. After the sampling is completed, the temporarily stored markers are inserted into the sampled data at one time, and the process returns to step 2. (4) If the data is not marked, no processing is done and the process returns to step 2; In step (3), Di is the delay error, ti is the sampling time point, Ni is the marking time point, and the time error has two major influencing factors: system response delay and process inherent delay. The controller starts multiple threads at the same time to complete data collection and marking tasks, reducing response delay and process inherent delay. The delay time Di is inferred, Di <T: The delay error Di is controlled within one sampling period, and the data mark Ni will not be mistakenly operated as the next mark Ni+1, that is, data shift will not occur.

7. A multimodal brain-computer interaction method, characterized in that: The following steps are involved: S1: A software client is provided, the human-computer interaction interface module of the client is connected to the signal acquisition module and the control module by wire or wirelessly, and the user wears the acquisition device to complete the acquisition preparation; S2: Setting parameters of the program in the human-computer interaction interface, including but not limited to the type of human-computer signal used for interaction, the signal prompt method, the requirements for command output, and the human-computer interaction paradigm; S3: Before the system starts running, a training operation is performed first. Specifically, the user looks at the training interface and uses the interaction set in S2 to perform corresponding operations. The acquisition device synchronously acquires the interactive signal, performs bandpass filtering on the acquisition, and intercepts the data into sampling points of a certain length of time; S4: Execute the recognition operation of human-computer interaction, prompt each virtual object or button in the GUI in a certain order, and when the user needs to select the virtual object or button, determine the target by the operation in S3, and the acquisition device synchronously acquires various modal signals, performs bandpass filtering on the acquisition, and intercepts the data into sampling points of a certain length of time; S5: Compare the feature vector generated by S4 with the training pattern using a certain algorithm. If the similarity exceeds a threshold, it is determined to be a target button, and the serial number of the target button is sent to the display module. S6: The client completes the recognition and identification of the target button, transmits the result to the human-computer interaction interface module, and executes the function of the option.

8. A multimodal brain-computer interaction method according to claim 7, characterized in that: In S2, when there are multiple user graphical interfaces GUI, the ID number of the currently displayed GUI is sent back to the control module to provide a reference for the client's signal recognition. After that, the GUI interface generates options with different functions for the user to choose from. Each option corresponds to a virtual object or button, and the user completes the corresponding operation by selecting the required virtual object or button.

9. A multimodal brain-computer interaction method according to claim 7, characterized in that: In S3, the sampling points are downsampled, and the downsampled data is used to form a feature vector, and the computer stores the feature vector corresponding to each interaction; The result obtained after the feature vector is averaged is used as the threshold to establish a training model for the user in preparation for subsequent recognition operations.

10. A multimodal brain-computer interaction method according to claim 7, characterized in that: In S4, the sampling points are downsampled and a waveform detection operation is performed. If its characteristics are detected through waveform, it is considered that there is a potential target, and a feature vector is formed by transformation using a certain algorithm; the computer stores the corresponding feature vector when each virtual object or button flashes.

Images