Control method and control device of brain-computer interface system based on multi-modal signal

Abstract

The invention discloses a control method and control device of a brain-computer interface system based on a multi-modal signal, the brain-computer interface system comprises a plurality of training normal forms, the training normal forms and to-be-collected signals have a corresponding relation, and the control method comprises the following steps: determining a brain-computer interface training normal form; acquiring a multi-dimensional physiological signal of the target user based on a brain-computer interface training normal form; determining a target model classification algorithm based on the multi-dimensional physiological signal, and analyzing the intention of the target user according to the target model classification algorithm; and outputting a control signal based on the intention of the target user, and controlling the external device based on the control signal. Therefore, according to the method, the multi-dimensional physiological signals are collected based on the brain-computer interface training normal form, and the multi-dimensional physiological signals are fused through the target model classification algorithm to determine the user intention so as to generate the control signal to control the external equipment, so that the analysis precision of the user intention and the control accuracy and response speed of the external equipment are improved; and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of brain-computer interface technology, and in particular to a control method for a brain-computer interface system based on multimodal signals, a control device for a brain-computer interface system based on multimodal signals, a computer-readable storage medium, and an electronic device. Background Art

[0002] Brain-computer interface systems are designed to establish a direct communication channel between the brain and a computer or other external device, enabling individuals to control external devices or applications through brain activity without relying on traditional muscle movements or the nervous system.

[0003] In related technologies, brain-computer interface systems capture and analyze electrophysiological signals from the brain, converting these signals into commands that can be recognized by computers or other external devices, thereby controlling external devices. However, the control commands generated by this technical solution are less accurate, which reduces the user experience. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present application is to propose a control method for a brain-computer interface system based on multimodal signals, collect multi-dimensional physiological signals based on the brain-computer interface training paradigm, and determine the user's intention by fusing the multi-dimensional physiological signals through the target model classification algorithm to generate a control signal to control the external device, thereby improving the accuracy of user intention analysis and the control accuracy and response speed of the external device, and improving the user experience.

[0005] The second objective of this application is to propose a control device for a brain-computer interface system based on multimodal signals.

[0006] The third object of the present application is to provide a computer-readable storage medium.

[0007] The fourth objective of the present application is to provide an electronic device.

[0008] To achieve the above-mentioned objectives, the first aspect embodiment of the present application proposes a control method for a brain-computer interface system based on multimodal signals, the brain-computer interface system based on multimodal signals includes multiple training paradigms, and the training paradigms have a corresponding relationship with the signals to be collected. The control method for the brain-computer interface system based on multimodal signals includes: determining the brain-computer interface training paradigm; collecting multi-dimensional physiological signals of the target user based on the brain-computer interface training paradigm; determining a target model classification algorithm based on the multi-dimensional physiological signals, and parsing the target user's intentions according to the target model classification algorithm; outputting a control signal based on the target user's intentions, and controlling external devices based on the control signal.

[0009] According to the control method of the brain-computer interface system based on multimodal signals of the embodiment of the present application, the brain-computer interface training paradigm is first determined, and then the multi-dimensional physiological signals of the target user are collected based on the brain-computer interface training paradigm, and the target model classification algorithm is determined based on the multi-dimensional physiological signals, the intention of the target user is analyzed according to the target model classification algorithm, and the control signal is output based on the intention of the target user, and the external device is controlled based on the control signal. Thus, the method collects multi-dimensional physiological signals based on the brain-computer interface training paradigm, and determines the user's intention by fusing the multi-dimensional physiological signals through the target model classification algorithm, so as to generate a control signal to control the external device, thereby improving the accuracy of user intention analysis and the accuracy and response speed of control of the external device, and improving the user experience.

[0010] In addition, the control method of the brain-computer interface system based on multimodal signals according to the above embodiment of the present application may also have the following additional technical features:

[0011] According to one embodiment of the present application, a target model classification algorithm is determined based on a multi-dimensional physiological signal, including: performing data dimensionality reduction on the multi-dimensional physiological signal by means of principal component analysis, and extracting the physiological signal corresponding to the maximum value in the weight coefficient of the multi-dimensional physiological signal; and determining the target model classification algorithm based on the physiological signal.

[0012] According to one embodiment of the present application, the model classification algorithm includes: at least one of an EEG signal classification algorithm, a classical machine learning classification algorithm, and a deep learning classification algorithm.

[0013] According to one embodiment of the present application, the physiological signal includes at least one of an electroencephalogram signal, a near-infrared spectrum signal, a blood oxygen saturation signal, a pulse wave signal, a skin electrode activity signal, an electrocardiogram signal, and a skin temperature signal.

[0014] According to one embodiment of the present application, before parsing the intention of the target user according to the target model classification algorithm, the control method of the brain-computer interface system based on multimodal signals also includes: preprocessing multi-dimensional physiological signals, wherein the preprocessing method of the EEG signal includes: filtering, artifact removal, and normalization; the preprocessing method of the near-infrared spectrum signal includes: signal denoising, baseline correction, and signal enhancement; the preprocessing method of the blood oxygen saturation signal includes: filtering, peak detection, and signal smoothing; the preprocessing method of the pulse wave signal includes: detrending, filtering, and heart rate variability analysis; the preprocessing method of the skin electrodermal activity signal includes: filtering, baseline correction, and signal standardization; the preprocessing method of the electrocardiogram signal includes: filtering, R wave detection, and heart rate analysis; the preprocessing method of the skin temperature signal includes: filtering, baseline drift correction, and temperature change rate calculation.

[0015] According to one embodiment of the present application, the control signal includes a visual signal, an auditory signal, a tactile signal and a motion control signal, wherein when the control signal includes a visual signal, the external device includes an LED (Light Emitting Diode) lamp and a display; when the control signal includes an auditory signal, the external device includes a buzzer and an earphone; when the control signal includes a tactile signal, the external device includes a vibration motor and an electric stimulator; when the control signal includes a motion control signal, the external device includes a wheelchair and a robot.

[0016] According to one embodiment of the present application, the control method of the brain-computer interface system based on multimodal signals also includes: sending the control signal to an external device by using one of a wired communication protocol, a wireless communication protocol, and an open source communication protocol; wherein the wired communication protocol includes at least one of USB (Universal Serial Bus), RS232, and Ethernet, the wireless communication protocol includes at least one of Wi-Fi, Bluetooth, ZigBee, and LoRa, and the open source communication protocol includes at least one of MQTT (Message Queuing Telemetry Transport), CoAP (Constrained Application Protocol), and WebSocket.

[0017] To achieve the above-mentioned purpose, the second aspect embodiment of the present application proposes a control device for a brain-computer interface system based on multimodal signals, wherein the brain-computer interface system based on multimodal signals includes multiple training paradigms, and the training paradigms have a corresponding relationship with the signals to be collected. The control device for the brain-computer interface system based on multimodal signals includes: a determination module, used to determine the brain-computer interface training paradigm; an acquisition module, used to collect multi-dimensional physiological signals of the target user based on the brain-computer interface training paradigm; an analysis module, used to determine a target model classification algorithm based on the multi-dimensional physiological signals, and analyze the target user's intention according to the target model classification algorithm; a control module, used to output a control signal based on the target user's intention, and control an external device based on the control signal.

[0018] According to the control device of the brain-computer interface system based on multimodal signals in the embodiment of the present application, the brain-computer interface training paradigm is determined by the determination module, the multi-dimensional physiological signals of the target user are collected based on the brain-computer interface training paradigm by the acquisition module, the target model classification algorithm is determined based on the multi-dimensional physiological signals by the analysis module, the intention of the target user is analyzed according to the target model classification algorithm, the control module outputs a control signal based on the intention of the target user, and controls the external device based on the control signal. The device collects multi-dimensional physiological signals based on the brain-computer interface training paradigm, and determines the user intention by fusing the multi-dimensional physiological signals through the target model classification algorithm, so as to generate a control signal to control the external device, thereby improving the accuracy of user intention analysis and the control accuracy and response speed of the external device, and improving the user experience.

[0019] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes a computer-readable storage medium, on which a control program for a brain-computer interface system based on multimodal signals is stored. When the control program for a brain-computer interface system based on multimodal signals is executed by a processor, the control method for a brain-computer interface system based on multimodal signals is implemented.

[0020] According to the computer-readable storage medium of the embodiment of the present application, the control program of the brain-computer interface system based on multimodal signals implements the above-mentioned control method of the brain-computer interface system based on multimodal signals when executed by the processor. Based on the above-mentioned control method of the brain-computer interface system based on multimodal signals, the accuracy of user intention analysis and the control accuracy and response speed of external devices are improved, thereby improving the user experience.

[0021] To achieve the above-mentioned objectives, the fourth aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a control program for a brain-computer interface system based on multimodal signals stored in the memory and executable on the processor. When the processor executes the control program for the brain-computer interface system based on multimodal signals, the control method for the brain-computer interface system based on multimodal signals is implemented.

[0022] According to the electronic device of the embodiment of the present application, when the processor executes the control program of the brain-computer interface system based on multimodal signals, the control method of the brain-computer interface system based on multimodal signals is implemented. Based on the control method of the brain-computer interface system based on multimodal signals, the accuracy of user intention analysis and the control accuracy and response speed of external devices are improved, thereby improving the user experience.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a control method of a brain-computer interface system based on multimodal signals according to an embodiment of the present application;

[0025] Figure 2 is a flow chart of a control method of a brain-computer interface system based on multimodal signals according to a specific embodiment of the present application;

[0026] Figure 3 is a connection diagram of a control device of a brain-computer interface system based on multimodal signals according to an embodiment of the present application;

[0027] Figure 4 Schematic diagram of a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes, with reference to the accompanying drawings, a control method for a brain-computer interface system based on multimodal signals, a control device for a brain-computer interface system based on multimodal signals, a computer-readable storage medium, and an electronic device proposed in an embodiment of the present application.

[0030] Figure 1 This is a flowchart of a control method for a brain-computer interface system based on multimodal signals according to an embodiment of the present application.

[0031] In one embodiment of the present application, the brain-computer interface system based on multimodal signals includes multiple training paradigms, and the training paradigms have a corresponding relationship with the signals to be collected.

[0032] That is to say, the brain-computer interface system based on multimodal signals is provided with a multimodal brain-computer interface paradigm library, and users can select the training paradigm in the multimodal brain-computer interface paradigm library according to their needs. The multimodal brain-computer interface paradigm library may include: P300 paradigm, steady-state visual evoked potential (SSVEP), motor imagery (MI), and user-defined hybrid paradigms, etc., without specific limitation.

[0033] Among them, the P300 paradigm: induces the user's P300 brain wave through visual or auditory stimulation, which can be used for interactive applications such as spellers and virtual keyboards, so the signals to be collected corresponding to the P300 paradigm include brain wave signals; Steady-State Visual Evoked Potentials (SSVEP): uses visual stimulation of a fixed frequency to capture the user's SSVEP signal, which is used to control cursor movement or selection operations, so the signals to be collected corresponding to the steady-state visual evoked potential paradigm include SSVEP signals; Motor Imagery (MI): induces brain wave changes by imagining the movement of specific body parts, which is used to control auxiliary equipment such as wheelchairs and robots, so the signals to be collected corresponding to the motor imagery paradigm include brain wave signals; Hybrid paradigm: the platform supports users to customize the selection of different paradigms or a combination of multiple paradigms, which can adapt to different ergonomic application needs and user habits.

[0034] like Figure 1 As shown, the control method of the brain-computer interface system based on multimodal signals in the embodiment of the present application may include:

[0035] S1, determine the BCI training paradigm;

[0036] S2, collects multi-dimensional physiological signals of target users based on the brain-computer interface training paradigm;

[0037] S3, determining a target model classification algorithm based on multi-dimensional physiological signals, and analyzing the target user's intention according to the target model classification algorithm;

[0038] S4, outputting a control signal based on the intention of the target user, and controlling the external device based on the control signal.

[0039] Specifically, the corresponding brain-computer interface training paradigm is selected according to different target users, so that through standardized brain-computer interface training paradigm, they can learn how to control external devices through thinking, establish personal characteristic models, and improve the adaptability of the brain-computer interface training paradigm to the target users.

[0040] According to the selected brain-computer interface training paradigm, the multi-dimensional physiological information of the target user is collected, wherein the physiological signal may include at least one of an EEG signal, a near-infrared spectrum signal, a blood oxygen saturation signal, a pulse wave signal, a skin electrode activity signal, an electrocardiogram signal, and a skin temperature signal, which can be specifically determined according to the selected brain-computer interface training paradigm. For example, when the P300 paradigm is selected, the EEG signal and pulse wave signal of the target user can be collected; when the mixed paradigm is selected, the EEG signal, blood oxygen saturation signal, skin electrode activity signal, and skin temperature signal of the target user can be collected based on the user's customization.

[0041] According to the type of multi-dimensional physiological signal acquired, the corresponding target model classification algorithm is determined to perform fusion empowerment processing on the multi-dimensional physiological signal through the target model classification algorithm, that is, the collected multi-dimensional physiological signal is used to train the model through the classification algorithm to identify the operator's intention, thereby improving the model classification accuracy and ensuring the quality of the intention analysis of the target user. For example, the model classification algorithm may include FBCCA (Filter Bank Canonical Correlation Analysis, filter bank canonical correlation analysis) classification algorithm, Random Forest (RF) classification algorithm, Convolutional Neural Networks (CNN), etc., to process multi-modal physiological signals in real time and improve signal processing efficiency. Then, the intention of the target user obtained by the analysis is converted into a control signal output to achieve precise control of external devices, improve the control reliability and stability of the brain-computer interface system based on multi-modal signals, and improve user experience.

[0042] In one embodiment of the present application, a target model classification algorithm is determined based on a multi-dimensional physiological signal, including: performing data dimensionality reduction on the multi-dimensional physiological signal by means of principal component analysis, and extracting the physiological signal corresponding to the maximum value in the weight coefficient of the multi-dimensional physiological signal; and determining the target model classification algorithm based on the physiological signal.

[0043] That is to say, if multi-dimensional physiological signals are selected, such as EEG, blood oxygen saturation and pulse wave signals, the data of the acquired EEG signals, blood oxygen saturation signals and pulse wave signals are first reduced in dimension through principal component analysis and other methods, and the physiological signal features with high weight rankings are extracted for training multiple deep learning algorithm models, and the model with the best effect is selected as the target model classification algorithm for the final classification calculation to analyze user intentions.

[0044] In one embodiment of the present application, the model classification algorithm includes: at least one of an EEG signal classification algorithm, a classical machine learning classification algorithm, and a deep learning classification algorithm.

[0045] Specifically, the system can have a variety of commonly used classification algorithms built in, supporting users to select single EEG signals and multi-dimensional physiological signals, and using the fusion of multi-dimensional physiological signals to empower EEG signal processing and improve the classification accuracy of the model. For example, the EEG signal classification algorithm can be FBCCA (Filter Bank Canonical Correlation Analysis), FBCSP (Filter Bank Common Spatial Patterns), etc., the classic machine learning classification algorithm can be Support Vector Machine (SVM), Random Forest, K-Nearest Neighbor (KNN), etc., and the deep learning classification algorithm can be Convolutional Neural Network, Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), etc.

[0046] Furthermore, if only EEG signals are used to analyze the target user's intentions, an untrained classical model can be selected according to the paradigm, such as the untrained FBCCA algorithm can be selected for the SSVEP paradigm; a trained model can also be selected according to the paradigm, such as the FBCSP algorithm that needs to be trained can be selected for the MI paradigm, and the trained model can be used to classify the signal, without specific restrictions.

[0047] This embodiment is based on the setting of the model classification algorithm and can process multi-dimensional physiological signals in real time to improve data processing efficiency.

[0048] In one embodiment of the present application, before parsing the intention of the target user according to the target model classification algorithm, the control method of the brain-computer interface system based on multimodal signals also includes: preprocessing the multi-dimensional physiological signals, wherein the preprocessing methods of the EEG signals include: filtering, artifact removal, and normalization; the preprocessing methods of the near-infrared spectrum signals include: signal denoising, baseline correction, and signal enhancement; the preprocessing methods of the blood oxygen saturation signals include: filtering, peak detection, and signal smoothing; the preprocessing methods of the pulse wave signals include: detrending, filtering, and heart rate variability analysis; the preprocessing methods of the skin electrodermal activity signals include: filtering, baseline correction, and signal standardization; the preprocessing methods of the electrocardiogram signals include: filtering, R wave detection, and heart rate analysis; the preprocessing methods of the skin temperature signals include: filtering, baseline drift correction, and temperature change rate calculation.

[0049] That is, after acquiring the multi-dimensional physiological signal, the multi-dimensional physiological signal is first preprocessed, and based on the preprocessed multi-dimensional physiological signal, the intention of the target user is analyzed according to the target model classification algorithm.

[0050] Among them, the preprocessing of EEG signals may include steps such as filtering, artifact removal, and normalization to improve signal quality; the preprocessing of near-infrared spectral signals may include steps such as signal denoising, baseline correction, and signal enhancement; the preprocessing of blood oxygen saturation signals may include steps such as filtering, peak detection, and signal smoothing; the preprocessing of pulse wave signals may include steps such as detrending, filtering, and heart rate variability analysis; the preprocessing of skin electrodermal activity signals may include steps such as filtering, baseline correction, and signal standardization; the preprocessing of electrocardiogram signals may include steps such as filtering, R wave detection, and heart rate analysis; the preprocessing of skin temperature signals may include steps such as filtering, baseline drift correction, and temperature change rate calculation.

[0051] This embodiment improves the signal quality through a preprocessing algorithm after collecting the multi-dimensional physiological signals of the target user.

[0052] In one embodiment of the present application, the control signal includes a visual signal, an auditory signal, a tactile signal and a motion control signal, wherein when the control signal includes a visual signal, the external device includes an LED light and a display; when the control signal includes an auditory signal, the external device includes a buzzer and an earphone; when the control signal includes a tactile signal, the external device includes a vibration motor and an electric stimulator; when the control signal includes a motion control signal, the external device includes a wheelchair and a robot.

[0053] Specifically, users can select specific standardized signals for output control through a graphical interface, thereby connecting the software and hardware devices required by the user. For example, when the user selects a visual signal, the LED light and screen display can be controlled; when the user selects an auditory signal, the buzzer and headphones can be controlled; when the user selects a tactile signal, the vibration motor and electrical stimulator can be controlled; when the user selects a motion control signal, the wheelchair and robot arm can be controlled.

[0054] In one embodiment of the present application, the control method of the brain-computer interface system based on multimodal signals also includes: sending the control signal to an external device by using one of a wired communication protocol, a wireless communication protocol, and an open source communication protocol; wherein the wired communication protocol includes at least one of USB, RS232, and Ethernet, the wireless communication protocol includes at least one of Wi-Fi, Bluetooth, ZigBee, and LoRa, and the open source communication protocol includes at least one of MQTT, CoAP, and WebSocket.

[0055] In other words, the brain-computer interface system based on multimodal signals is compatible with various types of signal input and output, ensuring the correct transmission of signals and the wide compatibility of equipment. In addition, through the wireless communication protocol, it can ensure that the signal transmission between the operator and the UGV is not limited by distance. Through the open source communication protocol, the system can be integrated with other open source hardware and software systems.

[0056] As a specific embodiment of the present application, Figure 2 As shown, the control method of the brain-computer interface system based on multimodal signals may include the following steps:

[0057] S101, determine the brain-computer interface training paradigm.

[0058] S102, collecting multi-dimensional physiological signals of the target user based on the brain-computer interface training paradigm.

[0059] S103, using principal component analysis to perform data dimensionality reduction on the multi-dimensional physiological signals.

[0060] S104, extracting the physiological signal corresponding to the maximum value of the weight coefficients in the multi-dimensional physiological signal.

[0061] S105: Determine a target model classification algorithm based on the physiological signal.

[0062] S106, collecting multi-dimensional physiological signals of the target user based on the brain-computer interface training paradigm.

[0063] S107, determining a target model classification algorithm based on the multi-dimensional physiological signal.

[0064] S108, preprocessing the multi-dimensional physiological signals.

[0065] S109, processing the pre-processed multi-dimensional physiological signal through a target model classification algorithm to analyze the intention of the target user.

[0066] S110, outputting a control signal based on the intention of the target user.

[0067] S111, using one of a wired communication protocol, a wireless communication protocol, and an open source communication protocol to send a control signal to an external device to control the external device.

[0068] The control method of the brain-computer interface system based on multimodal signals improves the accuracy and response speed of control operations, while reducing the cognitive burden of operators in complex environments and improving the safety and efficiency of task execution. In addition, the system's multimodal signal processing and open source communication protocol provide strong technical support and flexible application prospects for future research on human-computer interaction control of intelligent equipment.

[0069] In summary, according to the control method of the brain-computer interface system based on multimodal signals of the embodiment of the present application, the brain-computer interface training paradigm is first determined, and then the multi-dimensional physiological signals of the target user are collected based on the brain-computer interface training paradigm, and the target model classification algorithm is determined based on the multi-dimensional physiological signals, the intention of the target user is analyzed according to the target model classification algorithm, and the control signal is output based on the intention of the target user, and the external device is controlled based on the control signal. Therefore, the method collects multi-dimensional physiological signals based on the brain-computer interface training paradigm, and determines the user's intention by fusing the multi-dimensional physiological signals through the target model classification algorithm to generate a control signal to control the external device, thereby improving the accuracy of user intention analysis and the accuracy and response speed of control of external devices, and improving the user experience.

[0070] Corresponding to the above embodiments, the present application also proposes a control device for a brain-computer interface system based on multimodal signals.

[0071] According to one embodiment of the present application, a brain-computer interface system based on multimodal signals includes multiple training paradigms, and the training paradigms have a corresponding relationship with the signals to be collected.

[0072] like Figure 3 As shown, the control device of the brain-computer interface system based on multimodal signals according to an embodiment of the present application may include: a determination module 10 , a collection module 20 , an analysis module 30 and a control module 40 .

[0073] Among them, the determination module 10 is used to determine the brain-computer interface training paradigm. The acquisition module 20 is used to collect multi-dimensional physiological signals of the target user based on the brain-computer interface training paradigm. The analysis module 30 is used to determine the target model classification algorithm based on the multi-dimensional physiological signals, and analyze the intention of the target user according to the target model classification algorithm. The control module 40 is used to output a control signal based on the intention of the target user, and control the external device based on the control signal.

[0074] According to one embodiment of the present application, the analysis module 30 determines the target model classification algorithm based on the multi-dimensional physiological signal, and is specifically used for: performing data dimensionality reduction on the multi-dimensional physiological signal by means of principal component analysis, and extracting the physiological signal corresponding to the maximum value in the weight coefficient of the multi-dimensional physiological signal; determining the target model classification algorithm based on the physiological signal.

[0075] According to one embodiment of the present application, the model classification algorithm includes: at least one of an EEG signal classification algorithm, a classical machine learning classification algorithm, and a deep learning classification algorithm.

[0076] According to one embodiment of the present application, the physiological signal includes at least one of an electroencephalogram signal, a near-infrared spectrum signal, a blood oxygen saturation signal, a pulse wave signal, a skin electrode activity signal, an electrocardiogram signal, and a skin temperature signal.

[0077] According to one embodiment of the present application, before parsing the intention of the target user according to the target model classification algorithm, the parsing module 30 is also used to: pre-process the multi-dimensional physiological signals, wherein the pre-processing method of the EEG signal includes: filtering, artifact removal, and normalization; the pre-processing method of the near-infrared spectrum signal includes: signal denoising, baseline correction, and signal enhancement; the pre-processing method of the blood oxygen saturation signal includes: filtering, peak detection, and signal smoothing; the pre-processing method of the pulse wave signal includes: detrending, filtering, and heart rate variability analysis; the pre-processing method of the skin electrodermal activity signal includes: filtering, baseline correction, and signal standardization; the pre-processing method of the electrocardiogram signal includes: filtering, R wave detection, and heart rate analysis; the pre-processing method of the skin temperature signal includes: filtering, baseline drift correction, and temperature change rate calculation.

[0078] According to one embodiment of the present application, the control signal includes a visual signal, an auditory signal, a tactile signal and a motion control signal, wherein when the control signal includes a visual signal, the external device includes an LED lamp and a display; when the control signal includes an auditory signal, the external device includes a buzzer and an earphone; when the control signal includes a tactile signal, the external device includes a vibration motor and an electric stimulator; when the control signal includes a motion control signal, the external device includes a wheelchair and a robot.

[0079] According to one embodiment of the present application, the control module 40 is also used to: send a control signal to an external device using one of a wired communication protocol, a wireless communication protocol, and an open source communication protocol; wherein the wired communication protocol includes at least one of USB, RS232, and Ethernet, the wireless communication protocol includes at least one of Wi-Fi, Bluetooth, ZigBee, and LoRa, and the open source communication protocol includes at least one of MQTT, CoAP, and WebSocket.

[0080] It should be noted that for details not disclosed in the control device of the brain-computer interface system based on multimodal signals in the embodiment of the present application, please refer to the details disclosed in the control method of the brain-computer interface system based on multimodal signals in the above embodiment of the present application, and the details will not be repeated here.

[0081] According to the control device of the brain-computer interface system based on multimodal signals in the embodiment of the present application, the brain-computer interface training paradigm is determined by the determination module, the multi-dimensional physiological signals of the target user are collected based on the brain-computer interface training paradigm by the acquisition module, the target model classification algorithm is determined based on the multi-dimensional physiological signals by the analysis module, the intention of the target user is analyzed according to the target model classification algorithm, the control module outputs a control signal based on the intention of the target user, and controls the external device based on the control signal. The device collects multi-dimensional physiological signals based on the brain-computer interface training paradigm, and determines the user intention by fusing the multi-dimensional physiological signals through the target model classification algorithm, so as to generate a control signal to control the external device, thereby improving the accuracy of user intention analysis and the control accuracy and response speed of the external device, and improving the user experience.

[0082] Corresponding to the above embodiments, the present application also proposes a computer-readable storage medium.

[0083] The computer-readable storage medium of an embodiment of the present application stores a control program for a brain-computer interface system based on multimodal signals, and when the control program for a brain-computer interface system based on multimodal signals is executed by a processor, the control method for a brain-computer interface system based on multimodal signals is implemented.

[0084] According to the computer-readable storage medium of the embodiment of the present application, the control program of the brain-computer interface system based on multimodal signals implements the above-mentioned control method of the brain-computer interface system based on multimodal signals when executed by the processor. Based on the above-mentioned control method of the brain-computer interface system based on multimodal signals, the accuracy of user intention analysis and the control accuracy and response speed of external devices are improved, thereby improving the user experience.

[0085] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0086] like Figure 4 As shown, the electronic device 100 of an embodiment of the present application includes: a memory 110, a processor 120, and a control program for a brain-computer interface system based on multimodal signals stored in the memory 110 and executable on the processor 120. When the processor 120 executes the control program for the brain-computer interface system based on multimodal signals, the control method for the brain-computer interface system based on multimodal signals is implemented.

[0087] According to the electronic device of the embodiment of the present application, when the processor executes the control program of the brain-computer interface system based on multimodal signals, the control method of the brain-computer interface system based on multimodal signals is implemented. Based on the control method of the brain-computer interface system based on multimodal signals, the accuracy of user intention analysis and the control accuracy and response speed of external devices are improved, thereby improving the user experience.

[0088] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0089] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0090] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0091] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0092] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0093] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A control method for a brain-computer interface system based on multimodal signals, characterized in that: The system includes a plurality of training paradigms, the training paradigms have a corresponding relationship with the signals to be collected, and the method includes: Determine the BCI training paradigm; Collect multi-dimensional physiological signals of target users based on the brain-computer interface training paradigm; Determine a target model classification algorithm based on the multi-dimensional physiological signal, and analyze the intention of the target user according to the target model classification algorithm; A control signal is output based on the target user's intention, and an external device is controlled based on the control signal.

2. The control method of the brain-computer interface system based on multimodal signals according to claim 1 is characterized in that: Determining a target model classification algorithm based on the multi-dimensional physiological signal includes: Performing data dimensionality reduction on the multi-dimensional physiological signal by means of principal component analysis, and extracting the physiological signal corresponding to the maximum value of the weight coefficient in the multi-dimensional physiological signal; The target model classification algorithm is determined based on the physiological signal.

3. The control method of the brain-computer interface system based on multimodal signals according to claim 2 is characterized in that: The model classification algorithm includes: at least one of an electroencephalogram signal classification algorithm, a classical machine learning classification algorithm, and a deep learning classification algorithm.

4. The control method of the brain-computer interface system based on multimodal signals according to claim 1, characterized in that: The physiological signal includes: at least one of an electroencephalogram signal, a near-infrared spectrum signal, a blood oxygen saturation signal, a pulse wave signal, a skin electrode activity signal, an electrocardiogram signal and a skin temperature signal.

5. The control method of the brain-computer interface system based on multimodal signals according to claim 4 is characterized in that: Before parsing the intention of the target user according to the target model classification algorithm, the method further includes: The multi-dimensional physiological signals are preprocessed, wherein the preprocessing methods of the EEG signals include: filtering, artifact removal, and normalization; the preprocessing methods of the near-infrared spectrum signals include: signal denoising, baseline correction, and signal enhancement; the preprocessing methods of the blood oxygen saturation signals include: filtering, peak detection, and signal smoothing; the preprocessing methods of the pulse wave signals include: detrending, filtering, and heart rate variability analysis; the preprocessing methods of the electrocutaneous activity signals include: filtering, baseline correction, and signal standardization; the preprocessing methods of the electrocardiogram signals include: filtering, R-wave detection, and heart rate analysis; the preprocessing methods of the skin temperature signals include: filtering, baseline drift correction, and temperature change rate calculation.

6. The control method of the brain-computer interface system based on multimodal signals according to claim 1, characterized in that: The control signal includes a visual signal, an auditory signal, a tactile signal and a motion control signal, wherein: When the control signal includes a visual signal, the external device includes an LED lamp and a display; When the control signal includes an auditory signal, the external device includes a buzzer and an earphone; When the control signal includes a tactile signal, the external device includes a vibration motor and an electrical stimulator; When the control signal includes a motion control signal, the external device includes a wheelchair or a robot.

7. The control method of the brain-computer interface system based on multimodal signals according to claim 1, characterized in that: The method further comprises: Using one of a wired communication protocol, a wireless communication protocol, and an open source communication protocol to send the control signal to the external device; Among them, the wired communication protocol includes at least one of USB, RS232, and Ethernet, the wireless communication protocol includes at least one of Wi-Fi, Bluetooth, ZigBee, and LoRa, and the open source communication protocol includes at least one of MQTT, CoAP, and WebSocket.

8. A control device for a brain-computer interface system based on multimodal signals, characterized in that: The system includes a plurality of training paradigms, the training paradigms have a corresponding relationship with the signals to be collected, and the device includes: A determination module, used to determine the brain-computer interface training paradigm; An acquisition module, used to acquire multi-dimensional physiological signals of the target user based on the brain-computer interface training paradigm; A parsing module, used to determine a target model classification algorithm based on the multi-dimensional physiological signal, and parse the intention of the target user according to the target model classification algorithm; The control module is used to output a control signal based on the intention of the target user and control an external device based on the control signal.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the control method of the brain-computer interface system based on multimodal signals according to any one of claims 1-7 is implemented.

10. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, a control method for a brain-computer interface system based on multimodal signals is implemented according to any one of claims 1 to 7.

Images