Emergency state motion intention brain-computer interface system based on fusion features

Abstract

The application discloses an emergency state motion intention brain-computer interface system based on fusion features, and the system comprises the following steps: combining an active BCI and a reactive BCI, designing a new paradigm for inducing emergency state motion intention, and inducing ERP and MRCP combined features; building an electroencephalogram signal acquisition device, and recording electroencephalogram data of a user; extracting offline data features, and establishing an identification model; importing the identification model to conduct online experiments, classifying emergency states and non-emergency states by using linear discriminant analysis, and outputting control instructions for external equipment when detecting the emergency state. The application can effectively improve the classification and recognition accuracy of the BCI system, brings innovation in the paradigm and application scene for the development of the BCI, and is expected to realize a fast-response, reliable and stable BCI system, and brings considerable social and economic benefits.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interfaces (BCI), and more particularly to a brain-computer interface system for emergency movement intention based on fusion features. Background Technology

[0002] Brain-computer interface (BCI) refers to an artificially constructed pathway between the brain and computers or external devices, distinct from traditional brain-to-brain information transmission. It can replace, rebuild, strengthen, supplement, or improve the normal output of the central nervous system. Based on the characteristics of electroencephalogram (EEG) signals, it can be divided into active, reactive, and passive BCIs. Active BCIs do not depend on external events; their output control signals reflect the user's voluntary activities. MotorImagery BCI (MI-BCI) is a typical example of an active BCI. The user imagines movement in a specific part of their body, inducing a specific response in the corresponding brain region. The computer recognizes this response and translates the user's intention into control commands for external devices to complete a preset task. This includes: Event-Related Desynchronization / Synchronization (ERD / ERS) characteristics representing energy changes and Movement-Related Cortical Potentials (MRCPs) characteristics representing waveform changes. MRCPs, as characteristic potentials reflecting the motion preparation process, can predict the user's motion intention in advance during the preparation / planning phase of spontaneous motion, thereby making human-computer interaction more natural and efficient. Therefore, the detection and recognition of motion preparation response characteristics are of great significance.

[0003] Current research paradigms on motor imagery (MI) mainly focus on users imagining based on prompts given in experiments, without considering application scenarios in sudden or emergency situations.

[0004] Reactive brain-computer interfaces (BCIs) require external stimuli to induce changes in the brain at specific frequencies or waveforms. Based on different responses to neural signals, these changes are converted into corresponding command outputs. Typical passive BCIs include ERP-BCI and SSVEP-BCI (Steady-State Visual Evoked Potential BCI). Unlike ordinary evoked potentials, event-related potentials (ERPs) record the brain's response to information brought by stimuli. Based on attention, they are related to mental activities such as memory and recognition, reflecting the neurophysiological changes in the brain during cognitive processes. Classic ERP components include N2, P3, P1, N1, and P2; the first two are endogenous components, and the latter three are exogenous components. P3, also known as P300, is an endogenous ERP induced by low-probability events (visual, auditory, tactile, etc.). P300-based BCIs often utilize stimulus sequences of specific events to induce the user's P300 potential. Based on its lock-in characteristics, the user's conscious activity is determined by detecting the P300 potential.

[0005] Currently, the vast majority of BCIs are called "simple" BCIs, meaning they only use one of ERD / ERS, MRCPs, or P300 potentials as classification indicators. Summary of the Invention

[0006] This invention provides a brain-computer interface system for emergency state movement intention based on fusion features. Targeting emergency state or event application scenarios, this invention combines proactive and reactive BCI to design a new paradigm for inducing emergency state movement intention. This paradigm can induce joint features of ERP and MRCP (wherein, the visual presentation of the emergency state or event induces ERP features, and movement intention induces MRCP features). Finally, a brain-computer interface system for emergency state movement intention based on fusion features is constructed using pattern recognition algorithms, as detailed below:

[0007] An emergency state movement intention brain-computer interface system based on fusion features, the system comprising:

[0008] By combining proactive BCI and reactive BCI, a new paradigm for inducing motion intent in emergency situations is designed to induce joint features of ERP and MRCP.

[0009] Set up an EEG signal acquisition device to record the user's EEG data; extract offline data features and establish a recognition model;

[0010] The recognition model was imported for online experiments. Linear discriminant analysis was used to classify emergency and non-emergency states. When an emergency state was detected, control commands to external devices were output.

[0011] The motion intention induces MRCPs features; the visual presentation of the emergency state induces ERP features. By extracting the feature signals of neural electrical activity, fusing motion-related and event-related potential features, and using pattern recognition algorithms to identify the user's state, the motion intention is quickly identified by utilizing the time difference between the EEG response and the actual motion response.

[0012] Furthermore, the new paradigm is: based on VR technology, design a scene where a cup slides off the edge of a table, and use the sudden fall of the cup to simulate an emergency task or event to trigger ERP and MRCP.

[0013] ① Observation task: The user wears VR glasses and sits quietly in a chair, watching the cup fall in VR without having to perform any other tasks;

[0014] ②Button task: Wearing VR glasses, the user presses a button as quickly as possible the instant the cup falls, simulating the action of catching the cup, and the reaction time is recorded;

[0015] ③Imagination task: Similar to button task, imagination task does not produce actual movement, but only rehearses the action of catching the cup in the mind;

[0016] The timing of any cup falling in any of the three tasks is random, and a user will only execute one of the three tasks within a session.

[0017] The step of outputting control commands to external devices when an emergency is detected is as follows:

[0018] The results of real-time classification by the user are transmitted through the user data packet protocol, enabling command communication between MATLAB external devices and providing real-time feedback to the user.

[0019] Furthermore, the system defines the moment when the cup falls off the table as time zero, and uses the event code as a signal marker to segment the data from 1 second before time zero to 1 second after time zero.

[0020] Simultaneously, 2 seconds of non-task state data were extracted as EEG signals under non-urgent tasks to compare the differences in EEG responses between urgent and non-urgent tasks.

[0021] The system transmits data to Unity3D via UDP protocol, manipulating a virtual arm to catch a cup. When an emergency is detected, a prediction result is output and sent back to Unity to control the virtual arm and start the next task. When the detection result is a non-emergency task, the system will automatically extract the next set of data for analysis until a session ends.

[0022] Furthermore, the motion intent-induced MRCPs feature has the time-locking characteristic of ERP. The segmented data is superimposed and averaged by multiple trials according to the experimental conditions to finally obtain the ERP waveform or the "ERP+MRCP" fused waveform.

[0023] The system includes:

[0024] Input data from four sessions of the imagination task in the offline experiment, preprocess and extract features to obtain individual models and save them;

[0025] Input online data, perform preprocessing, feature extraction and pattern recognition in sequence, treat the offline model as the training set and the online data features as the test set, and use LDA to distinguish between emergency and non-emergency states.

[0026] The beneficial effects of the technical solution provided by this invention are:

[0027] 1. Compared with the single feature of the traditional BCI paradigm, the feature fusion can effectively improve the classification and recognition accuracy of the BCI system. This system brings innovation to the development of BCI in terms of paradigm and application scenarios, and is expected to realize a fast-response, reliable and stable BCI system, bringing considerable social and economic benefits.

[0028] 2. This invention combines active and reactive BCI paradigms, focusing on the "switch" application for emergency states or events. By inducing and fusing different EEG features, it can achieve rapid and accurate identification of emergency states or events. Compared with the single feature of the traditional BCI paradigm, it can effectively improve the classification and recognition accuracy of the BCI system.

[0029] 3. The VR technology used in this invention not only optimizes the experimental scenario but also enhances user participation and immersion, thus improving the system's usability. This system overcomes the limitations of existing BCI systems and is expected to provide innovative ideas and reliable technical support for the development of new BCI systems. Attached Figure Description

[0030] Figure 1 A schematic diagram of the brain-computer interface system framework for motor intention in emergency situations;

[0031] Figure 2 A flowchart for a VR-based paradigm for inducing movement intent in emergency situations;

[0032] Figure 3 This is a schematic diagram of the experimental scenario;

[0033] Figure 4 A schematic diagram of the time-domain waveform under the observation task;

[0034] Figure 5 This is a schematic diagram of the time-domain waveform under a key press task;

[0035] Figure 6 A schematic diagram of the time-domain waveform under the imaginative task;

[0036] Figure 7 Flowchart of an online experiment for a brain-computer interface system for motor intention in emergency situations. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0038] Compared to a single BCI system, a hybrid brain-computer interface (Hybrid BCI, hBCI) can effectively improve the classification accuracy and execution efficiency of a BCI system.

[0039] Virtual Reality (VR), as a human-computer interaction technology, constructs virtual environments through computer simulation, providing users with multi-sensory simulations characterized by interactivity, immersion, and imaginative possibilities. Research shows that compared to two-dimensional displays, three-dimensional environments create an immersive experience, allowing users to focus more on the experiment itself and thus enhancing the activation of the cerebral cortex. Simultaneously, due to the inherent appeal of VR, it can effectively alleviate fatigue and enhance the user experience.

[0040] Therefore, this embodiment of the invention uses VR technology to design a scenario where a cup slides off the edge of a table. The sudden drop of the cup simulates an emergency task or event to trigger ERP and MRCPs, thus expanding the paradigm and application scenarios of the existing BCI system by integrating proactive and reactive BCI.

[0041] Experimental scenarios based on VR technology are designed and developed to simulate emergency situations or events in daily life. Compared to non-emergency situations, emergency situations or events induce special neural electrical activity in the cerebral cortex. By extracting characteristic signals of neural electrical activity, fusing motion-related and event-related potential features, and using pattern recognition algorithms to identify the user's state, and by utilizing the time difference between EEG response and actual motion response, the user's movement intention can be quickly identified, thereby helping the user to avoid risks. This has significant application prospects in daily life and military fields.

[0042] Therefore, this invention combines BCI and VR technologies to design a new paradigm for inducing motion intent in emergency situations. It detects and identifies emergency situations or events through time-domain analysis and converts them into command outputs to achieve rapid response. By controlling external devices, it aims to reduce danger and protect personal safety.

[0043] The technical process is as follows: design a new paradigm for inducing motor intention in emergency situations, simultaneously inducing ERPs and MRCPs; build an EEG signal acquisition device to record the user's EEG data; extract offline data features and establish a recognition model; import the recognition model for online experiments, and use Linear Discriminant Analysis (LDA) to classify the two states (emergency and non-emergency states). When an emergency state or event is detected, output control commands to external devices.

[0044] I. System Architecture and Experimental Procedure

[0045] The overall system design of this invention embodiment is as follows: Figure 1 As shown, the system mainly comprises four parts: a stimulus presentation module, a data acquisition module, an EEG data processing module, and an online control module. The stimulus presentation module displays the specific experimental task in VR format, prompting subjects to respond at specified times. The data acquisition module utilizes a Neuroscan EEG acquisition device to synchronously acquire raw EEG signals and event tags (the event tags are sent by the stimulus presentation module and transmitted to the EEG acquisition device via a parallel port). The main functions of the EEG data processing module include preprocessing the raw EEG signals (acquired by the data acquisition module); extracting time-domain waveform features from the preprocessed EEG data and establishing a recognition model; classifying and recognizing offline data; and performing real-time classification and recognition in online experiments. The online control module transmits the user's real-time classification results via the User Datagram Protocol (UDP), enabling command communication between MATLAB and Unity3D or external devices (e.g., virtual arms, robotic arms, drones, etc.) and providing real-time feedback to the user (e.g., visual feedback).

[0046] The flowchart of the VR-based emergency motion intent induction paradigm is as follows: Figure 2As shown. The offline experiment included three tasks: ① Observation task: The user, wearing VR glasses, sat quietly in a chair and only needed to watch the cup fall in VR without performing any other tasks; ② Button task: Wearing VR glasses, the user needed to press a button as quickly as possible the instant the cup fell, simulating the action of catching the cup, and the reaction time was recorded; ③ Imaginary task: Similar to the button task, the user needed to react when the cup fell. The difference was that the imaginary task did not produce real movement, but only mentally rehearsed the action of catching the cup. The time of any cup falling in any of the three tasks was random (3-5 seconds) to eliminate the interference of psychological expectations on the experiment and to obtain the characteristic response of the brain in an emergency. The offline experiment contained 12 sessions, 4 sessions for each task, and 30 trials in each session (i.e., the cup falls 30 times). The user performed only one of the three tasks in a session, and a rest period was given to the subject after each session. The online experiment contained only one imaginary task. After recognizing the user's movement intention, the control command was fed back to Unity to realize the control of the virtual arm.

[0047] II. Functions of Each Module

[0048] (1) Stimulus Presentation Module

[0049] The stimulus presentation module primarily utilizes Unity3D software developed by Unity Technologies for 3D scene construction. Scene materials are modeled using 3ds Max, with appropriate materials applied, and the completed models are exported as FBX files for later use. Then, the exported FBX models are imported into Unity3D, and C# scripts written using Visual Studio are used to control the movement of the cups. Prefabs are generated for other objects in the scene for later use. Finally, lighting and camera angles are adjusted to complete the VR scene setup. To provide a more realistic experience, users need to wear VR glasses during the experiment. This system uses HTC VIVE, a VR headset jointly developed by HTC and Value, with technical support provided by Value and SteamVR. Therefore, in the final step, the Steam VR SDK files required for HTC VIVE need to be imported into Unity3D and configured simply to achieve an immersive VR experience (e.g., ...). Figure 3 ).

[0050] (2) Data Acquisition Module

[0051] EEG data acquisition used Neuroscan's Synamps. 2The amplifier amplifies the raw signal, and the accompanying SCAN software saves the data. Standard Ag / Agcl electrodes are used, positioned according to the international 10-20 system, with 64 leads. Acquisition parameters are set to a sampling rate of 1000Hz, a 0.1–200Hz bandpass filter, and a 50Hz notch filter to remove power frequency interference. During acquisition, the top of the head is used as a reference, the forehead is grounded, and the impedance between the scalp and the electrodes is maintained below 10KΩ. Subjects are required to remain as still as possible during the experiment, avoiding involuntary eye movements and minor movements unrelated to the task to ensure the reliability of the acquired data.

[0052] (3) EEG data processing module

[0053] a. EEG data preprocessing

[0054] This system uses EEGLAB, a brainwave processing toolbox developed based on MATLAB, to preprocess the raw EEG signals. Preprocessing operations include data format conversion, bandpass filtering, downsampling, and data segmentation. The steps are as follows:

[0055] 1) Data format conversion: Use the EEGLAB toolbox to convert the original data into the more common ".mat" format;

[0056] 2) Bandpass filtering: This system mainly explores the characteristics of N200, P300 and MRCP. For this purpose, a third-order Butterworth bandpass filter with a filtering range of 1 to 10 Hz is used to filter the data and remove extremely low and extremely high frequency interference in the EEG signal.

[0057] 3) Downsampling: Commonly used EEG information is mainly concentrated within 100Hz. In order to improve the efficiency of the operation, the original EEG signal is downsampled from 1000Hz to 200Hz without distorting the signal.

[0058] 4) Data Segmentation: This system defines the moment when the visual stimulus (i.e., the cup falling from the table) occurs as time zero. Using the event code as a signal marker, data from 1 second before time zero to 1 second after time zero is segmented for subsequent analysis. At the same time, 2 seconds of data in the non-task state are extracted as EEG signals under non-urgent tasks to compare the differences in EEG responses between urgent and non-urgent tasks.

[0059] b. Time-domain waveform analysis and classification

[0060] 1) Superimposed average

[0061] Unlike the random changes in spontaneous EEG, constant latency and constant waveform are two important characteristics of ERP. Therefore, multiple EEG signals induced by the same stimulus can be superimposed and averaged, so that irregular spontaneous EEG or noise can cancel each other out during the superposition process. When the number of superpositions is sufficient, the amplitude of ERP will continuously increase and thus become prominent. Similarly, the low-frequency MRCP induced by motor preparation also has the time-locking characteristics of ERP. Therefore, the segmented data are superimposed and averaged according to the experimental conditions for multiple trials, and finally the ERP waveform or "ERP+MRCP" fusion waveform is obtained. The calculation process is shown in formula (1):

[0062]

[0063] Among them, X i This represents the average value of the target data for the i-th task (i = 1, 2), and there are M trials for each task. x (m) N represents the m-th trial. c N represents the number of channels used to collect EEG data. t This indicates the length of the intercepted signal.

[0064] Current results are as follows Figure 4 , 5 As shown in Figure 6, the target (solid line) represents the waveform under urgent tasks, and the non-target (dashed line) represents the waveform under non-urgent tasks. Compared with the observation task, the P300 amplitude of both the key-pressing task and the imagination task is significantly reduced, while the N200 amplitude is somewhat increased. This is because motion execution / imagination can induce MRCP, thereby enhancing (N200) or weakening (P300) certain features, thus inducing the fusion of features.

[0065] 2) Feature extraction

[0066] Discriminative Canonical Pattern Matching (DCPM) comprises two spatial filtering processes: Discriminative Spatial Pattern Analysis (DSP) and Canonical Correlation Analysis (CCA). The overall approach is as follows: first, preprocessed training data is filtered through a spatial filter to construct a data template; then, test data is filtered through a spatial filter and matched against the template data that has also undergone spatial filtering; finally, a decision classification is performed.

[0067] First, the template signal is obtained by averaging the training set data. Suppose X iLet M be the set of training data for the i-th task (i = 1, 2), where each class of training data contains M samples. x (m) This indicates the m-th trial. For the test sample, N c N represents the number of channels used to collect EEG data. t Indicates the length of the intercepted signal:

[0068]

[0069] Then, a DSP spatial filter was built. b Let S be the inter-class scatter matrix. w The scatter matrix is ​​the intra-class scatter matrix. and The template signals for the two training sets are obtained by averaging all samples in the two training sets. The optimal solution U is a matrix. Feature vectors:

[0070]

[0071]

[0072]

[0073] Template matching is performed using Pearson correlation coefficient and CCA. First, the Pearson correlation coefficient between the DSP-filtered test data and the template signal is calculated:

[0074]

[0075]

[0076] The test data after DSP spatial filtering was subjected to CCA analysis with each template, and the Pearson correlation coefficient was recalculated in the new projection space.

[0077]

[0078]

[0079]

[0080]

[0081] By comparing the magnitudes of feature values, we determine which class of training samples the test sample is more closely matched with. Let:

[0082] ρ1=ρ 11 -ρ 21 ρ2=ρ 12 -ρ 22 #(12)

[0083] Two-dimensional features can be extracted, [ρ1, ρ2]. T

[0084] 3) Classification and Recognition

[0085] The basic idea of ​​LDA is: during training, the training samples are projected onto a straight line, which makes the projection points of samples of the same type as close as possible, while the projection points of samples of different types are as far apart as possible; during prediction, the data to be predicted is projected onto the straight line learned during training according to the projection matrix obtained during training, and the category is determined according to the position of the projection points. Therefore, LDA can be regarded as a process of feature vector dimensionality reduction, projecting multi-dimensional features into one dimension. The form of the Fisher discriminant model is shown in formula (13), where x is the feature vector:

[0086] f(x)=ω T x+b#(13)

[0087] The current results are shown in the table below. Ten-fold cross-validation was used, with data collected 300ms after the visual cue used for classification. Consistent with the research objective, fused features (i.e., key press and imagination tasks) performed better in classification compared to single features (observation task).

[0088] Table 1 Classification accuracy under three tasks

[0089]

[0090] (4) Online control module

[0091] The main function of the online control module is to identify the user's state in real time through a computer processing program. When it detects the user's intention to catch the cup, it outputs control commands. These commands are transmitted to Unity3D via UDP protocol, manipulating the virtual arm to complete the cup-catching action. The online experimental process is as follows: Figure 7 As shown, the process mainly consists of two parts: First, data from four sessions of the offline simulation task is input, preprocessed, and feature extracted to obtain individual models, which are then saved. Second, online data is input, and preprocessing, feature extraction, and pattern recognition are performed sequentially. Here, the offline model can be considered as the training set, and the online data features as the test set. LDA is used for binary classification prediction (emergency and non-emergency states). When an emergency state is identified, the prediction result is output and sent back to Unity to complete the control of the virtual arm, while starting the next task. When the identification result is a non-emergency task, the system will automatically extract the next set of data for analysis until the end of a session.

[0092] This invention presents a brain-computer interface (BCI) system for emergency movement intention based on fusion features. This innovative design utilizes a BCI coding paradigm that can simultaneously induce ERP and MRCP features, thereby improving the recognition accuracy and response time of traditional active BCI systems. It has broad application prospects in daily life, military battlefields, and sports, and is expected to yield considerable social and economic benefits.

[0093] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0094] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A brain-computer interface system for emergency movement intention based on fusion features, characterized in that, The system includes: By combining proactive BCI and reactive BCI, a new paradigm for inducing motion intent in emergency situations is designed to induce joint features of ERP and MRCP. Set up an EEG signal acquisition device to record the user's EEG data; extract offline data features and establish a recognition model; The recognition model was imported for online experiments. Linear discriminant analysis was used to classify emergency and non-emergency states. When an emergency state was detected, control commands to external devices were output. The motion intent induces MRCPs features; the visual presentation of the emergency state induces ERP features. By extracting feature signals of neural electrical activity, fusing motion-related and event-related potential features, and using pattern recognition algorithms to identify the user's state, the motion intent can be quickly identified by utilizing the time difference between EEG response and actual motion response. The new paradigm is: using VR technology to design a scene where a cup slides off the edge of a table, and using the sudden fall of the cup to simulate an emergency task or event to trigger ERP and MRCP. ① Observation task: The user wears VR glasses and sits quietly in a chair, watching the cup fall in VR without having to perform any other tasks; ②Button task: Wearing VR glasses, the user presses a button as quickly as possible the instant the cup falls, simulating the action of catching the cup, and the reaction time is recorded; ③Imagination task: Similar to button task, imagination task does not produce actual movement, but only rehearses the action of catching the cup in the mind; The timing of any cup falling in any of the three tasks is random, and a user will only execute one of the three tasks within a session.

2. The emergency state movement intention brain-computer interface system based on fusion features according to claim 1, characterized in that, The control command to external devices when an emergency is detected is as follows: The results of real-time classification by the user are transmitted through the user data packet protocol, enabling command communication between MATLAB external devices and providing real-time feedback to the user.

3. The emergency state movement intention brain-computer interface system based on fusion features according to claim 1, characterized in that, The system defines the moment when the cup falls off the table as time zero, and uses the event code as a signal marker to segment the data from 1 second before time zero to 1 second after time zero. Simultaneously, 2 seconds of non-task state data were extracted as EEG signals under non-urgent tasks to compare the differences in EEG responses between urgent and non-urgent tasks.

4. The emergency state movement intention brain-computer interface system based on fusion features according to claim 1, characterized in that, The system transmits data to Unity3D via UDP protocol, manipulating a virtual arm to catch a cup. When an emergency is detected, a prediction result is output and sent back to Unity to control the virtual arm, while starting the next task. When the detection result is a non-emergency task, the system will automatically extract the next set of data for analysis until a session ends.

5. The emergency state movement intention brain-computer interface system based on fusion features according to claim 1, characterized in that, The motion intent-induced MRCPs feature has the time-locking characteristic of ERP. The segmented data is superimposed and averaged according to the experimental conditions of multiple trials to finally obtain the ERP waveform or the "ERP+MRCP" fused waveform.

6. The emergency state movement intention brain-computer interface system based on fusion features according to claim 1, characterized in that, The system includes: Input data from four sessions of the imagination task in the offline experiment, preprocess and extract features to obtain individual models and save them; Input online data, perform preprocessing, feature extraction and pattern recognition in sequence, treat the offline model as the training set and the online data features as the test set, and use LDA to classify and predict emergency and non-emergency states.

Images