Control instruction generation method based on multi-mode electroencephalogram signals

Abstract

The invention discloses a control instruction generation method based on a multi-mode electroencephalogram signal, and relates to the technical field of man-machine interaction. According to the method, eight paths of electroencephalogram signal detection and three-axis gyroscope attitude detection are creatively fused, and the method is realized through multi-modal signal fusion and a unique algorithm, so that the problems of signal interpretation limitation, control instruction limitation, single information dimension and the like existing in pure electroencephalogram signal control and single sensor control in the prior art are solved. According to the method, the intention of the user can be more comprehensively and accurately interpreted, richer and more accurate control instructions can be generated, the control precision and flexibility are improved, the application scene is expanded, the user experience is improved, the system robustness is enhanced, and the method has important practical significance and application value and has remarkable progress compared with the prior art.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology and relates to a method for generating control commands based on multimodal electroencephalogram (EEG) signals. Background Technology

[0002] With the rapid development of technology, human-computer interaction technology is constantly evolving, aiming to achieve more natural, efficient, and convenient interaction methods. Brain-computer interface (BCI) technology, as a highly promising human-computer interaction technology, allows humans to interact directly with external devices through brain activity, bringing new possibilities to many fields.

[0003] Electroencephalography (EEG) signal detection, a key component of brain-computer interface (BCI) technology, captures the electrical signals generated by the activity of neurons in the brain. Electrodes placed on the scalp can detect EEG signals of different frequencies and characteristics, reflecting different states and activities of the brain. EEG detection technology has been widely applied in neuroscience research, clinical medical diagnosis, and brain-computer interfaces.

[0004] However, in existing technologies, EEG signals are very weak and complex, easily affected by environmental noise, physiological artifacts, and other interferences, leading to difficulties in signal interpretation. Furthermore, the characteristics of EEG signals vary among individuals; the same thought activity may produce different EEG signal patterns in different people, limiting the accuracy of general EEG signal recognition models. Control commands are also limited; existing EEG signal control methods can only implement a limited number of control commands, making it difficult to meet the diverse control command requirements of complex tasks.

[0005] In terms of single-sensor control, the information dimension is limited. When using only a single sensor such as a gyroscope for control, the information obtained is limited. Lacking the advantages of multimodal fusion, single-sensor control technology cannot fully utilize the complementary advantages of multiple information sources, resulting in poor adaptability and robustness in complex environments or tasks.

[0006] Therefore, developing a more intelligent, precise, and flexible multimodal human-computer interaction control method has significant practical implications and application value. Summary of the Invention

[0007] This invention aims to address the limitations of signal interpretation, limited control commands, single information dimension, and lack of multimodal fusion advantages in existing technologies such as simple EEG signal control and single sensor control, and provides a method for generating control commands based on multimodal EEG signals.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for generating control commands based on multimodal EEG signals includes two parts: hardware construction and software design, wherein: Hardware setup: An 8-channel EEG signal detection device is used, with electrodes placed at Fp7, Fp8, FPz, Fz, Cz, Tp9, Tp10, Pz, and Eog positions according to the international 10-20 system electrode layout; a 3-axis gyroscope is integrated for real-time detection of device attitude changes. Software design: The software is developed using C language. The algorithm implementation process includes the initialization stage, data acquisition stage, stillness detection stage, blink detection stage, head movement detection stage, lip shape and frowning action training and recognition stage, and control command generation and output stage.

[0009] Furthermore, the calming detection phase employs a 3-second calming cursor repositioning algorithm, which includes EEG signal preprocessing, feature extraction, calming state judgment, timing, and repositioning steps.

[0010] Furthermore, the blink detection stage employs a blink-based key operation algorithm, which includes blink feature extraction, blink detection, and key command mapping steps.

[0011] Furthermore, the head movement detection stage employs a micro-movement head movement cursor algorithm, which includes gyroscope data preprocessing, attitude calculation, cursor movement calculation, and cursor control output steps.

[0012] Furthermore, the lip shape and frowning action training and recognition stage adopts a lip shape and frowning action training and text input algorithm, which includes a training stage and a practical application stage.

[0013] The beneficial effects of this invention are as follows: Improved control precision and flexibility: Through multimodal signal fusion and unique algorithm design, it can more accurately interpret user intentions and generate richer and more precise control commands.

[0014] Expanding application scenarios: This patented technology can generate control commands applicable to various devices and scenarios, opening up a wider application space for multimodal EEG signal control technology.

[0015] Enhanced user experience: For users whose hands are occupied or whose physical mobility is limited, it provides a more natural and convenient interaction method, freeing them from the limitations of traditional manual operation.

[0016] Enhanced system robustness: The fusion of multimodal signals enables the system to acquire information from multiple dimensions, resulting in stronger adaptability and robustness when facing complex environments or individual differences.

[0017] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a flowchart of a software algorithm according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0021] 1. Hardware Setup The EEG signal detection section employed an 8-channel EEG signal detection device, with electrodes placed according to the international 10-20 system electrode layout at the following locations on the head: Fp7 (left frontal lobe), Fp8 (right frontal lobe), FPz (midfrontal lobe), Fz (midfrontal lobe), Cz (central midline), Tp9 (left posterior temporal lobe), Tp10 (right posterior temporal lobe), Pz (mid-parietal lobe), and Eog (electroophthalmos). These locations comprehensively collect electrical activity signals from different brain regions, providing a rich data source for subsequent EEG signal analysis.

[0022] Gyroscope section: Integrates a 3-axis gyroscope for real-time detection of device attitude changes (such as when wearing a helmet). The gyroscope can accurately measure angular velocity and acceleration along three axes (usually X, Y, and Z axes) and transmit this data to the subsequent processing unit in real time.

[0023] 2. Software Design Development environment: Software development is carried out using the C language. The C language has the characteristics of high efficiency, flexibility and strong portability, which can meet the needs of direct control of hardware resources and efficient data processing.

[0024] Algorithm implementation process: Initialization Phase: After system startup, the EEG signal detection device and gyroscope are first initialized and configured to ensure normal operation and accurate data acquisition. Simultaneously, system parameters and models are initialized, such as setting the default cursor position and loading pre-trained lip and frown recognition models.

[0025] Data acquisition phase: Continuous real-time acquisition of 8 channels of EEG signals and 3-axis gyroscope data. The EEG signal detection device amplifies and filters the acquired weak EEG signals before transmitting them to the data processing unit; the gyroscope measures the attitude changes of the device in real time and transmits the measurement data synchronously to the data processing unit.

[0026] The calming detection phase employs an algorithm that returns the cursor to its default position after 3 seconds of calming to analyze EEG signals in real time and detect whether the user is in a calming state. When the user is detected to be in a calming state for 3 consecutive seconds, the cursor is automatically returned to its default position.

[0027] Blink detection phase: The blink detection algorithm is used to accurately identify blink features in EEG signals and convert blinking actions into corresponding button control commands.

[0028] Head movement detection stage: The cursor is moved by a slight head movement algorithm based on the 3-axis data of the gyroscope.

[0029] Lip and frowning motion training and recognition stage: Lip and frowning motion training and text input algorithms are used. By training and learning lip and frowning motions, a mapping relationship between the corresponding motions and simple text input is established, realizing handless text input.

[0030] Control command generation and output stage: Based on the detection and recognition results of the above stages, corresponding control commands are generated according to different application scenarios and transmitted to the corresponding external devices to achieve precise control of the devices.

[0031] Supplementary Explanation of Algorithm Implementation Steps The steps for implementing the algorithm to reset the cursor to its default position after 3 seconds of quiet reflection are as follows: EEG signal preprocessing: After acquiring 8 channels of EEG signals, the signals are first bandpass filtered to remove 50Hz power frequency interference and other high-frequency and low-frequency noise, retaining the EEG signals in the 1-30Hz frequency band, and then the signals are normalized.

[0032] Feature extraction: The power spectral density (PSD) analysis method was used to calculate the power values ​​of the preprocessed EEG signal in different frequency bands, and the time-domain features such as the average amplitude and variance of the signal were extracted.

[0033] Calm state assessment: A classification model based on support vector machine (SVM) is established, using extracted EEG signal features as input, to classify brain states into calm and non-calm states.

[0034] Timing and Reset: When the model continuously determines that the brain is in a calm state, a timer is started. If the timer accumulates 3 seconds, a cursor reset operation is triggered, moving the cursor on the screen to a preset default position. If a non-calm state is detected during the timing process, the timer is reset.

[0035] The steps for implementing the blink-based button operation algorithm are as follows: Blink feature extraction: Time-domain analysis of EEG signals is performed to extract features such as the maximum and minimum amplitude of the signal, rise time, and fall time during the blinking process.

[0036] Blink detection: Using sliding window technology, with appropriate window length and step size set, the window slides across the EEG signal, and blink feature analysis is performed on the signal within each window. When the signal feature within the window meets the preset blink feature threshold, a blink action is determined to have been detected.

[0037] Key command mapping: Establish a mapping relationship between blinking actions and key commands. When a blinking action is detected, the corresponding key control command is generated according to the preset mapping relationship and sent to the external device for response.

[0038] Steps for implementing the algorithm of moving the cursor by slightly moving the head: Gyroscope data preprocessing: After acquiring the raw angular velocity data output by the 3-axis gyroscope, noise reduction processing is performed. The accelerometer data is fused using the Kalman filter algorithm to eliminate the drift error of the gyroscope. The sampling frequency of the processed data is adjusted to match the sampling frequency of the EEG signal.

[0039] Attitude calculation: Based on the 3-axis angular velocity data of the gyroscope, the rotation angle changes of the device in the three axes are calculated by integration. The attitude calculation is performed using the quaternion method or Euler angle method, and the angular velocity information is converted into the attitude angle of the device relative to the initial position.

[0040] Cursor movement calculation: Establish a mapping relationship between posture angle and cursor movement distance and direction. Based on the posture angle changes obtained in real time, calculate the cursor movement distance and direction in the screen coordinate system according to the preset mapping relationship, and filter the calculated cursor movement amount.

[0041] Cursor control output: The calculated cursor movement information is converted into control commands that conform to the external device interface protocol and sent to the device via USB interface or wireless communication module to achieve real-time control of the cursor position.

[0042] Steps for training lip shape and frowning movements and implementing text input algorithms: Training phase: Data acquisition: In training mode, the user is guided to make different lip shapes and frowning movements, while 8 channels of EEG signals and facial image information are collected.

[0043] Feature extraction: Time-frequency analysis was performed on the acquired EEG signals to extract the features of the EEG signals corresponding to different actions in the time and frequency domains; the acquired facial images were preprocessed, and convolutional neural networks were used to extract features related to lip shape and frowning actions in the facial images.

[0044] Model training: The extracted EEG signal features and facial image features are fused together as input data, and a deep learning model is used for training, with the user's action labels as output labels.

[0045] Practical application stage: Real-time data acquisition and processing: Real-time acquisition of 8 channels of EEG signals and facial image information, and the acquisition data undergoes the same preprocessing and feature extraction operations as in the training phase.

[0046] Action recognition and text input: The extracted real-time features are input into the trained model. The model analyzes and classifies the input features to determine the user's current lip-shaped or frowning action. Based on the model's recognition results, the action is converted into corresponding simple text or instructions according to a preset mapping relationship, and the generated text or instructions are sent to external devices.

[0047] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0048] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for generating control commands based on multimodal electroencephalogram (EEG) signals, characterized in that, It includes two parts: hardware setup and software design. The specific steps are as follows: Hardware setup: An 8-channel EEG signal detection device is used, with electrodes placed at Fp7, Fp8, FPz, Fz, Cz, Tp9, Tp10, Pz, and Eog positions according to the international 10-20 system electrode layout; a 3-axis gyroscope is integrated for real-time detection of device attitude changes. Software design: The software is developed using C language. The algorithm implementation process includes the initialization stage, data acquisition stage, stillness detection stage, blink detection stage, head movement detection stage, lip shape and frowning action training and recognition stage, and control command generation and output stage.

2. The method according to claim 1, characterized in that, The stillness detection phase employs a 3-second stillness cursor return-to-default position algorithm, which includes the following steps: EEG signal preprocessing: Bandpass filtering was performed on the 8 collected EEG signals to remove 50Hz power frequency interference and other high-frequency and low-frequency noise, retaining the EEG signals in the 1-30Hz frequency band, and then the signals were normalized. Feature extraction: Power spectral density analysis was used to calculate the power values ​​of the preprocessed EEG signal in different frequency bands, and the time-domain features such as the average amplitude and variance of the signal were extracted. Calm state assessment: A classification model based on support vector machine is established, using extracted EEG signal features as input, to classify brain states into calm and non-calm states; Timing and Reset: When the model continuously determines that the brain is in a calm state, a timer is started. If the timer accumulates to 3 seconds, a cursor reset operation is triggered, moving the cursor on the screen to the preset default position. If a non-calm state is detected during the timing process, the timer is reset.

3. The method according to claim 1, characterized in that, The blink detection phase uses a blink-based button operation algorithm. Includes the following steps: Blinking feature extraction: Time-domain analysis of EEG signals is performed to extract features such as the maximum and minimum amplitude of the signal, rise time, and fall time during blinking. Blink detection: Using sliding window technology, with appropriate window length and step size set, the window slides on the EEG signal, and blink feature analysis is performed on the signal in each window. When the signal feature in the window meets the preset blink feature threshold, it is determined that a blink action has been detected. Key command mapping: Establish a mapping relationship between blinking actions and key commands. When a blinking action is detected, the corresponding key control command is generated according to the preset mapping relationship and sent to the external device for response.

4. The method according to claim 1, characterized in that, The head movement detection stage employs a micro-movement head movement cursor movement algorithm, which includes the following steps: Gyroscope data preprocessing: After acquiring the raw angular velocity data output by the 3-axis gyroscope, noise reduction processing is performed. The Kalman filter algorithm is used to fuse the accelerometer data to eliminate the drift error of the gyroscope. The sampling frequency of the processed data is adjusted to match the sampling frequency of the EEG signal. Attitude calculation: Based on the 3-axis angular velocity data of the gyroscope, the rotation angle changes of the device in the three axes are calculated by integration. The attitude calculation is performed using the quaternion method or Euler angle method, and the angular velocity information is converted into the attitude angle of the device relative to the initial position. Cursor movement calculation: Establish a mapping relationship between posture angle and cursor movement distance and direction. Based on the posture angle changes obtained in real time, calculate the cursor movement distance and direction in the screen coordinate system according to the preset mapping relationship, and filter the calculated cursor movement amount. Cursor control output: The calculated cursor movement information is converted into control commands that conform to the external device interface protocol and sent to the device via USB interface or wireless communication module to achieve real-time control of the cursor position.

5. The method according to claim 1, characterized in that, The lip shape and frowning action training and recognition stage employs a lip shape and frowning action training and text input algorithm. This algorithm includes a training stage and a practical application stage, wherein the training stage includes: Data acquisition: In training mode, the user is guided to make different lip shapes and frowning movements, while 8 channels of EEG signals and facial image information are collected. Feature extraction: Time-frequency analysis was performed on the acquired EEG signals to extract the features of the EEG signals corresponding to different actions in the time and frequency domains. The acquired facial images were preprocessed, and convolutional neural networks were used to extract features related to lip shape and frowning actions in the facial images. Model training: Extracted EEG signal features and facial image features are fused as input data, and a deep learning model is used for training, with user-generated action labels as output labels; Practical application stages include: Real-time data acquisition and processing: Real-time acquisition of 8 channels of EEG signals and facial image information, and the acquisition data undergoes the same preprocessing and feature extraction operations as in the training phase; Action recognition and text input: The extracted real-time features are input into the trained model. The model analyzes and classifies the input features to determine the user's current lip-shaped or frowning action. Based on the model's recognition results, the action is converted into corresponding simple text or instructions according to a preset mapping relationship, and the generated text or instructions are sent to external devices.

Images