Device control system and method based on electroencephalogram signal

Abstract

The present invention discloses a device control system based on electroencephalogram signals, including an electroencephalogram signal acquisition module, a standard electroencephalogram signal generation module and a device control module; the device is controlled mainly by matching the acquired electroencephalogram signals with standard motor imagery electroencephalogram signals, thereby eliminating the need to train the system before each use, and effectively improving the control efficiency of the device. At the same time, the present invention can control the device in real time, and the user does not need to complete the motor imagery task within the prescribed task time during use, making it more convenient to use. Furthermore, the present invention can stably identify the user's motor imagery intention during long-term use, thereby greatly increasing the accuracy of the user's control of the device during use.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram signal processing, and in particular relates to an electroencephalogram signal-based device control system and method. Background Art

[0002] Brain-computer interface technology is an emerging technology developed based on brain science, and plays an increasingly important role in the fields of medical care, education, etc. Brain-computer interface technology includes multiple research directions such as motor imagery and steady-state visual evoked potentials, and the combination of motor imagery and the Internet of Things is increasingly being used to help patients with limb movement disorders achieve autonomous living. The current control systems based on brain-computer interfaces are mostly based on specific components in visually stimulated EEG signals, which are identified and converted into control commands, but the low-frequency flicker used for visual stimulation can easily cause visual fatigue and cannot be used for a long time.

[0003] Smart home control based on motor imagery generally involves: designing a motor imagery method, using an EEG cap to acquire brain waves, using algorithms to decode brain waves to identify the patient's motor intentions, and finally transmitting the identification signal to an external actuator to control the device.

[0004] There are many EEG signal decoding and recognition algorithms currently, most of which use machine learning methods to extract features, and then send the extracted features to the classifier for classification and decoding. Currently, there are many machine learning algorithms on the market that are based on clear task time and classify EEG signals within the task time. This paradigm requires a clear task time interval and requires training before each use. It cannot be applied to smart home control scenarios that require continuous response control. Summary of the invention

[0005] Purpose of the invention: In view of the problems existing in the prior art, the present invention proposes an EEG signal-based device control system that matches the collected EEG signals with standard motor imagery EEG signals and controls the device according to the matching results.

[0006] Technical solution: To achieve the above-mentioned purpose, the present invention provides a device control system based on EEG signals, including an EEG signal acquisition module, a standard EEG signal generation module and a device control module;

[0007] Among them, the EEG signal acquisition module is used to collect the EEG signals of the user when performing motor imagination;

[0008] A standard EEG signal generation module is used to generate standard EEG signals corresponding to different motor imagery and save them in a standard EEG signal set;

[0009] The device control module matches the currently collected EEG signal with each standard EEG signal in the standard EEG signal set in the standard EEG signal generation module; and finds the corresponding device control instruction according to the category of the successfully matched motor imagination, and controls the device according to the device control instruction.

[0010] Among them, the standard EEG signal generation module includes a first feature extraction model, which is trained using a first training data set. All EEG signals in the first training data set are mapped to the feature space through the trained first feature extraction model to form feature vectors; the heart-like feature vectors of feature vectors of different categories are calculated; the heart-like feature vectors of different categories are respectively input into the first feature extraction model for back propagation; and standard EEG signals corresponding to different categories of motor imagery are obtained.

[0011] Furthermore, the first feature extraction model includes a twin neural network and a Triplet loss function; randomly sampling a segment of EEG signal in the multi-category EEG signal data set as the Anchor EEG signal; randomly sampling a segment of EEG signal in the EEG signal of the same category as the Anchor EEG signal as the Positive EEG signal; randomly sampling a segment of EEG signal in the data of different categories from the Anchor as the Negative EEG signal; forming a ternary training sample, repeating continuously, combining all EEG signals in the multi-category EEG signal data set to form the first training data set. The first feature extraction model is trained using the first training data set, so that the features extracted by the first feature extraction model obtained in this way have better distinguishability, that is, the feature vectors of the same category are closer, and the feature vectors of different categories are farther away.

[0012] Further, the standard EEG signal generation module includes a calibration submodule and a user standard EEG signal generation module;

[0013] The calibration submodule uses the EEG signals of the user when performing the motor imagery task and the standard EEG signals in the standard EEG signal set as the second training data set, and uses the second training data set to perform migration training on the first feature extraction model to obtain the second feature extraction model;

[0014] The user standard EEG signal generation module maps all EEG signals in the second training data set to the feature space to form feature vectors through the trained second feature extraction model; calculates the different types of heart-like feature vectors; inputs the obtained different heart-like feature vectors into the second feature extraction model for back propagation; obtains the standard EEG signals corresponding to the different types of motor imagination of the user, and saves them to the standard EEG signal set. In this way, a standard EEG signal exclusive to the user can be obtained, which can effectively improve the accuracy of device control.

[0015] Furthermore, the device control module matches the EEG signal currently collected in the sliding time window with each standard EEG signal in the standard EEG signal set. In this way, the device can be continuously controlled without locking in the control time window, which greatly improves the control efficiency of the device; where locking means that a fixed time window can only give one prediction result.

[0016] Furthermore, the size of the sliding time window is consistent with the time length of the standard EEG signal, which can effectively improve the accuracy of the EEG signal matching result.

[0017] The present invention also provides a device control method based on EEG signals, comprising the following steps:

[0018] Matching the collected EEG signals of the user when performing motor imagery with the standard EEG signals in the standard EEG signal dataset;

[0019] By matching, the category of the motor imagination currently performed by the user is obtained;

[0020] Find the corresponding device control instruction according to the motor imagery category; send the control instruction to the device to complete the device control;

[0021] Among them, the method for obtaining standard EEG signals is: using a first training data set to train a first feature extraction model, the first training data set includes different categories of motor imagery EEG signals; mapping all EEG signals in the first training data set to the feature space to form feature vectors through the trained first feature extraction model; calculating the center-of-center feature vectors of different categories respectively; inputting the distribution center of the obtained feature vectors of different categories into the trained first feature extraction model for back propagation, thereby obtaining standard EEG signals of motor imagery of different categories to form a standard EEG signal set.

[0022] Furthermore, the method for acquiring the standard EEG signal also includes a calibration part and a user standard EEG signal generation part.

[0023] In the calibration part, the EEG signals of the user when performing the motor imagery task and the standard EEG signals in the standard EEG signal set are used as the second training data set, and the first feature extraction model is transferred and trained using the second training data set to obtain the second feature extraction model;

[0024] The user standard EEG signal generation part maps all the EEG signals in the second training data set to the feature space to form feature vectors through the trained second feature extraction model; calculates the center-like feature vectors of different categories of feature vectors; inputs the obtained different center-like feature vectors into the second feature extraction model for back propagation; obtains the standard EEG signals corresponding to different categories of motor imagination of the user, and the standard EEG signal set.

[0025] The present invention also provides a home control system based on the above-mentioned device control system of the EEG signal, including an EEG signal acquisition module, a standard EEG signal generation module and a device control module, a visual guidance module, a SSVEP_BCI module and an Internet of Things module;

[0026] The visual guidance module is used to display a list of all controllable devices connected to the Internet of Things and guide the motor imagination task; the visual guidance module displays the list of controllable devices in the form of tiles, and each tile flashes at a fixed frequency;

[0027] The SSVEP_BCI module stimulates the steady-state visual evoked potential in the user's EEG signal through the list of control devices displayed in the visual guidance module to determine the device selected by the user;

[0028] The EEG signal acquisition module is used to collect EEG signals when the user performs motor imagery tasks;

[0029] A standard EEG signal generation module is used to generate standard EEG signals corresponding to different motor imagery and save them in a standard EEG signal set;

[0030] The device control module matches the currently collected EEG signal with each standard EEG signal in the standard EEG signal set in the standard EEG signal generation module; finds the corresponding device control instruction according to the category of the successfully matched motor imagery, and sends the device control instruction to the device selected by the user through the Internet of Things module to complete the device control;

[0031] Among them, the standard EEG signal generation module includes a first feature extraction model, which is trained using a first training data set. All EEG signals in the first training data set are mapped to the feature space through the trained first feature extraction model to form feature vectors; the heart-like feature vectors of feature vectors of different categories are calculated; the obtained heart-like feature vectors of different categories are respectively input into the first feature extraction model for back propagation; and standard EEG signals corresponding to different categories of motor imagery are obtained.

[0032] The present invention also provides a computer-readable medium storing software, wherein the software includes instructions that can be executed by one or more computers, and the instructions enable the one or more computers to perform operations through such execution, and the operations include the process of the above-mentioned device control method based on EEG signals.

[0033] The present invention also provides a computer system, comprising:

[0034] one or more processors;

[0035] A memory stores operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations, wherein the operations include the process of the above-mentioned device control method based on EEG signals.

[0036] Beneficial effects: Compared with the prior art, the present invention matches the collected EEG signals with standard motor imagery EEG signals and controls the device according to the matching results, thereby eliminating the need to train the system before each use, thereby effectively improving the control efficiency of the device.

[0037] At the same time, the present invention can control the device when it is not locked. During use, the user does not need to complete the motor imagination task within the prescribed task time, which is more convenient to use.

[0038] Furthermore, the present invention can stably identify the user's motor imagination intention during long-term use, thereby greatly increasing the accuracy of the user's control over the device during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the method for acquiring multi-category motor imagery standard EEG signals disclosed in Example 1;

[0040] Figure 2 This is a flow chart of the method for acquiring standard EEG signals of multi-category motor imagery of a user disclosed in Example 2;

[0041] Figure 3 This is a schematic diagram of the structure of the device control system based on EEG signals disclosed in Example 3;

[0042] Figure 4 This is a schematic diagram of the structure of a home control system based on EEG signals disclosed in Example 4;

[0043] Figure 5 A schematic diagram of a display format of a control device list in a visual guidance module;

[0044] Figure 6 It is a schematic diagram of another display form of the control device list in the visual guidance module;

[0045] Figure 7 Schematic diagram of the display format of device control instructions in the vision guidance module. DETAILED DESCRIPTION

[0046] The following will be combined with the accompanying drawings in the examples of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Embodiment 1:

[0048] like Figure 1 As shown, this embodiment provides a method for acquiring multi-category motor imagery standard EEG signals, which mainly includes the following steps:

[0049] Step A1: training a first feature extraction model using a first training data set;

[0050] The first training data set includes a large number of multi-category motor imagery signals, and the first training data set is mainly sourced from EEG signals disclosed through various channels.

[0051] In this embodiment, the first feature extraction model is mainly for 4 types of motor imagery signals, namely tongue movement, foot movement, left hand movement and right hand movement. Therefore, there are 4 types of EEG signals for motor imagery tasks in the general training data set, and the number of EEG signals of each category is N.

[0052] During the training process, first, randomly sample a segment of EEG signals from N segments of EEG signals and set it as the Anchor EEG signal; secondly, randomly sample a segment of EEG signals from the data of the same category as the Anchor EEG signal and set it as the Positive EEG signal; thirdly, randomly sample a segment of EEG signals from the data of a different category from the Anchor and set it as the Negative EEG signal, forming a (A, P, N) ternary training sample, wherein the length of each segment of EEG signals in the ternary training sample is the same; repeat this process continuously, combine all the data in the data set according to the above requirements, and form the first training data set, the scale of the first training data set is C*N*(N-1)*(N*(C-1)), which can effectively increase the scale of the data set, wherein C represents the number of categories of motor imagination performed, in this embodiment, C is 4, and N represents the number of EEG signal segments generated by each category of motor imagination. Finally, the first training data set is input into the first feature extraction model for training.

[0053] In this embodiment, the first feature extraction model adopts a twin neural network and a Triplet loss function; the first feature extraction model after training can map the EEG signals performing different categories of motor imagery tasks to a feature space with greater discrimination; specifically, the distance between the feature vectors after mapping the same category of signals is small, and the distance between the feature vectors after mapping different categories of signals is large.

[0054] Among them, Triplet loss function:

[0055]

[0056] in, Represents the feature vector of the Anchor EEG signal after being mapped by the twin neural network; Represents the feature vector of the Positive EEG signal after being mapped by the twin neural network; Represents the eigenvector of the Negative EEG signal after being mapped by the twin neural network; α is the set minimum interval of the eigenvector, α is obtained by grid parameter adjustment, and is searched between 0.01, 0.1, 1, 10, and 100. In this embodiment, the value of α is 1.

[0057] Step A2: input the first training data set into the feature extraction model trained in step A1; map all EEG signals in the first training data set to the feature space to form a feature vector;

[0058] Step A3: respectively obtaining the centroid feature vectors of feature vectors of different categories; mainly by calculating the mean of feature vectors of the same category to obtain the centroid feature vector of the same category;

[0059] Step A4: Input the four types of motor imagery feature vectors obtained in step A3 into the first feature extraction model for back propagation, obtain four types of motor imagery standard EEG signals, and form the four types of motor imagery standard EEG signals into a first standard EEG signal set. The back propagation process is to find the partial derivative of the model parameters, use the gradient descent algorithm, and iterate the model parameters. The specific back propagation mainly includes the following steps:

[0060] Step A3-1: randomly generate a matrix with the same dimension as the EEG signal input during training, hereinafter referred to as random signal;

[0061] Step A3-2: using the generated random signal as the input of the first feature extraction model and mapping it into a feature vector;

[0062] Step A3-3: Compare the difference between the mapped feature vector and the given motion imagery centroid feature vector through the mean square error loss function;

[0063] Step A3-4: Use the gradient descent algorithm to continuously iterate and optimize the random signal; ultimately, minimize the loss function between the feature vector of the iterative random signal after model mapping and the centroid feature vector, that is, minimize the difference; thus, the obtained random signal is extremely consistent with the standard EEG signal of motor imagery.

[0064] Embodiment 2:

[0065] like Figure 2 As shown, this embodiment provides a method for obtaining a user's multi-category motor imagery standard EEG signal, which adds a calibration part and a user standard EEG signal generation part based on the method disclosed in Example 1, and specifically includes the following steps:

[0066] Step A1: training a first feature extraction model using a first training data set;

[0067] The first training data set includes a large number of multi-category motor imagery signals, and the first training data set is mainly sourced from EEG signals disclosed through various channels.

[0068] In this embodiment, the first feature extraction model is mainly for 4 types of motor imagery signals, namely tongue movement, foot movement, left hand movement and right hand movement. Therefore, there are 4 types of EEG signals for motor imagery tasks in the general training data set, and the number of EEG signals of each category is N.

[0069] During the training process, first, randomly sample a segment of EEG signals from N segments of EEG signals and set it as the Anchor EEG signal; secondly, randomly sample a segment of EEG signals from the data of the same category as the Anchor EEG signal and set it as the Positive EEG signal; thirdly, randomly sample a segment of EEG signals from the data of a different category from the Anchor and set it as the Negative EEG signal; repeat this process continuously, combine all the data in the data set according to the above requirements, and form the first training data set. The scale of the first training data set is C*N*(N-1)*(N*(C-1)), where C represents the number of categories of motor imagination performed, and in this embodiment, C is 4, and N represents the number of EEG signal segments generated by each category of motor imagination. Finally, the first training data set is input into the first feature extraction model for training.

[0070] In this embodiment, the first feature extraction model adopts a twin neural network and a Triplet loss function; the first feature extraction model after training can map the EEG signals performing different categories of motor imagery tasks to a feature space with greater discrimination; specifically, the distance between the feature vectors after mapping the same category of signals is small, and the distance between the feature vectors after mapping different categories of signals is large.

[0071] Among them, Triplet loss function:

[0072]

[0073] in, Represents the feature vector of the Anchor EEG signal after being mapped by the twin neural network; Represents the feature vector of the Positive EEG signal after being mapped by the twin neural network; It represents the feature vector of Negative EEG signal after being mapped by the twin neural network; α is the set minimum interval of the feature vector.

[0074] Step A2: input the first training data set into the feature extraction model trained in step A1; map all EEG signals in the first training data set to the feature space to form a feature vector;

[0075] Step A3: respectively obtaining the centroid feature vectors of feature vectors of different categories; mainly by calculating the mean of feature vectors of the same category to obtain the centroid feature vector of the same category;

[0076] Step A4: Input the four types of motor imagery-like feature vectors obtained in step A3 into the first feature extraction model for back propagation to obtain four types of motor imagery standard EEG signals, and compose the four types of motor imagery standard EEG signals into a first standard EEG signal set.

[0077] Step A5: prompting the user to perform four types of motor imagery tasks n times each, collecting EEG signals of the user when performing the four types of motor imagery tasks respectively; and marking the EEG signals generated by performing different types of motor imagery;

[0078] Step A6: Use the second training data set to perform migration training on the first feature extraction model trained in step A1; obtain the second feature extraction model. That is, fix the parameters of the shallow layer of the model, and only train the parameters of the output layer or the last few layers in the first feature extraction model; in this embodiment, specifically, in the migration training process, the first 95% of the parameters of the model do not participate in the iteration, and only the gradient descent algorithm is used to iterate the remaining 5% of the parameters.

[0079] Among them, the second training data set includes the first standard EEG signal and the EEG signals of the user performing 4 types of motor imagery tasks respectively collected in step A5; a segment of the EEG signal in the first standard EEG signal is set as the Anchor EEG signal; secondly, a segment of the EEG signal is randomly sampled from the data of the same category as the Anchor EEG signal and set as the Positive EEG signal; thirdly, a segment of the EEG signal is randomly sampled from the data of different category from the Anchor EEG signal and set as the Negative EEG signal; finally, this process is repeated continuously, and all the data in the data set are combined according to the above requirements to form the second training data set. The scale of the second training data set is C*n*(n*(C-1)), where n is the number of times the user performs each category of motor imagery, that is, each category of motor imagery generates n segments of EEG signals.

[0080] Step A7: Input the EEG signal collected in step A5 into the second feature extraction model trained in step A6, and map the collected EEG signal to the feature space to form a feature vector.

[0081] Step A8: respectively calculating the centroid feature vectors of the feature vectors of different categories obtained in step A7; mainly by calculating the mean of the feature vectors of the same category to obtain the centroid feature vectors of the same category;

[0082] Step A9: Input the heart-like feature vectors of the four types of motor imagery obtained in step A8 into the second feature extraction model for back propagation to obtain the four types of motor imagery standard EEG signals based on the user, and form the four types of motor imagery standard EEG signals for the user into a second standard EEG signal set.

[0083] The back propagation method in this embodiment is the same as that in Embodiment 1. For the back propagation in step A9, the generated random signal is used as the input of the second feature extraction model to perform subsequent steps.

[0084] Embodiment 3:

[0085] like Figure 3 As shown, this embodiment provides a device control system based on EEG signals, including an EEG signal acquisition module, a standard EEG signal generation module and a device control module;

[0086] Among them, the EEG signal acquisition module is used to collect EEG signals when the user performs motor imagination; and send the collected EEG signals to the device control module; in this embodiment, the device control module includes a brain-computer interface module based on motor phenomena (hereinafter referred to as MI_BCI module);

[0087] The standard EEG signal generation module is used to generate standard EEG signals corresponding to different motor imagery and save them in the standard EEG signal set. The standard EEG signal generation method in the standard EEG signal generation module in this embodiment mainly adopts the method disclosed in Example 1. That is, the first standard EEG signal set obtained in Example 1. In this embodiment, the standard signal database is based on the four types of motor imagery standard EEG signals of the user, which are the EEG signals corresponding to moving the tongue, the EEG signals corresponding to moving the feet, the EEG signals corresponding to moving the left hand, and the EEG signals corresponding to moving the right hand.

[0088] The device control module matches the currently collected EEG signal with each standard EEG signal in the standard EEG signal set in the standard EEG signal generation module; and finds the corresponding device control instruction according to the category of the successfully matched motor imagination, and controls the device according to the device control instruction.

[0089] The device control system based on EEG signals disclosed in this embodiment is used to control electric lights. The control method of the device control system based on EEG signals mainly includes the following steps:

[0090] Step B1: The EEG signal acquisition module acquires the EEG signals of the user when performing the motor imagery task, and sends the acquired EEG signals to the MI_BCI module. In this embodiment, the correspondence between the motor imagery task and the device control instruction is informed to the user in the device manual. In this embodiment, the motor imagery corresponding to turning on the light is moving the tongue, the motor imagery corresponding to turning off the light is moving the feet, the motor imagery corresponding to dimming the light is moving the left hand, and the motor imagery corresponding to brightening the light is moving the right hand.

[0091] Step B2: The MI_BCI module matches the EEG signals received during the motor imagery task with each segment of the standard EEG signals in the standard EEG signal set in turn; through the matching, the motor imagery category performed by the current user is obtained;

[0092] Among them, the matching is continued with the standard EEG signals based on the four categories of motor imagery of the user in a sliding time window manner. The size of the sliding time window is consistent with the time length of the standard EEG signal. In this embodiment, the correlation coefficient formula is used for the matching process:

[0093]

[0094] Where y(t) is the standard EEG signal; x(t) is the currently received EEG signal, and dt represents the derivative with respect to time t. In this embodiment, t is the length of the sliding time window, and ρ xy represents the correlation coefficient between the currently received EEG signal and the standard EEG signal in the standard signal database. xy When it is not less than the matching threshold, it means that the category of motor imagery performed by the currently received EEG signal and the standard EEG signal is the same. In this embodiment, the matching threshold is 0.7.

[0095] Among them, the method for matching the EEG signal of the user during the motor imagery task with each standard EEG signal in the standard EEG signal set can also use the KL divergence method, the mean square error method, the spectrum similarity method, the wavelet decomposition method, etc.

[0096] Step B3: The MI_BCI module finds the corresponding device control instruction according to the motor imagery category; sends the control instruction to the device to complete the device control;

[0097] If the MI_BCI module recognizes after matching that the user is performing tongue movement imagination, the MI_BCI module will send a "turn on the light" command to the light, and the light will turn on according to the command.

[0098] After receiving the device control command, the IoT module executes the specified control on the corresponding device to complete the operation. After receiving the "turn on the light" command, the IoT module turns on the light through the network.

[0099] Embodiment 4:

[0100] This embodiment discloses a home control system based on EEG signals, including an EEG signal acquisition module, a standard EEG signal acquisition module, a device control module, a wake-up module, a visual guidance module, a brain-computer interface module based on steady-state visual evoked potential (hereinafter referred to as SSVEP_BCI module) and an Internet of Things module;

[0101] The wake-up module is triggered by the user performing a specific wake-up task. After being triggered, the visual guidance module is activated and the signal acquisition module starts to collect the user's EEG signals. The wake-up module can effectively avoid the phenomenon of misoperation that is easy to occur in long-term use;

[0102] The visual guidance module includes a display, AR glasses and other mobile terminals that can be used for display; after receiving the activation command of the wake-up module, the mobile terminal displays a list of all controllable devices connected to the Internet of Things. Specifically, Figures 5-6 As shown, the control device list is displayed in the form of tiles, where each tile flashes at a fixed frequency (8-12Hz), and can also be synchronously displayed in the AR glasses to stimulate the SSVEP in the user's EEG signal, so that the user can select the device to be controlled by looking at it;

[0103] The SSVEP_BCI module stimulates the steady-state visual evoked potential in the user's EEG signal through the list of control devices displayed in the visual guidance module to determine the device selected by the user to control; it receives the EEG signal during the user's device selection, identifies the specific device the user wants to control based on the principle of steady-state visual evoked potential (SSVEP), and returns it to the visual guidance module;

[0104] After receiving the returned control device information, the mobile terminal displays the preset control device command and synchronously displays it in the AR glasses to guide the user to perform the corresponding motor imagery task. Figure 7 As shown, the movement corresponding to turning on the light is imagined as moving the tongue, the movement corresponding to turning off the light is imagined as moving the feet, the movement corresponding to dimming the light is imagined as moving the left hand, and the movement corresponding to brightening the light is imagined as moving the right hand.

[0105] The EEG signal acquisition module acquires the EEG signals when the user performs the motor imagery task; and sends the acquired EEG signals to the device control module;

[0106] The standard EEG signal generation module is used to generate standard EEG signals corresponding to different motor imagery and save them in the standard EEG signal set. The standard EEG signal generation method in the standard EEG signal generation module in this embodiment mainly adopts the method disclosed in Example 2. That is, the second standard EEG signal set obtained in Example 2. In this embodiment, the standard signal database is based on the four types of motor imagery standard EEG signals of the user, which are the EEG signals corresponding to moving the tongue, the EEG signals corresponding to moving the feet, the EEG signals corresponding to moving the left hand, and the EEG signals corresponding to moving the right hand.

[0107] The device control module matches the currently collected EEG signal with each standard EEG signal in the standard EEG signal set in the standard EEG signal generation module; and finds the corresponding device control instruction according to the category of the successfully matched motor imagination, and sends the device control instruction to the specific device through the Internet of Things module to complete the device control.

[0108] The present invention also provides a computer system, comprising: one or more processors; a memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations, wherein the operations include the process of the control method of the above-mentioned device control system based on EEG signals.

[0109] It should be understood that the examples of the control method of the device control system based on EEG signals of the present invention can be in any computer system including data storage and data processing, and the aforementioned computer system can be at least one electronic processing system or electronic device including a processor and a memory, such as a PC, whether it is a personal PC, a commercial PC, or a PC for graphics processing, or a server-level PC. These PCs can realize wired and / or wireless data transmission, especially image data, by having a data interface and / or a network interface.

[0110] In other embodiments, the computer system may also be a server, especially a cloud server, having data storage, processing and network communication functions.

[0111] An exemplary computer system generally includes at least one processor, a memory, and a network interface connected by a system bus. The network interface is used to communicate with other devices / systems.

[0112] The processor is used to provide system calculations and control.

[0113] The memory includes non-volatile memory and cache.

[0114] Non-volatile memory generally has a massive storage capacity and can store operating systems and computer programs. These computer programs may include operable instructions that, when executed by one or more processors, enable one or more processors to execute the process of the control method of the device control system based on EEG signals of the aforementioned embodiment of the present invention.

[0115] In necessary or reasonable implementations, the aforementioned computer system, whether a PC device or a server, may also include more or fewer components than those shown in the diagram, or a combination, or use different hardware, software or other components or different deployment methods.

Claims

1. A device control system based on EEG signals, characterized in that: It includes an EEG signal acquisition module, a standard EEG signal generation module and a device control module; The EEG signal acquisition module is used to acquire EEG signals when the user performs motor imagery; The standard EEG signal generation module is used to generate standard EEG signals corresponding to different motor imagery and save them into a standard EEG signal set; The device control module matches the currently collected EEG signal with each standard EEG signal in the standard EEG signal set in the standard EEG signal generation module; finds the corresponding device control instruction according to the category of the successfully matched motor imagery, and controls the device according to the device control instruction; The standard EEG signal generation module includes a first feature extraction model, a first training data set is used to train the first feature extraction model, all EEG signals in the first training data set are mapped to the feature space through the trained first feature extraction model to form feature vectors; different categories of heart-like feature vectors are calculated; the obtained heart-like feature vectors of different categories are respectively input into the first feature extraction model for back propagation; and standard EEG signals corresponding to different categories of motor imagery are obtained; Backpropagation consists of the following steps: Step A3-1: randomly generate a matrix with the same dimension as the EEG signal input during training, hereinafter referred to as random signal; Step A3-2: using the generated random signal as the input of the first feature extraction model and mapping it into a feature vector; Step A3-3: Compare the difference between the mapped feature vector and the given motion imagery centroid feature vector through the mean square error loss function; Step A3-4: Use the gradient descent algorithm to continuously iterate and optimize the random signal; ultimately, minimize the loss function between the feature vector of the iterative random signal mapped by the model and the centroid feature vector; the random signal thus obtained is the standard EEG signal corresponding to motor imagery.

2. The device control system based on EEG signals according to claim 1, characterized in that: The first feature extraction model includes a twin neural network and a triplet loss function; randomly sampling a segment of EEG signal in a multi-category EEG signal data set as an Anchor EEG signal; randomly sampling a segment of EEG signal in the EEG signal of the same category as the Anchor EEG signal as a Positive EEG signal; randomly sampling a segment of EEG signal in data of a different category from the Anchor as a Negative EEG signal to form a ternary training sample; repeating the process continuously to combine all EEG signals in the multi-category EEG signal data set to form a first training data set.

3. The device control system based on EEG signals according to claim 1, characterized in that: The standard EEG signal generation module includes a calibration submodule and a user standard EEG signal generation module; The calibration submodule uses the EEG signals of the user when performing the motor imagery task and the standard EEG signals in the standard EEG signal set as the second training data set, and uses the second training data set to perform migration training on the first feature extraction model to obtain the second feature extraction model; The user standard EEG signal generation module maps all EEG signals in the second training data set to the feature space to form feature vectors through the trained second feature extraction model; calculates the heart-like feature vectors of different categories; inputs the obtained different heart-like feature vectors into the second feature extraction model for back propagation; obtains standard EEG signals corresponding to different categories of motor imagination of the user, and saves them in the standard EEG signal set.

4. The device control system based on EEG signals according to claim 1, characterized in that: The device control module matches the currently collected EEG signal within the sliding time window with each standard EEG signal in the standard EEG signal set.

5. The device control system based on EEG signals according to claim 4, characterized in that: The size of the sliding time window is consistent with the time length of the standard EEG signal.

6. A device control method based on EEG signals, characterized in that: The following steps are involved: Matching the collected EEG signals of the user performing motor imagery with the standard EEG signals in the standard EEG signal data set; obtaining the motor imagery category performed by the current user; Find the corresponding device control instructions according to the motor imagery category; Send control instructions to the device to complete device control; The method for acquiring the standard EEG signal is as follows: using a first training data set to train a first feature extraction model, the first training data set includes different categories of motor imagery EEG signals; mapping all EEG signals in the first training data set to a feature space to form a feature vector through the trained first feature extraction model; respectively calculating the center-of-center feature vectors of different categories; inputting the distribution centers of the obtained feature vectors of different categories into the trained first feature extraction model for back propagation, thereby obtaining standard EEG signals of motor imagery of different categories to form a standard EEG signal set; Backpropagation consists of the following steps: Step A3-1: randomly generate a matrix with the same dimension as the EEG signal input during training, hereinafter referred to as random signal; Step A3-2: using the generated random signal as the input of the first feature extraction model and mapping it into a feature vector; Step A3-3: Compare the difference between the mapped feature vector and the given motion imagery centroid feature vector through the mean square error loss function; Step A3-4: Use the gradient descent algorithm to continuously iterate and optimize the random signal; ultimately, minimize the loss function between the feature vector of the iterative random signal mapped by the model and the centroid feature vector; the random signal thus obtained is the standard EEG signal corresponding to motor imagery.

7. The device control method based on EEG signals according to claim 6, characterized in that: The method for acquiring the standard EEG signal also includes a calibration part and a user standard EEG signal generation part. The calibration part uses the EEG signals of the user when performing the motor imagery task and the standard EEG signals in the standard EEG signal set as the second training data set, and uses the second training data set to perform migration training on the first feature extraction model to obtain the second feature extraction model; The user standard EEG signal generation part maps all the EEG signals in the second training data set to the feature space to form feature vectors through the trained second feature extraction model; calculates the center-like feature vectors of different categories of feature vectors; inputs the obtained different center-like feature vectors into the second feature extraction model for back propagation; obtains the standard EEG signals corresponding to different categories of motor imagination of the user, and the standard EEG signal set.

8. A home control system based on the device control system of the electroencephalogram signal according to claim 1, characterized in that: It includes EEG signal acquisition module, standard EEG signal generation module and device control module, visual guidance module, SSVEP_BCI module and Internet of Things module; The visual guidance module is used to display a list of all controllable devices connected to the Internet of Things and guide the motor imagination task; the visual guidance module displays the list of controllable devices in the form of tiles, and each tile flashes at a fixed frequency; The SSVEP_BCI module stimulates the steady-state visual evoked potential in the user's EEG signal through the list of control devices displayed in the visual guidance module to determine the device selected by the user; The EEG signal acquisition module is used to collect EEG signals when the user performs motor imagery tasks; A standard EEG signal generation module is used to generate standard EEG signals corresponding to different motor imagery and save them in a standard EEG signal set; The device control module matches the currently collected EEG signal with each standard EEG signal in the standard EEG signal set in the standard EEG signal generation module; finds the corresponding device control instruction according to the category of the successfully matched motor imagery, and sends the device control instruction to the device selected by the user through the Internet of Things module to complete the device control; The standard EEG signal generation module includes a first feature extraction model, a first training data set is used to train the first feature extraction model, all EEG signals in the first training data set are mapped to the feature space through the trained first feature extraction model to form feature vectors; the heart-like feature vectors of different categories of feature vectors are calculated; the obtained different heart-like feature vectors are respectively input into the first feature extraction model for back propagation; and the standard EEG signals corresponding to different categories of motor imagery are obtained; Backpropagation consists of the following steps: Step A3-1: randomly generate a matrix with the same dimension as the EEG signal input during training, hereinafter referred to as random signal; Step A3-2: using the generated random signal as the input of the first feature extraction model and mapping it into a feature vector; Step A3-3: Compare the difference between the mapped feature vector and the given motion imagery centroid feature vector through the mean square error loss function; Step A3-4: Use the gradient descent algorithm to continuously iterate and optimize the random signal; ultimately, minimize the loss function between the feature vector of the iterative random signal mapped by the model and the centroid feature vector; the random signal thus obtained is the standard EEG signal corresponding to motor imagery.

9. A computer-readable medium storing software, characterized in that: The software includes instructions that can be executed by one or more computers, and the instructions enable the one or more computers to perform operations through such execution, and the operations include the process of any one of claims 6-7 based on the above-mentioned EEG signal-based device control method.

10. A computer system, characterized in that: include: one or more processors; A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations, wherein the operations include the process of the device control method based on the above-mentioned EEG signal as described in any one of claims 6 to 7.

Images