A method, device and system for controlling a drone based on an EEG acquisition headband

Abstract

The present invention provides a method, device and system for controlling a drone based on an electroencephalogram (EEG) acquisition headband. The method comprises: obtaining motor imagery EEG signals of at least two users and hand motion data labels corresponding to the motor imagery EEG signals; performing frequency domain filtering processing on the motor imagery EEG signals using filters with multiple different passbands to obtain multiple frequency bands and EEG data corresponding to each frequency band, calculating mutual information, and sorting the frequency bands and EEG data according to the mutual information to obtain an input sample set; inputting the input sample set into a motor imagery neural network model to train the motor imagery neural network model; obtaining an EEG signal of a drone operator, and inputting the EEG signal of the drone operator into the trained motor imagery neural network model; obtaining a flight instruction output by the motor imagery neural network model; and controlling the flight of the drone according to the flight instruction.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interface, and in particular relates to a method, device and system for controlling a drone based on an electroencephalogram (EEG) acquisition headband. Background Art

[0002] Unmanned aerial vehicle technology is developing and being applied at a very fast pace. UAV equipment was originally envisioned for military applications, but has now been widely adopted by government and corporate entities to provide a wide range of services, including traffic management, cargo delivery, inventory tracking, disaster management, wireless networks, etc. Under the premise of the rapid development of drone technology, the existing drones interact with people mainly through traditional remote controls. However, as drone control becomes more and more complicated, the mapping methods such as joysticks on traditional remote controls also put forward higher requirements for drone operators. When operating at high speed in complex environments, accidents often occur due to overly complex operations. Even regular flights require a lot of training for operators to ensure stable flight. In addition, in special working environments, such as high-altitude operations and disaster relief, there are also problems such as how to control drones for auxiliary operations while freeing hands to carry out other important activities. Therefore, how to find a new drone control method that can replace traditional methods has gradually become a major demand.

[0003] The brain, as the control center of human physiological activities, contains huge exploration and utilization value. At present, there are four main means of obtaining information from the brain: MRI, fMRI, FNIRS, and EEG. EEG is electroencephalogram, and its signal is mainly formed by the mutual conduction of discharges of neuronal networks. Among these four methods, MRI and fMRI equipment are expensive and cannot be miniaturized, FNIRS has a low time resolution, and EEG has the characteristics of being relatively easy to receive and having a high time resolution.

[0004] However, in the field of brain-computer interfaces, the EEG recognition accuracy of different users on the same device varies greatly. In order to improve the general adaptability of the same device to different users, a large amount of training in the use of the device is needed, which will cause the equipment to be overloaded with training during use and cannot be conveniently applied in practice. When combining the brain-computer interface with the drone flight control, this will seriously affect the applicability of the entire drone flight control system, as well as the stability and accuracy of the drone during flight, making it difficult to actually apply brain-controlled drones. Summary of the invention

[0005] In view of this, the present invention aims to propose a drone brain control method, device and system based on an EEG acquisition headband to solve the problem of poor drone flight control accuracy caused by the poor applicability of existing drone brain control systems to different users.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] In a first aspect, the present invention provides a method for controlling a drone based on an EEG acquisition headband, comprising:

[0008] Obtaining motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals of at least two users, wherein each user collects at least two sets of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals, and each set of motor imagery EEG signals has six hand motion data labels;

[0009] The motor imagery EEG signal is subjected to frequency domain filtering processing using filters with multiple different passbands to obtain EEG data corresponding to each frequency band, and then the mutual information between the EEG data corresponding to each frequency band and the hand movement data label is calculated, and the frequency bands and their corresponding EEG data are sorted according to the mutual information to obtain an input sample set; wherein, the greater the mutual information, the greater the contribution of the EEG data corresponding to the frequency band to the information amount used for motor imagery action recognition, and the higher the ranking of the frequency band and its corresponding EEG data;

[0010] Inputting the input sample set into the motor imagery neural network model to train the motor imagery neural network model; wherein the motor imagery neural network model includes a feature fusion convolutional neural network, a fully connected layer, a multi-level cascade feature fusion layer, a multi-rotation forest classifier layer, and a voting output layer connected in sequence;

[0011] Obtaining the EEG signal of the drone operator and inputting the EEG signal of the drone operator into the trained motor imagery neural network model;

[0012] Obtaining the flight instructions output by the motor imagery neural network model; wherein the flight instructions of the drone correspond one-to-one to the hand motion data labels of the motor imagery EEG signals;

[0013] The UAV is controlled to fly according to the flight instructions.

[0014] Further, motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals of at least two users are obtained, and each user collects at least two sets of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals, and each set of motor imagery EEG signals has six hand motion data labels, including:

[0015] Acquire multiple groups of motor imagery EEG signals of the user and hand movement data labels corresponding to the motor imagery EEG signals; when acquiring each group of motor imagery EEG signals of the user, the user forms the hand movement data labels by performing six hand movements, and the motor imagery EEG signal collection of each hand movement lasts for a certain period of time;

[0016] Obtain motor imagery EEG signals of multiple users and hand motion data labels corresponding to the motor imagery EEG signals.

[0017] Furthermore, the six hand motion data labels respectively represent the left wrist raised upward, the left hand clenched into a fist, the five fingers of the left hand spread out, the right wrist raised upward, the right hand clenched into a fist, and the five fingers of the right hand spread out.

[0018] Furthermore, the method further comprises:

[0019] Constructing the motor imagery neural network model using the motor imagery EEG signal and the hand movement data label corresponding to the motor imagery EEG signal;

[0020] The step of constructing the motor imagery neural network model comprises:

[0021] The motor imagery EEG signal is randomly scattered to form a sample space;

[0022] 60% of the sample space is used for frequency domain filtering to obtain an input sample set, and the learning rate of the motor imagery neural network model is set to 0.001, and a total of 100 rounds of cyclic training are performed with a batch size of 32 to obtain an initial network model;

[0023] Taking 20% ​​of the remaining sample space to input into the initial network model for calibration, and adjusting the hyperparameters of the initial network model according to the result to obtain a trained motor imagery neural network model;

[0024] The last 20% of the remaining sample space is used to input the trained motor imagery neural network model for effect verification.

[0025] Further, the motor imagery EEG signal is subjected to frequency domain filtering processing using filters with multiple different passbands to obtain EEG data corresponding to each frequency band, and then the mutual information between the EEG data corresponding to each frequency band and the hand movement data label is calculated, and the frequency bands and their corresponding EEG data are sorted according to the mutual information to obtain an input sample set, including:

[0026] Using a frequency domain filter group to perform frequency domain filtering on the motor imagery EEG signal to obtain EEG data corresponding to each frequency band;

[0027] Using the standard common spatial pattern algorithm, the spatial filter banks corresponding to the EEG data in each frequency band were obtained;

[0028] The feature vector of each frequency band is obtained by matrix multiplication of the EEG data corresponding to each frequency band and the spatial filter bank;

[0029] The mutual information is calculated using the feature vectors of each frequency band and the hand motion data labels of the EEG data corresponding to each frequency band to obtain the mutual information corresponding to each frequency band;

[0030] According to the mutual information, each frequency band, the EEG data corresponding to each frequency band, and the spatial filter group corresponding to the EEG data are reordered, and the covariance matrix of the EEG data and the spatial filter group is calculated to obtain an input sample set.

[0031] Furthermore, the feature fusion convolutional neural network includes at least two branches, each of which includes:

[0032] The first feature fusion convolution layer has a convolution kernel size of 3×3, 10 convolution kernels, and the ReLU function is selected as the activation function;

[0033] The second feature fusion convolution layer has a convolution kernel size of 3×3, 14 convolution kernels, and the ReLU function is selected as the activation function;

[0034] The third feature fusion convolution layer has a convolution kernel size of 3×3 and 18 convolution kernels, and the ReLU function is selected as the activation function;

[0035] The Flatten layer is used to convert the multidimensional data of each branch into one dimension.

[0036] Furthermore, the multi-rotation forest classifier layer includes at least two rotation forest classifiers, and each rotation forest classifier randomly takes a part of the output of the multi-level cascade feature fusion layer as input.

[0037] In a second aspect, an embodiment of the present invention further provides a drone brain control device based on an EEG acquisition headband, comprising:

[0038] An acquisition module is used to acquire motor imagery EEG signals of at least two users and hand motion data labels corresponding to the motor imagery EEG signals. Each user collects at least two groups of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals. Each group of motor imagery EEG signals has six hand motion data labels.

[0039] An obtaining module is used to perform frequency domain filtering processing on the motor imagery EEG signal using filters with multiple different passbands to obtain multiple frequency bands and EEG data corresponding to each frequency band, calculate the mutual information between the EEG data corresponding to each frequency band and the information amount of the motor imagery EEG signal, and sort the frequency bands and EEG data according to the mutual information to obtain an input sample set;

[0040] A training module, used for inputting an input sample set into a motor imagery neural network model to train the motor imagery neural network model;

[0041] An acquisition module is used to obtain the EEG signal of the drone operator and input the EEG signal of the drone operator into the trained motor imagery neural network model;

[0042] An output module, used to obtain the flight instructions output by the motor imagery neural network model;

[0043] The control module is used to control the flight of the UAV according to the flight instructions.

[0044] Furthermore, the device also includes:

[0045] A construction module, used to construct the motor imagery neural network model using the motor imagery EEG signal and the hand movement data label corresponding to the motor imagery EEG signal;

[0046] The building blocks are used to:

[0047] The motor imagery EEG signal is randomly scattered to form a sample space;

[0048] 60% of the sample space is used for frequency domain filtering to obtain an input sample set, and the learning rate of the motor imagery neural network model is set to 0.001, and a total of 100 rounds of cyclic training are performed with a batch size of 32 to obtain an initial network model;

[0049] Taking 20% ​​of the remaining sample space to input into the initial network model for calibration, and adjusting the hyperparameters of the initial network model according to the result to obtain a trained motor imagery neural network model;

[0050] The last 20% of the remaining sample space is used to input the trained motor imagery neural network model for effect verification.

[0051] In a third aspect, an embodiment of the present invention further provides a drone brain control system based on an EEG acquisition headband, comprising:

[0052] Any of the above embodiments provided hereinabove, a drone brain control device based on an EEG acquisition headband;

[0053] EEG signal acquisition device and UAV flight device.

[0054] Compared with the prior art, the method, device and system for controlling a drone based on an EEG acquisition headband described in the present invention have the following advantages:

[0055] The technical problem to be solved by the present invention is to provide a new portable brain-controlled drone system which is highly portable, can accurately identify the EEG signals of the drone operator and parse the corresponding instructions, and control the drone to complete certain tasks.

[0056] The technical solution adopted by the present invention is to provide a solution for using physiological electrical signals of the human brain for drone control, to achieve accurate acquisition, effective identification and correct classification of EEG signals, and to directly control the drone through the brain-computer interface BCI, completing a huge leap in the rapid interaction between humans and drones, thereby enabling drone operators to have a larger operating space and complete more complex flight missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0058] Figure 1 A flowchart of a method for controlling a drone based on an EEG acquisition headband provided in Embodiment 1 of the present invention;

[0059] Figure 2 A schematic diagram of the structure of an EEG signal acquisition headband device in a drone brain control method based on an EEG acquisition headband provided in the first embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of a motor imagery neural network model in a drone brain control method based on an EEG acquisition headband provided in the first embodiment of the present invention;

[0061] Figure 4 A block diagram of a multi-rotation forest analysis method of a multi-rotation forest classifier layer in a drone brain control method based on an EEG acquisition headband provided in the first embodiment of the present invention;

[0062] Figure 5 A schematic diagram of the structure of a UAV flight device in a UAV brain control method based on an EEG acquisition headband provided in the first embodiment of the present invention;

[0063] Figure 6 A flow chart of a method for controlling a drone based on an EEG acquisition headband provided in Embodiment 2 of the present invention;

[0064] Figure 7 A schematic diagram of the structure of a drone brain control device based on an EEG acquisition headband provided in Embodiment 3 of the present invention;

[0065] Figure 8 This is a schematic diagram of the structure of a drone brain control system based on an EEG acquisition headband provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0067] Embodiment 1

[0068] Figure 1 This is a flow chart of a drone brain control method based on an EEG acquisition headband provided in the first embodiment of the present invention. This embodiment can be applied to the situation where the motor imagery EEG signals acquired by the EEG acquisition device are used to identify the operator's hand movements, thereby realizing the drone flight control. The method can be executed by a drone brain control device based on an EEG acquisition headband, and specifically includes the following steps:

[0069] S110, obtaining motor imagery EEG signals of at least two users and hand motion data labels corresponding to the motor imagery EEG signals, each user collecting at least two sets of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals, each set of motor imagery EEG signals having six hand motion data labels;

[0070] Exemplarily, an EEG signal collection headband device may be used to acquire the user's motor imagery EEG signals. Figure 2 This is a structural diagram of the EEG signal acquisition headband device in the drone brain control method based on the EEG acquisition headband provided in the first embodiment of the present invention, see Figure 2 The EEG signal acquisition headband device includes: a retractable headband shell for carrying content chips and circuits, a headband main control chip for controlling EEG acquisition processing and forwarding, an EEG signal acquisition and amplifier circuit, a headband power supply voltage conversion circuit, a headband wireless communication interface circuit, and an EEG acquisition electrode interface circuit.

[0071] The retractable headband shell includes a behind-the-ear fixing device, a retractable component, an elastic lining, a shell and an electrode opening. It can be worn on a person's head and fixed behind the ear by the fixing device. The retractable component can be used to adjust the tightness of the headband for people with different head shapes. The shell is provided with four electrode openings for placing four electrodes to contact the human body. The four electrodes are respectively located on the forehead and behind the ears.

[0072] The headband main control chip is used to control and receive the motor imagery EEG data received by the EEG acquisition electrode interface circuit. It customizes a communication protocol through data packaging, adds data headers and data tails every 100 potential changes to enhance data reliability, and forwards the packaged data to the headband wireless communication interface circuit through the Wi-Fi AP mode. Wi-Fi works under the 802.11b protocol with a maximum transmission rate of 54Mbps. Since the EEG signal is very weak (10-20μV), it needs to be amplified before A / D conversion. The EEG signal acquisition and amplifier circuit first amplifies the EEG signal and then performs A / D conversion. The circuit uses a 24-bit synchronous sampling analog-to-digital converter (ADC), a built-in programmable gain amplifier (PGA), an internal reference voltage source and an onboard oscillator, and uses differential input. The common mode rejection ratio (CMRR) can be as high as 110dB.

[0073] The headband power supply voltage conversion circuit is powered by a rechargeable lithium battery, and the lithium battery is charged by a USB port through a charging chip. Alternatively, the system can be powered directly by a USB port without using a lithium battery, and the working voltage of different chips in the system is provided by a voltage conversion module; different devices in the system require different power supply voltages, and a total of three power supply modes, 3.3v, ±2.5v, and 5v, are required. The voltage provided by the USB port to the circuit is +5V, and for 3.3, the +5V voltage must be converted into 3.3V voltage through the TL533 voltage regulator chip. The magnetic beads can also be replaced by inductors or 0 ohm resistors, but this has a slight effect on voltage stability. When the power indicator light is on, it indicates that the circuit is powered on and each power supply voltage is normal. The 5V voltage is filtered by the magnetic beads and capacitors to eliminate noise, and then input into the TL533 chip. The -2.5V voltage conversion first converts the 5.0V voltage into -5.0V voltage through the TL632 voltage reverse chip, and then converts the -5.0V voltage into -2.5V voltage through the TL625 voltage regulator chip.

[0074] The headband wireless communication interface circuit is connected to the headband main control chip via UART, and the serial port rate is 460800bps. After receiving the packaged EEG data, in AP mode, the maximum transmission rate is 54Mbps, and the EEG signal data is transmitted to the drone brain control device.

[0075] The EEG acquisition electrode interface circuit is connected to the patch electrode lead wire via the PJ313 model headphone interface, and a small filter circuit is introduced to perform 1-40 Hz bandpass filtering and 50 Hz notch processing on the collected EEG signal.

[0076] S120, performing frequency domain filtering processing on the motor imagery EEG signal using filters of multiple different passbands to obtain EEG data corresponding to each frequency band, calculating the mutual information between the EEG data corresponding to each frequency band and the hand movement data label, and sorting the frequency bands and their corresponding EEG data according to the mutual information to obtain an input sample set; wherein, the greater the mutual information, the greater the contribution of the EEG data corresponding to the frequency band to the information amount used for motor imagery action recognition, and the higher the frequency band and its corresponding EEG data are sorted;

[0077] In the field of brain-computer interfaces, the EEG recognition accuracy of different users on the same device varies greatly. One of the reasons is that the EEG frequency band energies of different users are different. In order to improve the universal adaptability of the same device to different users and reduce the training burden during device use, it is necessary to use mutual information indicators to calculate the contribution of different frequency bands to the amount of EEG signal information and sort the frequency bands. By re-sorting, frequency bands and EEG data with high contribution to the amount of EEG signal information can be obtained, so that redundant information can be discarded, the instability of the solution space of the subsequent neural network can be reduced, and the generalization ability of the motor imagery neural network model for different users can be improved. S130, input the input sample set into the motor imagery neural network model to train the motor imagery neural network model; wherein the motor imagery neural network model includes a feature fusion convolutional neural network, a fully connected layer, a multi-stage cascade feature fusion layer, a multi-rotation forest classifier layer, and a voting output layer connected in sequence;

[0078] Exemplarily, the motor imagery EEG signal and the hand motion data label corresponding to the motor imagery EEG signal may be used to construct the motor imagery neural network model;

[0079] For the EEG time series signal sample set formed after segmentation that can be used to train and test the motor imagery neural network model, in order to ensure that there is both relatively sufficient data for model training and relatively sufficient data for model testing, the motor imagery EEG signal can be randomly scattered to form a sample space; 60% of the sample space is used for frequency domain filtering to obtain the input sample set, i.e., the training set, and the learning rate of the motor imagery neural network model is set to 0.001, and a total of 100 rounds of cyclic training are performed with a Batchsize of 32 to obtain the initial network model; 20% of the remaining sample space is used to input the initial network model for calibration, and based on the results, the hyperparameters of the initial network model are adjusted to obtain the trained motor imagery neural network model; the last 20% of the remaining sample space is used to input the trained motor imagery neural network model for effect verification, and the effect verification uses the PR curve and the F1 indicator.

[0080] Figure 3This is a structural diagram of the motor imagery neural network model in the drone brain control method based on the EEG acquisition headband provided in the first embodiment of the present invention, see Figure 3 , the motor imagery neural network model includes:

[0081] Feature fusion convolutional neural network, the feature fusion convolutional neural network includes P branches, each branch takes the data in the input sample set as input. Specifically, the first feature fusion convolutional layer, the convolution kernel size is 3×3, the number of convolution kernels is 10, and the ReLU function is selected as the activation function; the second feature fusion convolutional layer, the convolution kernel size is 3×3, the number of convolution kernels is 14, and the ReLU function is selected as the activation function; the third feature fusion convolutional layer, the convolution kernel size is 3×3, the number of convolution kernels is 18, and the ReLU function is selected as the activation function; Flatten layer, used to convert the multi-dimensional data of each branch into one dimension;

[0082] For the fully connected layer, ReLU is selected as the activation function, L2 norm is selected as the regularization term, and the L2 norm is set to 0.001;

[0083] Multi-level cascade feature fusion layer, which fuses the information output by the fully connected layer and the information obtained in the previous convolution stage. Since the convolution process is a feature fusion process, some detailed features may be lost in the convolution process. In order to minimize information loss and obtain data features of multiple training stages of the neural network, it is necessary to add a multi-level cascade feature fusion layer to achieve better classification results. Since the information obtained in the convolution stage has a high dimensionality, it is necessary to reduce the dimensionality through the principal component analysis (PCA) method;

[0084] A multi-rotation forest classifier layer, the multi-rotation forest classifier layer includes at least two rotation forest classifiers, each of which randomly takes a part of the output of the multi-level cascade feature fusion layer as input;

[0085] The voting output layer uses all probability outputs in the multi-rotation forest classifier layer as voters to vote and obtain the final classification result.

[0086] Among them, the multi-rotation forest classifier layer and the voting output layer are the classification part networks, and the parameters of the first half of the network need to be fixed during training.

[0087] When the motor imagery neural network model is trained, the sorted EEG data are respectively input into the parallel multi-branch feature fusion convolutional neural network, and then feature extraction is performed through the multi-branch feature fusion convolutional neural network respectively, and finally all features are automatically fused in the fully connected layer; this method can obtain ordered EEG data after sorting the EEG data, and during the training process, the weights of the ordered EEG data will be automatically adjusted in the fully connected layer and the subsequent multi-rotation forest classifier layer, so as to automatically obtain better classification effect; this method can achieve better classification effect by pre-sorting the EEG data and combining the subsequent multi-branch feature fusion convolutional neural network, fully connected layer, and multi-rotation forest classifier layer, so as to facilitate the voting output layer to obtain more accurate classification results, that is, the motor imagery action data label recognition results. Figure 4 This is a block diagram of a multi-rotation forest analysis method for a multi-rotation forest classifier layer in a drone brain control method based on an EEG acquisition headband provided in Embodiment 1 of the present invention, see Figure 4 , L1 is the total number of decision trees.

[0088] In practical applications, the training set X can be first divided into K subsets [Xn, n∈(1, K)] according to the feature dimension. For each subset Xn, 75% of the samples are selected as Xn'. Then, the principal component analysis (PCA) is used to reduce the dimension of Xn'. In order to obtain the principal component Tn, it is necessary to construct the corresponding rotation matrix R, and then use the transformed data [X*Rj, j∈(1, L1)] to construct its corresponding decision tree Ti to obtain the predicted probability.

[0089] Input training data set x i , label y i , the initial weights are:

[0090] w k,i =1 / N,k∈[1,M]

[0091] For each k∈[1,M], train the rotation forest F k (x i ), the loss is calculated as:

[0092]

[0093] The weights are calculated as:

[0094]

[0095] Update the weights to:

[0096] w k+1,i ←w k,i *exp(a k *I(y i ≠Fk (x i )))

[0097] Then continue to update w k+1,i , and finally the output is:

[0098]

[0099] S140, obtaining an EEG signal of the drone operator, and inputting the EEG signal of the drone operator into the trained motor imagery neural network model;

[0100] When the operator actually uses the drone to fly, the EEG signal collection headband device is used to collect the operator's motor imagery EEG signals, and the trained motor imagery neural network model is used to distinguish the above motor imagery EEG signals.

[0101] S150, obtaining the flight instructions output by the motor imagery neural network model; wherein the flight instructions of the drone correspond one-to-one to the hand motion data labels of the motor imagery EEG signals;

[0102] Currently, the six hand motion data labels represent left wrist raised upward, left hand clenched into a fist, left hand five fingers spread out, right wrist raised upward, right hand clenched into a fist, and right hand five fingers spread out.

[0103] In actual application, raising the left wrist, clenching the left hand into a fist, spreading the five fingers of the left hand, raising the right wrist, clenching the right hand into a fist, and spreading the five fingers of the right hand can respectively correspond to the ascent command, take-off unlock command, descent command, forward command, horizontal rotation command, and backward command. When the drone needs to turn left or right, it can first rotate horizontally, and then move forward or backward after rotating horizontally to a certain angle. In this way, the operator can achieve more accurate control of the drone by controlling the angle of horizontal rotation, which is conducive to more complex control flight of the drone.

[0104] Raising the left wrist, clenching the left fist, spreading the five fingers of the left hand, raising the right wrist, clenching the right fist, and spreading the five fingers of the right hand can also correspond to the ascending command, descending command, left move command, right move command, forward command, and backward command respectively. Technical personnel in this field can also set the flight instructions of other drones according to the hand motion data labels according to actual needs, as long as the flight instructions of the drone can be made to correspond one-to-one with the hand motion data labels of the motor imagery EEG signals, thereby realizing the operator's control over the flight of the drone.

[0105] S160: Control the flight of the UAV according to the flight instruction.

[0106] For example, a UAV flight device may be used to receive flight control instructions from a UAV brain control device to achieve control of the UAV flight; Figure 5 This is a structural diagram of the UAV flight device in the UAV brain control method based on the EEG acquisition headband provided in the first embodiment of the present invention, see Figure 5 , the UAV flight device includes: communication and flight main control module, voltage conversion module, power output module. The communication and flight main control module uses a single chip with a Wi-Fi wireless communication protocol stack and a control terminal function at a high level of integration. The load current in the working state is 59mA, and the maximum Wi-Fi transmission rate is 54Mbps. It receives the flight control instructions sent by the UAV brain control device and parses them through the internal program. After unpacking, it obtains the corresponding flight action instructions and executes the corresponding flight actions under the control of the internal program. It can also control the LED on the UAV to flash different colors through the internal program to feedback the current movement information of the UAV to the human eye. In the actual application process, the UAV flight device can also feedback the action information to the UAV brain control device through wireless transmission to ensure that the action is correct. Each time the command is sent, the UAV action corresponding to the command lasts for 1S, and then maintains a self-stabilizing state. For power output, it is necessary to control the voltage conversion module to convert it into different voltage outputs.

[0107] The voltage conversion module adopts PWM control output mode, converts the PWM wave sent by the communication and flight main control module into different voltages through the internal power electronic switch device, and loads it to both ends of the power output module;

[0108] The power output module adopts a DC hollow cup motor with a rated input voltage of 3.7V. A total of four motors are arranged at the four corners of the unmanned aerial vehicle to provide flight power and control flight movements.

[0109] This embodiment obtains multiple groups of motor imagery EEG signals from multiple users, as well as hand movement data labels corresponding to the motor imagery EEG signals, and performs frequency domain filtering processing on the motor imagery EEG signals using multiple filters with different passbands to obtain multiple frequency bands and EEG data corresponding to each frequency band. The mutual information index is then used to calculate the contribution of different frequency bands to the amount of information in the EEG signal, and the frequency bands are sorted. By re-sorting, frequency bands and EEG data with a high contribution to the amount of information in the EEG signal can be obtained, thereby discarding redundant information, reducing the instability of the solution space of the subsequent neural network, and improving the generalization ability of the motor imagery neural network model for different users.

[0110] Embodiment 2

[0111] Figure 6A flowchart of a method for controlling a drone based on an EEG acquisition headband provided in the second embodiment of the present invention. This embodiment is optimized based on the above embodiment. In this embodiment, the motor imagery EEG signals of at least two users and the hand movement data labels corresponding to the motor imagery EEG signals are acquired. Each user acquires at least two groups of motor imagery EEG signals and the hand movement data labels corresponding to the motor imagery EEG signals. Each group of motor imagery EEG signals has six hand movement data labels. The specific optimization is as follows: acquiring multiple groups of motor imagery EEG signals of the user and the hand movement data labels corresponding to the motor imagery EEG signals; when acquiring each group of motor imagery EEG signals of the user, the user forms the hand movement data labels by performing six hand movements, and the motor imagery EEG signal acquisition of each hand movement lasts for a certain period of time; acquiring the motor imagery EEG signals of multiple users and the hand movement data labels corresponding to the motor imagery EEG signals.

[0112] Accordingly, the drone brain control method based on the EEG acquisition headband provided in this embodiment specifically includes:

[0113] S210, obtaining multiple groups of motor imagery EEG signals of the user, and hand movement data labels corresponding to the motor imagery EEG signals; when obtaining each group of motor imagery EEG signals of the user, the user forms the hand movement data labels by performing six hand movements, and the motor imagery EEG signal collection of each hand movement lasts for a certain period of time, for example, it can last for 3-6 seconds, to ensure that sufficient EEG signal changes can be collected for subsequent classification. In actual application, it may also be necessary to adjust the continuous collection time of the motor imagery EEG signal of each hand movement to ensure that sufficient EEG signal changes can be obtained to meet the subsequent classification and training of the motor imagery neural network model.

[0114] Exemplarily, for each user, an EEG experiment needs to be conducted first for initial signal collection. A total of n groups of motor imagery EEG need to be collected. Each group of motor imagery has six different movements, namely, lifting the left wrist upward, clenching the left hand into a fist, spreading the five fingers of the left hand, lifting the right wrist upward, clenching the right hand into a fist, and spreading the five fingers of the right hand. The advantages of these movements are that they have good distinction in EEG and are easy to imagine. Each motor imagery EEG collection lasts for 4 seconds, and sufficient EEG signal changes can be collected for subsequent classification.

[0115] S220, obtaining motor imagery EEG signals of multiple users and hand motion data labels corresponding to the motor imagery EEG signals;

[0116] For example, collecting EEG signals of N users, n EEG experiments can be expressed as:

[0117]

[0118] The corresponding hand motion data label is:

[0119]

[0120] Where n is the number of experiments, T is the number of sampling points, and M is the number of channels.

[0121] All the data are stitched together, and the stitched EEG data is represented as:

[0122] X source =concat{X1,X2,…,X N}

[0123] The corresponding hand motion data label is expressed as:

[0124] Y source =concat{Y1,Y2,…,Y N}

[0125] Where N is the number of experimenters.

[0126] By splicing, a motor imagery EEG signal dataset and a hand movement data label dataset can be formed for subsequent data processing and calculation.

[0127] S230, performing frequency domain filtering processing on the motor imagery EEG signal using filters of multiple different passbands to obtain multiple frequency bands and EEG data corresponding to each frequency band, calculating the mutual information between the EEG data corresponding to each frequency band and the information amount of the motor imagery EEG signal, and sorting the frequency bands and EEG data according to the mutual information to obtain an input sample set; wherein, the greater the mutual information, the greater the contribution of the frequency band to the information amount of the motor imagery EEG signal, and the higher the sorting of the frequency band and its corresponding EEG data;

[0128] Exemplarily, a frequency domain filter bank may be used to perform frequency domain filtering on the motor imagery EEG signal to obtain EEG data corresponding to each frequency band;

[0129] Using the standard common spatial pattern algorithm, the spatial filter banks corresponding to the EEG data in each frequency band were obtained;

[0130] The feature vector of each frequency band is obtained by matrix multiplication of the EEG data corresponding to each frequency band and the spatial filter bank;

[0131] The mutual information is calculated using the feature vectors of each frequency band and the hand motion data labels of the EEG data corresponding to each frequency band to obtain the mutual information corresponding to each frequency band;

[0132] According to the mutual information, each frequency band, the EEG data corresponding to each frequency band, and the spatial filter group corresponding to the EEG data are reordered, and the covariance matrix of the EEG data and the spatial filter group is calculated to obtain an input sample set.

[0133] Specifically, in order to obtain the input sample set, two stages of data processing can be performed in sequence, namely, the frequency domain filtering stage and the mutual information measurement stage, which specifically include:

[0134] In the frequency domain filtering stage, multiple frequency bands are predefined and a frequency domain filter bank b is used. k , where k = 1, 2, ..., K, K is the total number of filters, for X source To filter:

[0135]

[0136] in represents the filtering operation, E k is the EEG data after filtering;

[0137] Using the standard co-spatial pattern algorithm, we get k The spatial filter bank:

[0138] W k ∈i U×U

[0139] Where U is the number of spatial filters;

[0140] In the mutual information measurement stage, the filtered EEG data E k and the spatial filter bank W k Matrix multiplication between them to obtain each frequency band b k The eigenvector of:

[0141]

[0142] Where var() represents the variance operation. Since it is uncertain which frequency bands can constitute separable EEG features, the frequency band b is used k and feature V k The mutual information on encodes uncertainty. Mutual information is a measure of the mutual dependence between two random variables and the uncertainty of the random variables. The mutual information is calculated as follows:

[0143] I(V k ; Y source )=H(V k )-H(V k |Y source )

[0144] Among them, H() is entropy, H(|) is conditional entropy, and I() is mutual information. The contribution of each frequency band to the amount of EEG information is calculated. According to the size of mutual information, the contribution of frequency band b k and the filtered EEG data E k Rearrange and get b p , E p and W p , p=1,2,…,P. Calculate the covariance matrix:

[0145]

[0146] Calculate P times and get the input sample set C = {C1, C2, ..., C p}∈R U×U×P ;

[0147] S240, inputting the input sample set into the motor imagery neural network model, and training the motor imagery neural network model; wherein the motor imagery neural network model includes a feature fusion convolutional neural network, a fully connected layer, a multi-level cascade feature fusion layer, a multi-rotation forest classifier layer, and a voting output layer connected in sequence;

[0148] S250, obtaining an EEG signal of the drone operator, and inputting the EEG signal of the drone operator into the trained motor imagery neural network model;

[0149] S260, obtaining the flight instructions output by the motor imagery neural network model; wherein the flight instructions of the drone correspond one-to-one to the hand motion data labels of the motor imagery EEG signals;

[0150] S270: Control the flight of the UAV according to the flight instruction.

[0151] This embodiment obtains multiple groups of motor imagery EEG signals of the user, and hand movement data labels corresponding to the motor imagery EEG signals; when obtaining each group of motor imagery EEG signals of the user, the user forms the hand movement data labels by performing six hand movements, and the motor imagery EEG signal collection of each hand movement lasts for a certain period of time; and obtains motor imagery EEG signals of multiple users, and hand movement data labels corresponding to the motor imagery EEG signals, and trains the motor imagery neural network model using multiple groups of motor imagery EEG signals of multiple users, and hand movement data labels corresponding to the motor imagery EEG signals, thereby further improving the applicability of the motor imagery neural network model to different users, and facilitating improving the accuracy of the motor imagery neural network model outputting flight instructions.

[0152] Embodiment 3

[0153] Figure 7This is a schematic diagram of the structure of a UAV brain control device based on an EEG acquisition headband provided in Embodiment 3 of the present invention. This UAV brain control device analyzes the operator's EEG data through the motor imagery neural network model in the above embodiment, and sends the parsed flight instructions to the wireless receiving part in the UAV flight device. The UAV flight device is parsed into corresponding flight actions by the main control to realize the UAV flight control, such as Figure 7 As shown, the device comprises:

[0154] The acquisition module 310 is used to acquire motor imagery EEG signals of at least two users and hand motion data labels corresponding to the motor imagery EEG signals. Each user collects at least two groups of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals. Each group of motor imagery EEG signals has six hand motion data labels.

[0155] Obtaining module 320, for performing frequency domain filtering processing on the motor imagery EEG signal using filters with multiple different passbands, obtaining multiple frequency bands and EEG data corresponding to each frequency band, calculating the mutual information between the EEG data corresponding to each frequency band and the information amount of the motor imagery EEG signal, and sorting the frequency bands and EEG data according to the mutual information to obtain an input sample set;

[0156] A training module 330, for inputting an input sample set into a motor imagery neural network model to train the motor imagery neural network model;

[0157] An acquisition module 340 is used to acquire the EEG signal of the drone operator and input the EEG signal of the drone operator into the trained motor imagery neural network model;

[0158] An output module 350 is used to obtain the flight instructions output by the motor imagery neural network model;

[0159] The control module 360 ​​is used to control the flight of the UAV according to the flight instruction.

[0160] Based on the above embodiments, the device further includes:

[0161] A construction module, used to construct the motor imagery neural network model using the motor imagery EEG signal and the hand movement data label corresponding to the motor imagery EEG signal;

[0162] The building blocks are used to:

[0163] The motor imagery EEG signal is randomly scattered to form a sample space;

[0164] 60% of the sample space is used for frequency domain filtering to obtain an input sample set, and the learning rate of the motor imagery neural network model is set to 0.001, and a total of 100 rounds of cyclic training are performed with a batch size of 32 to obtain an initial network model;

[0165] Taking 20% ​​of the remaining sample space to input into the initial network model for calibration, and adjusting the hyperparameters of the initial network model according to the result to obtain a trained motor imagery neural network model;

[0166] The last 20% of the remaining sample space is used to input the trained motor imagery neural network model for effect verification.

[0167] Exemplarily, each of the above modules can be run in a host computer to achieve data reception, processing and distribution.

[0168] The drone brain control device based on the EEG acquisition headband provided in this embodiment obtains multiple groups of motor imagery EEG signals of at least two users, and hand movement data labels corresponding to the motor imagery EEG signals; performs frequency domain filtering processing and mutual information calculation on the motor imagery EEG signals, and sorts the frequency bands and EEG data according to the size of the mutual information to obtain an input sample set; then inputs the input sample set into a motor imagery neural network model to train the motor imagery neural network model; finally, obtains the EEG signal of the drone operator, and inputs the EEG signal of the drone operator into the trained motor imagery neural network model, thereby obtaining the flight instructions output by the motor imagery neural network model, and controls the flight of the drone according to the flight instructions

[0169] The drone brain control device based on the EEG acquisition headband provided in the embodiment of the present invention can execute the drone brain control method based on the EEG acquisition headband provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0170] Embodiment 4

[0171] Figure 8 This is a schematic diagram of the structure of the UAV brain control system based on the EEG acquisition headband provided in the fourth embodiment of the present invention, see Figure 8 The drone brain control system based on the EEG acquisition headband includes: any of the drone brain control devices based on the EEG acquisition headband provided in the above embodiments; an EEG signal acquisition device and a drone flight device; wherein the EEG signal acquisition device realizes data connection with the drone flight device through the drone brain control device.

[0172] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for controlling a drone based on an EEG acquisition headband, characterized in that: include: Obtaining motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals of at least two users, wherein each user collects at least two sets of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals, and each set of motor imagery EEG signals has six hand motion data labels; The motor imagery EEG signal is subjected to frequency domain filtering processing using filters with multiple different passbands to obtain EEG data corresponding to each frequency band, and then the mutual information between the EEG data corresponding to each frequency band and the hand movement data label is calculated, and the frequency bands and their corresponding EEG data are sorted according to the mutual information to obtain an input sample set; wherein, the greater the mutual information, the greater the contribution of the EEG data corresponding to the frequency band to the information amount used for motor imagery action recognition, and the higher the ranking of the frequency band and its corresponding EEG data; Inputting the input sample set into the motor imagery neural network model to train the motor imagery neural network model; wherein the motor imagery neural network model comprises a feature fusion convolutional neural network, a fully connected layer, a multi-level cascade feature fusion layer, a multi-rotation forest classifier layer, and a voting output layer connected in sequence; the multi-rotation forest classifier layer comprises at least two rotation forest classifiers, each of which randomly takes a part of the output of the multi-level cascade feature fusion layer as input; Obtaining the EEG signal of the drone operator and inputting the EEG signal of the drone operator into the trained motor imagery neural network model; Obtaining the flight instructions output by the motor imagery neural network model; wherein the flight instructions of the drone correspond one-to-one to the hand motion data labels of the motor imagery EEG signals; The UAV is controlled to fly according to the flight instructions.

2. The method according to claim 1, characterized in that Obtain motor imagery EEG signals of at least two users and hand motion data labels corresponding to the motor imagery EEG signals. Each user collects at least two sets of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals. Each set of motor imagery EEG signals has six hand motion data labels, including: Acquire multiple groups of motor imagery EEG signals of the user and hand movement data labels corresponding to the motor imagery EEG signals; when acquiring each group of motor imagery EEG signals of the user, the user forms the hand movement data labels by performing six hand movements, and the motor imagery EEG signal collection of each hand movement lasts for a certain period of time; Obtain motor imagery EEG signals of multiple users and hand motion data labels corresponding to the motor imagery EEG signals.

3. The method according to claim 1, characterized in that: The six hand motion data labels respectively represent the left wrist being raised upward, the left hand being clenched into a fist, the five fingers of the left hand being spread out, the right wrist being raised upward, the right hand being clenched into a fist, and the five fingers of the right hand being spread out.

4. The method according to claim 1, characterized in that: The method further comprises: Constructing the motor imagery neural network model using the motor imagery EEG signal and the hand movement data label corresponding to the motor imagery EEG signal; The step of constructing the motor imagery neural network model comprises: The motor imagery EEG signal is randomly scattered to form a sample space; 60% of the sample space is used for frequency domain filtering to obtain an input sample set, and the learning rate of the motor imagery neural network model is set to 0.001, and a total of 100 rounds of cyclic training are performed with a batch size of 32 to obtain an initial network model; 20% of the remaining sample space is used to input the initial network model for calibration, and according to the result, the hyperparameters of the initial network model are adjusted to obtain a trained motor imagery neural network model; The last 20% of the remaining sample space is used to input into the trained motor imagery neural network model for effect verification.

5. The method according to claim 1, characterized in that The motor imagery EEG signal is subjected to frequency domain filtering processing using filters with multiple different passbands to obtain EEG data corresponding to each frequency band, and then the mutual information between the EEG data corresponding to each frequency band and the hand movement data label is calculated, and the frequency bands and their corresponding EEG data are sorted according to the mutual information to obtain an input sample set, including: Using a frequency domain filter group to perform frequency domain filtering on the motor imagery EEG signal to obtain EEG data corresponding to each frequency band; Using the standard common spatial pattern algorithm, the spatial filter banks corresponding to the EEG data in each frequency band were obtained; The feature vector of each frequency band is obtained by matrix multiplication of the EEG data corresponding to each frequency band and the spatial filter bank; The mutual information is calculated using the feature vectors of each frequency band and the hand motion data labels of the EEG data corresponding to each frequency band to obtain the mutual information corresponding to each frequency band; According to the mutual information, each frequency band, the EEG data corresponding to each frequency band, and the spatial filter group corresponding to the EEG data are sorted, and the covariance matrix of the EEG data and the spatial filter group is calculated to obtain an input sample set.

6. The method according to claim 1, characterized in that The feature fusion convolutional neural network includes at least two branches, each of which includes: The first feature fusion convolution layer has a convolution kernel size of 3×3, 10 convolution kernels, and the ReLU function is selected as the activation function; The second feature fusion convolution layer has a convolution kernel size of 3×3, 14 convolution kernels, and the ReLU function is selected as the activation function; The third feature fusion convolution layer has a convolution kernel size of 3×3 and 18 convolution kernels, and the ReLU function is selected as the activation function; The Flatten layer is used to convert the multidimensional data of each branch into one dimension.

7. A UAV brain control device based on an EEG acquisition headband, characterized in that: include: An acquisition module is used to acquire motor imagery EEG signals of at least two users and hand motion data labels corresponding to the motor imagery EEG signals. Each user collects at least two groups of motor imagery EEG signals and hand motion data labels corresponding to the motor imagery EEG signals. Each group of motor imagery EEG signals has six hand motion data labels. A module is provided for performing frequency domain filtering processing on the motor imagery EEG signal by using filters of multiple different passbands, obtaining multiple frequency bands and EEG data corresponding to each frequency band, calculating the mutual information between the EEG data corresponding to each frequency band and the information amount of the motor imagery EEG signal, and sorting the frequency bands and EEG data according to the mutual information to obtain an input sample set; wherein, the greater the mutual information, the greater the contribution of the EEG data corresponding to the frequency band to the information amount used for motor imagery action recognition, and the higher the frequency band and its corresponding EEG data are sorted; A training module is used to input an input sample set into a motor imagery neural network model to train the motor imagery neural network model; wherein the motor imagery neural network model includes a feature fusion convolutional neural network, a fully connected layer, a multi-level cascade feature fusion layer, a multi-rotation forest classifier layer, and a voting output layer connected in sequence; the multi-rotation forest classifier layer includes at least two rotation forest classifiers, each of which randomly takes a part of the output of the multi-level cascade feature fusion layer as input; An acquisition module is used to obtain the EEG signal of the drone operator and input the EEG signal of the drone operator into the trained motor imagery neural network model; An output module, used to obtain the flight instructions output by the motor imagery neural network model; The control module is used to control the flight of the UAV according to the flight instructions.

8. The device according to claim 7, characterized in that The device also includes: A construction module, used to construct the motor imagery neural network model using the motor imagery EEG signal and the hand movement data label corresponding to the motor imagery EEG signal; The building blocks are used to: The motor imagery EEG signal is randomly scattered to form a sample space; 60% of the sample space is used for frequency domain filtering to obtain an input sample set, and the learning rate of the motor imagery neural network model is set to 0.001, and a total of 100 rounds of cyclic training are performed with a batch size of 32 to obtain an initial network model; 20% of the remaining sample space is used to input the initial network model for calibration, and according to the result, the hyperparameters of the initial network model are adjusted to obtain a trained motor imagery neural network model; The last 20% of the remaining sample space is used to input into the trained motor imagery neural network model for effect verification.

9. A UAV brain control system based on an EEG acquisition headband, characterized in that: include: The UAV brain control device based on the EEG acquisition headband as described in claim 7 or 8; EEG signal acquisition device and UAV flight device.

Images