A brain-controlled drone system based on LED stereo interactive interface

Abstract

The present invention discloses a brain-controlled drone system based on an LED stereoscopic interactive interface. The system includes an LED-based stereoscopic interactive module, an electroencephalogram signal acquisition module, an electroencephalogram signal preprocessing module 1, an electroencephalogram signal preprocessing module 2, an electroencephalogram signal analysis module 1, an electroencephalogram signal analysis module 2, a positioning module, and a lower computer. The present invention proposes for the first time a brain-controlled drone system based on an LED stereoscopic interactive interface, which enables drone controllers to control the drone more intuitively. In addition, the system realizes speed control of the drone, which is beneficial to improving drone control efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a brain-controlled unmanned aerial vehicle system based on an LED stereoscopic interactive interface. Background Art

[0002] The flight control of drones is carried out in three-dimensional space, and so is brain-controlled drones. The EEG signals used by the brain-controlled drone system include P300 EEG signals, Steady State Visual Evoked Potential (SSVEP) EEG signals, and motor imagery EEG signals. Among them, P300 EEG signals and Steady State Visual Evoked Potential EEG signals require a visual stimulation interface to induce the corresponding EEG signals. Considering ergonomics, it is best to use a three-dimensional interactive interface to control objects that move in three dimensions, which is more in line with people's operating habits and more intuitive and effective. At present, in terms of brain-controlled drones, the interactive interface between the controller and the drone is flat, such as a screen. When there are less than 4 brain-controlled commands (such as forward, backward, left, and right), the visual stimulation display on the plane can be intuitively corresponding to the objects moving in three-dimensional space. When there are more than 4 brain control commands, for example 6 commands (forward, backward, left, right, up, down), although the visual stimulation display on the plane can still correspond to the objects moving in three-dimensional space, it is not intuitive. If there are more brain control commands, the intuitiveness will be greatly reduced. In addition, in the process of brain-controlled drone three-dimensional space control, existing research has less control over the speed of the drone.

[0003] In summary, there is no brain-controlled drone control three-dimensional interactive system in the prior art; most of the brain-controlled drones in the current literature move at a constant speed, and there is no research on the control of speed changes. Summary of the invention

[0004] The purpose of the present invention is to propose a brain-controlled drone system based on an LED three-dimensional interactive interface, which is conducive to the control personnel to control the brain-controlled drone more intuitively and naturally, and realize the control personnel's control of the speed, thereby improving the drone control efficiency.

[0005] A brain-controlled drone system based on an LED stereoscopic interactive interface comprises an LED-based stereoscopic interactive module, an electroencephalogram signal acquisition module, an electroencephalogram signal preprocessing module 1, an electroencephalogram signal preprocessing module 2, an electroencephalogram signal analysis module 1, an electroencephalogram signal analysis module 2, a positioning module, and a lower computer.

[0006] The LED-based stereoscopic interactive module is composed of a three-dimensional orthogonal bracket and 13 LED lights fixed on the bracket, and is used to realize the display of visual stimulation.

[0007] The EEG signal acquisition module includes an EEG amplifier, brain electrodes, an EEG cap and EEG acquisition software. The EEG signal sampling frequency is fs=1000 Hz, which is used to realize the acquisition of EEG signals.

[0008] The EEG signal preprocessing module 1 and the EEG signal preprocessing module 2 preprocess the EEG signals according to the corresponding EEG patterns, respectively, so as to further analyze the EEG signals.

[0009] The EEG signal analysis module 1 recognizes three consecutive rapid eye blinks of the controller by analyzing EEG signals, thereby realizing command control of take-off / landing and normal flight status.

[0010] The EEG signal analysis module 2 realizes the analysis of the control instructions of the drone.

[0011] The positioning module includes an optical positioning smart camera and a camera gimbal, which are used to collect the position information of the drone in real time.

[0012] The lower computer receives the position information of the UAV and converts the UAV formation control instruction or the single-machine control instruction into the UAV underlying mechanism execution instruction and transmits the instruction to the UAV to realize the execution of the control personnel's intention.

[0013] Beneficial effects of the present invention: The present invention proposes a brain-controlled drone system based on an LED stereoscopic interactive interface, which enables the control personnel to control the brain-controlled drone system more intuitively and naturally, and realizes the control of the speed of the drone in six directions, thereby improving the control efficiency of the brain-controlled drone. The present invention proposes a brain-controlled drone system based on an LED stereoscopic interactive interface, and the control instructions reach 16 (1 low-speed right shift, 2 high-speed right shift, 3 low-speed left shift, 4 high-speed left shift, 5 low-speed forward, 6 high-speed forward, 7 low-speed backward, 8 high-speed backward, 9 low-speed rise, 10 high-speed rise, 11 low-speed descent, 12 high-speed descent, 13 hover, 14 hold, 15 take-off, 16 landing). The present invention proposes for the first time a brain-controlled drone system based on an LED stereoscopic interactive interface, which enables the drone control personnel to control the drone more intuitively. In addition, the speed control of the drone is realized, which is conducive to improving the control efficiency of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a structural block diagram of the brain-controlled drone system based on the LED three-dimensional interactive interface of the present invention.

[0015] Figure 2 It is a schematic diagram of a stereoscopic interactive module based on LED;

[0016] In the picture, there are 13 LED lights in total, each of which flashes at its own frequency and phase. When the drone controller looks at a specific square, it will induce a corresponding EEG pattern.

[0017] Figure 3 The channels used for EEG signal collection (black marks).

[0018] Figure 4 This is the algorithm flow chart of EEG signal analysis module 2. DETAILED DESCRIPTION

[0019] In order to facilitate understanding of the present invention, the present invention will be described more fully below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0020] This embodiment includes a LED-based stereoscopic interaction module, an EEG signal acquisition module, an EEG signal preprocessing module 1, an EEG signal analysis module 1, an EEG signal preprocessing module 2, an EEG signal analysis module 2, a lower computer, and a positioning module. The system block diagram is shown in FIG. Figure 1 shown.

[0021] The LED-based stereoscopic interactive module realizes the display of visual stimulation. The EEG signal acquisition module realizes the acquisition of EEG signals. The EEG signal preprocessing module 1 and the EEG signal preprocessing module 2 preprocess the EEG signals according to the corresponding EEG patterns respectively, so as to further analyze the EEG signals. The EEG signal analysis module 1 recognizes the three consecutive rapid blinks of the controller by analyzing the EEG signals, and then realizes the command control of take-off / landing and normal flight status. The EEG signal analysis module 2 realizes the control command analysis of the UAV. The positioning module collects the location information of the UAV in real time. The lower computer receives the location information of the UAV and converts the UAV formation control command or the single-machine control command into the execution command of the UAV underlying mechanism and transmits the command to the UAV to realize the execution of the controller's intention.

[0022] LED-based stereo interactive modules such as Figure 2 As shown, it consists of a three-dimensional orthogonal bracket and 13 LED lights fixed on the bracket. Lights 2, 4, 6, 8, 10, and 12 are distributed at the end positions, lights 1, 3, 5, 7, 9, and 11 are distributed at the midpoint positions, and light 13 is at the center position. The flashing frequency and phase of the 13 LED lights are fk / Фk (k=1, 2, …, 13), which are used to induce 13 EEG patterns. The 13 EEG patterns correspond to 13 control instructions of the drone. When the controller is not looking at the interface, the 14th EEG pattern will be induced and correspond to instruction 14.

[0023] The EEG signal acquisition module includes an EEG amplifier, brain electrodes, an EEG cap and EEG acquisition software. The EEG signal sampling frequency is fs = 1000Hz. This patent uses 23 channels, of which 15 channels (Pz, P1, P2, P3, P4, P5, P6, POz, PO3, PO4, PO7, PO8, Oz, O1, O2) are distributed in and around the visual area of ​​the brain to identify steady-state visual evoked potentials, and the other 8 channels (FPz, FP1, FP2, AFz, AF3, AF4, AF7, AF8) are distributed in the frontal lobe area to identify 3 consecutive rapid blinking activities. Specific channels are as follows: Figure 3 shown.

[0024] The purpose of the EEG signal preprocessing module 1 is to preprocess the EEG signal before the EEG signal analysis module 1, using 8 channels in the frontal lobe area. The main work includes filtering and downsampling. The EEG signal corresponding to the time length T can be expressed as N = T * fs represents the sampling points in the time dimension, and L = 8 represents the number of channels. First, a bandpass filter is used for filtering, with a high-pass cutoff frequency of 1 Hz and a low-pass cutoff frequency of 10 Hz. Then downsampling is performed, and the downsampling factor is τ = 10. The EEG signal after preprocessing can be expressed as N1=N / τ.

[0025] The purpose of the EEG signal analysis module 1 is to identify the three consecutive rapid eye blinks of the control personnel, so as to realize the control of take-off / landing. After collecting the EEG signal of time length T, the support vector machine is used to identify the presence or absence of three consecutive rapid eye blinks. When three consecutive rapid eye blinks are detected, the take-off / landing control is performed; otherwise, the flight is normal. The EEG signal analysis module 1 uses the EEG signals of 8 channels in the frontal lobe area. The EEG signal is converted into a vector Then, principal component analysis is performed to extract features from EEG data. The extracted features can be expressed as

[0026] Θ=H·X

[0027] represents the principal component analysis conversion matrix, Q = 50 represents the feature dimension after compression, which contains 95% of the original feature information. Then the support vector machine is applied for classification. The support vector machine model can be expressed as

[0028]

[0029] n represents the number of support vectors, w i represents the weight of the i-th support vector, Θ i represents the i-th support vector of the classifier, δ represents the hyperparameter, and ε represents the bias of the classifier. Output of EEG signal analysis module 1

[0030]

[0031] Among them, S1=0 means that take-off / landing control is not performed, and S1=1 means that take-off / landing control is performed.

[0032] The purpose of the EEG signal preprocessing module 2 is to preprocess the EEG signal before the EEG signal analysis module 2. The EEG signal of 15 channels in the visual area is used, and the main work includes filtering. The EEG signal corresponding to the time length T can be expressed as N = T * fs represents the sampling points in the time dimension, and L = 15 represents the number of channels. A bandpass filter is used for filtering, and the high-pass cutoff frequency is f L =min{f1,f2,...,f k ,...,f K-1 ,f K}-2Hz, K=13, low-pass cutoff frequency is f H =max{f1,f2,...,f k ,...,f K-1 ,f K}+2Hz, K=13.

[0033] The purpose of EEG signal analysis module 2 is to identify the control intention of the controller to the UAV. After collecting EEG signals of time length T, the canonical correlation analysis algorithm is used to match 13 EEG patterns. The control instructions corresponding to the matched EEG patterns are the control intentions of the controller to the entire cluster. EEG signal analysis module 2 uses EEG signals of 15 channels in the visual area. Figure 4 It is the algorithm flow chart of EEG signal analysis module 2. The EEG signal corresponding to the time length T can be expressed as N = T * fs represents the sampling points in the time dimension, and L = 15 represents the channel. The EEG signal analysis module 2 has a total of 13 templates, which can be expressed as

[0034]

[0035] Where k = 1, 2, .., 13, D=120,

[0036]

[0037] Calculate x and M k The typical correlation coefficient ρ k =canoncorr(x,M k ), the output of EEG signal analysis module 2

[0038]

[0039] Among them, K=13, η is a pre-set threshold.

[0040] The control instructions of the system can be expressed as

[0041]

[0042] Among them, S1 represents the output of EEG analysis module 1, S2 represents the output of EEG analysis module 2, and z represents the flight altitude of the drone.

[0043] The lower computer includes the UAV position acquisition module, the control personnel intention acquisition module, the UAV control module, and the UAV position, speed, and attitude control algorithm. It ultimately outputs the speed, angle and other instructions of the UAV, and converts them into execution instructions for the UAV's underlying mechanism and transmits the instructions to the UAV to realize the execution of human intentions.

[0044] The positioning system consists of an optical positioning smart camera, a camera gimbal and supporting software, which is used to record the position of the drone in real time.

[0045] Working process: First, the drone stops on the ground. At this time, z=0. The drone controller performs three consecutive rapid eye blinks. The output of the EEG analysis module 1 is S1=1. At this time, the system control command is C=15, and the drone takes off. After that, the drone controller controls the drone according to the real-time status of the drone. According to his decision-making instructions, the controller stares at the corresponding LED for a duration of T, which will induce the corresponding EEG signal; then the EEG analysis module 2 analyzes the collected EEG signal to obtain the EEG pattern corresponding to the current time T, and the output of the EEG analysis module 2 is S2; at this time, the output of the EEG analysis module 1 is S1=0. During the flight, z>0, and the system control command is C=S2. When the controller does not need to output the control command, there is no need to stare at the flashing square of the stimulation module. At this time, it will correspond to the 14th EEG pattern and output the "hold" command. The drone will keep the command of the previous moment and execute the command of the previous moment; until the drone receives the command 11-13, otherwise the drone control command will remain unchanged. During the flight of the drone, z>0. When the controller needs to land the drone, he / she performs three consecutive rapid eye blinks, and the output of the EEG analysis module 1 is S1=1. At this time, the system control instruction is C=16, and the drone lands.

[0046] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A brain-controlled drone system based on an LED stereoscopic interactive interface, characterized in that: It includes a LED-based stereoscopic interaction module, an EEG signal acquisition module, an EEG signal preprocessing module 1, an EEG signal preprocessing module 2, an EEG signal analysis module 1, an EEG signal analysis module 2, a positioning module, and a lower computer; The EEG signal preprocessing module 1 and the EEG signal preprocessing module 2 preprocess the EEG signals according to the corresponding EEG patterns, respectively, so as to further analyze the EEG signals; The EEG signal analysis module 1 uses the EEG signals of 8 channels in the frontal lobe area; converts the EEG signals into a vector X=[x1, x2, ..., x NL ], and then perform principal component analysis to extract features from the EEG data; the extracted features can be expressed as Θ = Η·X represents the principal component analysis conversion matrix, Q = 50 represents the feature dimension after compression, which contains 95% of the original feature information; then the support vector machine is applied for classification, and the support vector machine model can be expressed as n represents the number of support vectors, w i represents the weight of the i-th support vector, Θ i represents the i-th support vector of the classifier, δ represents the hyperparameter, and ε represents the bias of the classifier; Output of EEG signal analysis module 1 Wherein, S1=0 means no take-off / landing control, and S1=1 means take-off / landing control; The EEG signal analysis module 2 uses 15 channels of EEG signals in the visual area; the EEG signal corresponding to the time length T can be expressed as N = T*fs represents the sampling points in the time dimension, L = 15 represents the channel; EEG signal analysis module 2 has a total of 13 templates, which can be expressed as in, Calculate x and M k The typical correlation coefficient ρ k =canoncorr(x,M k ), the output of EEG signal analysis module 2 Among them, K = 13, η is the threshold value set in advance; The control instructions of the system can be expressed as Among them, S1 represents the output of EEG analysis module 1, S2 represents the output of EEG analysis module 2, and z represents the flight altitude of the drone; The EEG signal analysis module 1 recognizes three consecutive rapid eye blinks of the control personnel by analyzing the EEG signals, thereby realizing command control of take-off / landing and normal flight status; The EEG signal analysis module 2 realizes the analysis of the control instructions of the drone.

2. According to claim 1, a brain-controlled drone system based on an LED stereoscopic interactive interface is characterized in that: The LED-based stereoscopic interactive module is composed of a three-dimensional orthogonal bracket and 13 LED lights fixed on the bracket, and is used to realize the display of visual stimulation.

3. According to claim 1, a brain-controlled drone system based on an LED stereoscopic interactive interface is characterized in that: The EEG signal acquisition module includes an EEG amplifier, brain electrodes, an EEG cap and EEG acquisition software. The EEG signal sampling frequency is fs=1000 Hz, which is used to realize the acquisition of EEG signals.

4. According to claim 1, a brain-controlled drone system based on an LED stereoscopic interactive interface is characterized in that: The positioning module includes an optical positioning smart camera and a camera gimbal, which are used to collect the position information of the drone in real time.

5. According to claim 1, a brain-controlled drone system based on an LED stereoscopic interactive interface is characterized in that: The lower computer receives the position information of the UAV and converts the UAV formation control instruction or the single-machine control instruction into the UAV underlying mechanism execution instruction and transmits the instruction to the UAV to realize the execution of the control personnel's intention.

Images