Method and system for controlling unmanned aerial vehicle based on brain imagination and eye movement tracking thought

Through the idea control method based on brain imagination and eye movement tracking, combined with eye movement signals and EEG signals, the problem of complexity of drone control and difficulty in eye-hand coordination is solved, and natural and intuitive drone control is achieved, especially Convenient operation for the elderly and people with movement disorders.

CN118942146AActive Publication Date: 2024-11-12NORTHWESTERN POLYTECHNICAL UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410991322.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

The existing drone control methods are complex and difficult to achieve eye-hand coordination, especially for ordinary users, the elderly and people with movement disorders. Traditional methods rely on physical joysticks or screen virtual joysticks, and cannot achieve natural and intuitive control.

Method used

Using the idea control method based on brain imagination and eye movement tracking, by training a personalized brain imagination decoding model, combining eye movement signals and EEG signals, it uses eye movement patterns that conform to human intuition and EEG signals that synchronize brain imagination to control none Human-machine flight and provide real-time first-view video streaming feedback to achieve an immersive operating experience.

Benefits of technology

Liberating your hands greatly facilitates the elderly, physically inconvenient, quadriplegic or with movement disorders, providing more convenient and flexible drone control methods, getting rid of the dependence on limb movement, and lowering the operating threshold.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942146A_ABST
    Figure CN118942146A_ABST
Patent Text Reader

Abstract

The invention discloses a method and a system for controlling an unmanned aerial vehicle based on brain imagination and eye movement tracking thought, provides an eye movement mode conforming to human intuition and electroencephalogram signals synchronizing brain imagination to control the flight of the unmanned aerial vehicle, and also provides a good operation and interaction interface. The first view angle video stream of the unmanned aerial vehicle is fed back in real time on an interface, and immersive unmanned aerial vehicle control is provided for an operator. Two hands can be liberated, and the unmanned aerial vehicle can be greatly conveniently controlled by the old, people with physical inconvenience, people with limb paralysis or people with other dyskinesia and other diseases.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a method and system for controlling unmanned aerial vehicles by mind based on brain imagination and eye movement tracking. Background Art

[0002] Drone control is gradually being widely used in various fields, such as aerial photography and transportation. However, current drone control still relies on traditional human-computer interaction, which is complex and requires a variety of manipulation methods. It requires a physical joystick or a screen-based virtual joystick, which makes it difficult to achieve eye-hand coordination and requires certain technical thresholds for users. This makes it difficult for ordinary users, especially the elderly and people with motor disabilities, to master it.

[0003] With the development of brain-computer interface technology, decoding the captured electrical signals of brain activity and controlling machines through sounds and actions imagined in the brain has gradually become a reality. Therefore, it is easy to think of using brain-computer interface technology to achieve simple and natural drone control.

[0004] EEG and eye movement are two signals commonly used in brain-computer interface technology. EEG signals are signals generated by the electrical activity of brain neurons, with extremely high temporal resolution, which can reflect the individual's cognitive state and intention; while eye movement signals are signals obtained by tracking the trajectory of eye movements, which directly reflect the individual's visual attention area. In the interaction with drones, combining EEG and eye movement signals can achieve a more intuitive and efficient control method.

[0005] Most of the current control methods that integrate EEG and eye movement are still based on blinking, visual stimulation or attention, that is, users need to blink, concentrate or focus their eyes on visual stimulation in a specified frequency range to achieve control. Although these methods free the user's hands, they are still a passive trigger control method. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a method and system for controlling drones with the mind based on brain imagination and eye tracking, and proposes to control the flight of drones using eye movement patterns that conform to human intuition and EEG signals that synchronize brain imagination. It also provides a good operation and interaction interface, which provides real-time feedback of the drone's first-person video stream, and provides the operator with immersive drone control. The present invention can free hands, greatly facilitating the control of drones by the elderly, physically disabled, quadriplegic, or people with other movement disorders.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] Step 1: Train a personalized brain imagination decoding model;

[0009] Step 1-1: Collect the natural state;

[0010] The fixation point a1s is presented, and then the user starts to maintain a natural relaxed state a2s. After the end, the user chooses to accept or reject this round of collection according to the user's state; the collection is repeated for n rounds, and when n≥5, the next state of collection is selected;

[0011] Step 1-2: Collect the forward state;

[0012] Present the fixation point a3s, and then the user starts to imagine pushing the block forward with both hands while silently saying "forward"; after the end, choose to accept or reject this round of collection according to the user's state; repeat the collection for n rounds or enter the next state of collection;

[0013] Step 1-3: Collect the back state;

[0014] Present the fixation point a4s, and then the user starts to imagine pulling the block backwards with both hands while silently saying "backward"; after the end, the user chooses to accept or reject this round of collection according to the user's state; repeat the collection for n rounds or enter the next state of collection;

[0015] Step 1-4: Collection is completed;

[0016] An individual independent imagination sample library is established based on the collected results. The data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile is used in each subsequent drone control.

[0017] Step 2: The computer terminal obtains the first-person video stream returned by the drone in real time, and presents an intuitive control interface based on the first-person video stream;

[0018] Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals; the eye tracker is connected to the computer terminal, the eye tracker is turned on, and the eye movement signals are collected; the EEG device is connected to the computer terminal and the EEG signals are collected;

[0019] Step 4: The computer terminal obtains the eye movement signal and the EEG signal and performs preprocessing respectively;

[0020] For eye movement signals, the eye movement signal processing module first subscribes to the eye movement signals collected by the eye tracker, selects the eye movement coordinates, and normalizes the eye movement coordinates to the (0,1) interval, where (0,0) represents the upper left corner of the screen and (1,1) represents the lower right corner of the screen;

[0021] Then, a sliding window mechanism is used to fill missing values ​​and smooth the eye movement signal within a specified time interval. Specifically, the center point coordinates (centerx ,center y ) fill missing values ​​(nan x ,nan y ); The sliding window mechanism uses a mean smoothing filter and uses the coordinate mean in the time window as an eye movement feature. The calculation formula is as follows:

[0022]

[0023] Where t represents the total number of eye gaze point coordinates in the time window, I x Indicates the horizontal coordinate of the eye movement fixation point, I y Indicates the vertical coordinate of the eye movement fixation point;

[0024] The eye movement signal processing module sends the pre-processed eye movement coordinates to the terminal control module;

[0025] For EEG signals, the EEG signal processing module first obtains the EEG signals collected by the EEG device; then pre-processes the EEG signals within a specified time interval;

[0026] Step 5: Calculate the eye movement features and EEG features into drone control commands;

[0027] In the terminal control module, the eye movement signal is first monitored, and the eye movement coordinates are determined once within the specified time interval to determine whether they fall within the control area. If they do, the corresponding control command is sent to the drone according to the eye movement coordinates. The corresponding relationship between the eye movement coordinates and the drone control command is as follows:

[0028] The eye movement coordinates are (0≤x≤0.1,≤0.1≤y≤0.9), and the drone control command is left;

[0029] The eye movement coordinates are (0.9≤x≤1.0,.01≤y≤0.9), and the drone control command is right;

[0030] The eye movement coordinates are (0.1≤x≤0.9, 0.9≤y≤1.0), and the drone control command is upward;

[0031] The eye movement coordinates are (0.1≤x≤0.9, 0.9≤y≤1.0), and the drone control command is downward;

[0032] The eye movement coordinates are other, and the drone control command is suspended;

[0033] When the eye movement coordinates are within other ranges, the terminal control module switches to the EEG signal monitoring mode; at this time, the terminal control module extracts the EEG features of the specified time step from the EEG signal processing module, loads the user configuration file, and uses the trained model in the configuration file to identify which speech motor imagery pattern best matches the user's current EEG features; selects the best matching pattern and converts it into the corresponding drone control command, as follows:

[0034] When the brain imagines the action of pushing forward and silently says "forward", the drone control command is to move forward;

[0035] When the brain imagines the action of pulling backwards and silently says "backward", the drone control command is to back away;

[0036] When the brain imagines something else, the drone control command is suspended;

[0037] Step 6: The terminal control module sends control commands to the drone via the wireless network to control the flight of the drone.

[0038] Preferably, a1=3, a2=8, a3=3, and a4=3.

[0039] Preferably, the preprocessing in step 4 includes locating channel positions, filtering, resampling, removing artifacts, independent component analysis, calculating frequency domain features and time domain features, and standardization.

[0040] A mind-controlled drone system based on brain imagination and eye tracking, including:

[0041] Module 1: Terminal display module, used to display the drone's first-person video stream and a friendly human-computer interaction prompt interface in real time;

[0042] Module 2: Eye movement signal acquisition module, used to collect eye movement signals in real time;

[0043] Module 3: EEG signal acquisition module, used to collect EEG signals in real time.

[0044] Module 4: Eye movement signal processing module, which receives the data stream transmitted by the eye movement signal acquisition module and pre-processes the eye movement signal;

[0045] Module 5: EEG signal processing module, used for the data stream transmitted by the real-time EEG signal acquisition module and pre-processing the EEG signal;

[0046] Module 6: Terminal control module, used to load user configuration files and convert eye movement signals and EEG signals into drone control commands;

[0047] Module 7: UAV communication module, used to transmit the UAV first-person video stream to the terminal display module and receive control commands from the terminal control module to realize UAV control;

[0048] Module 8: Personalized brain imagination decoding model training module; used to generate user profiles, collect user brain imagination data before officially controlling the drone flight, and use the pattern recognition model for training to obtain the user's personalized brain imagination pre-trained pattern recognition model.

[0049] The beneficial effects of the present invention are as follows:

[0050] The present invention utilizes intuitive eye movement characteristics and EEG characteristics that reflect brain imagination to realize mind control of drones, which is not only conducive to the operation of ordinary users, but also can provide a more convenient and flexible drone control method for special groups such as the elderly and people with movement disorders, getting rid of dependence on limb movements, so that more people can enjoy the convenience brought by drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 This is the thought control flow chart of the present invention;

[0053] Figure 3 This is a schematic diagram of the human-computer interaction prompt interface of the present invention;

[0054] Figure 4 It is a schematic diagram of the interaction of system modules of the present invention;

[0055] Figure 5 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] The present invention provides a method and system for controlling drones with the mind based on brain imagination and eye tracking, and proposes to control the flight of drones using eye movement patterns that are consistent with human intuition and EEG signals that are synchronized with brain imagination. It also provides a good operation and interaction interface, which provides real-time feedback of the drone's first-person video stream, and provides the operator with immersive drone control. The present invention can free hands, greatly facilitating the control of drones by the elderly, physically disabled, quadriplegic, or people with other movement disorders.

[0058] The present invention proposes a method for controlling a drone based on eye movement and brain speech movement imagination, comprising the following steps:

[0059] Step 1: Train a personalized brain image decoding model. First, pre-train the user's brain imagery. The present invention provides a novel and highly arousing data acquisition paradigm that combines verbal imagery and motor imagery. The specific process is as follows:

[0060] Step 1-1: Collect the natural state. Present the fixation point for 3 seconds, and then the user begins to maintain a natural and relaxed state for 8 seconds. After the end, the user can choose to accept or reject this round of collection according to the user's state. Repeat the collection for n rounds and enter the next state of collection.

[0061] Step 1-2: Collect the forward state. Present the fixation point for 3 seconds, and then the user begins to imagine pushing the block forward with both hands while silently saying "forward". After the end, the user can choose to accept or reject this round of collection based on the user's state. Repeat the collection for n rounds or enter the next state collection.

[0062] Step 1-3: Collect the back state. Present the fixation point for 3 seconds, and then the user starts to imagine pulling the block back with both hands while silently saying "back". After the end, the user can choose to accept or reject this round of collection according to the state. Repeat the collection for n rounds or enter the next state of collection.

[0063] Step 1-4: Collection is completed. Based on the collection results, an independent imagination sample library is established, and the data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile can be used in each subsequent drone control.

[0064] Step 2: The computer terminal obtains the first-person video stream returned by the drone in real time, and presents an intuitive control interface based on the video stream. The specific design of the terminal display module's operation interface is as follows: Figure 4 As shown in the figure, when the user's eyes look up, down, left, and right, it means that the user wants to broaden his or her field of vision. According to this intuition, the operation interface provides friendly human-computer interaction prompts. The drone's first-person video stream is fully covered on the terminal interface, which is conducive to the user's immersive experience of remote control of the drone. There is a direction control area that occupies one-tenth of the screen width in the top, bottom, left, and right sides of the interface, and there are arrows in the center of the direction control area to indicate the direction. When the user's line of sight falls within the range of the direction control area, the line of sight coordinates will be visualized on the screen in the form of a glowing red hollow circle, and the direction control area will be hidden.

[0065] Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals. Connect the eye tracker to the computer terminal, turn on the eye tracker, and start collecting eye movement signals; connect the EEG device to the computer terminal and start collecting EEG signals.

[0066] Step 4: The computer terminal obtains the eye movement signal and EEG signal and pre-processes them respectively. For the eye movement signal, the eye movement signal processing module first subscribes to the eye movement signal collected by the eye tracker, selects the eye movement coordinates, and normalizes the eye movement coordinates to the (0,1) interval, where (0,0) represents the upper left corner of the screen and (1,1) represents the lower right corner of the screen. Then, a sliding window mechanism is used to fill in missing values ​​and smooth the eye movement signal within a specified time interval. Specifically, the center point coordinates (center x ,center y ) fill missing values ​​(nan x ,nan y ), the reason comes from the physiological explanation that when people look at natural stimuli, their intuition is to focus on the center of the picture. The smoothing mechanism uses a mean smoothing filter, using the coordinate mean in the time window as an eye movement feature, and the calculation formula is as follows:

[0067]

[0068] The eye movement signal processing module sends the pre-processed eye movement coordinates to the terminal control module.

[0069] For EEG signals, the EEG signal processing module first obtains the EEG signals collected by the EEG device. Then, the EEG signals are preprocessed within a specified time interval. The preprocessing process includes locating channel positions, filtering, resampling, artifact removal, independent component analysis, calculating frequency domain features and time domain features, and standardization.

[0070] Step 5: Calculate the eye movement features and EEG features into drone control commands. In the terminal control module, the eye movement signals are first monitored to determine whether the eye movement coordinates fall within the control area within a specified time interval. If they fall within the control area, the corresponding control command is sent to the drone according to the eye movement coordinates. The corresponding relationship between the eye movement coordinates and the drone control commands is shown in Table 1 below:

[0071] Table 1 Conversion between eye movement coordinates and drone control commands

[0072] Eye movement coordinates Drone control commands (0≤x≤0.1, 0.1≤y≤0.9) left (0.9≤x≤1.0,0.1≤y≤0.9) To the right (0.1≤x≤0.9, 0.9≤y≤1.0) up (0.1≤x≤0.9,0.0≤y≤0.1) down other Dangling

[0073] When the eye movement coordinates are within other ranges, the terminal control module switches to the EEG signal monitoring mode. At this time, the terminal control module extracts the EEG features of the specified time step from the EEG signal processing module, loads the user configuration file, and uses the trained model in the configuration file to identify which speech motor imagery mode the user's current EEG features best match. Select the best matching mode and convert it into the corresponding drone control command, as shown in Table 2:

[0074] Table 2 Brain imagination and drone control command conversion

[0075]

[0076]

[0077] Step 6: The terminal control module sends control commands to the drone via the wireless network to control the flight of the drone.

[0078] The present invention proposes a consciousness-controlled drone system based on motor imagery EEG and eye movement signals, which includes the following submodules:

[0079] Module 1: Terminal display module. Used to display the drone’s first-person video stream in real time and a friendly human-computer interaction prompt interface.

[0080] Module 2: Eye movement signal acquisition module. Used to collect eye movement signals in real time.

[0081] Module 3: EEG signal acquisition module. Used to collect EEG signals in real time.

[0082] Module 4: Eye movement signal processing module. Receives the data stream transmitted by the eye movement signal acquisition module and pre-processes the eye movement signal.

[0083] Module 5: EEG signal processing module. It is used for the data stream transmitted by the real-time EEG signal acquisition module and pre-processes the EEG signal.

[0084] Module 6: Terminal control module. Used to load user configuration files and convert eye movement signals and EEG signals into drone control commands.

[0085] Module 7: UAV communication module. It is used to transmit the UAV first-person video stream to the terminal display module and receive control commands from the terminal control module to realize UAV control.

[0086] Module 8: Personalized brain imagination decoding model training module. It is used to generate user profiles. Before officially controlling the flight of the drone, it collects the user's brain imagination data and uses the pattern recognition model for training to obtain the user's personalized brain imagination pre-training pattern recognition model.

[0087] Example:

[0088] See also Figure 1 The method of the present invention specifically comprises the following steps:

[0089] Step 1: Train a personalized brain image decoding model. Specifically, for user A, perform speech motor imagery pre-training. The present invention provides a novel and highly arousing data acquisition paradigm, and the specific process is as follows:

[0090] Step 1-1: Collect the natural state. Present the fixation point for 3 seconds, and then the user begins to maintain a natural and relaxed state for 8 seconds. After the end, you can choose to accept or reject this round of collection according to the user's state. Repeat the collection for n rounds or enter the next state collection. There are many methods for judging the user's state. This embodiment judges the user's state by actively asking, and collects 10 rounds of samples for the natural state.

[0091] Step 1-2: Collect the forward state. Present the fixation point for 3 seconds, and then the user begins to imagine pushing the block forward with both hands while silently saying "forward". After the end, the user can choose to accept or reject this round of collection according to the user's state. Repeat the collection for n rounds or proceed to the next state collection. There are many ways to determine the user's state. This embodiment determines the user's state by actively asking, and collects 10 rounds of samples for the forward state.

[0092] Step 1-3: Collect the back state. Present the fixation point for 3 seconds, and then the user starts to imagine pulling the block backward with both hands while silently saying "back". After the end, the user can choose to accept or reject this round of collection according to the user's state. Repeat the collection n or enter the next state collection. There are many ways to judge the user's state. This embodiment judges the user's state by actively asking, and collects 10 samples of the back state.

[0093] Step 1-4: End of collection. According to the collection results, an individual independent imagination sample library is established, and the data in the sample library is trained using the pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile can be used in each drone control by user A. The pattern recognition models include: support vector machine, decision tree, random forest, multi-layer perceptron, convolutional neural network and Transformer. Taking the support vector machine as an example, the EEG feature pattern recognition algorithm is based on the following process to implement EEG signal feature recognition: Let the separable data set D = {(x i ,y i )|i=1,2,...n}, where the input vector x i ∈R d , R d is the d-dimensional real plane, y i ∈{-1, +1}, -1 represents a negative example, +1 represents a positive example. The separable data set D is the EEG feature, which is the data collected and preprocessed by the EEG device. The inner product and function uses a nonlinear kernel function (such as radial basis function RBF, polynomial kernel, etc.) to map the data to a high-dimensional space. This embodiment uses the radial basis function RBF, and its calculation formula is:

[0094] K(x i , x j ) = exp(-γ||x i -x j ||2 )

[0095] Where γ controls the width of the kernel function. The optimization problem of nonlinear SVM can be converted into a dual problem using the Lagrange multiplier method to solve it. After solving it, the classification decision function is obtained:

[0096]

[0097] Where b is the bias term, which can be determined by the support vector, α i is the Lagrange multiplier, and k(·) is the kernel function. For the three categories (natural state, forward state, and backward state), three independent two-classification support vector machine models are established, and each model distinguishes one class from the other two classes.

[0098] Step 2: The computer terminal obtains the first-person video stream returned by the drone in real time, and presents an intuitive control interface based on the video stream. Specifically, the terminal communicates with the drone via a wireless network, and the drone camera is turned on and the real-time video stream is started on the terminal. The terminal reads the video stream from the drone frame by frame and displays it on the operation interface in an adaptive screen width and height. At the same time, the operation interface presents friendly human-computer interaction prompts in real time at the frame rate of the video stream, such as Figure 2 As shown. Specifically, the operation interface consists of four parts: the drone's first-person video stream, the control area covering the video stream, and the indicator arrows and visualized eye coordinates located in the center of each control area. The operation interface will interact with users in an intuitive and friendly way. For example, when the user wants to widen the drone's field of view to the left, he only needs to move his eyes to the left naturally, so that his attention moves to the control area on the left side of the operation interface. At this time, the control area will be hidden and present the user's eye coordinates in a red glowing circle to prompt that the drone is now moving to the left. Similarly, when the user wants to widen the field of view to the right, up or down, he only needs to move his eyes naturally to control the drone. When the user wants to enjoy the video shot by the drone, he only needs to browse the video area at will. At this time, the control area plays a role in preventing accidental touches.

[0099] Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals. Specifically, while the user is watching the video stream, the eye tracker collects the user's eye movement signals, and the EEG device collects the user's EEG signals. The eye movement signals include the coordinates of the left and right eye gaze points on the screen, the unit vectors of the left and right eye gaze directions, and the pupil diameters of the left and right eyes. Depending on the EEG device, EEG signals of different channels can be collected. This embodiment uses an EEG device with 14 electrodes, which can collect EEG signals of 14 channels. The 14 electrodes are distributed in the frontal lobe, temporal lobe, and parietal lobe of the brain.

[0100] Step 4: The computer terminal obtains the eye movement signal and the EEG signal and preprocesses them respectively. Specifically, when the drone video stream starts, the eye movement signal processing module starts to subscribe to the eye movement signal, selects the eye movement coordinates therein, and normalizes the eye movement coordinates to the (0,1) interval. The eye movement coordinates are composed of a (x,y) tuple, where x represents width, y represents length, the coordinate (0,0) is located in the upper left corner of the screen, and the coordinate (1,1) is located in the lower right corner of the screen. For eye movement coordinates with missing values, the center coordinates (0.5,0.5) are used instead. In order to reduce the noise of the eye movement coordinates and smooth the eye movement coordinates, a mean smoothing filter is used to replace the original coordinates with the coordinate average within the specified sliding window step size. In this embodiment, the sliding window step size is specified to be 0.3 seconds. For EEG signals, the EEG signals are preprocessed within the sliding window step size. The preprocessing process includes locating the channel position, filtering, resampling, removing artifacts, independent component analysis, calculating frequency domain features and time domain features, and standardizing. Specifically, a bandpass filter is used to retain the signal in the range of 0.5-70 Hz, a notch filter is used to remove the power frequency interference in the range of 48-52 Hz, the signal is resampled to 128 Hz, the electrooculogram and electromyography artifacts are removed through independent component analysis, the power spectral density characteristics of the signal are calculated using fast Fourier transform, and finally the extracted features are standardized.

[0101] Step 5: Decode the eye movement features and EEG features into drone control commands. Specifically, Figure 2 As shown, in the terminal control module, a control command is prepared to be sent to the drone every specified time step. In this embodiment, the specified time step is 0.3 seconds. That is, every 0.3 seconds, the terminal control module will first take out a set of coordinates from the eye movement signal processing module, and then determine whether the coordinates are located in the control area. If they are located in the control area, the corresponding control command is obtained according to Table 1. For example, when the coordinates taken out are (0.08, 0.45), it means that it is located in the left control area. At this time, the control command of the drone is set to left. If it is not located in the control area, the terminal control module switches to the EEG signal monitoring mode, that is, it takes a period of EEG features with a duration of 0.3s from the EEG signal processing module, and loads the user configuration file at the same time. All three classifiers are applied to the extracted EEG features to obtain three prediction scores, which are finally attributed to the category with the highest score, and the corresponding control command is obtained according to Table 2.

[0102] Step 6: The terminal control module sends a control command to the drone through the wireless network to control the drone's flight. Specifically, the terminal control module sends the control command obtained from step 5 to the drone through the wireless LAN to control the drone's flight. For example, when the control command obtained from step 5 is to the left, it means that the user wants to widen the drone's field of view to the left, so the terminal control module sends a command to fly left to the drone. When the control command obtained from step 5 is to hover, it means that the user does not want to do any control now, so the terminal control module sends a hover command to the drone, and the drone hovers at the current position.

[0103] Although in this embodiment, a command is set to be sent to the drone every 0.3 seconds, in a specific environment, it can be flexibly adjusted according to the user's status. The drone control method provided in this application provides 7 degrees of freedom drone control, namely "up", "down", "left", "right", "forward", "backward" and "hover". These 7 commands can meet the basic scene drone flight. At the same time, this method uses eye movement signals and EEG signals to solve the user's intentions and control the drone according to the user's intentions. The method of this application uses EEG and eye movement signals to realize direct mind control of drones, which is not only conducive to the operation of ordinary users, but also can provide a more convenient and flexible drone control method for special groups such as the elderly, people with motor disabilities, etc., get rid of the dependence on limb movement, so that more people can enjoy the convenience brought by drones.

[0104] The present application also provides a drone control system based on EEG and eye movement signals, such as Figure 4 , Figure 5 As shown:

[0105] The user wears the EEG device and sits in front of the computer terminal, with the eye tracker placed below the computer screen. The EEG device is connected to the computer terminal via Bluetooth, the eye tracker is connected to the computer terminal via a wired connection, and the drone is connected to the computer terminal via a wireless LAN. The placement of the drone is determined by the coverage of the wireless LAN.

[0106] The drone includes a wireless communication module for transmitting the first-person video stream to a computer terminal and for receiving control commands sent by a control module of the computer terminal.

[0107] The eye tracker includes an eye movement signal acquisition module, which is used to collect the user's eye movement signals and transmit the eye movement signals to the eye movement signal processing module of the computer terminal.

[0108] The EEG device includes an EEG signal acquisition module, which collects the user's EEG signals and transmits the EEG signals to the EEG signal processing module of the computer terminal.

[0109] The computer terminal includes an eye movement signal processing module, which is used to subscribe to the eye movement signals transmitted by the eye tracker, to filter the signals, to process missing values, and to smooth the signals, and to store pre-processed eye movement coordinates.

[0110] The computer terminal also includes an EEG signal processing module for receiving EEG signals transmitted by the EEG device, performing pre-processing operations such as filtering on the EEG signals, and classifying the EEG signals.

[0111] The computer terminal also includes a terminal control module for sending control commands to the drone and for solving eye movement coordinates and EEG signal classification results into drone control commands.

[0112] The computer terminal also includes a terminal display module for receiving and displaying the first-person perspective video stream of the drone and for displaying a friendly human-computer interaction prompt interface.

[0113] The computer terminal also includes a personalized brain imagination decoding model training module, which is used to generate a user profile, collect the user's speech and motor imagination data before officially controlling the drone flight, and use the pattern recognition model for training to obtain the user's independent pre-trained EEG feature recognition model.

[0114] Figure 5 A specific application scenario of the system of the present invention is shown. In the figure, a user sitting in a wheelchair suffers from motor impairment and cannot control the keyboard, mouse, joystick and other devices with his hands or feet. The system can help such users to achieve convenient and simple drone control, helping users enjoy the fun of drone control while appreciating the scenery captured by the drone.

Claims

1. A method for controlling a drone with the mind based on brain imagination and eye tracking, characterized in that: The steps include: Step 1: Train a personalized brain imagination decoding model; Step 1-1: Collect the natural state; The fixation point a1s is presented, and then the user starts to maintain a natural relaxed state a2s. After the end, the user chooses to accept or reject this round of collection according to the user's state; the collection is repeated for n rounds, and when n≥5, the next state of collection is selected; Step 1-2: Collect the forward state; Present the fixation point a3s, and then the user starts to imagine pushing the block forward with both hands while silently saying "forward"; after the end, choose to accept or reject this round of collection according to the user's state; repeat the collection for n rounds or enter the next state of collection; Step 1-3: Collect the back state; Present the fixation point a4s, and then the user starts to imagine pulling the block backwards with both hands while silently saying "backward"; after the end, the user chooses to accept or reject this round of collection according to the user's state; repeat the collection for n rounds or enter the next state of collection; Step 1-4: Collection is completed; An individual independent imagination sample library is established based on the collected results. The data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile is used in each subsequent drone control. Step 2: The computer terminal obtains the first-person video stream returned by the drone in real time, and presents an intuitive control interface based on the first-person video stream; Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals; The eye tracker is connected to the computer terminal, turned on, and the eye movement signal is collected; the EEG device is connected to the computer terminal to collect EEG signals; Step 4: The computer terminal obtains the eye movement signal and the EEG signal and performs preprocessing respectively; For eye movement signals, the eye movement signal processing module first subscribes to the eye movement signals collected by the eye tracker, selects the eye movement coordinates, and normalizes the eye movement coordinates to the (0,1) interval, where (0,0) represents the upper left corner of the screen and (1,1) represents the lower right corner of the screen; Then, a sliding window mechanism is used to fill missing values ​​and smooth the eye movement signal within a specified time interval. Specifically, the center point coordinates (center x , center y ) fill missing values ​​(nan x ,nan y ); The sliding window mechanism uses a mean smoothing filter and uses the coordinate mean in the time window as an eye movement feature. The calculation formula is as follows: Where t represents the total number of eye gaze point coordinates in the time window, I x Indicates the horizontal coordinate of the eye movement fixation point, I y Indicates the vertical coordinate of the eye movement fixation point; The eye movement signal processing module sends the pre-processed eye movement coordinates to the terminal control module; For EEG signals, the EEG signal processing module first obtains the EEG signals collected by the EEG device; then pre-processes the EEG signals within a specified time interval; Step 5: Calculate the eye movement features and EEG features into drone control commands; In the terminal control module, the eye movement signal is first monitored, and the eye movement coordinates are determined once within the specified time interval to determine whether they fall within the control area. If they do, the corresponding control command is sent to the drone according to the eye movement coordinates. The corresponding relationship between the eye movement coordinates and the drone control command is as follows: The eye movement coordinates are (0≤x≤0.1, 0.1≤y≤0.9), and the drone control command is left; The eye movement coordinates are (0.9≤x≤1.0, 0.1≤g≤0.9), and the drone control command is right; The eye movement coordinates are (0.1≤x≤0.9, 0.9≤y≤1.0), and the drone control command is upward; The eye movement coordinates are (0.1≤x≤0.9, 0.0≤y≤0.1), and the drone control command is downward; The eye movement coordinates are other, and the drone control command is suspended; When the eye movement coordinates are within other ranges, the terminal control module switches to the EEG signal monitoring mode; at this time, the terminal control module extracts the EEG features of the specified time step from the EEG signal processing module, loads the user configuration file, and uses the trained model in the configuration file to identify which speech motor imagery pattern best matches the user's current EEG features; selects the best matching pattern and converts it into the corresponding drone control command, as follows: When the brain imagines the action of pushing forward and silently says "forward", the drone control command is to move forward; When the brain imagines the action of pulling back and silently says "back", the drone control command is back; When the brain imagines something else, the drone control command is suspended; Step 6: The terminal control module sends control commands to the drone via the wireless network to control the flight of the drone.

2. The method for controlling a drone by mind based on brain imagination and eye tracking according to claim 1, characterized in that: Said a1=3, a2=8, a3=3, a4=3.

3. The method for controlling a drone by mind based on brain imagination and eye tracking according to claim 2, characterized in that: The preprocessing in step 4 includes locating channel positions, filtering, resampling, removing artifacts, independent component analysis, calculating frequency domain features and time domain features, and standardization.

4. A drone system using the method as claimed in claim 1, characterized in that: include: Module 1: Terminal display module, used to display the drone's first-person video stream in real time and a friendly human-computer interaction prompt interface; Module 2: Eye movement signal acquisition module, used to collect eye movement signals in real time; Module 3: EEG signal acquisition module, used to collect EEG signals in real time; Module 4: Eye movement signal processing module, which receives the data stream transmitted by the eye movement signal acquisition module and pre-processes the eye movement signal; Module 5: EEG signal processing module, used for the data stream transmitted by the real-time EEG signal acquisition module, and pre-processing the EEG signal; Module 6: Terminal control module, used to load user configuration files and convert eye movement signals and EEG signals into drone control commands; Module 7: UAV communication module, used to transmit the UAV first-person video stream to the terminal display module and receive control commands from the terminal control module to realize UAV control; Module 8: Personalized brain imagination decoding model training module; It is used to generate a user profile. Before officially controlling the flight of the drone, it collects the user's brain imagination data and uses the pattern recognition model for training to obtain a user-personalized brain imagination pre-trained pattern recognition model.

Citation Information

Patent Citations

  • Method and system for unmanned aerial vehicles to execute tasks

    CN106656310A

  • Unmanned aerial vehicle reconnaissance system based on hybrid control technology

    CN107357311A

  • Unmanned aerial vehicle cluster formation interactive simulation verification system and implementation method thereof

    CN108845802A

  • Task scene associated unmanned aerial vehicle pilot visual attention distribution mode extraction method

    CN111951637A

  • Electroencephalogram-eye movement hybrid teleoperation robot control method, system and device

    CN117873330A