Mind-controlled drone method and system based on brain imagination and eye tracking

Abstract

This invention discloses a method and system for mind-controlled drones based on brain imagery and eye tracking. This system utilizes intuitive eye movement patterns and electroencephalogram (EEG) signals synchronized with brain imagery to control drone flight. It also provides a user-friendly interface that displays the drone's first-person perspective video stream in real time, providing the operator with immersive drone control. This system frees up hands, greatly facilitating drone control for the elderly, those with disabilities, quadriplegia, or other motor impairments.

Description

Technical Field

[0001] The present invention belongs to the technical field of drones, and in particular relates to a method and system for mind-controlled drones based on brain imagination and eye tracking. Background Art

[0002] Drone control is gradually gaining widespread application in various fields, such as aerial photography and transportation. However, current drone control still relies on traditional human-computer interaction, resulting in complex operation and numerous control methods. These methods require either a physical joystick or a screen-based virtual joystick, making eye-hand coordination difficult and requiring a high level of user skill. This makes it difficult for ordinary users, especially the elderly and those with motor disabilities, to master drones.

[0003] With the development of brain-computer interface technology, decoding captured brain activity signals and controlling machines through imagined sounds and movements has gradually become a reality. Therefore, it is easy to imagine using brain-computer interface technology to achieve simple and natural drone control.

[0004] EEG and eye movement signals are two commonly used signals in brain-computer interface technology. EEG signals, generated by the electrical activity of brain neurons, have extremely high temporal resolution and can reflect an individual's cognitive state and intentions. Eye movement signals, on the other hand, are obtained by tracking eye movements and directly reflect an individual's visual attention area. Combining EEG and eye movement signals in drone interactions can achieve more intuitive and efficient control.

[0005] Most current control methods that integrate EEG and eye movement are still based on blinking, visual stimulation, or attention. This means that users need to blink, focus their attention, or focus their gaze on visual stimulation within a specified frequency range to achieve control. While these methods free the user's hands, they are still passive trigger control methods. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a method and system for mind-controlled drones based on brain imagery and eye tracking. This system utilizes intuitive eye movement patterns and EEG signals synchronized with brain imagery to control drone flight. It also provides a user-friendly interface that provides real-time first-person video feeds of the drone, providing immersive drone control. This system frees up hands, greatly facilitating drone control for the elderly, those with disabilities, quadriplegia, or other motor impairments.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] Step 1: Train a personalized brain image decoding model;

[0009] Step 1-1: Collect natural state;

[0010] The gaze point a1s is presented, and the user then begins to maintain a natural and relaxed state a2s. After the end, the user chooses to accept or reject this round of acquisition based on the user's state. Repeat the acquisition for n rounds, and when n ≥ 5, select the next state of acquisition.

[0011] Step 1-2: Collect forward status;

[0012] The user is presented with the fixation point a3s, and then begins to imagine pushing the block forward with both hands while silently saying "forward." After this, the user chooses to accept or reject this round of acquisition based on the user's state. The acquisition is repeated for n rounds or proceeds to the next state of acquisition.

[0013] Step 1-3: Collect the back state;

[0014] Present the fixation point a4s, and then the user begins to imagine pulling the block backward with both hands while silently saying "back." After the end, the user chooses to accept or reject the current round of acquisition based on the user's state. Repeat the acquisition for n rounds or proceed to the next state of acquisition.

[0015] Step 1-4: Collection is completed;

[0016] Based on the collected results, an independent imagination sample library is established. The data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile is used in each subsequent drone control.

[0017] Step 2: The computer terminal obtains the real-time first-person video stream returned by the drone and presents an intuitive control interface based on the first-person video stream;

[0018] Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals; the eye tracker is connected to the computer terminal, turned on, and collects eye movement signals; the EEG device is connected to the computer terminal and collects EEG signals;

[0019] Step 4: The computer terminal obtains eye movement signals and EEG signals and pre-processes them respectively;

[0020] For eye movement signals, the eye movement signal processing module first subscribes to the eye movement signals collected by the eye tracker, selects the eye movement coordinates, and normalizes the eye movement coordinates to the (0,1) interval, where (0,0) represents the upper left corner of the screen and (1,1) represents the lower right corner of the screen;

[0021] Then, a sliding window mechanism is used to fill missing values ​​and smooth the eye movement signal within a specified time interval. Specifically, the center point coordinates (centerx ,center y )Fill missing values ​​(nan x ,nan y ); The sliding window mechanism uses a mean smoothing filter and uses the coordinate mean within the time window as an eye movement feature. The calculation formula is as follows:

[0022]

[0023] Where t represents the total number of eye gaze point coordinates within the time window, I x Indicates the horizontal coordinate of the eye movement gaze point, I y Indicates the vertical coordinate of the eye movement fixation point;

[0024] The eye movement signal processing module sends the pre-processed eye movement coordinates to the terminal control module;

[0025] For EEG signals, the EEG signal processing module first obtains the EEG signals collected by the EEG device; then pre-processes the EEG signals within a specified time interval;

[0026] Step 5: Decode eye movement features and EEG features into drone control commands;

[0027] In the terminal control module, the eye movement signal is first monitored. Within a specified time interval, it is determined whether the eye movement coordinates fall within the control area. If they do, the corresponding control command is sent to the drone according to the eye movement coordinates. The correspondence between the eye movement coordinates and the drone control command is as follows:

[0028] The eye movement coordinates are (0≤x≤0.1,≤0.1≤y≤0.9), and the drone control command is to move left;

[0029] The eye movement coordinates are (0.9≤x≤1.0, .01≤y≤0.9), and the drone control command is right;

[0030] The eye movement coordinates are (0.1≤x≤0.9, 0.9≤y≤1.0), and the drone control command is upward;

[0031] The eye movement coordinates are (0.1≤x≤0.9, 0.9≤y≤1.0), and the drone control command is downward;

[0032] The eye movement coordinates are other and the drone control command is suspended;

[0033] When the eye movement coordinates are within other ranges, the terminal control module switches to the EEG signal monitoring mode. At this time, the terminal control module extracts the EEG features of the specified time step from the EEG signal processing module, loads the user configuration file, and uses the trained model in the configuration file to identify which speech motor imagery pattern best matches the user's current EEG features. The module selects the best matching pattern and converts it into the corresponding drone control command, as follows:

[0034] When the brain imagines the action of pushing forward and silently says "forward", the drone control command is forward;

[0035] When the brain imagines the action of pulling back and silently says "back", the drone control command is back;

[0036] When the brain imagines something else, the drone control command is suspended;

[0037] Step 6: The terminal control module sends control commands to the drone via the wireless network to control the drone's flight.

[0038] Preferably, a1=3, a2=8, a3=3, and a4=3.

[0039] Preferably, the preprocessing in step 4 includes locating channel positions, filtering, resampling, artifact removal, independent component analysis, calculating frequency domain features and time domain features, and standardization.

[0040] A mind-controlled drone system based on brain imagination and eye tracking, including:

[0041] Module 1: Terminal display module, used to display the drone's first-person video stream in real time and a friendly human-computer interaction prompt interface;

[0042] Module 2: Eye movement signal acquisition module, used to collect eye movement signals in real time;

[0043] Module 3: EEG signal acquisition module, used to collect EEG signals in real time.

[0044] Module 4: Eye movement signal processing module, which receives the data stream transmitted by the eye movement signal acquisition module and pre-processes the eye movement signal;

[0045] Module 5: EEG signal processing module, which is used to process the data stream transmitted by the real-time EEG signal acquisition module and pre-process the EEG signal;

[0046] Module 6: Terminal control module, used to load user configuration files and convert eye movement signals and EEG signals into drone control commands;

[0047] Module 7: UAV communication module, used to transmit the UAV's first-person perspective video stream to the terminal display module and receive control commands from the terminal control module to realize UAV control;

[0048] Module 8: Personalized brain imagination decoding model training module; used to generate user profiles. Before officially controlling the drone flight, it collects user brain imagination data and uses the pattern recognition model for training to obtain the user's personalized brain imagination pre-trained pattern recognition model.

[0049] The beneficial effects of the present invention are as follows:

[0050] The present invention uses intuitive eye movement characteristics and EEG characteristics that reflect brain imagination to realize mind-controlled drones. This is not only conducive to the operation of ordinary users, but also can provide a more convenient and flexible drone control method for special groups such as the middle-aged and elderly people and people with movement disorders, getting rid of dependence on limb movement, so that more people can enjoy the convenience brought by drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flow chart of the method of the present invention;

[0052] Figure 2 This is the thought control flow chart of the present invention;

[0053] Figure 3 This is a schematic diagram of the human-computer interaction prompt interface of the present invention;

[0054] Figure 4 This is a schematic diagram of the interaction between the system modules of the present invention;

[0055] Figure 5 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below with reference to the accompanying drawings and examples.

[0057] This invention provides a method and system for mind-controlled drones based on brain imagery and eye tracking. This system utilizes intuitive eye movement patterns and EEG signals synchronized with brain imagery to control drone flight. It also provides a user-friendly interface that displays the drone's first-person perspective video stream in real time, providing immersive drone control. This system frees up hands, greatly facilitating drone control for the elderly, those with disabilities, quadriplegia, or other movement disorders.

[0058] The present invention proposes a method for controlling a drone with mind based on eye movement and brain speech motor imagery, comprising the following steps:

[0059] Step 1: Train a personalized brain imagery decoding model. First, pre-train the user's brain imagery. This invention provides a novel and highly arousing data acquisition paradigm that integrates verbal imagery and motor imagery. The specific process is as follows:

[0060] Step 1-1: Collect in a natural state. Present the gaze point for 3 seconds, then allow the user to maintain a natural, relaxed state for 8 seconds. Afterward, the user can accept or reject the collection round based on their state. Repeat this collection round n times before proceeding to the next state.

[0061] Step 1-2: Collect the forward state. Present the fixation point for 3 seconds, then let the user imagine pushing the block forward with both hands while silently saying "forward." Afterward, the user can accept or reject the collection based on their state. Repeat for n rounds or proceed to the next state.

[0062] Steps 1-3: Collect the backward state. Present the fixation point for 3 seconds, then let the user imagine pulling the block backward with both hands while silently saying "backward." Afterward, the user can choose to accept or reject this round of collection based on their state. Repeat this for n rounds or proceed to the next state.

[0063] Steps 1-4: Collection is complete. Based on the collection results, an independent imaginary sample library is established. The data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile can be used in each subsequent drone control.

[0064] Step 2: The computer terminal obtains the first-person video stream returned by the drone in real time, and presents an intuitive control interface based on the video stream. The specific design of the terminal display module's operation interface is as follows: Figure 4 As shown, when the user's eyes move up, down, left, or right, it indicates that they want to expand their field of view. Based on this intuition, the user interface provides friendly human-computer interaction prompts. The drone's first-person video stream is overlaid on the terminal interface, facilitating an immersive experience of remotely controlling the drone. There are directional control areas in each of the top, bottom, left, and right corners of the interface, each occupying one-tenth of the screen width. Arrows indicate the direction in the center of each directional control area. When the user's gaze falls within the directional control area, the gaze coordinates are visualized on the screen as a glowing red hollow circle, and the directional control area is hidden.

[0065] Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals. Connect the eye tracker to the computer terminal, turn on the eye tracker, and start collecting eye movement signals; connect the EEG device to the computer terminal and start collecting EEG signals.

[0066] Step 4: The computer terminal obtains the eye movement signal and EEG signal and pre-processes them separately. For the eye movement signal, the eye movement signal processing module first subscribes to the eye movement signal collected by the eye tracker, selects the eye movement coordinates, and normalizes the eye movement coordinates to the (0,1) interval, where (0,0) represents the upper left corner of the screen and (1,1) represents the lower right corner of the screen. Then, a sliding window mechanism is used to fill in the missing values ​​of the eye movement signal within the specified time interval and smooth it. Specifically, the center point coordinates (center x ,center y )Fill missing values ​​(nan x ,nan y ) The reason for this is physiological: when people view natural stimuli, their instinct is to focus their attention on the center of the image. The smoothing mechanism uses a mean smoothing filter, using the mean of the coordinates within the time window as an eye movement feature. The calculation formula is as follows:

[0067]

[0068] The eye movement signal processing module sends the pre-processed eye movement coordinates to the terminal control module.

[0069] For EEG signals, the EEG signal processing module first obtains the EEG signals collected by the EEG device. It then preprocesses the EEG signals within a specified time interval. The preprocessing process includes channel location, filtering, resampling, artifact removal, independent component analysis, calculation of frequency and time domain features, and normalization.

[0070] Step 5: Calculate the eye movement features and EEG features into drone control commands. In the terminal control module, the eye movement signal is first monitored and the eye movement coordinates are determined to determine whether they fall within the control area within a specified time interval. If they do, the corresponding control command is sent to the drone according to the eye movement coordinates. The corresponding relationship between eye movement coordinates and drone control commands is shown in Table 1 below:

[0071] Table 1 Conversion between eye movement coordinates and drone control commands

[0072] Eye movement coordinates Drone control commands (0≤x≤0.1, 0.1≤y≤0.9) left (0.9≤x≤1.0,0.1≤y≤0.9) To the right (0.1≤x≤0.9, 0.9≤y≤1.0) up (0.1≤x≤0.9,0.0≤y≤0.1) down other Dangling

[0073] When the eye movement coordinates are within other ranges, the terminal control module switches to EEG signal monitoring mode. At this point, the terminal control module extracts EEG features of the specified time step from the EEG signal processing module, loads the user configuration file, and uses the trained model in the configuration file to identify which speech motor imagery pattern best matches the user's current EEG features. The best matching pattern is selected and converted into the corresponding drone control command, as shown in Table 2:

[0074] Table 2 Brain imagination and drone control command conversion

[0075]

[0076]

[0077] Step 6: The terminal control module sends control commands to the drone via the wireless network to control the drone's flight.

[0078] This paper proposes a consciousness-controlled drone system based on motor imagery EEG and eye movement signals, which includes the following submodules:

[0079] Module 1: Terminal display module. Used to display the drone's first-person video stream in real time and provide a user-friendly human-computer interaction prompt interface.

[0080] Module 2: Eye movement signal acquisition module. Used to collect eye movement signals in real time.

[0081] Module 3: EEG signal acquisition module. Used to collect EEG signals in real time.

[0082] Module 4: Eye Movement Signal Processing Module: Receives the data stream transmitted by the eye movement signal acquisition module and pre-processes the eye movement signal.

[0083] Module 5: EEG signal processing module. It is used to process the data stream transmitted by the real-time EEG signal acquisition module and pre-process the EEG signal.

[0084] Module 6: Terminal control module. Used to load user configuration files and convert eye movement signals and EEG signals into drone control commands.

[0085] Module 7: UAV communication module. It is used to transmit the UAV's first-person perspective video stream to the terminal display module and receive control commands from the terminal control module to realize UAV control.

[0086] Module 8: Personalized Brain Imagination Decoding Model Training Module. This module is used to generate a user profile. Before officially controlling the drone flight, it collects the user's brain image data and trains it using a pattern recognition model to obtain a personalized pre-trained pattern recognition model.

[0087] Example:

[0088] See also Figure 1 , the method of the present invention specifically comprises the following steps:

[0089] Step 1: Train a personalized brain imagery decoding model. Specifically, perform speech motor imagery pre-training for user A. This invention provides a novel and highly arousing data acquisition paradigm. The specific process is as follows:

[0090] Step 1-1: Capture the user's natural state. Present the gaze point for 3 seconds, then the user begins to maintain a natural, relaxed state for 8 seconds. After this, the user can choose to accept or reject the current round of capture based on their state. Repeat for n rounds or proceed to the next state. There are various methods for determining the user's state. In this embodiment, active questioning is used to determine the user's state, and 10 rounds of samples of the natural state are collected.

[0091] Step 1-2: Collect the forward state. Present the fixation point for 3 seconds, then let the user imagine pushing the block forward with both hands while silently saying "forward." Afterward, the user can accept or reject this round of collection based on their state. Repeat for n rounds or proceed to the next state. While various methods can be used to determine the user's state, this example uses active querying to determine the user's state and collects 10 rounds of samples for the forward state.

[0092] Step 1-3: Capture the backward state. Present the fixation point for 3 seconds, then let the user imagine pulling the block backward with both hands while silently saying "backward." Afterward, the user can choose to accept or reject this round of capture based on their state. Repeat the capture cycle or proceed to the next state. While various methods can be used to determine the user's state, this embodiment uses active querying to determine the user's state and collects 10 samples of the backward state.

[0093] Step 1-4: End of collection. Based on the collection results, an independent imagination sample library is established. The data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile can be used in each drone control by user A. The pattern recognition models include: support vector machine, decision tree, random forest, multi-layer perceptron, convolutional neural network and Transformer. Taking the support vector machine as an example, the EEG feature pattern recognition algorithm is based on the following process to realize the EEG signal feature recognition: Let the separable data set D = {(x i ,y i )|i=1,2,...n}, where the input vector x i ∈R d , R d is the d-dimensional real plane, y i ∈{-1, +1}, -1 represents a negative example, and +1 represents a positive example. The separable data set D is the EEG feature, which is the data collected and preprocessed by the EEG device. The inner product sum function uses a nonlinear kernel function (such as radial basis function RBF, polynomial kernel, etc.) to map the data to a high-dimensional space. This embodiment uses the radial basis function RBF, and its calculation formula is:

[0094] K(x i , x j )=exp(-γ||x i -x j ||2 )

[0095] Where γ controls the width of the kernel function. The optimization problem of nonlinear SVM can be converted into a dual problem using the Lagrange multiplier method to solve it. After solving it, the classification decision function is obtained:

[0096]

[0097] Where b is the bias term, which can be determined by the support vector, α i is the Lagrange multiplier, and k(·) is the kernel function. For the three categories (natural state, forward state, and backward state), three independent two-class support vector machine models are established, and each model distinguishes one class from the other two classes.

[0098] Step 2: The computer terminal obtains the first-person video stream returned by the drone in real time, and presents an intuitive control interface based on the video stream. Specifically, the terminal communicates with the drone via a wireless network. Operations on the terminal open the drone camera and start real-time video streaming. The terminal reads the video stream from the drone frame by frame and displays it on the operation interface in an adaptive screen width and height. At the same time, the operation interface presents friendly human-computer interaction prompts in real time at the frame rate of the video stream, such as Figure 2 As shown. Specifically, the operation interface consists of four parts: the drone's first-person video stream, the control area covering the video stream, and the indicator arrows and visual eye coordinates located in the center of each control area. The operation interface will interact with the user in an intuitive and friendly way. For example, when the user wants to widen the drone's field of view to the left, he only needs to move his eyes naturally to the left, so that his attention moves to the control area on the left side of the operation interface. At this time, the control area will be hidden and the user's eye coordinates will be presented in a red glowing circle to prompt that the drone is now moving to the left. Similarly, when the user wants to widen the field of view to the right, up or down, he only needs to move his eyes naturally to control the drone. When the user wants to enjoy the video shot by the drone, he only needs to browse the video area at will. At this time, the control area plays a role in preventing accidental touches.

[0099] Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals. Specifically, while the user is watching the video stream, the eye tracker collects the user's eye movement signals, and the EEG device collects the user's EEG signals. The eye movement signals include the coordinates of the left and right eye gaze points on the screen, the unit vectors of the left and right eye gaze directions, and the pupil diameters of the left and right eyes. Depending on the EEG device, EEG signals from different channels can be collected. This embodiment uses a 14-electrode EEG device that can collect EEG signals from 14 channels. The 14 electrodes are distributed across the frontal, temporal, and parietal lobes of the brain.

[0100] Step 4: The computer terminal obtains the eye movement signal and the EEG signal and pre-processes them separately. Specifically, when the drone video stream starts, the eye movement signal processing module starts to subscribe to the eye movement signal, selects the eye movement coordinates therein, and normalizes the eye movement coordinates to the (0,1) interval. The eye movement coordinates are composed of a (x,y) tuple, where x represents width and y represents length. The coordinate (0,0) is located in the upper left corner of the screen, and the coordinate (1,1) is located in the lower right corner of the screen. For eye movement coordinates with missing values, the center coordinates (0.5,0.5) are used instead. In order to reduce the noise of the eye movement coordinates and smooth the eye movement coordinates, a mean smoothing filter is used, and the original coordinates are replaced by the average value of the coordinates within the specified sliding window step. In this embodiment, the sliding window step is specified to be 0.3 seconds. For the EEG signal, the EEG signal is pre-processed within the sliding window step. The pre-processing process includes locating the channel position, filtering, resampling, removing artifacts, independent component analysis, calculating frequency domain features and time domain features, and standardization. Specifically, bandpass filtering is used to retain signals in the range of 0.5-70 Hz, notch filtering is used to remove power frequency interference in the range of 48-52 Hz, the signal is resampled to 128 Hz, independent component analysis is used to remove electrooculographic artifacts and electromyographic artifacts, and fast Fourier transform is used to calculate the power spectral density characteristics of the signal. Finally, the extracted features are standardized.

[0101] Step 5: Calculate the eye movement features and EEG features into drone control commands. Specifically, Figure 2 As shown, the terminal control module is ready to send a control command to the drone every specified time step. In this embodiment, the specified time step is 0.3 seconds. That is, every 0.3 seconds, the terminal control module will first extract a set of coordinates from the eye movement signal processing module, and then determine whether the coordinates are located in the control area. If they are located in the control area, the corresponding control command is obtained according to Table 1. For example, when the extracted coordinates are (0.08, 0.45), it means that it is located in the left control area. At this time, the control command of the drone is set to left. If it is not located in the control area, the terminal control module switches to the EEG signal monitoring mode, that is, extracts EEG features of a duration of 0.3s from the EEG signal processing module, and loads the user profile at the same time. All three classifiers are applied to the extracted EEG features to obtain three prediction scores, which are finally assigned to the category with the highest score, and the corresponding control command is obtained according to Table 2.

[0102] Step 6: The terminal control module sends a control command to the drone via the wireless network to control its flight. Specifically, the terminal control module transmits the control command obtained in Step 5 to the drone via the wireless local area network to control its flight. For example, if the control command calculated in Step 5 is "left," indicating that the user wishes to widen the drone's field of view to the left, the terminal control module sends a command to the drone to fly left. If the control command calculated in Step 5 is "hover," indicating that the user does not wish to perform any control operations at this time, the terminal control module sends a command to hover, causing the drone to hover at its current position.

[0103] Although it is set in this embodiment to send a command to the drone every 0.3 seconds, in specific environments, it can be flexibly adjusted according to the user status. The drone control method provided in this application provides 7 degrees of freedom drone control, namely "up", "down", "left", "right", "forward", "backward" and "hover". These 7 commands can meet the basic scenarios of drone flight. At the same time, this method uses eye movement signals and EEG signals to solve the user's intentions and control the drone according to the user's intentions. The method of this application uses EEG and eye movement signals to realize direct mind control of drones, which is not only conducive to the operation of ordinary users, but also can provide special groups such as middle-aged and elderly people, people with motor disabilities, etc. with a more convenient and flexible drone control method, get rid of the dependence on limb movement, so that more people can enjoy the convenience brought by drones.

[0104] The present application also provides a drone control system based on EEG and eye movement signals, such as Figure 4 、 Figure 5 As shown:

[0105] The user wears an EEG headset and sits in front of a computer terminal, with the eye tracker placed below the computer screen. The EEG headset and computer terminal are connected via Bluetooth, the eye tracker and computer terminal are connected via a wired connection, and the drone and computer terminal are connected via a wireless LAN. The drone's placement is determined by the coverage area of ​​the wireless LAN.

[0106] The drone includes a wireless communication module for transmitting the first-person video stream to the computer terminal and receiving control commands sent by the computer terminal control module.

[0107] The eye tracker includes an eye movement signal acquisition module, which is used to collect the user's eye movement signals and transmit the eye movement signals to the eye movement signal processing module of the computer terminal.

[0108] The EEG device includes an EEG signal acquisition module, which collects the user's EEG signals and transmits the EEG signals to the EEG signal processing module of the computer terminal.

[0109] The computer terminal includes an eye movement signal processing module, which is used to subscribe to the eye movement signals transmitted by the eye tracker, to filter, process missing values, and smooth the signals, and to store pre-processed eye movement coordinates.

[0110] The computer terminal also includes an EEG signal processing module for receiving EEG signals transmitted by the EEG device, performing pre-processing operations such as filtering on the EEG signals, and classifying the EEG signals.

[0111] The computer terminal also includes a terminal control module for sending control commands to the drone and for converting eye movement coordinates and EEG signal classification results into drone control commands.

[0112] The computer terminal also includes a terminal display module for receiving and displaying the drone's first-person perspective video stream and for displaying a friendly human-computer interaction prompt interface.

[0113] The computer terminal also includes a personalized brain imagery decoding model training module. This module is used to generate a user profile. Before officially controlling the drone flight, it collects the user's speech and motor imagery data and trains it using a pattern recognition model to obtain a user-specific pre-trained EEG feature recognition model.

[0114] Figure 5 This example illustrates a specific application scenario for the system presented in this paper. The figure depicts a wheelchair user with a motor impairment who is unable to control devices such as a keyboard, mouse, or joystick with their hands or feet. This system can help such users easily and conveniently control a drone, allowing them to enjoy the thrill of drone control while also appreciating the scenery captured by the drone.

Claims

1. A method for controlling a drone with the mind based on brain imagination and eye tracking, characterized in that: The steps include: Step 1: Train a personalized brain image decoding model; Step 1-1: Collect natural state; The gaze point a1s is presented, and the user then begins to maintain a natural and relaxed state a2s. After the end, the user chooses to accept or reject this round of acquisition based on the user's state. Repeat the acquisition for n rounds, and when n ≥ 5, select the next state of acquisition. Step 1-2: Collect forward status; The user is presented with the fixation point a3s, and then begins to imagine pushing the block forward with both hands while silently saying "forward." After this, the user chooses to accept or reject this round of acquisition based on the user's state. The acquisition is repeated for n rounds or proceeds to the next state of acquisition. Step 1-3: Collect the back state; Present the fixation point a4s, and then the user begins to imagine pulling the block backward with both hands while silently saying "back." After this, the user chooses to accept or reject this round of acquisition based on the user's state. Repeat the acquisition for n rounds or proceed to the next state of acquisition. Step 1-4: Collection is completed; Based on the collected results, an independent imagination sample library is established. The data in the sample library is trained using a pattern recognition algorithm to obtain a personalized classification model for the individual, which is used as the user's profile. This profile is used in each subsequent drone control. Step 2: The computer terminal obtains the real-time first-person video stream returned by the drone and presents an intuitive control interface based on the first-person video stream; Step 3: The eye tracker collects eye movement signals, and the EEG device collects EEG signals; The eye tracker is connected to the computer terminal, turned on, and collects eye movement signals; the EEG device is connected to the computer terminal and collects EEG signals; Step 4: The computer terminal obtains eye movement signals and EEG signals and pre-processes them respectively; For eye movement signals, the eye movement signal processing module first subscribes to the eye movement signals collected by the eye tracker, selects the eye movement coordinates, and normalizes the eye movement coordinates to the (0,1) interval, where (0,0) represents the upper left corner of the screen and (1,1) represents the lower right corner of the screen; Then, a sliding window mechanism is used to fill missing values ​​and smooth the eye movement signal within a specified time interval. Specifically, the center point coordinates (center x , center y )Fill missing values ​​(nan x ,nan y ); The sliding window mechanism uses a mean smoothing filter and uses the coordinate mean within the time window as an eye movement feature. The calculation formula is as follows: Where t represents the total number of eye gaze point coordinates within the time window, I x Indicates the horizontal coordinate of the eye movement fixation point, I y Indicates the vertical coordinate of the eye movement fixation point; The eye movement signal processing module sends the pre-processed eye movement coordinates to the terminal control module; For EEG signals, the EEG signal processing module first obtains the EEG signals collected by the EEG device; then pre-processes the EEG signals within a specified time interval; Step 5: Decode eye movement features and EEG features into drone control commands; In the terminal control module, the eye movement signal is first monitored. Within a specified time interval, it is determined whether the eye movement coordinates fall within the control area. If they do, the corresponding control command is sent to the drone according to the eye movement coordinates. The correspondence between the eye movement coordinates and the drone control command is as follows: The eye movement coordinates are (0≤x≤0.1, 0.1≤y≤0.9), and the drone control command is to move left; The eye movement coordinates are (0.9≤x≤1.0, 0.1≤g≤0.9), and the drone control command is right; The eye movement coordinates are (0.1≤x≤0.9, 0.9≤y≤1.0), and the drone control command is upward; The eye movement coordinates are (0.1≤x≤0.9, 0.0≤y≤0.1), and the drone control command is downward; The eye movement coordinates are other and the drone control command is suspended; When the eye movement coordinates are within other ranges, the terminal control module switches to the EEG signal monitoring mode. At this time, the terminal control module extracts the EEG features of the specified time step from the EEG signal processing module, loads the user configuration file, and uses the trained model in the configuration file to identify which speech motor imagery pattern best matches the user's current EEG features. The module selects the best matching pattern and converts it into the corresponding drone control command, as follows: When the brain imagines the action of pushing forward and silently says "forward", the drone control command is forward; When the brain imagines the action of pulling back and silently says "back", the drone control command is back; When the brain imagines something else, the drone control command is suspended; Step 6: The terminal control module sends control commands to the drone via the wireless network to control the drone's flight.

2. The method of controlling a drone with the mind based on brain imagination and eye tracking according to claim 1, characterized in that: Said a1=3, a2=8, a3=3, a4=3.

3. The method of controlling a drone with the mind based on brain imagination and eye tracking according to claim 2, characterized in that: The preprocessing in step 4 includes locating channel positions, filtering, resampling, artifact removal, independent component analysis, calculating frequency domain features and time domain features, and standardization.

4. A drone system using the method according to claim 1, characterized in that: include: Module 1: Terminal display module, used to display the drone's first-person video stream in real time and a friendly human-computer interaction prompt interface; Module 2: Eye movement signal acquisition module, used to collect eye movement signals in real time; Module 3: EEG signal acquisition module, used to collect EEG signals in real time; Module 4: Eye movement signal processing module, which receives the data stream transmitted by the eye movement signal acquisition module and pre-processes the eye movement signal; Module 5: EEG signal processing module, which is used to process the data stream transmitted by the real-time EEG signal acquisition module and pre-process the EEG signal; Module 6: Terminal control module, used to load user configuration files and convert eye movement signals and EEG signals into drone control commands; Module 7: UAV communication module, used to transmit the UAV's first-person perspective video stream to the terminal display module and receive control commands from the terminal control module to realize UAV control; Module 8: Personalized brain imagination decoding model training module; It is used to generate user profiles. Before officially controlling the flight of the drone, it collects the user's brain imagination data and uses the pattern recognition model for training to obtain the user's personalized brain imagination pre-training pattern recognition model.

Images