A UAV control method and device based on EEG and joystick control

By combining EEG and rocker control methods, using EEG data and rocker instructions to generate drone control signals, the problem of refined attitude control in quadrotor drone control is solved, and the human-computer interaction efficiency is significantly improved.

CN114859782BActive Publication Date: 2024-11-26BEIJING MECHANICAL EQUIP INST
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210435520.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-24
Publication Date
2024-11-26
Estimated Expiration
2042-04-24

AI Technical Summary

Technical Problem

It is difficult for existing four-rotor drones to achieve refined quantitative control of drone attitude during control, resulting in low human-computer interaction efficiency. A method is needed to improve the interaction efficiency between operators and drones.

Method used

Using the control method based on EEG and rocker, EEG data is collected through EEG acquisition equipment and augmented reality glasses, combined with the joystick control equipment, and used preset discrimination rules to generate brain control instructions, and sent control through the drone API interface. Signal to the drone to achieve refined control of the drone's attitude.

Benefits of technology

A single operator can simultaneously have the ability to quickly control and refine and quantitatively control the quadrotor drone on a large scale, greatly improving the interaction efficiency between the operator and the quadrotor drone.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114859782B_ABST
    Figure CN114859782B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for controlling a drone based on brain waves and joystick control. The method utilizes the API interface of the drone to send control commands through the joystick to simulate the control of the drone by a remote controller, collects and analyzes the EEG signals of the operator and uses augmented reality glasses to induce EEG signals to send commands for quantitatively controlling the attitude of the drone, so that a single operator has the ability to quickly control a large range of quad-rotor drones and finely control the quad-rotor drones, greatly improving the interaction efficiency between the operator and the quad-rotor drone.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] A quad-rotor drone is a drone with real-time image and data transmission functions, and is often used in high-altitude photography, target search in complex environments, and hoisting and rescue of materials. Usually, operators use the remote control that comes with the quad-rotor drone for remote control. This is a large-scale and rapid control method, and it is difficult to achieve precise quantitative control of the drone's attitude. Additional operators are required to cooperate with the drone ground station system to perform quantitative control related operations, resulting in low efficiency of human-computer interaction.

[0003] Therefore, one or more methods are needed to solve the above problems.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a drone control method, device, electronic device and computer-readable storage medium based on brain waves and joystick control, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0006] According to one aspect of the present disclosure, a method for controlling a drone based on EEG and joystick control is provided, comprising:

[0007] The drone sends the drone attitude and status data to the drone data processing server subsystem on the first server. After receiving the drone attitude and status data, the drone data processing server subsystem forwards the drone attitude and status data to the drone ground station subsystem on the third server; receives the drone control signal sent by the drone ground station subsystem on the third server, and implements the control operation of the drone based on the drone control signal;

[0008] Based on the stimulation paradigm program on the second server, EEG data is collected through the EEG acquisition device and the augmented reality glasses, and the EEG data is sent to the EEG signal processing program on the second server. The EEG signal processing program on the second server converts the EEG data into brain control instructions based on preset discrimination rules, and forwards the brain control instructions to the drone ground station subsystem on the third server;

[0009] After receiving the UAV attitude and status data, the UAV ground station subsystem on the third server presents and displays the UAV attitude and status data on the display device of the third server; after receiving the brain control instruction and the joystick instruction generated by the joystick control device, the UAV ground station subsystem on the third server generates a UAV control signal based on the preset UAV instruction encoding rule for the brain control instruction and the joystick instruction, and sends the UAV control signal to the UAV.

[0010] In an exemplary embodiment of the present disclosure, the method further includes:

[0011] The UAV directly sends the video signal collected by the UAV to the UAV ground station subsystem on the third server based on a dedicated video transmission antenna;

[0012] After receiving the video signal, the UAV ground station subsystem on the third server presents and displays the video signal on the display device of the third server.

[0013] In an exemplary embodiment of the present disclosure, the method further includes:

[0014] After receiving the brain control instruction and the joystick instruction, the UAV ground station subsystem on the third server generates a UAV control signal based on the preset UAV instruction encoding rule for the brain control instruction and the joystick instruction, and sends the UAV control signal to the UAV based on a general data transmission antenna communicating with the UAV ground station subsystem on the third server.

[0015] In an exemplary embodiment of the present disclosure, the method further includes:

[0016] The electroencephalogram signal processing program on the second server obtains real-time electroencephalogram data from the electroencephalogram acquisition device through a serial port or a network port, and sends the electroencephalogram stimulation paradigm parameters to the augmented reality glasses through the network port based on the stimulation paradigm program on the second server. After receiving the parameters, the augmented reality glasses parse and generate and display the stimulation paradigm, and the electroencephalogram acquisition device acquires the electroencephalogram data generated by the user under the stimulation of the stimulation paradigm.

[0017] In an exemplary embodiment of the present disclosure, the preset discrimination rule for converting the electroencephalogram data into a brain control instruction based on a preset discrimination rule includes:

[0018] For the Nth judgment result at the start of program operation, when 0 < N < 3, no brain control instruction is generated;

[0019] When N = 3, if the judgment results of the first, second, and third times are the same, and the correlation values calculated three times are all greater than or equal to the threshold, a brain control instruction corresponding to the third judgment result is generated, and no brain control instruction is generated when the above conditions are not met;

[0020] When N>3, a two-step judgment process is performed. In the first step, if the results of the Nth, N-1th and N-2th judgments are the same, the results of the N-3th and N-2th judgments are different, and the correlation values ​​calculated for the Nth, N-1th and N-2th times are all greater than or equal to the threshold, then a brain control instruction corresponding to the Nth judgment result is generated. If the conditions are not met, the second step of judgment is performed. If the results of the Nth, N-1th, N-2nd and N-3rd judgments are the same, and the correlation values ​​calculated for the Nth, N-1th and N-2th times are all greater than or equal to the threshold, and the correlation value calculated for the N-3th time is less than the threshold, then a brain control instruction corresponding to the Nth judgment result is generated. If the above conditions are not met, no brain control instruction is generated.

[0021] In an exemplary embodiment of the present disclosure, the drone ground station subsystem on the third server in the method further includes:

[0022] An external data processing thread, the external data processing thread is used to receive the drone attitude and status data and the brain control command, and present the drone attitude and status data on the display device of the third server, and generate a drone control signal based on the brain control command based on a preset drone command coding rule, and send the drone control signal to the drone through a universal data transmission antenna.

[0023] The joystick data processing thread is used to receive and forward the control command data generated by the joystick control device through the third server, and generate a drone control signal based on the control command data based on a preset drone command coding rule, and send the drone control signal to the drone through a universal data transmission antenna.

[0024] The image display thread receives the video signal sent by the drone and presents the video signal on the display device of the third server.

[0025] In one aspect of the present disclosure, a drone control device based on brain wave and joystick control is provided, comprising:

[0026] A drone, used to send the drone attitude and status data to the drone data processing server subsystem on the first server, and the drone data processing server subsystem forwards the drone attitude and status data to the drone ground station subsystem on the third server after receiving the drone attitude and status data; receive the drone control signal sent by the drone ground station subsystem on the third server, and implement the control operation of the drone based on the drone control signal;

[0027] The brain control subsystem is used to collect EEG data through the EEG acquisition device and the augmented reality glasses based on the stimulation paradigm program on the second server, and send the EEG data to the EEG signal processing program on the second server. The EEG signal processing program on the second server converts the EEG data into brain control instructions based on preset discrimination rules, and forwards the brain control instructions to the drone ground station subsystem on the third server;

[0028] The ground station subsystem is used for the drone ground station subsystem on the third server to present the drone attitude and status data on the display device of the third server after receiving the drone attitude and status data; the drone ground station subsystem on the third server generates a drone control signal based on the preset drone instruction coding rules after receiving the brain control instructions and the joystick instructions generated by the joystick control device, and sends the drone control signal to the drone.

[0029] In one aspect of the present disclosure, there is provided an electronic device, comprising:

[0030] Processor; and

[0031] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of the above items.

[0032] In one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the above items is implemented.

[0033] A drone control method based on EEG and joystick control is disclosed in an exemplary embodiment. The method utilizes the API interface of the drone to simulate the control of the drone by a remote controller by sending control commands through a joystick. By collecting and analyzing the EEG signals of the operator and using augmented reality glasses to induce EEG signals, commands for quantitatively controlling the attitude of the drone are sent, so that a single operator has the ability to quickly control a large range of quad-rotor drones and finely control quantitatively the quad-rotor drones, greatly improving the interaction efficiency between the operator and the quad-rotor drone.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other features and advantages of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the attached drawings.

[0036] Figure 1A flowchart of a drone control method based on EEG and joystick control according to an exemplary embodiment of the present disclosure is shown;

[0037] Figure 2 The system composition and workflow diagram of a drone control method based on EEG and joystick control according to an exemplary embodiment of the present disclosure are shown;

[0038] Figure 3 A flowchart for implementing brain control command generation and discrimination rules of a drone control method based on EEG and joystick control according to an exemplary embodiment of the present disclosure is shown;

[0039] Figure 4 A schematic block diagram of a drone control device based on EEG and joystick control according to an exemplary embodiment of the present disclosure is shown;

[0040] Figure 5 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure; and

[0041] Figure 6 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0043] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, materials, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0045] In this example embodiment, a drone control method based on EEG and joystick control is first provided; Figure 1 As shown in , the drone control method based on EEG and joystick control may include the following steps:

[0046] Step S110, the drone sends the drone attitude and status data to the drone data processing server subsystem on the first server. After receiving the drone attitude and status data, the drone data processing server subsystem forwards the drone attitude and status data to the drone ground station subsystem on the third server; receives the drone control signal sent by the drone ground station subsystem on the third server, and implements the control operation of the drone based on the drone control signal;

[0047] Step S120, based on the stimulation paradigm program on the second server, collect EEG data through the EEG acquisition device and the augmented reality glasses, and send the EEG data to the EEG signal processing program on the second server. The EEG signal processing program on the second server converts the EEG data into brain control instructions based on preset discrimination rules, and forwards the brain control instructions to the drone ground station subsystem on the third server;

[0048] Step S130, after receiving the drone attitude and status data, the drone ground station subsystem on the third server presents the drone attitude and status data on the display device of the third server; after receiving the brain control command and the joystick command generated by the joystick control device, the drone ground station subsystem on the third server generates a drone control signal based on the preset drone command coding rules according to the brain control command and the joystick command, and sends the drone control signal to the drone.

[0049] A drone control method based on EEG and joystick control is disclosed in an exemplary embodiment. The method utilizes the API interface of the drone to simulate the control of the drone by a remote controller by sending control commands through a joystick. By collecting and analyzing the EEG signals of the operator and using augmented reality glasses to induce EEG signals, commands for quantitatively controlling the attitude of the drone are sent, so that a single operator has the ability to quickly control a large range of quad-rotor drones and finely control quantitatively the quad-rotor drones, greatly improving the interaction efficiency between the operator and the quad-rotor drone.

[0050] Next, a drone control method based on EEG and joystick control in this example embodiment will be further described.

[0051] Embodiment 1: Figure 2 The figure shows the composition and workflow of the UAV control system based on EEG and joystick control.

[0052] In step S110, the drone may send the drone attitude and status data to the drone data processing server subsystem on the first server. After receiving the drone attitude and status data, the drone data processing server subsystem forwards the drone attitude and status data to the drone ground station subsystem on the third server; receives the drone control signal sent by the drone ground station subsystem on the third server, and implements the control operation of the drone based on the drone control signal;

[0053] In the embodiment of this example, the method further includes:

[0054] The drone directly sends the video signal collected by the drone to the drone ground station subsystem on the third server based on the dedicated image transmission antenna;

[0055] After receiving the video signal, the drone ground station subsystem on the third server presents the video signal on the display device of the third server.

[0056] In the embodiment of this example, the method further includes:

[0057] After receiving the brain control instructions and joystick instructions, the drone ground station subsystem on the third server generates a drone control signal based on the preset drone instruction coding rules, and sends the drone control signal to the drone based on the universal data transmission antenna that communicates with the drone ground station subsystem on the third server.

[0058] In the embodiment of this example, the drone data processing server subsystem has the function of receiving and controlling the flow forwarding of the drone's real-time attitude and status data. The drone data processing server subsystem consists of a PC, which is named PC No. 1. The attitude and status data of the drone are transmitted through a pair of dedicated data transmission antennas, one of which is located on the drone body, and the other is connected to PC No. 1 via USB. PC No. 1 runs the drone data processing program. When the drone is powered on, the drone data processing program sends specific instructions to the antenna serial port, and then the drone will return data to the same serial port at a fixed frequency. The program controls the flow of the returned data, and finally sends the controlled data to the drone ground station subsystem through the network port.

[0059] In step S120, based on the stimulation paradigm program on the second server, electroencephalogram (EEG) data can be collected through an EEG acquisition device and an augmented reality (AR) glasses, and the EEG data is sent to the EEG signal processing program on the second server. The EEG signal processing program on the second server converts and generates a brain control instruction based on a preset discrimination rule, and forwards the brain control instruction to the drone ground station subsystem on the third server;

[0060] In the embodiment of this example, the method further includes:

[0061] The EEG signal processing program on the second server obtains real-time EEG data from the EEG acquisition device through a serial port or a network port, and sends the EEG stimulation paradigm parameters to the AR glasses through the network port based on the stimulation paradigm program on the second server. After receiving the parameters, the AR glasses parse and generate and display the stimulation paradigm, and the EEG acquisition device collects the EEG data generated by the user under the stimulation of the stimulation paradigm.

[0062] In the embodiment of this example, the preset discrimination rule for converting the EEG data into a brain control instruction based on a preset discrimination rule includes:

[0063] For the Nth judgment result at the start of program operation, when 0 < N < 3, no brain control instruction is generated;

[0064] When N = 3, if the judgment results of the 1st, 2nd, and 3rd times are the same, and the correlation values calculated three times are all greater than or equal to the threshold value, then a brain control instruction corresponding to the 3rd judgment result is generated. When the above conditions are not met, no brain control instruction is generated;

[0065] When N > 3, a two-step judgment process is experienced. In the first step of judgment, if the judgment results of the Nth, (N - 1)th, and (N - 2)th times are the same, the judgment result of the (N - 3)th time is different from that of the (N - 2)th time, and the correlation values calculated for the Nth, (N - 1)th, and (N - 2)th times are all greater than or equal to the threshold value, then a brain control instruction corresponding to the Nth judgment result is generated. If the conditions are not met, enter the second step of judgment. If the judgment results of the Nth, (N - 1)th, (N - 2)th, and (N - 3)th times are the same, and the correlation values calculated for the Nth, (N - 1)th, and (N - 2)th times are all greater than or equal to the threshold value, and the correlation value calculated for the (N - 3)th time is less than the threshold value, then a brain control instruction corresponding to the Nth judgment result is generated. When the above conditions are not met, no brain control instruction is generated.

[0066] In the embodiment of this example, the brain control subsystem functions to collect EEG signals, set and display steady-state visual evoked potential (SSVEP) stimulation paradigm parameters, process EEG signal data, generate and send brain control instructions;

[0067] In the embodiment of this example, the brain control subsystem consists of an electroencephalogram (EEG) acquisition device, an augmented reality (AR) glasses, and a PC named PC No. 2. The EEG acquisition device collects EEG data in real time by having a person wear an EEG cap, and is connected to PC No. 2 via USB or WIFI. PC No. 2 runs an EEG signal processing program and an SSVEP stimulation paradigm setting program. The EEG signal processing program obtains real-time EEG data from the EEG acquisition device via a serial port or a network port. The EEG stimulation paradigm program sends the relevant parameters of the EEG stimulation paradigm to the AR glasses via the network port. After receiving the parameter settings, the AR glasses perform parsing and then display the stimulation paradigm. Under the stimulation of the stimulation paradigm, a person will generate SSVEP EEG signals. The EEG signal processing program parses the acquired EEG signals, can continuously generate multiple judgment results, uses discrimination rules to generate brain control commands for the judgment results, and finally sends the brain control commands to the UAV ground station subsystem via the network port;

[0068] As Figure 3 shown, the discrimination rule for generating brain control commands is that in the SSVEP stimulation paradigm of the asynchronous mode, the EEG signal processing program intercepts data in the form of a sliding window. For each intercepted segment of data, a judgment result and a correlation value are calculated. A threshold is set in the EEG signal processing program. The process from the judgment result to the generation of a brain control command is as follows: Assume that the latest judgment result is the Nth judgment result since the program starts running. When 0 < N < 3, no brain control command is generated; when N = 3, if the judgment results of the 1st, 2nd, and 3rd times are the same, and the correlation values calculated for the 3 times are all greater than or equal to the threshold, then a brain control command corresponding to the 3rd judgment result is generated. When the above conditions are not met, no brain control command is generated; when N > 3, a two-step judgment process is experienced. In the first step of judgment, if the judgment results of the Nth, N - 1th, and N - 2th times are the same, the judgment result of the N - 3th time is different from that of the N - 2th time, and the correlation values calculated for the Nth, N - 1th, and N - 2th times are all greater than or equal to the threshold, then a brain control command corresponding to the Nth judgment result is generated. If the conditions are not met, enter the second step of judgment. If the judgment results of the Nth, N - 1th, N - 2th, and N - 3th times are the same, and the correlation values calculated for the Nth, N - 1th, and N - 2th times are all greater than or equal to the threshold, and the correlation value calculated for the N - 3th time is less than the threshold, then a brain control command corresponding to the Nth judgment result is generated. When the above conditions are not met, no brain control command is generated.

[0069] In step S130, after receiving the drone attitude and status data, the drone ground station subsystem on the third server can present the drone attitude and status data on the display device of the third server; after receiving the brain control instructions and the joystick instructions generated by the joystick control device, the drone ground station subsystem on the third server can generate a drone control signal based on the preset drone instruction coding rules according to the brain control instructions and the joystick instructions, and send the drone control signal to the drone.

[0070] In the embodiment of this example, the drone ground station subsystem on the third server in the method further includes:

[0071] An external data processing thread, the external data processing thread is used to receive the drone attitude and status data and the brain control command, and present the drone attitude and status data on the display device of the third server, and generate a drone control signal based on the brain control command based on a preset drone command coding rule, and send the drone control signal to the drone through a universal data transmission antenna.

[0072] The joystick data processing thread is used to receive and forward the control command data generated by the joystick control device through the third server, and generate a drone control signal based on the control command data based on a preset drone command coding rule, and send the drone control signal to the drone through a universal data transmission antenna.

[0073] The image display thread receives the video signal sent by the drone and presents the video signal on the display device of the third server.

[0074] In the embodiment of this example, the drone ground station subsystem has the functions of receiving and locally presenting the real-time feedback images from the drone camera module, receiving and locally presenting the real-time attitude and status data of the drone forwarded by the drone data processing server subsystem, generating and sending joystick control instructions, receiving brain control instructions sent by the brain control subsystem, and translating and sending corresponding drone control instructions.

[0075] In the embodiment of this example, the drone ground station subsystem consists of a PC and a joystick control device, where the PC is named PC No. 3. PC No. 3 runs a comprehensive data processing program, a drone ground station program, and an image display program. The comprehensive data processing program includes two threads: external data processing and joystick data processing. The external data processing thread can simultaneously receive the brain control instructions sent by the brain control subsystem and the drone attitude and status data after flow control sent by the drone data processing server subsystem through the network port. The brain control instructions are translated according to the drone instruction coding rules according to their meanings, and then sent to the drone through the general data transmission antenna; the drone data after flow control is sent to the drone ground station program through the serial port and presented locally. The control instruction data generated by the joystick control device is first sent to the drone ground station program through the serial port, and after being translated by the ground station program according to the drone instruction coding rules, it is sent to the joystick data processing thread through the serial port. The joystick data processing thread controls the data flow, and then sends it to the drone through the general data transmission antenna. The image display program uses a dedicated image transmission antenna to receive real-time video streams through the network port and present them locally. There is a pair of general data transmission antennas and dedicated image transmission antennas. One antenna is located on the drone body, and the other is connected to PC No. 3 via USB.

[0076] Embodiment 2:

[0077] S1. UAV data processing server subsystem, whose function is to receive and control the flow forwarding of real-time attitude and status data of UAV;

[0078] Prepare a PC for running the drone data processing program, which is named PC No. 1. The drone uses the KP880X4A quad-rotor drone produced by Kunshan Kunpeng Drone Technology Co., Ltd. The drone is equipped with a pair of dedicated data transmission antennas, one of which is connected to the flight control device of the drone body, and the other antenna is connected to PC No. 1 via USB. After the drone is powered on and the drone data processing program sets the parameters required for serial port and network communication, start running the program;

[0079] The drone data processing program first sends the hexadecimal command "aa 01 00 2001 cc 97" through the dedicated data transmission antenna serial port to start the transmission of the drone's attitude and status data. The drone then sends data at a frequency of 25Hz. The program receives the drone data through the same serial port and controls the data flow. Finally, the controlled data is sent to the drone ground station subsystem at a frequency of 10Hz through the UDP protocol.

[0080] S2. Brain control subsystem, whose functions include the collection of EEG signals, setting and displaying parameters of SSVEP (steady-state visual evoked potential) stimulation paradigm, data processing of EEG signals, and generation and sending of brain control commands;

[0081] Prepare a PC for running the EEG signal processing program and the SSVEP stimulation paradigm setting program. The PC is named PC No. 2. The EEG acquisition device is SynApms2 produced by Neuroscan, Australia. The device is connected to PC No. 2 via USB. People wear a special EEG cap to collect EEG signals. The augmented reality glasses are HoloLens2 produced by Microsoft, USA. The device is connected to PC No. 2 via wireless LAN.

[0082] The paradigm parameters set by the SSVEP stimulation paradigm setting program are as follows: asynchronous mode, the number of stimulation targets is 10, the flickering frequency of each stimulation target is (9, 9.25, 9.5, 9.75, 10.25, 10.5, 10.75, 11, 11.25, 11.5), the flickering phase is all 0, and the text (up, down, left rotation, right rotation, forward, backward, left shift, right shift, locked, unlocked) is displayed on each stimulation target. After the SSVEP stimulation paradigm setting program starts running, the above paradigm parameter settings will be sent to HoloLens via UDP. After HoloLens receives the parameters and successfully parses them, it displays the SSVEP stimulation paradigm;

[0083] The parameters of the EEG signal processing program are as follows: the data capture sliding window length is 2s, the sliding window interval is 0.2s, and the initial threshold is set to 1. If it is not appropriate, it can be adjusted according to the relevant values ​​during the operation of the system. After the EEG signal processing program starts running, the EEG data is received through the SynApms2 serial port and then processed. During the data processing process, each time the data capture sliding window slides, a judgment result can be calculated from the 10 stimulation targets. When multiple judgment results meet the judgment rules, the program generates the corresponding brain control command and sends it to the UAV ground station subsystem via UDP;

[0084] The discrimination rule for brain - controlled instruction generation is as follows: in the SSVEP stimulation paradigm of the asynchronous mode, the EEG signal processing program intercepts data in the form of a sliding window. For each intercepted data segment, a judgment result and a correlation value are calculated. A threshold is set in the EEG signal processing program. The process of mapping from the judgment result to the brain - controlled instruction is as follows: Assume that the latest judgment result is the Nth judgment result since the program starts running. When 0 < N < 3, no brain - controlled instruction is generated; when N = 3, if the judgment results of the 1st, 2nd, and 3rd times are the same, and the correlation values calculated in the 3 times are all greater than or equal to the threshold, then a brain - controlled instruction corresponding to the 3rd judgment result is generated. When the above conditions are not met, no brain - controlled instruction is generated; when N > 3, a two - step judgment process is experienced. In the first step of judgment, if the judgment results of the Nth, (N - 1)th, and (N - 2)th times are the same, the judgment result of the (N - 3)th time is different from that of the (N - 2)th time, and the correlation values calculated in the Nth, (N - 1)th, and (N - 2)th times are all greater than or equal to the threshold, then a brain - controlled instruction corresponding to the Nth judgment result is generated. If the conditions are not met, enter the second step of judgment. If the judgment results of the Nth, (N - 1)th, (N - 2)th, and (N - 3)th times are the same, the correlation values calculated in the Nth, (N - 1)th, and (N - 2)th times are all greater than or equal to the threshold, and the correlation value calculated in the (N - 3)th time is less than the threshold, then a brain - controlled instruction corresponding to the Nth judgment result is generated. When the above conditions are not met, no brain - controlled instruction is generated.

[0085] S3. The UAV ground station subsystem, whose functions include receiving and locally presenting the images real - time transmitted by the UAV camera module, receiving and locally presenting the real - time attitude and state data of the UAV controlled by the UAV data processing server subsystem for flow control and forwarding, generating and sending joystick control instructions, receiving the brain - controlled instructions sent by the brain - controlled subsystem, and translating and sending the corresponding UAV control instructions;

[0086] Prepare a PC for running the comprehensive data processing program, the UAV ground station program, and the image display program. This PC is named PC No. 3. The joystick control device selects the eF - SAOII flight simulation cabin produced by Sichuan雏鹰未来航空科技有限公司 (Chuying Future Aviation Technology Co., Ltd. in Chinese), and this simulation cabin integrates the joystick control device and PC No. 3. The general data transmission antenna selects a pair of C10X wireless communication modules produced by Shenzhen Lingchengxin Electronics Co., Ltd. One antenna is connected to the flight control device of the UAV body through a Dupont wire, and the other antenna is connected to PC No. 3 through a Dupont wire and a USB - to - TTL module. The dedicated video transmission antenna is a pair of supporting antennas of the UAV. One antenna is connected to the flight control device of the UAV body, and the other antenna is connected to PC No. 3 through a network cable;

[0087] After the comprehensive data processing program sets the parameters required for serial port and network communication, it starts running. The external data processing thread simultaneously monitors the drone data after flow control sent by the drone data processing server subsystem and the brain control instructions sent by the brain control subsystem. The two types of data can be distinguished according to the data type. The drone data after flow control received by the external data processing thread continues to be sent to the drone ground station program through the virtual serial port. The drone ground station program receives the data and parses it, and finally presents it visually. There are 10 types of brain control instructions, and their meanings are (rise 1 meter, drop 1 meter, rotate 5 degrees to the left, rotate 5 degrees to the right, move forward 1 meter, move back 1 meter, move left 1 meter, move right 1 meter, cut off brain control, and restore brain control). When the external data processing thread receives the first 8 types of brain control instructions, it translates the meaning of the brain control instructions into control instructions recognized by the drone according to the drone instruction encoding rules, and then sends them to the drone through the C10X antenna serial port; when receiving the last two instructions, it switches and restores the translation of brain control instructions respectively;

[0088] The drone ground station program receives the commands sent by the joystick through the joystick serial port, and then translates the meaning of the joystick commands into control commands recognized by the drone according to the drone command encoding rules, and sends them to the joystick data processing thread through the virtual serial port. After the joystick data processing thread receives the translated joystick control commands, it controls the data flow and then sends it to the drone through the C10X antenna serial port at a frequency of 10Hz. The drone flight control device distinguishes between brain control and joystick translation commands by receiving the function words in the commands. When the two commands appear at the same time, the joystick has a higher priority than brain control;

[0089] The image display program parses the video stream sent by the dedicated image transmission antenna through the RSTP protocol and presents it locally. PC No. 3 requires two display screens, one for displaying the drone data presented in real time by the drone ground station program, and the other for displaying the images captured by the drone camera module presented in real time by the image display program.

[0090] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0091] In addition, in this exemplary embodiment, a drone control device based on EEG and joystick control is also provided. Figure 4 As shown, the drone control device 400 based on brain waves and joystick control may include: a drone 410, a brain control subsystem 420, and a ground station subsystem 430. Among them:

[0092] UAV 410 is used to send the UAV attitude and status data to the UAV data processing server subsystem on the first server, and the UAV data processing server subsystem forwards the UAV attitude and status data to the UAV ground station subsystem on the third server after receiving the UAV attitude and status data; receive the UAV control signal sent by the UAV ground station subsystem on the third server, and implement the control operation of the UAV based on the UAV control signal;

[0093] The brain control subsystem 420 is used to collect EEG data through the EEG acquisition device and the augmented reality glasses based on the stimulation paradigm program on the second server, and send the EEG data to the EEG signal processing program on the second server. The EEG signal processing program on the second server converts the EEG data into brain control instructions based on preset discrimination rules, and forwards the brain control instructions to the drone ground station subsystem on the third server;

[0094] The ground station subsystem 430 is used to present the drone attitude and status data on the display device of the third server after the drone ground station subsystem on the third server receives the drone attitude and status data; after the drone ground station subsystem on the third server receives the brain control instructions and the joystick instructions generated by the joystick control device, it generates a drone control signal based on the preset drone instruction coding rules and sends the drone control signal to the drone.

[0095] The specific details of each of the above-mentioned drone control device modules based on EEG and joystick control have been described in detail in the corresponding drone control method based on EEG and joystick control, so they will not be repeated here.

[0096] It should be noted that although several modules or units of a drone control device 400 based on EEG and joystick control are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for embodiment.

[0097] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0098] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0099] Refer to the following Figure 5 An electronic device 500 according to such an embodiment of the present invention is described. Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0100] like Figure 5 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510), and a display unit 540.

[0101] The storage unit stores program codes, which can be executed by the processing unit 510, so that the processing unit 510 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 510 can perform the following steps: Figure 1 Steps S110 to S130 shown in FIG.

[0102] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0103] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5203, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0104] Bus 550 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0105] The electronic device 500 may also communicate with one or more external devices 570 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 560. As shown, the network adapter 560 communicates with other modules of the electronic device 500 via a bus 550. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0106] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0107] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0108] refer to Figure 6 As shown, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0109] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0111] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0112] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0113] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0114] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0115] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A drone control method based on EEG and joystick control, characterized in that: The method includes: The drone sends the drone attitude and status data to the drone data processing server subsystem on the first server. After receiving the drone attitude and status data, the drone data processing server subsystem forwards the drone attitude and status data to the drone ground station subsystem on the third server; receives the drone control signal sent by the drone ground station subsystem on the third server, and implements the control operation of the drone based on the drone control signal; Based on the stimulation paradigm program on the second server, electroencephalogram (EEG) data is collected through an EEG acquisition device and an augmented reality (AR) glasses, and the EEG data is converted into a brain control instruction based on a preset discrimination rule, and the brain control instruction is sent to the brain control subsystem on the second server. After receiving the brain control instruction, the brain control subsystem forwards the brain control instruction to the drone ground station subsystem on the third server; the preset discrimination rule for converting the EEG data into a brain control instruction in the method includes: At the Nth judgment result at the start of program operation, when 0 < N < 3, no brain control instruction is generated; When N = 3, if the judgment results of the 1st, 2nd, and 3rd times are the same, and the correlation values calculated three times are all greater than or equal to the threshold, then a brain control instruction corresponding to the 3rd judgment result is generated. When the above conditions are not met, no brain control instruction is generated; When N > 3, a two-step judgment process is experienced. In the first step of judgment, if the judgment results of the Nth, N - 1th, and N - 2th times are the same, the judgment result of the N - 3th time is different from that of the N - 2th time, and the correlation values calculated for the Nth, N - 1th, and N - 2th times are all greater than or equal to the threshold, then a brain control instruction corresponding to the Nth judgment result is generated. If the conditions are not met, enter the second step of judgment. If the judgment results of the Nth, N - 1th, N - 2th, and N - 3th times are the same, and the correlation values calculated for the Nth, N - 1th, and N - 2th times are all greater than or equal to the threshold, and the correlation value calculated for the N - 3th time is less than the threshold, then a brain control instruction corresponding to the Nth judgment result is generated. When the above conditions are not met, no brain control instruction is generated; After receiving the drone attitude and status data, the drone ground station subsystem on the third server presents and displays the drone attitude and status data on the display device of the third server; after receiving the brain control instruction, the drone ground station subsystem on the third server generates a drone control signal based on the preset drone instruction coding rule, and sends the drone control signal to the drone; The drone ground station subsystem on the third server in the method further includes: An external data processing thread, which is used to receive the drone attitude and status data and the brain control instruction, generate a drone control signal based on the preset drone instruction coding rule for the drone attitude and status data and the brain control instruction, and send the drone control signal to the drone through a general purpose data transmission antenna; A rocker data processing thread, which is used to receive and forward the control instruction data generated by the rocker control device through the third server, generate a drone control signal based on the preset drone instruction encoding rule for the control instruction data, and send the drone control signal to the drone through a general data transmission antenna; An image display thread, which receives the video signal sent by the drone and presents the video signal on the display device of the third server.

2. The method according to claim 1, characterized in that The method further includes: The drone directly sends the video signal collected by the drone to the drone ground station subsystem on the third server based on a dedicated video transmission antenna; After receiving the video signal, the drone ground station subsystem on the third server presents and displays the video signal on the display device of the third server.

3. The method according to claim 1, characterized in that The method further includes: After receiving the brain control instruction, the drone ground station subsystem on the third server generates a drone control signal based on the preset drone instruction encoding rule for the brain control instruction, and sends the drone control signal to the drone based on the general data transmission antenna communicating with the drone ground station subsystem on the third server.

4. The method according to claim 1, characterized in that The method further includes: The electroencephalogram signal processing program on the second server obtains real-time electroencephalogram data from the electroencephalogram acquisition device through a serial port or a network port, and sends the electroencephalogram stimulation paradigm parameters to the augmented reality glasses through the network port based on the stimulation paradigm program on the second server. After receiving the parameters, the augmented reality glasses parse and generate and display the stimulation paradigm, and the electroencephalogram acquisition device acquires the electroencephalogram data generated by the user under the stimulation of the stimulation paradigm.

5. A drone control device based on EEG and joystick control, characterized in that: The device includes: A drone, which is used to send the drone attitude and status data to the drone data processing server subsystem on the first server. After receiving the drone attitude and status data, the drone data processing server subsystem forwards the drone attitude and status data to the drone ground station subsystem on the third server; receive the drone control signal sent by the drone ground station subsystem on the third server, and implement the control operation of the drone based on the drone control signal; A brain control subsystem, which is used to collect electroencephalogram data through an electroencephalogram acquisition device and augmented reality glasses based on the stimulation paradigm program on the second server, convert and generate a brain control instruction based on a preset discrimination rule for the electroencephalogram data, and send the brain control instruction to the brain control subsystem on the second server. After receiving the brain control instruction, the brain control subsystem forwards the brain control instruction to the drone ground station subsystem on the third server; wherein, the preset discrimination rule for converting the electroencephalogram data into a brain control instruction based on a preset discrimination rule includes: The Nth judgment result at the start of program operation. When 0 < N < 3, no brain control instruction is generated; When N = 3, if the judgment results of the 1st, 2nd, and 3rd times are the same, and the relevant values calculated 3 times are all greater than or equal to the threshold, then generate a brain control instruction corresponding to the 3rd judgment result. When the above conditions are not met, no brain control instruction is generated; When N>3, a two-step judgment process is performed. In the first step, if the Nth, N-1th and N-2th judgment results are the same, the N-3th and N-2th judgment results are different, and the correlation values ​​calculated for the Nth, N-1th and N-2th times are all greater than or equal to the threshold, then a brain control instruction corresponding to the Nth judgment result is generated. If the conditions are not met, the second step is entered. If the Nth, N-1th, N-2nd and N-3rd judgment results are the same, and the correlation values ​​calculated for the Nth, N-1th and N-2th times are all greater than or equal to the threshold, and the correlation value calculated for the N-3th time is less than the threshold, then a brain control instruction corresponding to the Nth judgment result is generated. If the above conditions are not met, no brain control instruction is generated; The ground station subsystem is used for the drone ground station subsystem on the third server to present the drone attitude and status data on the display device of the third server after receiving the drone attitude and status data; after receiving the brain control instruction, the drone ground station subsystem on the third server generates a drone control signal based on the preset drone instruction coding rule, and sends the drone control signal to the drone; wherein the drone ground station subsystem on the third server also includes: An external data processing thread, the external data processing thread is used to receive the drone attitude and status data and the brain control command, and generate a drone control signal based on the drone attitude and status data and the brain control command based on a preset drone command coding rule, and send the drone control signal to the drone through a universal data transmission antenna; A joystick data processing thread, used for receiving and forwarding control instruction data generated by the joystick control device through the third server, generating a drone control signal based on the control instruction data based on a preset drone instruction coding rule, and sending the drone control signal to the drone through a universal data transmission antenna; The image display thread receives the video signal sent by the drone and presents the video signal on the display device of the third server.

6. An electronic device, characterized in that: include Processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

Citation Information

Patent Citations

  • Multi-channel induced brain wave controlled unmanned aerial vehicle system and control method

    CN108113670A

  • Control work efficiency analysis method, device and system based on brain load

    CN112256123A

  • Multi-mode control system for ornithopter

    CN113253750A

  • Unmanned aerial vehicle suite, unmanned aerial vehicle control device and control method

    WO2018112847A1